Extended reality for productivity

Abstract

To provide an approach for providing mobile environment that enables a user to experience comfort of a static workspace at a desired location by providing a virtual desktop-like screen using extended reality (XR).SOLUTION: An integrated computing interface device may include a portable housing having a key region and a non-key region, a keyboard associated with the key region of the housing, and a holder associated with the non-key region of the housing. The holder may be configured to selectively engage with and disengage from a wearable extended reality apparatus, so that when the wearable extended reality apparatus is selectively engaged with the housing via the holder, the wearable extended reality apparatus is transportable together with the housing.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 147,051, filed on February 8, 2021; U.S. Provisional Patent Application No. 63 / 157,768, filed on March 7, 2021; U.S. Provisional Patent Application No. 63 / 173,095, filed on April 9, 2021; U.S. Provisional Patent Application No. 63 / 213,019, filed on June 21, 2021; U.S. Provisional Patent Application No. 63 / 215,500, filed on June 27, 2021; U.S. Provisional Patent Application No. 63 / 216,335, filed on June 29, 2021; U.S. Provisional Patent Application No. 63 / 226,977, filed on July 29, 2021; U.S. Provisional Patent Application No. 63 / 300,005, filed on January 16, 2022; U.S. Provisional Patent Application No. 63 / 307,207, filed on February 7, 2022; U.S. Provisional Patent Application No. 63 / 307,203, filed on February 7, 2022; and U.S. Provisional Patent Application No. 63 / 307,217, filed on February 7, 2022, all of which are hereby incorporated by reference in their entirety.

[0002] Technical Field The present disclosure generally relates to the field of extended reality. More specifically, the present disclosure relates to systems, methods, and devices for providing productivity applications using extended reality environments.

Background Art

[0003] For years, PC users have faced a productivity dilemma of either sacrificing mobility (when choosing a desktop computer) or screen size (when choosing a laptop computer). One partial solution to this problem is to use a docking station. A docking station is an interface device for connecting a laptop computer to other devices. By plugging the laptop computer into the docking station, laptop users can enjoy the improved visibility provided by a larger monitor. However, since the large monitor is stationary, the user's mobility, while improved, is still limited. For example, even a laptop user with a docking station does not have the freedom to use two 32-inch screens at a desired location.

[0004] Some of the disclosed embodiments are directed to a new approach for solving the productivity dilemma, which uses extended reality (XR) to provide a mobile environment that enables a user to experience the comfort of a stationary work space at a desired location by providing a virtual desktop-like screen. SUMMARY OF THE INVENTION

[0005] Embodiments consistent with the present disclosure provide a system, method, and device for providing and supporting productivity applications using an extended reality environment.

[0006] Some of the disclosed embodiments may include an integrated computing interface device, the integrated computing interface device may include a portable housing having a key area and a non-key area, a keyboard associated with the key area of the housing, and a holder associated with the non-key area of the housing. The holder may be configured to selectively engage and disengage with a wearable extended reality device, such that when the wearable extended reality device selectively engages with the housing via the holder, the wearable extended reality device becomes transportable with the housing.

[0007] Some of the disclosed embodiments may include an integrated computing interface device including a housing, at least one image sensor, and a foldable protective cover. The housing may have a key area and a non-key area, and a keyboard associated with the key area. The foldable protective cover incorporates at least one image sensor. The protective cover may be configured to be operable in a plurality of folding configurations including a first folding configuration and a second folding configuration. In the first folding configuration, the protective cover may be configured to accommodate at least a portion of the key area and the non-key area. In the second folding configuration, the protective cover may be configured to stand such that the optical axis of at least one image sensor faces substantially the user of the integrated computing interface device while the user is typing on the keyboard.

[0008] Some of the disclosed embodiments can include a case for an integrated computing interface device that includes at least one image sensor and a foldable protective cover incorporating the at least one image sensor. The protective cover may be configured to be manipulated into a plurality of folded configurations. In a first folded configuration, the protective cover may be configured to house a housing of the integrated computing interface device having a key region and a non-key region. In a second folded configuration, the protective cover may be configured to stand such that, while a user types on a keyboard associated with the key region, the protective cover generally directs an optical axis of the at least one image sensor toward the user of the integrated computing interface device.

[0009] Some of the disclosed embodiments can include a system, method, and non-transitory computer-readable medium for changing a display of virtual content based on temperature. Some of these embodiments include displaying virtual content via a wearable extended reality device, wherein heat is generated by at least one component of the wearable extended reality device during the display of the virtual content; receiving information indicative of a temperature associated with the wearable extended reality device; determining, based on the received information, a need to change a display setting of the virtual content; and changing the display setting of the virtual content to achieve a target temperature, based on the determination.

[0010] Some of the disclosed embodiments may include a system, a method, and a non-transitory computer-readable medium for implementing hybrid virtual keys in an extended reality environment. Some of these embodiments include receiving, during a first period, a first signal corresponding to positions on a touch-sensitive surface of a plurality of virtual activatable elements virtually projected by a wearable extended reality device on the touch-sensitive surface; determining, from the first signal, positions of the plurality of virtual activatable elements on the touch-sensitive surface; receiving a touch input from a user via the touch-sensitive surface, the touch input including a second signal generated as a result of an interaction with at least one sensor within the touch-sensitive surface; determining a coordinate position associated with the touch input based on the second signal generated as a result of an interaction with at least one sensor within the touch-sensitive surface; comparing the coordinate position of the touch input with at least one of the determined positions to identify one of the plurality of virtual activatable elements corresponding to the touch input; and causing a change in virtual content associated with the wearable extended reality device, the change corresponding to the identified one of the plurality of virtual activatable elements.

[0011] Some of the disclosed embodiments can include a system, a method, and a non-transitory computer-readable medium for controlling a virtual display using a combination of a keyboard and a wearable extended reality device. Some of these embodiments include receiving, from a first hand position sensor associated with the wearable extended reality device, a first signal representing movement of a first hand; receiving, from a second hand position sensor associated with the keyboard, a second signal representing movement of a second hand, the movement of the second hand including actions other than an interaction with a feedback component; and controlling the virtual display based on the first signal and the second signal.

[0012] Some of the disclosed embodiments can include a system, method, and non-transitory computer-readable medium for integrating a movable input device with a virtual display projected via a wearable extended reality device. Some of these embodiments include receiving a motion signal associated with the movable input device, where the motion signal reflects a physical movement of the movable input device, and during a first period, outputting a first display signal to the wearable extended reality device, where the first display signal is configured to cause the wearable extended reality device to virtually present content in a first orientation, and during a second period different from the first period, outputting a second display signal to the wearable extended reality device, where the second display signal is configured to cause the wearable extended reality device to virtually present content in a second orientation different from the first orientation, and switching between outputting the first display signal and outputting the second display signal based on the received motion signal of the movable input device.

[0013] Some of the disclosed embodiments may include a system, method, and non-transitory computer-readable medium for virtually extending a physical keyboard. Some of these embodiments include receiving image data from an image sensor associated with a wearable extended reality device, the image data representing a keyboard disposed on a surface; determining that the keyboard is paired with the wearable extended reality device; receiving an input for causing a display of a virtual controller in conjunction with the keyboard; displaying, via the wearable extended reality device, the virtual controller at a first position on the surface, the virtual controller having an original spatial orientation relative to the keyboard at the first position; detecting movement of the keyboard to a different position on the surface; and presenting the virtual controller at a second position on the surface in response to the detected movement of the keyboard, the virtual controller having a subsequent spatial orientation relative to the keyboard at the second position corresponding to the original spatial orientation.

[0014] Some of the disclosed embodiments can include a system, a method, and a non-transitory computer-readable medium for adjusting virtual content display according to a mobility status. Some of these embodiments include accessing rules that associate a plurality of user mobility statuses with a plurality of display modes for presenting virtual content via a wearable extended reality device; receiving first sensor data from at least one sensor associated with the wearable extended reality device, the first sensor data reflecting a user mobility status of the wearable extended reality device during a first period; determining, based on the first sensor data, that the user of the wearable extended reality device is associated with a first mobility status during the first period; implementing at least the first accessed rule to generate a first display of virtual content via the wearable extended reality device associated with the first mobility status; receiving second sensor data from at least one sensor, the second sensor data reflecting a user mobility status during a second period; determining, based on the second sensor data, that the user of the wearable extended reality device is associated with a second mobility status during the second period; and implementing at least the second accessed rule to generate a second display of virtual content via the wearable extended reality device associated with the second mobility status, the second display of virtual content being different from the first display of virtual content.

[0015] Some of the disclosed embodiments can include a system, method, and non-transitory computer-readable medium for changing the display of virtual objects docked to a movable input device. Some of these embodiments include receiving image data from an image sensor associated with a wearable extended reality device, the image data representing an input device disposed at a first position on a support surface; causing the wearable extended reality device to generate a presentation of at least one virtual object proximate to the first position; docking at least one virtual object to the input device; determining that the input device is at a second position on the support surface; in response to determining that the input device is at the second position, updating the presentation of the at least one virtual object such that the at least one virtual object appears proximate to the second position; determining that the input device is at a third position removed from the support surface; and in response to determining that the input device is removed from the support surface, changing the presentation of the at least one virtual object.

[0016] Some of the disclosed embodiments can include a system, method, and non-transitory computer-readable medium for docking virtual objects to a virtual display screen in an extended reality environment. Some of these embodiments include generating virtual content for presentation via a wearable extended reality device, the virtual content including a virtual display and a plurality of virtual objects located outside the virtual display, receiving a selection of at least one virtual object from the plurality of virtual objects, docking the at least one virtual object to the virtual display, after docking the at least one virtual object to the virtual display, receiving an input indicating an intention to change the position of the virtual display without an expression of an intention to move the at least one virtual object, and changing the position of the virtual display in response to the input. Changing the position of the virtual display moves the at least one virtual object together with the virtual display as a result of the docking of the at least one virtual object to the virtual display.

[0017] Some of the disclosed embodiments can include a system, a method, and a non-transitory computer-readable medium for performing selective virtual object display changes. Some of these embodiments include generating an extended reality environment via a wearable extended reality device, the extended reality environment including a first virtual plane associated with a physical object and a second virtual plane associated with an item, the second virtual plane extending in a direction transverse to the first virtual plane; accessing a first instruction for docking a first set of virtual objects at a first position associated with the first virtual plane; accessing a second instruction for docking a second set of virtual objects at a second position associated with the second virtual plane; receiving a first input associated with movement of the physical object; in response to receiving the first input, causing a change in the display of the first set of virtual objects in a manner corresponding to the movement of the physical object while maintaining the second set of virtual objects at the second position; receiving a second input associated with movement of the item; and in response to receiving the second input, causing a change in the display of the second set of virtual objects in a manner corresponding to the movement of the item while maintaining the first position of the first set of virtual objects.

[0018] Some of the disclosed embodiments can include a system, method, and non-transitory computer-readable medium for determining a display configuration for presenting virtual content. Some of these embodiments include receiving image data from an image sensor associated with a wearable extended reality device, the wearable extended reality device being configured to be paired with a plurality of input devices, each input device being associated with a default display setting; analyzing the image data to detect a particular input device disposed on a surface; determining a value of at least one usage parameter for the particular input device; retrieving from memory a default display setting associated with the particular input device; determining a display configuration for presenting virtual content based on the value of the at least one usage parameter and the retrieved default display setting; and causing presentation of the virtual content via the wearable extended reality device according to the determined display configuration.

[0019] Some of the disclosed embodiments can include systems, methods, and non-transitory computer-readable media for extending a physical display with a virtual display. Some of these embodiments include receiving a first signal representing a first object presented entirely on a physical display, receiving a second signal representing a second object having a first portion presented on the physical display and a second portion extending beyond a boundary of the physical display, receiving a third signal representing a third object first presented on the physical display and then moving entirely beyond the boundary of the physical display, in response to receiving the second signal, presenting the second portion of the second object via a wearable extended reality device in a virtual space while the first portion of the second object is presented on the physical display, and in response to receiving the third signal, presenting the third object entirely via a wearable extended reality device in a virtual space following the third object being presented entirely on the physical display.

[0020] Consistent with other disclosed embodiments, a non-transitory computer-readable storage medium can store program instructions that are executed by at least one processing device to perform any of the methods described herein.

[0021] The foregoing general description and the following detailed description are merely exemplary and explanatory and are not restrictive of the claims.

[0022] The accompanying drawings, incorporated in and constituting a part of this disclosure, illustrate various disclosed embodiments.

Brief Description of the Drawings

[0023]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41

Figure 42

Figure 43

Figure 44

Figure 45A

Figure 45B

Figure 46

Figure 47

Figure 48

Figure 49

Figure 50

Figure 51

Figure 52

Figure 53

Figure 54

Figure 55

Figure 56

Figure 57

Figure 58

Figure 59A

Figure 59B

Figure 60A

Figure 60B

Figure 61A

Figure 61B

Figure 62A

Figure 62B

Figure 63

Figure 64

Figure 65

Figure 66

Figure 67

Figure 68

Figure 69

Figure 70

Figure 71

Figure 72

Figure 73

Figure 74

Figure 75

Figure 76

DETAILED DESCRIPTION OF THE INVENTION

[0024] The following detailed description refers to the accompanying drawings. Whenever possible, the same or similar parts are designated by the same reference numerals in the drawings and the following description. Although several exemplary embodiments are described herein, modifications, adaptations, and other embodiments are possible. For example, substitutions, additions, or modifications can be made to the components shown in the drawings, and the exemplary methods described herein can be modified by substituting, rearranging, deleting, or adding steps to the disclosed methods. Accordingly, the following detailed description is not limited to specific embodiments and examples, and includes the general principles described herein and shown in the drawings, in addition to the general principles encompassed by the appended claims.

[0025] The present disclosure relates to systems and methods for providing an extended reality environment to a user. The term "extended reality environment," which may also be referred to as an "extended reality space," "extended reality," or "augmented environment," refers to all types of reality-virtual composite environments and human-machine interactions that are at least partially generated by computer technology. The extended reality environment may be a fully simulated virtual environment or a combination of a real environment and a virtual environment that a user can perceive from different perspectives. In some examples, the user can interact with elements of the extended reality environment. One non-limiting example of an extended reality environment may be a virtual reality environment, also known as "virtual reality" or "virtual environment." An immersive virtual reality environment may be a simulated non-physical environment that provides the user with the perception of being present in the virtual environment. Another non-limiting example of an extended reality environment may be an augmented reality environment, also known as "augmented reality" or "augmented environment." An augmented reality environment may include a direct or indirect view of the live physical real-world environment enhanced with virtual computer-generated perceptual information such as virtual objects with which the user can interact. Another non-limiting example of an extended reality environment is a mixed reality environment, also known as "mixed reality" or "mixed environment." A mixed reality environment may be a hybrid of a physical real-world environment and a virtual environment, in which physical objects and virtual objects coexist and can interact in real time. In some examples, both augmented reality environments and mixed reality environments can include a combination of the real world and the virtual world, real-time interaction, and accurate 3D registration of virtual objects and real objects. In some examples, both augmented reality environments and mixed reality environments may include constructive overlaid sensory information that can be added to the physical environment. In other examples, both augmented reality environments and mixed reality environments may include destructive virtual content that can mask at least a portion of the physical environment.

[0026] In some embodiments, the system and method can provide an extended reality environment using extended reality devices. The term "extended reality device" can include any type of device or system that enables a user to perceive and / or interact with an extended reality environment. An extended reality device can enable a user to perceive and / or interact with an extended reality environment through one or more sensory modalities. Some non-limiting examples of such sensory modalities may include vision, hearing, touch, proprioception, and smell. An example of an extended reality device is a virtual reality device that enables a user to perceive and / or interact with a virtual reality environment. Another example of an extended reality device is an augmented reality device that enables a user to perceive and / or interact with an augmented reality environment. Yet another example of an extended reality device is a mixed reality device that enables a user to perceive and / or interact with a mixed reality environment.

[0027] According to one aspect of the present disclosure, the extended reality device may be a wearable device such as a head-mounted device, for example, smart glasses, smart contact lenses, a headset, or any other device that a human wears for the purpose of presenting extended reality to the human. Other extended reality devices may include a holographic projector, or any other device or system capable of providing augmented reality (AR), virtual reality (VR), mixed reality (MR), or any immersive experience. Typical components of a wearable extended reality device may include a stereoscopic head-mounted display, a stereoscopic head-mounted sound system, head motion tracking sensors (e.g., gyroscopes, accelerometers, magnetometers, image sensors, structured light sensors, etc.), a head-mounted projector, eye tracking sensors, and at least one of the additional components described below. According to another aspect of the present disclosure, the extended reality device may be a non-wearable extended reality device. Specifically, the non-wearable extended reality device may include a multi-projection environment device. In some embodiments, the extended reality device may be configured to change the viewing perspective of the extended reality environment in response to the movement of the user, particularly the movement of the user's head. In one example, a wearable extended reality device may change the viewing field of the extended reality environment in response to a change in the user's head pose, such as by changing the spatial orientation without changing the user's spatial position in the extended reality environment. In another example, a non-wearable extended reality device may change the user's spatial position in the extended reality environment in response to a change in the user's position in the real world, such as by changing the user's spatial position in the extended reality environment without changing the direction of the viewing field with respect to the spatial position.

[0028] According to some embodiments, the extended reality device may include a digital communication device configured to perform at least one of receiving virtual content data configured to enable presentation of virtual content, transmitting virtual content for sharing with at least one external device, receiving context data from at least one external device, transmitting context data to at least one external device, transmitting usage data indicating use of the extended reality device, and transmitting data based on information captured using at least one sensor included in the extended reality device. In additional embodiments, the extended reality device may include a memory for storing at least one of virtual data configured to enable presentation of virtual content, context data, usage data indicating use of the extended reality device, sensor data based on information captured using at least one sensor included in the extended reality device, software instructions configured to cause a processing device to present virtual content, software instructions configured to cause the processing device to collect and analyze context data, software instructions configured to cause the processing device to collect and analyze usage data, and software instructions configured to cause the processing device to collect and analyze sensor data. In additional embodiments, the extended reality device may include a processing device configured to perform at least one of rendering virtual content, collecting and analyzing context data, collecting and analyzing usage data, and collecting and analyzing sensor data. In additional embodiments, the extended reality device may include one or more sensors.One or more sensors can include one or more image sensors (e.g., configured to capture images and / or videos of the user of the device or the user's environment), one or more motion sensors (e.g., accelerometers, gyroscopes, magnetometers, etc.), one or more positioning sensors (e.g., GPS, outdoor positioning sensors, indoor positioning sensors, etc.), one or more temperature sensors (e.g., configured to measure the temperature of at least a part of the device and / or the environment), one or more contact sensors, one or more proximity sensors (e.g., configured to detect whether the device is currently being worn), one or more electrical impedance sensors (e.g., configured to measure the electrical impedance of the user), one or more eye-tracking sensors, such as eye detectors, optical trackers, potential trackers (e.g., electrooculogram (EOG) sensors), video-based eye trackers, infrared / near-infrared sensors, passive light sensors, or any other technology that can determine where a person is looking or gazing.

[0029] In some embodiments, the system and method can use an input device to interact with an extended reality device. The term input device can include any physical device configured to receive input from a user or the user's environment and provide data to a computing device. The data provided to the computing device may be in digital and / or analog form. In one embodiment, the input device can store input received from the user in a memory device accessible by a processing device, and the processing device can access the data stored for analysis. In another embodiment, the input device can directly provide data to the processing device, for example, via a bus or via another communication system configured to transfer data from the input device to the processing device. In some examples, the input received by the input device can include key presses, tactile input data, motion data, position data, gesture-based input data, direction data, or any other data for providing for computation. Some examples of input devices can include buttons, keys, keyboards, computer mice, touch pads, touch screens, joysticks, or another mechanism capable of receiving input. Another example of an input device can include an integrated computing interface device that includes at least one physical component for receiving input from a user. The integrated computing interface device can include at least memory, a processing device, and at least one physical component for receiving input from a user. In one example, the integrated computing interface device may further include a digital network interface that enables digital communication with other computing devices. In one example, the integrated computing interface device may further include a physical component for outputting information to the user. In some examples, all components of the integrated computing interface device may be included in a single housing, and in other examples, the components may be distributed across two or more housings.Some non - limiting examples of physical components for receiving input from a user that can be included in an integrated computing interface device can include at least one of buttons, keys, keyboards, touchpads, touchscreens, joysticks, or any other mechanism or sensor capable of receiving computing information. Some non - limiting examples of physical components for outputting information to a user can include at least one of optical indicators (such as LED indicators), screens, touchscreens, buzzers, voice speakers, or any other audio, video, or tactile device that can provide a human - perceivable output.

[0030] In some embodiments, one or more image sensors can be used to capture image data. In some examples, the image sensor may be included in an extended reality device, a wearable device, a wearable extended reality device, an input device, the user's environment, etc. In some examples, the image data may be read from memory, received from an external device, or generated (e.g., using a generative model), etc. Some non - limiting examples of image data can include images, grayscale images, color images, 2D images, 3D images, videos, 2D videos, 3D videos, frames, images, data derived from other image data, etc. In some examples, the image data may be encoded in any analog or digital format. Some non - limiting examples of such formats can include raw formats, compressed formats, uncompressed formats, irreversible formats, reversible formats, JPEG, GIF, PNG, TIFF, BMP, NTSC, PAL, SECAM, MPEG, MPEG - 4 Part 14, MOV, WMV, FLV, AVI, AVCHD, WebM, MKV, etc.

[0031] In some embodiments, the extended reality device can receive a digital signal from, for example, an input device. The term digital signal refers to a discrete series of digital values over time. The digital signal can represent, for example, sensor data, text data, audio data, video data, virtual data, or any other form of data that provides perceptible information. Consistent with the present disclosure, the digital signal can be configured to cause the extended reality device to present virtual content. In one embodiment, the virtual content may be presented in a selected orientation. In this embodiment, the digital signal can indicate the position and angle of the viewpoint in an environment such as an extended reality environment. Specifically, the digital signal can include an encoding of the position and angle in six degrees of freedom coordinates (e.g., forward / backward, up / down, left / right, yaw, pitch, and roll). In another embodiment, the digital signal can include an encoding of the position as three-dimensional coordinates (e.g., x, y, and z), and an encoding of the angle as a vector resulting from the encoded position. Specifically, the digital signal can indicate the orientation and angle of the presented virtual content in the absolute coordinates of the environment, for example, by encoding the yaw, pitch, and roll of the virtual content relative to a standard default angle. In another embodiment, the digital signal can indicate the orientation and angle of the presented virtual content relative to the viewpoint of another object (e.g., a virtual object, a physical object, etc.), for example, by encoding the yaw, pitch, and roll of the virtual content relative to the direction corresponding to the viewpoint or the direction corresponding to another object. In another embodiment, such a digital signal can include one or more projections of the virtual content, for example, in a format (e.g., an image, a video, etc.) that is ready for presentation. For example, each such projection can correspond to a specific orientation or a specific angle. In another embodiment, the digital signal can include a representation of the virtual content, for example, by encoding an object in a three-dimensional array of voxels, a polygon mesh, or any other format that can present the virtual content.

[0032] In some embodiments, the digital signal may be configured to cause an extended reality device to present virtual content. The term virtual content can include any type of data representation that can be presented to a user by an extended reality device. Virtual content can include virtual objects, inanimate virtual content, animate virtual content configured to change over time or in response to a trigger, virtual two-dimensional content, virtual three-dimensional content, virtual overlays on a portion of a physical environment or a physical object, virtual additions to a physical environment or a physical object, virtual promotional content, virtual representations of physical objects, virtual representations of physical environments, virtual documents, virtual characters or personas, virtual computer screens, virtual widgets, or any other format for presenting information virtually. Consistent with the present disclosure, virtual content can include any visual presentation rendered by a computer or processing device. In one embodiment, the virtual content includes virtual objects that are visual presentations rendered by a computer within a limited area and configured to represent a particular type of object (e.g., inanimate virtual objects, animate virtual objects, virtual furniture, virtual decorative objects, virtual widgets, or other virtual representations, etc.). The rendered visual presentation can be changed, for example, to mimic a change in the appearance of a physical object, to reflect a change to a state object, or to change the viewing angle of the object. In another embodiment, the virtual content can include a virtual display (also referred to herein as a "virtual display screen" or "virtual screen"), such as a virtual computer screen, a virtual tablet screen, or a virtual smartphone screen, configured to display information generated by an operating system, and the operating system can be configured to receive text data from a physical keyboard and / or a virtual keyboard and cause the display of text content on the virtual display screen. In one example, as shown in FIG. 1, the virtual content may include a virtual environment that includes a virtual computer screen and a plurality of virtual objects.In some examples, the virtual display may be a virtual object that mimics and / or extends the functionality of a physical display screen. For example, the virtual display may be presented in an extended reality environment (e.g., a mixed reality environment, an augmented reality environment, a virtual reality environment, etc.) using an extended reality device. In one example, the virtual display can present content generated by a normal operating system that can be presented equally on a physical display screen. In one example, text content input using a keyboard (e.g., using a physical keyboard, using a virtual keyboard, etc.) can be presented on the virtual display in real time as the text content is typed. In one example, a virtual cursor may be presented on the virtual display, and the virtual cursor may be controlled by a pointing device (a physical pointing device, a virtual pointing device, a computer mouse, a joystick, a touchpad, a physical touch controller, etc.). In one example, one or more windows of a graphical user interface operating system can be presented on the virtual display. In another example, the content presented on the virtual display may be interactive, i.e., the reaction to the user's action may change. In yet another example, the presentation of the virtual display may or may not include the presentation of a screen frame.

[0033] Some of the disclosed embodiments include a data structure or database and / or can access a data structure or database. The terms data structure and database consistent with this disclosure can include any collection of data values and the relationships between them. The data can be stored linearly, horizontally, hierarchically, relationally, non-relationally, unidimensionally, multi-dimensionally, operationally, in an ordered manner, in an unordered manner, object-oriented, centrally, non-centrally, distributively, customarily, or in any manner that enables data access. By way of non-limiting example, a data structure can include an array, an associative array, a linked list, a binary tree, a balanced tree, a heap, a stack, a queue, a set, a hash table, a record, a tagged union, an entity-relationship model, a graph, a hypergraph, a matrix, a tensor, etc. For example, a data structure can include an XML database, an RDBMS database, a SQL database, or a NoSQL alternative for data storage / search, such as MongoDB, Redis, Couchbase, Datastax Enterprise Graph, Elastic Search, Splunk, Solr, Cassandra, Amazon DynamoDB, Scylla, HBase, and Neo4J. A data structure can be a component of the disclosed system or a remote computing component (e.g., a cloud-based data structure). The data within a data structure can be stored in contiguous or non-contiguous memory. Further, a data structure does not require that information be located in the same place. This can be distributed, for example, across multiple servers that can be owned or operated by the same or different entities. Thus, the term data structure in the singular includes multiple data structures.

[0034] In some embodiments, the system can determine a reliability level for the received input or any determined value. The term "reliability level" refers to a numerical value or any other indication of a level (e.g., within a predetermined range) that indicates the amount of reliability the system has in the determined data. For example, the confidence level may have a value between 1 and 10. Alternatively, the confidence level may be expressed as a percentage or any other numerical or non-numerical indication. In some cases, the system can compare the reliability level to a threshold. The term "threshold" can indicate a reference value, level, point, or range of values. During operation, if the reliability level of the determined data exceeds (or, depending on the particular use case, is below) the threshold, the system can follow a first course of action, and if the reliability level is below (or, depending on the particular use case, exceeds) it, the system can follow a second course of action. The value of the threshold may be predetermined for each type of object being investigated or may be dynamically selected based on different considerations.

[0035] System Overview Referring now to FIG. 1, which shows a user using an exemplary extended reality system consistent with various embodiments of the present disclosure. It should be understood that FIG. 1 is a typical representation of only one embodiment, and that some of the illustrated elements may be omitted and other elements may be added within the scope of the present disclosure. As shown, user 100 is sitting behind table 102 and is supporting keyboard 104 and mouse 106. Keyboard 104 is connected by wire 108 to a wearable extended reality device 110 that displays virtual content to user 100. Instead of or in addition to wire 108, keyboard 104 can be connected wirelessly to wearable extended reality device 110. For illustrative purposes, the wearable extended reality device is shown as a pair of smart glasses, but as described above, wearable extended reality device 110 can be any type of head-mounted device used to present extended reality to user 100. The virtual content displayed by wearable extended reality device 110 includes virtual screen 112 (also referred to herein as the "virtual display screen" or "virtual display") and a plurality of virtual widgets 114. Virtual widgets 114A - 114D are displayed adjacent to virtual screen 112, and virtual widget 114E is displayed on table 102. User 100 can use keyboard 104 to enter text into document 116 displayed on virtual screen 112 and can use mouse 106 to control virtual cursor 118. In one example, virtual cursor 118 can move anywhere within virtual screen 112. In another example, virtual cursor 118 can move anywhere within virtual screen 112 and can also move to any of virtual widgets 114A - 114D, but cannot move to virtual widget 114E. In yet another example, virtual cursor 118 can move anywhere within virtual screen 112 and can also move to any of virtual widgets 114A - 114E.In an additional example, the virtual cursor 118 can move anywhere in the extended reality environment including the virtual screen 112 and the virtual widgets 114A - 114E. In yet another example, the virtual cursor can move only on all available surfaces (i.e., virtual or physical surfaces) or on selected surfaces within the extended reality environment. Alternatively or additionally, the user 100 can use hand gestures recognized by the wearable extended reality device 110 to interact with any of the virtual widgets 114A - 114E or a selected virtual widget. For example, the virtual widget 114E can be an interactive widget (e.g., a virtual slider controller) that can be operated with hand gestures.

[0036] FIG. 2 shows an example of a system 200 that provides an extended reality (XR) experience to a user such as user 100. It should be understood that FIG. 2 is a typical representation of just one embodiment, and within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added. System 200 may be computer-based and may include computer system components, wearable devices, workstations, tablets, handheld computing devices, memory devices, and / or an internal network connecting the components. System 200 may include or be connected to various network computing resources (e.g., servers, routers, switches, network connections, storage devices, etc.) to support the services provided by system 200. Consistent with the present disclosure, system 200 may include an input unit 202, an XR unit 204, a mobile communication device 206, and a remote processing unit 208. The remote processing unit 208 may include a server 210 coupled to one or more physical or virtual storage devices such as data structure 212. System 200 may also include or be connected to a communication network 214 that facilitates communication and data exchange between different system components and different entities associated with system 200.

[0037] In accordance with the present disclosure, the input unit 202 can include one or more devices capable of receiving input from the user 100. In one embodiment, the input unit 202 can include a text input device such as a keyboard 104. The text input device can include all possible types of devices and mechanisms for inputting text information into the system 200. Examples of text input devices can include mechanical keyboards, membrane keyboards, flexible keyboards, QWERTY keyboards, Dvorak keyboards, Colemak keyboards, coded keyboards, wireless keyboards, keypads, key-based control panels, or other arrangements of control keys, visual input devices, or any other mechanism for inputting text (regardless of whether the mechanism is provided in a physical form or presented virtually). In one embodiment, the input unit 202 can also include a pointing input device such as a mouse 106. The pointing input device can include all possible types of devices and mechanisms for inputting two-dimensional or three-dimensional information into the system 200. In one example, two-dimensional input from the pointing input device may be used to interact with virtual content presented via the XR unit 204. Examples of pointing input devices can include computer mice, trackballs, touchpads, trackpads, touchscreens, joysticks, pointing sticks, styli, light pens, or any other physical or virtual input mechanism. In one embodiment, the input unit 202 can also include a graphical input device such as a touch screen configured to detect contact, movement, or interruption of movement. The graphical input device can use any of a plurality of touch sensitivity technologies including, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more contact points. In one embodiment, the input unit 202 can also include one or more voice input devices such as a microphone.The voice input device can include all possible types of devices and mechanisms for inputting voice data to facilitate voice-related functions such as voice recognition, voice replication, digital recording, and telephone functions. In one embodiment, the input unit 202 may also include one or more image input devices, such as an image sensor, configured to capture image data. In one embodiment, the input unit 202 may also include one or more tactile gloves configured to capture hand movement and gesture data. In one embodiment, the input unit 202 may also include one or more proximity sensors configured to detect the presence and / or movement of objects within a selected area near the sensor.

[0038] According to some embodiments, the system can include at least one sensor configured to detect and / or measure characteristics associated with a user, the user's actions, or the user's environment. An example of the at least one sensor is sensor 216 included in input unit 202. Sensor 216 can be a motion sensor, a touch sensor, an optical sensor, an infrared sensor, an audio sensor, an image sensor, a proximity sensor, a positioning sensor, a gyroscope, a temperature sensor, a biometric sensor, or any other sensing device to facilitate related functions. Sensor 216 can be integrated with the input device, connected to the input device, or separated from the input device. In one example, a thermometer can be included in mouse 106 to determine the body temperature of user 100. In another example, a positioning sensor can be integrated with keyboard 104 to determine the movement of user 100 with respect to keyboard 104. Such a positioning sensor can be implemented using one of Global Positioning System (GPS), GLObal NAvigation Satellite System (GLONASS), Galileo Global Navigation Satellite System, BeiDou Navigation Satellite System, other Global Navigation Satellite Systems (GNSS), Indian Regional Navigation Satellite System (IRNSS), Local Positioning System (LPS), Real-Time Location System (RTLS), Indoor Positioning System (IPS), Wi-Fi based positioning system, cellular triangulation, image-based positioning technology, indoor positioning technology, outdoor positioning technology, or any other positioning technology.

[0039] According to some embodiments, the system can include one or more sensors for identifying the position and / or movement of physical devices (such as physical input devices, physical computing devices, keyboard 104, mouse 106, wearable extended reality device 110, etc.). The one or more sensors may be included in the physical device or may be external to the physical device. In some examples, an image sensor external to the physical device (e.g., an image sensor included in another physical device) can be used to capture image data of the physical device, and the image data can be analyzed to identify the position and / or movement of the physical device. For example, the image data may be analyzed using a visual object tracking algorithm to identify the movement of the physical device, and may be analyzed using a visual object detection algorithm to identify the position of the physical device (e.g., relative to the image sensor in a global coordinate system, etc.). In some examples, an image sensor included in the physical device can be used to capture image data, and the image data can be analyzed to identify the position and / or movement of the physical device. For example, the image data may be analyzed using a visual odometry algorithm to identify the position of the physical device, and may be analyzed using an egomotion algorithm to identify the movement of the physical device, etc. In some examples, a positioning sensor such as an indoor positioning sensor or an outdoor positioning sensor may be included in the physical device and used to determine the position of the physical device. In some examples, a motion sensor such as an accelerometer or a gyroscope may be included in the physical device and used to determine the movement of the physical device. In some examples, a physical device such as a keyboard or a mouse may be configured to be placed on a physical surface. Such a physical device can include an optical mouse sensor (also known as a non-mechanical tracking engine) directed towards the physical surface, and the output of the optical mouse sensor can be analyzed to determine the movement of the physical device relative to the physical surface.

[0040] In accordance with the present disclosure, the XR unit 204 may include a wearable extended reality device configured to present virtual content to the user 100. An example of a wearable extended reality device is the wearable extended reality device 110. Further examples of wearable extended reality devices can include virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices, or any other device capable of generating extended reality content. Some non-limiting examples of such devices can include Nreal Light, Magic Leap One, Varjo, Quest 1 / 2, Vive, and the like. In some embodiments, the XR unit 204 can present virtual content to the user 100. Generally, extended reality devices can include all real and virtual composite environments generated by computer technology and wearability, as well as human-machine interactions. As described above, the term "extended reality" (XR) refers to a superset that includes the entire spectrum from "complete reality" to "complete virtuality". It includes representative forms such as augmented reality (AR), mixed reality (MR), virtual reality (VR), and the areas interpolated between them. Therefore, it should be noted that the terms "XR device", "AR device", "VR device", and "MR device" can be used interchangeably herein and can refer to any of the various devices listed above.

[0041] In accordance with the present disclosure, the system can exchange data with various communication devices associated with a user, such as mobile communication device 206. The term "communication device" is intended to include all possible types of devices that can exchange data using a digital communication network, an analog communication network, or any other communication network configured to transmit data. In some examples, the communication device may include smartphones, tablets, smartwatches, personal digital assistants, desktop computers, laptop computers, IoT devices, dedicated terminals, wearable communication devices, and any other device that enables data communication. In some cases, mobile communication device 206 can supplement or replace input unit 202. Specifically, mobile communication device 206 may be associated with a physical touch controller that can function as a pointing input device. Additionally, mobile communication device 206 may also be used, for example, to implement a virtual keyboard and replace a text input device. For example, if user 100 leaves table 102 and walks into the break room with their smart glasses, they can receive an email that requires a quick response. In this case, the user can choose to use their smartwatch as an input device and type a response to the email while it is being virtually presented by the smart glasses.

[0042] In accordance with the present disclosure, embodiments of the system can include the use of a cloud server. The term "cloud server" refers to a computer platform that provides services via a network such as the Internet. In the exemplary embodiment shown in FIG. 2, the server 210 can use virtual machines that may not correspond to individual hardware. For example, computing and / or memory capabilities may be implemented by allocating an appropriate portion of the desired computing / memory power from a scalable repository such as a data center or a distributed computing environment. Specifically, in one embodiment, the remote processing unit 208 can be used with the XR unit 204 to provide virtual content to the user 100. In one configuration example, the server 210 may be a cloud server that functions as an operating system (OS) of the wearable extended reality device. In one example, the server 210 can implement the methods described herein using custom hardwired logic, one or more application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), firmware, and / or program logic that combines with a computer system to make the server 210 a dedicated machine.

[0043] In some embodiments, server 210 can access data structure 212 to determine, for example, virtual content for displaying user 100. Data structure 212 can utilize volatile or non-volatile, magnetic, semiconductor, tape, optical, removable, non-removable, other types of storage devices or tangible or non-transitory computer-readable media, or any medium or mechanism for storing information. As shown, data structure 212 may be part of server 210 or may be separate from server 210. If data structure 212 is not part of server 210, server 210 can exchange data with data structure 212 via a communication link. Data structure 212 can include one or more memory devices that store data and instructions used to execute one or more features of the disclosed method. In one embodiment, data structure 212 can include any of a plurality of suitable data structures, from small data structures hosted on a workstation to large data structures distributed among data centers. Data structure 212 can also include any combination of one or more data structures controlled by a memory controller device (e.g., a server) or software.

[0044] In accordance with the present disclosure, a communication network can be any type of network (including infrastructure) that supports communication, exchanges information, and / or facilitates the exchange of information between components of a system. For example, communication network 214 within system 200 can include, for example, a telephone network, an extranet, an intranet, the Internet, satellite communication, offline communication, wireless communication, transponder communication, a local area network (LAN), a wireless network (e.g., a Wi-Fi / 802.11 network), a wide area network (WAN), a virtual private network (VPN), a digital communication network, an analog communication network, or any other mechanism or combination of mechanisms that enables data transmission.

[0045] The components and arrangements of the system 200 shown in FIG. 2 are intended to be illustrative only, as system components used to implement the disclosed processes and features may vary, and are not intended to limit any embodiments.

[0046] FIG. 3 is a block diagram showing a configuration example of the input unit 202. It should be understood that FIG. 3 is a typical representation of just one embodiment, and within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added. In the embodiment of FIG. 3, the input unit 202 can directly or indirectly access a bus 300 (or other communication mechanism) that interconnects subsystems and components for transferring information within the input unit 202. For example, the bus 300 can interconnect a memory interface 310, a network interface 320, an input interface 330, a power supply 340, an output interface 350, a processing device 360, a sensor interface 370, and a database 380.

[0047] The memory interface 310 shown in FIG. 3 can be used to access software products and / or data stored in a non-transitory computer-readable medium. Generally, a non-transitory computer-readable storage medium refers to any type of physical memory that can store information or data readable by at least one processor. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, any other optical data storage medium, any physical medium having a pattern of holes, PROM, EPROM, FLASH-EPROM or any other flash memory, NVRAM, cache, registers, any other memory chip or cartridge, and network versions thereof. The terms “memory” and “computer-readable storage medium” can refer to multiple structures such as multiple memories or computer-readable storage media located within the input unit or at a remote location. Further, one or more computer-readable storage media can be utilized when implementing a method implemented by a computer. Thus, the term computer-readable storage medium should be understood to include tangible items and to exclude carrier waves and transient signals. In the particular embodiment shown in FIG. 3, the memory interface 310 can be used to access software products and / or data stored in a memory device such as the memory device 311. The memory device 311 may include high-speed random access memory and / or non-volatile memory, such as one or more magnetic disk storage devices, one or more optical storage devices, and / or flash memory (e.g., NAND, NOR). Consistent with the present disclosure, the components of the memory device 311 may be distributed across two or more units of the system 200 and / or two or more memory devices.

[0048] The memory device 311 shown in FIG. 3 can include software modules for executing processes consistent with the present disclosure. In particular, the memory device 311 can include an input determination module 312, an output determination module 313, a sensor communication module 314, a virtual content determination module 315, a virtual content communication module 316, and a database access module 317. Modules 312 - 317 can include software instructions for execution by at least one processor (e.g., processing device 360) associated with the input unit 202. The input determination module 312, the output determination module 313, the sensor communication module 314, the virtual content determination module 315, the virtual content communication module 316, and the database access module 317 can cooperate to perform various operations. For example, the input determination module 312 may determine text using data received from, for example, the keyboard 104. Thereafter, the output determination module 313 can cause the most recently input text to be presented on, for example, a dedicated display 352 physically or wirelessly coupled to the keyboard 104. In this way, when the user 100 inputs, a preview of the typed text can be viewed without constantly moving the head up and down to view the virtual screen 112. The sensor communication module 314 can receive data from different sensors to determine the state of the user 100. Thereafter, the virtual content determination module 315 can determine the virtual content to be displayed based on the received input and the determined status of the user 100. For example, the determined virtual content may be a virtual presentation of the most recently input text on a virtual screen disposed virtually adjacent to the keyboard 104. The virtual content communication module 316 may obtain virtual content not determined by the virtual content determination module 315 (e.g., an avatar of another user). The search for virtual content may be from the database 380, from the remote processing unit 208, or from any other source.

[0049] In some embodiments, the input determination module 312 can adjust the operation of the input interface 330 to receive pointer input 331, text input 332, voice input 333, and XR-related input 334. Details of pointer input, text input, and voice input have been described above. The term "XR-related input" can include any type of data that may cause a change in the virtual content displayed to the user 100. In one embodiment, the XR-related input 334 may include image data of the user 100, a wearable extended reality device (e.g., a detected hand gesture of the user 100). In another embodiment, the XR-related input 334 may include wireless communication indicating the presence of another user in proximity to the user 100. Consistent with the present disclosure, the input determination module 312 can receive different types of input data simultaneously. Thereafter, the input determination module 312 can further apply different rules based on the type of input detected. For example, pointer input may be prioritized over voice input.

[0050] In some embodiments, the output determination module 313 can adjust the operation of the output interface 350 to generate an output using the optical indicator 351, the display 352, and / or the speaker 353. Generally, the output generated by the output determination module 313 does not include virtual content presented by the wearable extended reality device. Instead, the output generated by the output determination module 313 includes various outputs related to the operation of the input unit 202 and / or the XR unit 204. In one embodiment, the optical indicator 351 can include an optical indicator indicating the state of the wearable extended reality device. For example, the optical indicator can display green light when the wearable extended reality device 110 is connected to the keyboard 104, and can blink when the battery of the wearable extended reality device 110 is low. In another embodiment, the display 352 can be used to display operation information. For example, the display can present an error message when the wearable extended reality device is inoperable. In another embodiment, the speaker 353 can be used to output sound, for example, when the user 100 wants to play music for another user.

[0051] In some embodiments, the sensor communication module 314 can adjust the operation of the sensor interface 370 to receive sensor data from one or more sensors integrated with or connected to the input device. The one or more sensors can include an audio sensor 371, an image sensor 372, a motion sensor 373, an environmental sensor 374 (e.g., a temperature sensor, an ambient light detector, etc.), and other sensors 375. In one embodiment, the physical orientation of the input device can be determined using the data received from the sensor communication module 314. The physical orientation of the input device can indicate the user's state and can be determined based on a combination of tilt motion, roll motion, and lateral motion. Thereafter, the physical orientation of the input device can be used by the virtual content determination module 315 to modify the display parameters of the virtual content to match the user's state (e.g., attention, sleepy, active, sitting, standing, leaning back, leaning forward, walking, moving, riding, etc.).

[0052] In some embodiments, the virtual content determination module 315 can determine the virtual content to be displayed by the wearable extended reality device. The virtual content can be determined based on data from the input determination module 312, the sensor communication module 314, and other sources (e.g., the database 380). In some embodiments, determining the virtual content can include determining the distance, size, and orientation of the virtual object. The determination of the position of the virtual object may be determined based on the type of the virtual object. Specifically, with respect to the example shown in FIG. 1, since the virtual widget 114E is a virtual controller (e.g., a volume bar), the virtual content determination module 315 can determine to arrange four virtual widgets 114A-114D on both sides of the virtual screen 112 and arrange the virtual widget 114E on the table 102. The determination of the position of the virtual object may be further determined based on the user's preference. For example, for a left-handed user, the virtual content determination module 315 can determine to arrange the virtual volume bar to the left of the keyboard 104, and for a right-handed user, the virtual content determination module 315 can determine to arrange the virtual volume bar to the right of the keyboard 104.

[0053] In some embodiments, the virtual content communication module 316 may adjust the operation of the network interface 320 to obtain data from one or more sources to be presented to the user 100 as virtual content. The one or more sources may include other XR units 204, the user's mobile communication device 206, remote processing units 208, publicly available information, and the like. In one embodiment, the virtual content communication module 316 can communicate with the mobile communication device 206 to provide a virtual representation of the mobile communication device 206. For example, the virtual representation can enable the user 100 to read messages and interact with applications installed on the mobile communication device 206. The virtual content communication module 316 can also adjust the operation of the network interface 320 to share virtual content with other users. In one example, the virtual content communication module 316 uses data from the input determination module to identify a trigger (e.g., the trigger may include a user gesture) and transfer the content from the virtual display to a physical display (e.g., a TV) or a virtual display of a different user.

[0054] In some embodiments, the database access module 317 can cooperate with the database 380 to retrieve stored data. The retrieved data can include, for example, privacy levels associated with different virtual objects, relationships between virtual objects and physical objects, user preferences, the user's past behavior, and the like. As described above, the virtual content determination module 315 can use the data stored in the database 380 to determine virtual content. The database 380 can include separate databases such as, for example, a vector database, a raster database, a tile database, a viewport database, and / or a user input database. The data stored in the database 380 may be received from the modules 314-317 or other components of the system 200. Further, the data stored in the database 380 may be provided as input using data input, data transfer, or data upload.

[0055] Modules 312 - 317 may be implemented in software, hardware, firmware, or any combination thereof. In some embodiments, any one or more of modules 312 - 317 and the data associated with database 380 may be stored in XR unit 204, mobile communication device 206, or remote processing unit 208. The processing device of system 200 may be configured to execute the instructions of modules 312 - 317. In some embodiments, aspects of modules 312 - 317 may be executable by one or more processors, alone or in various combinations with each other, in hardware, software (including one or more signal processing and / or application specific integrated circuits), firmware, or any combination thereof. Specifically, modules 312 - 317 may be configured to interact with each other and / or with other modules of system 200 to perform functions consistent with some of the disclosed embodiments. For example, input unit 202 may execute instructions including an image processing algorithm on data from XR unit 204 to determine the movement of user 100's head. Further, throughout this specification, each function described with respect to input unit 202, or with respect to the components of input unit 202, may correspond to a set of instructions for performing the function. These instructions need not be implemented as separate software programs, procedures, or modules. Memory device 311 may include additional modules and instructions or fewer modules and instructions. For example, memory device 311 may store an operating system such as ANDROID, iOS, UNIX, OSX, WINDOWS, DARWIN, RTXC, LINUX, or an embedded operating system such as VXWorkS. The operating system may process basic system services and may include instructions for performing hardware - dependent tasks.

[0056] The network interface 320 shown in FIG. 3 can provide bidirectional data communication to a network such as the communication network 214. In one embodiment, the network interface 320 can include an Integrated Services Digital Network (ISDN) card, a cellular modem, a satellite modem, or a modem to provide a data communication connection via the Internet. As another example, the network interface 320 can include a Wireless Local Area Network (WLAN) card. In another embodiment, the network interface 320 can include an Ethernet port connected to a radio frequency receiver and transmitter and / or an optical (e.g., infrared) receiver and transmitter. The specific design and implementation of the network interface 320 can depend on the communication network in which the input unit 202 is intended to operate. For example, in some embodiments, the input unit 202 can include a network interface 320 designed to operate via a GSM network, a GPRS network, an EDGE network, a Wi-Fi or WiMax network, and a Bluetooth network. In any such embodiment, the network interface 320 may be configured to transmit and receive electrical, electromagnetic, or optical signals that carry digital data streams or digital signals representing various types of information.

[0057] The input interface 330 shown in FIG. 3 can receive input from various input devices, such as a keyboard, a mouse, a touchpad, a touch screen, one or more buttons, a joystick, a microphone, an image sensor, and any other device configured to detect physical or virtual input. The received input may be in at least one form of text, voice, sound, hand gesture, body gesture, tactile information, and any other type of physical or virtual input generated by the user. In the illustrated embodiment, the input interface 330 can receive pointer input 331, text input 332, voice input 333, and XR-related input 334. In an additional embodiment, the input interface 330 may be an integrated circuit that can function as a bridge between the processing device 360 and any of the above input devices.

[0058] The power supply 340 shown in FIG. 3 can supply electrical energy to the power input unit 202 and optionally also supply power to the XR unit 204. In general, the power supply included in any device or system of the present disclosure can be one or more batteries (e.g., lead-acid battery, lithium-ion battery, nickel-metal hydride battery, nickel-cadmium battery), one or more capacitors, one or more connections to an external power source, one or more power converters, or any device capable of repeatedly storing, distributing, or transmitting power, including but not limited to these. Referring to the example shown in FIG. 3, the power supply may be portable, which means that the input unit 202 can be easily carried by hand (e.g., the total weight of the power supply 340 can be less than 1 pound). The portability of the power supply enables the user 100 to use the input unit 202 in various situations. In other embodiments, the power supply 340 may be associated with a connection to an external power source (such as a power grid) that can be used to charge the power supply 340. Further, the power supply 340 may be configured to charge one or more batteries included in the XR unit 204. For example, a pair of extended reality glasses (e.g., wearable extended reality device 110) can be charged (e.g., wirelessly or non-wirelessly) when they are placed on or near the input unit 202.

[0059] The output interface 350 shown in FIG. 3 can generate outputs from various output devices, for example, using an optical indicator 351, a display 352, and / or a speaker 353. In one embodiment, the output interface 350 may be an integrated circuit that can function as a bridge between the processing device 360 and at least one of the above output devices. The optical indicator 351 can include one or more light sources, such as an LED array associated with different colors, for example. The display 352 can include a screen (e.g., an LCD or dot matrix screen) or a touch screen. The speaker 353 can include audio headphones, a hearing aid type device, a speaker, bone conduction headphones, an interface that provides tactile cues, a vibration stimulation device, and the like.

[0060] The processing device 360 shown in FIG. 3 can include at least one processor configured to execute a computer program, application, method, process, or other software to implement the embodiments described in the present disclosure. Generally, the processing device included in any device or system of the present disclosure can include one or more integrated circuits, microchips, microcontrollers, microprocessors, central processing devices (CPUs), graphics processing devices (GPUs), digital signal processors (DSPs), all or part of a field programmable gate array (FPGA), or other circuits suitable for executing instructions or performing logical operations. The processing device can include at least one processor configured to implement the functions of the disclosed method, such as a microprocessor manufactured by Intel (trademark). The processing device can include a single core or a multi-core processor that simultaneously executes parallel processing. In one example, the processing device can be a single core processor configured with virtual processing technology. The processing device can implement virtual machine technology or other technologies to provide the ability to execute, control, execute, operate, store, etc. multiple software processes, applications, programs, etc. In another example, the processing device can include a multi-core processor configuration (e.g., dual, quad core, etc.) configured to provide a parallel processing function that enables devices associated with the processing device to execute multiple processes simultaneously. It should be understood that other types of processor configurations can be implemented to provide the functions disclosed herein.

[0061] The sensor interface 370 shown in FIG. 3 can acquire sensor data from various sensors, such as a voice sensor 371, an image sensor 372, a motion sensor 373, an environmental sensor 374, and other sensors 375. In one embodiment, the sensor interface 370 can be an integrated circuit that can function as a bridge between the processing device 360 and at least one of the above sensors.

[0062] The audio sensor 371 may include one or more audio sensors configured to capture audio by converting the audio into digital information. Some examples of audio sensors can include microphones, unidirectional microphones, bidirectional microphones, cardioid microphones, omnidirectional microphones, on-board microphones, wired microphones, wireless microphones, or any combination of the above. Consistent with the present disclosure, the processing device 360 can modify the presentation of virtual content based on data received from the audio sensor 371 (e.g., an audio command).

[0063] The image sensor 372 can include one or more image sensors configured to capture visual information by converting light into image data. Consistent with the present disclosure, the image sensor may be included in any device or system in the present disclosure and may be any device capable of detecting optical signals in the near-infrared, infrared, visible, and ultraviolet spectra and converting them into electrical signals. Examples of image sensors can include digital cameras, phone cameras, charge-coupled devices (CCDs), complementary metal-oxide-semiconductor (CMOS), or active pixel sensors of N-type metal-oxide-semiconductor (NMOS, live MOS). The electrical signals can be used to generate image data. Consistent with the present disclosure, the image data can include pixel data streams, digital images, digital video streams, data derived from captured images, and data that can be used to construct one or more 3D images, sequences of 3D images, 3D videos, or virtual 3D representations. The image data acquired by the image sensor 372 can be transmitted to any processing device of the system 200 by wired or wireless transmission. For example, the image data can be processed for object detection, event detection, action detection, face detection, person detection, recognition of known persons, or any other information that can be used by the system 200. Consistent with the present disclosure, the processing device 360 can modify the presentation of virtual content based on the image data received from the image sensor 372.

[0064] The motion sensor 373 can include one or more motion sensors configured to measure the movement of the input unit 202 or the movement of an object within the environment of the input unit 202. Specifically, the motion sensor can perform at least one of detecting the movement of an object within the environment of the input unit 202, measuring the speed of an object within the environment of the input unit 202, measuring the acceleration of an object within the environment of the input unit 202, detecting the movement of the input unit 202, measuring the speed of the input unit 202, measuring the acceleration of the input unit 202, and the like. In some embodiments, the motion sensor 373 can include one or more accelerometers configured to detect and / or measure a suitable change in acceleration of the input unit 202. In other embodiments, the motion sensor 373 can include one or more gyroscopes configured to detect a change in orientation of the input unit 202 and / or measure information regarding the orientation of the input unit 202. In other embodiments, the motion sensor 373 can include one or more that use an image sensor, a lidar sensor, a radar sensor, or a proximity sensor. For example, by analyzing the captured image, the processing device can determine the movement of the input unit 202, for example, using an egomotion algorithm. Further, the processing device can determine the movement of an object within the environment of the input unit 202, for example, using an object tracking algorithm. Consistent with the present disclosure, the processing device 360 can modify the presentation of virtual content based on the determined movement of the input unit 202 or the determined movement of an object within the environment of the input unit 202. For example, cause the virtual display to follow the movement of the input unit 202.

[0065] The environmental sensor 374 can include one or more sensors of different types configured to capture data reflecting the environment of the input unit 202. In some embodiments, the environmental sensor 374 is configured to perform at least one of measuring chemical properties in the environment of the input unit 202, measuring changes in chemical properties in the environment of the input unit 202, detecting the presence of chemical substances in the environment of the input unit 202, and measuring the concentration of chemical substances in the environment of the input unit 202, and can include one or more chemical sensors. Examples of such chemical properties can include pH level, toxicity, and temperature. Examples of such chemical substances can include electrolytes, certain enzymes, certain hormones, certain proteins, smoke, carbon dioxide, carbon monoxide, oxygen, ozone, hydrogen, and hydrogen sulfide. In other embodiments, the environmental sensor 374 can include one or more temperature sensors configured to detect changes in the temperature of the environment of the input unit 202 and / or measure the temperature of the environment of the input unit 202. In other embodiments, the environmental sensor 374 can include one or more barometers configured to detect changes in the air pressure in the environment of the input unit 202 and / or measure the air pressure in the environment of the input unit 202. In other embodiments, the environmental sensor 374 can include one or more light sensors configured to detect changes in ambient light in the environment of the input unit 202. Consistent with the present disclosure, the processing device 360 can modify the presentation of virtual content based on the input from the environmental sensor 374. For example, automatically lowering the brightness of virtual content when the environment of user 100 becomes dark.

[0066] The other sensor 375 can include a weight sensor, a light sensor, a resistance sensor, an ultrasonic sensor, a proximity sensor, a biometric sensor, or other detection devices to facilitate related functions. In certain embodiments, the other sensor 375 can include one or more positioning sensors configured to obtain positioning information of the input unit 202, detect a change in the position of the input unit 202, and / or measure the position of the input unit 202. Alternatively, the GPS software can enable the input unit 202 to access an external GPS receiver (e.g., connected via a serial port or Bluetooth). Consistent with the present disclosure, the processing device 360 can modify the presentation of the virtual content based on the input from the other sensor 375. For example, personal information is presented only after the user 100 is identified using data from a biometric sensor.

[0067] The components and arrangements shown in FIG. 3 are not intended to limit any embodiments. As will be understood by those having skill in the art with the benefit of this disclosure, numerous variations and / or modifications can be made to the illustrated configuration of the input unit 202. For example, not all components are essential for the operation of the input unit. Any component can be arranged in any suitable part of the input unit, and the components can be rearranged in various configurations while providing the functions of the various embodiments. For example, some input units may not include all of the elements as shown in the input unit 202.

[0068] FIG. 4 is a block diagram showing a configuration example of the XR unit 204. It should be understood that FIG. 4 is a typical representation of just one embodiment, and within the scope of the present disclosure, some of the illustrated elements can be omitted and other elements can be added. In the embodiment of FIG. 4, the XR unit 204 can directly or indirectly access a bus 400 (or other communication mechanism) that interconnects subsystems and components for transferring information within the XR unit 204. For example, the bus 400 can interconnect a memory interface 410, a network interface 420, an input interface 430, a power supply 440, an output interface 450, a processing device 460, a sensor interface 470, and a database 480.

[0069] The memory interface 410 shown in FIG. 4 is assumed to have the same functions as those of the memory interface 310 described in detail above. The memory interface 410 can be used to access software products and / or data stored in a non-transitory computer-readable medium or a memory device such as a memory device 411. The memory device 411 can include software modules for executing processes consistent with the present disclosure. In particular, the memory device 411 can include an input determination module 412, an output determination module 413, a sensor communication module 414, a virtual content determination module 415, a virtual content communication module 416, and a database access module 417. Modules 412 to 417 may include software instructions for execution by at least one processor (e.g., processing device 460) associated with the XR unit 204. The input determination module 412, the output determination module 413, the sensor communication module 414, the virtual content determination module 415, the virtual content communication module 416, and the database access module 417 can cooperate to perform various operations. For example, the input determination module 412 can determine a user interface (UI) input received from the input unit 202. At the same time, the sensor communication module 414 can receive data from different sensors to determine the state of the user 100. The virtual content determination module 415 can determine virtual content to be displayed based on the received input and the determined state of the user 100. The virtual content communication module 416 can search for virtual content not determined by the virtual content determination module 415. The search for virtual content may be from the database 380, the database 480, the mobile communication device 206, or the remote processing unit 208. Based on the output of the virtual content determination module 415, the output determination module 413 can cause a change in the virtual content displayed to the user 100 by the projector 454.

[0070] In some embodiments, the input determination module 412 can adjust the operation of the input interface 430 to receive gesture input 431, virtual input 432, voice input 433, and UI input 434. Consistent with the present disclosure, the input determination module 412 can receive different types of input data simultaneously. In one embodiment, the input determination module 412 can apply different rules based on the type of detected input. For example, gesture input can be prioritized over virtual input. In some embodiments, the output determination module 413 can adjust the operation of the output interface 450 to generate an output using the optical indicator 451, the display 452, the speaker 453, and the projector 454. In one embodiment, the optical indicator 451 can include an optical indicator indicating the state of the wearable extended reality device. For example, the optical indicator can display green light when the wearable extended reality device 110 is connected to the input unit 202, and can blink when the battery of the wearable extended reality device 110 is low. In another embodiment, the display 452 can be used to display operation information. In another embodiment, the speaker 453 can include bone conduction headphones used to output sound to the user 100. In another embodiment, the projector 454 can present virtual content to the user 100.

[0071] The operations of the sensor communication module, the virtual content determination module, the virtual content communication module, and the database access module have been described above with reference to FIG. 3, and the details are not repeated here. The modules 412-417 may be implemented in software, hardware, firmware, a mixture of any of them, and the like.

[0072] The network interface 420 shown in FIG. 4 is assumed to have the same functions as the network interface 320 described in detail above. The specific design and implementation of the network interface 420 may depend on the communication network in which the XR unit 204 is intended to operate. For example, in some embodiments, the XR unit 204 is configured to be selectively connectable by wire to the input unit 202. When connected by wire, the network interface 420 can enable communication with the input unit 202, and when not connected by wire, the network interface 420 can enable communication with the mobile communication device 206.

[0073] The input interface 430 shown in FIG. 4 is assumed to have the same functions as the input interface 330 described in detail above. In this case, the input interface 430 may communicate with an image sensor to obtain a gesture input 431 (e.g., the finger of the user 100 pointing to a virtual object), communicate with other XR units 204 to obtain a virtual input 432 (e.g., a virtual object shared with the XR unit 204, or a gesture of an avatar detected in the virtual environment), communicate with a microphone to obtain an audio input 433 (e.g., an audio command), and communicate with the input unit 202 to obtain a UI input 434 (e.g., virtual content determined by the virtual content determination module 315).

[0074] The power supply 440 shown in FIG. 4 is assumed to have the same functions as the power supply 340 described above, and supplies electrical energy to supply power to the XR unit 204. In some embodiments, the power supply 440 may be charged by the power supply 340. For example, the power supply 440 may be wirelessly changed when the XR unit 204 is placed on or near the input unit 202.

[0075] The output interface 450 shown in FIG. 4 is assumed to have the same functions as the output interface 350 described in detail above. In this case, the output interface 450 can generate outputs from the optical indicator 451, the display 452, the speaker 453, and the projector 454. The projector 454 can be any device, apparatus, instrument, etc. that can project (or direct) light to display virtual content on a surface. The surface may be part of the XR unit 204, part of the user 100's eye, or part of an object proximate to the user 100. In one embodiment, the projector 454 can include an illumination unit that condenses light within a limited solid angle by one or more mirrors and lenses and provides a high-value luminous intensity in a defined direction.

[0076] The processing device 460 shown in FIG. 4 is assumed to have the same functions as the processing device 360 described in detail above. When the XR unit 204 is connected to the input unit 202, the processing device 460 may cooperate with the processing device 360. Specifically, the processing device 460 can implement virtual machine technology or other technologies to provide the ability to execute, control, run, operate, store, etc. a plurality of software processes, applications, programs, etc. It should be understood that other types of processor configurations can be implemented to provide the capabilities disclosed herein.

[0077] The sensor interface 470 shown in FIG. 4 is assumed to have the same functions as the sensor interface 370 described in detail above. Specifically, the sensor interface 470 can communicate with the voice sensor 471, the image sensor 472, the motion sensor 473, the environmental sensor 474, and other sensors 475. The operations of the voice sensor, the image sensor, the motion sensor, the environmental sensor, and other sensors have been described above with reference to FIG. 3, and the details are not repeated here. It is understood that other types and combinations of sensors may be used to provide the capabilities disclosed herein.

[0078] The components and arrangements shown in FIG. 4 are not intended to limit any embodiment. As will be understood by those of ordinary skill in the art having the benefit of this disclosure, numerous variations and / or modifications can be made to the illustrated configuration of XR unit 204. For example, not all components are necessarily essential for the operation of XR unit 204 in all cases. Any component may be located in any suitable part of system 200, and the components may be rearranged in various configurations while providing the functions of the various embodiments. For example, some XR units may not include all of the elements within XR unit 204 (e.g., the wearable extended reality device 110 may not have the optical indicator 451).

[0079] FIG. 5 is a block diagram showing an example configuration of remote processing unit 208. It should be understood that FIG. 5 is a typical representation of only one embodiment, and that some of the illustrated elements may be omitted and other elements may be added within the scope of this disclosure. In the embodiment of FIG. 5, remote processing unit 208 can include a server 210 that directly or indirectly accesses a bus 500 (or other communication mechanism) that interconnects subsystems and components for transferring information within server 210. For example, bus 500 can interconnect a memory interface 510, a network interface 520, a power supply 540, a processing device 560, and a database 580. Remote processing unit 208 can also include one or more data structures. For example, data structures 212A, 212B, and 212C.

[0080] The memory interface 510 shown in FIG. 5 is assumed to have the same functions as those of the memory interface 310 described above in detail. The memory interface 510 can be used to access software products and / or data stored in a non-transitory computer-readable medium or other memory devices such as the memory devices 311, 411, 511, or the data structures 212A, 212B, and 212C. The memory device 511 can include software modules for executing a process consistent with the present disclosure. In particular, the memory device 511 can include a shared memory module 512, a node registration module 513, a load distribution module 514, one or more computing nodes 515, an internal communication module 516, an external communication module 517, and a database access module (not shown). The modules 512 to 517 can include software instructions for execution by at least one processor (e.g., the processing device 560) associated with the remote processing unit 208. The shared memory module 512, the node registration module 513, the load distribution module 514, the computing module 515, and the external communication module 517 can cooperate to perform various operations.

[0081] The shared memory module 512 can enable information sharing between the remote processing unit 208 and other components of the system 200. In some embodiments, the shared memory module 512 may be configured to allow the processing device 560 (and other processing devices within the system 200) to access, search for, and store data. For example, using the shared memory module 512, the processing device 560 can perform at least one of the steps of executing a software program stored in the memory device 511, the database 580, or the data structures 212A - C, storing information in the memory device 511, the database 580, or the data structures 212A - C, or searching for information from the memory device 511, the database 580, or the data structures 212A - C.

[0082] The node registration module 513 can be configured to track the availability of one or more computing nodes 515. In some examples, the node registration module 513 may be implemented as a software program, such as a software program executed by one or more computing nodes 515, a hardware solution, or a combined software and hardware solution. In some embodiments, the node registration module 513 can communicate with one or more computing nodes 515 using, for example, the internal communication module 516. In some examples, one or more computing nodes 515 can notify the node registration module 513 of their status by, for example, sending a message at startup, shutdown, at regular intervals, at a selected time, in response to a query received from the node registration module 513, or at any other determined time. In some examples, the node registration module 513 can query the status of one or more computing nodes 515 by, for example, sending a message at startup, at regular intervals, at a selected time, or at any other determined time.

[0083] The load distribution module 514 can be configured to divide the workload among one or more computing nodes 515. In some examples, the load distribution module 514 may be implemented as a software program, such as a software program executed by one or more of the computing nodes 515, a hardware solution, or a combined software and hardware solution. In some embodiments, the load distribution module 514 can interact with the node registration module 513 to obtain information regarding the availability of one or more computing nodes 515. In some embodiments, the load distribution module 514 can communicate with one or more computing nodes 515, for example, using the internal communication module 516. In some examples, one or more computing nodes 515 can notify the load distribution module 514 of their status by, for example, responding to a query received from the load distribution module 514 at startup, shutdown, at regular intervals, at a selected time, or by sending a message at any other determined time. In some examples, the load distribution module 514 can query the status of one or more computing nodes 515 by, for example, sending a message at startup, at regular intervals, at a preselected time, or at any other determined time.

[0084] The internal communication module 516 may be configured to receive and / or transmit information from one or more components of the remote processing unit 208. For example, control signals and / or synchronization signals can be transmitted and / or received via the internal communication module 516. In one embodiment, input information of a computer program, output information of a computer program, and / or intermediate information of a computer program can be transmitted and / or received via the internal communication module 516. In another embodiment, information received via the internal communication module 516 may be stored in the memory device 511, the database 580, the data structures 212A-C, or other memory devices within the system 200. For example, information retrieved from the data structure 212A may be transmitted using the internal communication module 516. In another example, input data may be received using the internal communication module 516 and stored in the data structure 212B.

[0085] The external communication module 517 may be configured to receive and / or transmit information from one or more components of the system 200. For example, control signals can be transmitted and / or received via the external communication module 517. In one embodiment, information received via the external communication module 517 can be stored in the memory device 511, the database 580, the data structures 212A-C, and / or any memory device within the system 200. In another embodiment, information retrieved from any of the data structures 212A-C may be transmitted to the XR unit 204 using the external communication module 517. In another embodiment, input data can be transmitted and / or received using the external communication module 517. Examples of such input data can include data received from the input unit 202, information captured from the environment of the user 100 using one or more sensors (e.g., the audio sensor 471, the image sensor 472, the motion sensor 473, the environmental sensor 474, other sensors 475), etc.

[0086] In some embodiments, aspects of modules 512-517 may be implemented in hardware, software (including one or more signal processing and / or application specific integrated circuits), firmware, or any combination thereof, and are executable by one or more processors, either alone or in various combinations with each other. Specifically, modules 512-517 may be configured to interact with each other and / or with other modules of system 200 to perform functions consistent with embodiments of the present disclosure. Memory device 511 may include additional modules and instructions or fewer modules and instructions.

[0087] Network interface 520, power supply 540, processing device 560, and database 580 shown in FIG. 5 are assumed to have functions similar to those of similar elements described above with reference to FIGS. 4 and 5. The specific design and implementation of the above-described components may vary based on the embodiment of system 200. Further, remote processing unit 208 may include more or fewer components. For example, remote processing unit 208 may include an input interface configured to receive input directly from one or more input devices.

[0088] In accordance with the present disclosure, a processing device of the system 200 (e.g., a processor within the mobile communication device 206, a processor within the server 210, a processor within a wearable extended reality device such as the wearable extended reality device 110, and / or a processor within an input device associated with the wearable extended reality device 110 such as the keyboard 104) can use a machine learning algorithm to implement any of the methods disclosed herein. In some embodiments, a machine learning algorithm (also referred to as a machine learning model in the present disclosure) can be trained using training examples, for example as described below. Some non-limiting examples of such machine learning algorithms include classification algorithms, data regression algorithms, image segmentation algorithms, visual detection algorithms (such as object detectors, face detectors, person detectors, motion detectors, edge detectors, etc.), visual recognition algorithms (such as face recognition, person recognition, object recognition, etc.), speech recognition algorithms, mathematical embedding algorithms, natural language processing algorithms, support vector machines, random forests, nearest neighbor algorithms, deep learning algorithms, artificial neural network algorithms, convolutional neural network algorithms, recurrent neural network algorithms, linear machine learning models, non-linear machine learning models, ensemble algorithms, and the like. For example, a trained machine learning algorithm may include an inference model such as a prediction model, a classification model, a data regression model, a clustering model, a segmentation model, an artificial neural network (e.g., a deep neural network, a convolutional neural network, a recurrent neural network, etc.), a random forest, a support vector machine, and the like. In some examples, a training example can include an input of the example and a desired output corresponding to the input of the example. Further, in some examples, a machine learning algorithm trained using training examples can generate a trained machine learning algorithm, and the trained machine learning algorithm can be used to estimate an output for an input not included in the training examples.In some examples, the engineers, scientists, processes, and machines that train machine learning algorithms can further use validation examples and / or test examples. For example, the validation examples and / or test examples can include exemplary inputs along with desired outputs corresponding to the exemplary inputs, and the trained machine learning algorithm and / or the intermediate trained machine learning algorithm can be used to estimate the outputs of the exemplary inputs of the validation examples and / or test examples, the estimated outputs can be compared to the corresponding desired outputs, and the trained machine learning algorithm and / or the intermediate trained machine learning algorithm can be evaluated based on the results of the comparison. In some examples, the machine learning algorithm can have parameters and hyperparameters, the hyperparameters can be set manually by a person or automatically by a process external to the machine learning algorithm (such as a hyperparameter search algorithm), and the parameters of the machine learning algorithm can be set by the machine learning algorithm based on the training examples. In some embodiments, the hyperparameters may be set based on the training examples and validation examples, and the parameters may be set based on the training examples and the selected hyperparameters. For example, given the hyperparameters, the parameters may be conditionally independent from the validation examples.

[0089] In some embodiments, a trained machine learning algorithm (also referred to herein as a machine learning model and a trained machine learning model) can be used to analyze an input and generate an output, for example, as described below. In some examples, a trained machine learning algorithm can be used as an inference model that generates an inferred output when an input is provided. For example, a trained machine learning algorithm can include a classification algorithm, the input can include a sample, and the inferred output can include a classification of the sample (e.g., a predicted label, a predicted tag, etc.). In another example, a trained machine learning algorithm can include a regression model, the input can include a sample, and the inferred output can include an inferred value corresponding to the sample. In yet another example, a trained machine learning algorithm can include a clustering model, the input can include a sample, and the inferred output can include an assignment of the sample to at least one cluster. In an additional example, a trained machine learning algorithm can include a classification algorithm, the input can include an image, and the inferred output can include a classification of an item depicted in the image. In yet another example, a trained machine learning algorithm can include a regression model, the input can include an image, and the inferred output can include an inferred value corresponding to an item depicted in the image (e.g., an estimated characteristic of an item such as the size, volume, age of a person depicted in the image, the distance from an item depicted in the image, etc.). In an additional example, a trained machine learning algorithm can include an image segmentation model, the input can include an image, and the inferred output can include a segmentation of the image. In yet another example, a trained machine learning algorithm can include an object detector, the input can include an image, and the inferred output can include one or more detected objects in the image and / or one or more positions of an object in the image.In some examples, a trained machine learning algorithm can include one or more formulas and / or one or more functions and / or one or more rules and / or one or more procedures, the input can be used as an input to the formula and / or function and / or rule and / or procedure, and the inferred output can be based on the output of the formula and / or function and / or rule and / or procedure (e.g., selecting one of the outputs of the formula and / or function and / or rule and / or procedure, using a statistical measure of the output of the formula and / or function and / or rule and / or procedure, etc.).

[0090] In accordance with the present disclosure, the processing device of the system 200 can analyze image data captured by an image sensor (e.g., image sensor 372, image sensor 472, or any other image sensor) to implement any of the methods disclosed herein. In some embodiments, analyzing the image data can include analyzing the image data to obtain pre-processed image data and then analyzing the image data and / or the pre-processed image data to obtain a desired result. Those skilled in the art will recognize that the following are examples, and that the image data can be pre-processed using other types of pre-processing methods. In some examples, the image data can be pre-processed by using a conversion function to convert the image data to obtain converted image data, and the pre-processed image data can include the converted image data. For example, the converted image data can include one or more convolutions of the image data. For example, the conversion function can include one or more image filters such as a low-pass filter, a high-pass filter, a band-pass filter, an all-pass filter, etc. In some examples, the conversion function can include a non-linear function. In some examples, the image data can be pre-processed by smoothing at least a portion of the image data, such as using Gaussian convolution, using a median filter, etc. In some examples, the image data can be pre-processed to obtain a different representation of the image data. For example, the pre-processed image data can include a representation of at least a portion of the image data in the frequency domain, a discrete Fourier transform of at least a portion of the image data, a discrete wavelet transform of at least a portion of the image data, a time / frequency representation of at least a portion of the image data, a low-dimensional representation of at least a portion of the image data, an irreversible representation of at least a portion of the image data, a reversible representation of at least a portion of the image data, any of the above chronological sequences, and any combination of the above, etc. In some examples, the image data can be pre-processed to extract edges, and the pre-processed image data can include information based on and / or related to the extracted edges. In some examples, the image data can be pre-processed to extract image features from the image data.Some non-limiting examples of such image features can include and / or be related to information based on edges, corners, blobs, ridges, Scale-Invariant Feature Transform (SIFT) features, temporal features, and the like. In some examples, analyzing the image data can include calculating at least one convolution of at least a portion of the image data and using the at least one calculated convolution to calculate at least one resultant value and / or perform determinations, identifications, recognitions, classifications, and the like.

[0091] In accordance with another aspect of the present disclosure, the processing device of the system 200 can analyze image data to implement any of the methods disclosed herein. In some embodiments, analyzing the image can include analyzing the image data and / or pre-processed image data using one or more rules, functions, procedures, artificial neural networks, object detection algorithms, face detection algorithms, visual event detection algorithms, action detection algorithms, motion detection algorithms, background subtraction algorithms, inference models, and the like. Some non-limiting examples of such inference models can include results for training examples of training algorithms such as manually pre-programmed inference models, classification models, regression models, machine learning algorithms, and / or deep learning algorithms, where the training examples can include examples of data instances, and in some cases, the data instances may be labeled with corresponding desired labels and / or results, and the like. In some embodiments, analyzing the image data (e.g., by the methods, steps, and modules described herein) can include analyzing pixels, voxels, point clouds, distance data, etc. included in the image data.

[0092] Convolution may include convolutions of any dimension. A one-dimensional convolution is a function that transforms an original sequence of numbers into a transformed sequence of numbers. A one-dimensional convolution can be defined by a series of scalars. Each particular value within the transformed sequence can be determined by calculating a linear combination of the values within a subsequence of the original sequence that corresponds to the particular value. The resulting value of the calculated convolution can include any value within the transformed sequence. Similarly, an n-dimensional convolution is a function that transforms an original n-dimensional array into a transformed array. An n-dimensional convolution can be defined by an n-dimensional array of scalars (known as the kernel of the n-dimensional convolution). Each particular value within the transformed array can be determined by calculating a linear combination of the values within an n-dimensional region of the original array that corresponds to the particular value. The resulting value of the calculated convolution can include any value within the transformed array. In some examples, an image can include one or more components (e.g., color components, depth components, etc.), and each component can include a two-dimensional array of pixel values. In one example, calculating the convolution of an image can include calculating a two-dimensional convolution for one or more components of the image. In another example, calculating the convolution of an image can include stacking arrays from different components to create a three-dimensional array and calculating a three-dimensional convolution on the resulting three-dimensional array. In some examples, a video can include one or more components (e.g., color components, depth components, etc.), and each component can include a three-dimensional array of pixel values (having two spatial axes and one temporal axis). In one example, calculating the convolution of a video can include calculating a three-dimensional convolution on one or more components of the video. In another example, calculating the convolution of a video can include stacking arrays from different components to create a four-dimensional array and calculating a four-dimensional convolution on the resulting four-dimensional array.

[0093] The following detailed description refers to the accompanying drawings. Whenever possible, the same reference numbers are used in the drawings and the following description to refer to the same or like parts. Although several exemplary embodiments are described herein, modifications, adaptations, and other embodiments are possible. For example, substitutions, additions, or modifications can be made to the components shown in the drawings, and the exemplary methods described herein can be modified by substituting, rearranging, deleting, or adding steps to the disclosed methods. Accordingly, the following detailed description is not limited to the disclosed embodiments and examples, and includes the general principles described herein and shown in the drawings, in addition to the general principles encompassed by the appended claims.

[0094] The present disclosure relates to systems and methods for providing a user with an extended reality environment. The term "extended reality environment," which may also be referred to as an "extended reality space," "extended reality," or "augmented environment," refers to all types of real-virtual composite environments and human-machine interactions that are at least partially generated by computer technology. The extended reality environment may be a fully simulated virtual environment or a combination of a real environment and a virtual environment that a user can perceive from different perspectives. In some examples, a user can interact with elements of the extended reality environment. One non-limiting example of an extended reality environment can be a virtual reality environment, also known as "virtual reality" or "virtual environment." An immersive virtual reality environment can be a simulated non-physical environment that provides a user with the perception of being present in the virtual environment. Another non-limiting example of an extended reality environment can be an augmented reality environment, also known as "augmented reality" or "augmented environment." An augmented reality environment can include a live direct or indirect view of a physical real-world environment enhanced with virtual computer-generated perceptual information, such as virtual objects with which a user can interact. Another non-limiting example of an extended reality environment is a mixed reality environment, also known as "mixed reality" or "mixed environment." A mixed reality environment may be a hybrid of a physical real-world environment and a virtual environment, in which physical objects and virtual objects coexist and can interact in real time. In some examples, both augmented reality environments and mixed reality environments can include a combination of the real world and the virtual world, real-time interaction, and accurate 3D registration of virtual objects and real objects. In some examples, both augmented reality environments and mixed reality environments may include constructive overlaid sensory information that can be added to the physical environment. In other examples, both augmented reality environments and mixed reality environments can include destructive virtual content that can mask at least a portion of the physical environment.

[0095] In some embodiments, the system and method can provide an extended reality environment using an extended reality device. The term "extended reality device" can include any type of device or system that enables a user to perceive and / or interact with an extended reality environment. The extended reality device can enable a user to perceive and / or interact with the extended reality environment through one or more sensory modalities. Some non-limiting examples of such sensory modalities may include vision, hearing, touch, proprioception, and smell. An example of an extended reality device is a virtual reality device that enables a user to perceive and / or interact with a virtual reality environment. Another example of an extended reality device is an augmented reality device that enables a user to perceive and / or interact with an augmented reality environment. Yet another example of an extended reality device is a mixed reality device that enables a user to perceive and / or interact with a mixed reality environment.

[0096] According to one aspect of the present disclosure, the extended reality device may be a wearable device such as a head-mounted device, for example, smart glasses, smart contact lenses, a headset, or any other device worn by a human for the purpose of presenting extended reality to the human. Other extended reality devices may include a holographic projector, or any other device or system capable of providing augmented reality (AR), virtual reality (VR), mixed reality (MR), or any immersive experience. Typical components of a wearable extended reality device may include a stereoscopic head-mounted display, a stereoscopic head-mounted sound system, head motion tracking sensors (e.g., gyroscopes, accelerometers, magnetometers, image sensors, structured light sensors, etc.), a head-mounted projector, eye tracking sensors, and at least one of the additional components described below. According to another aspect of the present disclosure, the extended reality device may be a non-wearable extended reality device. Specifically, the non-wearable extended reality device may include a multi-projection environment device. In some embodiments, the extended reality device may be configured to change the viewing perspective of the extended reality environment in response to the movement of the user, particularly the movement of the user's head. In one example, a wearable extended reality device may change the viewing field of the extended reality environment in response to a change in the user's head posture, such as by changing the spatial orientation without changing the user's spatial position in the extended reality environment. In another example, a non-wearable extended reality device may change the user's spatial position in the extended reality environment in response to a change in the user's position in the real world, such as by changing the user's spatial position in the extended reality environment without changing the direction of the viewing field with respect to the spatial position.

[0097] According to some embodiments, the extended reality device may include a digital communication device configured to perform at least one of: receiving virtual content data configured to enable presentation of virtual content; transmitting virtual content for sharing with at least one external device; receiving context data from at least one external device; transmitting context data to at least one external device; transmitting usage data indicating usage of the extended reality device; and transmitting data based on information captured using at least one sensor included in the extended reality device. In additional embodiments, the extended reality device may include a memory for storing at least one of: virtual data configured to enable presentation of virtual content, context data, usage data indicating usage of the extended reality device, sensor data based on information captured using at least one sensor included in a wearable extended reality device, software instructions configured to cause a processing device to present virtual content, software instructions configured to cause a processing device to collect and analyze context data, software instructions configured to cause a processing device to collect and analyze usage data, and software instructions configured to cause a processing device to collect and analyze sensor data. In additional embodiments, the extended reality device may include a processing device configured to perform at least one of: rendering virtual content, collecting and analyzing context data, collecting and analyzing usage data, and collecting and analyzing sensor data. In additional embodiments, the extended reality device may include one or more sensors.One or more sensors can include one or more image sensors (e.g., configured to capture images and / or videos of a user of the device or the user's environment), one or more motion sensors (e.g., accelerometers, gyroscopes, magnetometers, etc.), one or more positioning sensors (e.g., GPS, outdoor positioning sensors, indoor positioning sensors, etc.), one or more temperature sensors (e.g., configured to measure the temperature of at least a part of the device and / or the environment), one or more contact sensors, one or more proximity sensors (e.g., configured to detect whether the device is currently being worn), one or more electrical impedance sensors (e.g., configured to measure the electrical impedance of a user), one or more gaze tracking sensors, such as gaze detectors, optical trackers, potential trackers (e.g., electrooculogram (EOG) sensors), video-based gaze trackers, infrared / near-infrared sensors, passive light sensors, or any other technology capable of determining where a person is looking or fixating.

[0098] In some embodiments, the system and method can use an input device to interact with an extended reality device. The term input device can include any physical device configured to receive input from a user or the user's environment and provide data to a computing device. The data provided to the computing device may be in digital and / or analog form. In one embodiment, the input device can store input received from the user in a memory device accessible by a processing device, and the processing device can access the data stored for analysis. In another embodiment, the input device can directly provide data to the processing device, for example via a bus or via another communication system configured to transfer data from the input device to the processing device. In some examples, the input received by the input device can include key presses, tactile input data, motion data, position data, gesture-based input data, direction data, or any other data for supply for computation. Some examples of input devices can include buttons, keys, keyboards, computer mice, touch pads, touch screens, joysticks, or another mechanism capable of receiving input. Another example of an input device can include an integrated computing interface device that includes at least one physical component for receiving input from a user. The integrated computing interface device can include at least memory, a processing device, and at least one physical component for receiving input from a user. In one example, the integrated computing interface device may further include a digital network interface that enables digital communication with other computing devices. In one example, the integrated computing interface device may further include a physical component for outputting information to the user. In some examples, all components of the integrated computing interface device may be included in a single housing, and in other examples, the components may be distributed across two or more housings.Some non-limiting examples of physical components for receiving input from a user that can be included in an integrated computing interface device can include at least one of buttons, keys, keyboards, touch pads, touch screens, joysticks, or any other mechanism or sensor capable of receiving computing information. Some non-limiting examples of physical components for outputting information to a user can include at least one of optical indicators (such as LED indicators), screens, touch screens, buzzers, audio speakers, or any other audio, video, or tactile device that can provide a perceptible output to humans.

[0099] In some embodiments, one or more image sensors can be used to capture image data. In some examples, the image sensors may be included in extended reality devices, wearable devices, wearable extended reality devices, input devices, the user's environment, etc. In some examples, the image data may be read from memory, received from an external device, or generated (e.g., using a generative model), etc. Some non-limiting examples of image data can include images, grayscale images, color images, 2D images, 3D images, videos, 2D videos, 3D videos, frames, images, data derived from other image data, etc. In some examples, the image data may be encoded in any analog or digital format. Some non-limiting examples of such formats can include raw formats, compressed formats, uncompressed formats, irreversible formats, reversible formats, JPEG, GIF, PNG, TIFF, BMP, NTSC, PAL, SECAM, MPEG, MPEG-4 Part 14, MOV, WMV, FLV, AVI, AVCHD, WebM, MKV, etc.

[0100] In some embodiments, the extended reality device can receive digital signals from, for example, an input device. The term digital signal can refer to a temporally discrete series of digital values. The digital signal can represent, for example, sensor data, text data, audio data, video data, virtual data, or any other form of data that provides perceivable information. Consistent with the present disclosure, the digital signal can be configured to cause the extended reality device to present virtual content. In one embodiment, the virtual content may be presented in a selected orientation. In this embodiment, the digital signal can indicate the position and angle of a viewpoint in an environment such as an extended reality environment. Specifically, the digital signal can include an encoding of the position and angle in six degrees of freedom coordinates (e.g., forward / backward, up / down, left / right, yaw, pitch, and roll). In another embodiment, the digital signal can include an encoding of the position as three-dimensional coordinates (e.g., x, y, and z), and an encoding of the angle as a vector resulting from the encoded position. Specifically, the digital signal can indicate the orientation and angle of the presented virtual content in the absolute coordinates of the environment, for example, by encoding the yaw, pitch, and roll of the virtual content relative to a standard default angle. In another embodiment, the digital signal can indicate the orientation and angle of the presented virtual content with respect to the viewpoint of another object (e.g., a virtual object, a physical object, etc.), for example, by encoding the yaw, pitch, and roll of the virtual content with respect to the direction corresponding to the viewpoint or the direction corresponding to another object. In another embodiment, such a digital signal can include one or more projections of the virtual content, for example, in a format (e.g., an image, a video, etc.) that is ready for presentation. For example, each such projection can correspond to a particular orientation or a particular angle.In another embodiment, the digital signal can include a representation of virtual content by encoding an object in, for example, a three-dimensional array of voxels, a polygon mesh, or any other format capable of presenting virtual content.

[0101] In some embodiments, the digital signal may be configured to cause an extended reality device to present virtual content. The term virtual content can include any type of data representation that can be presented to a user by an extended reality device. Virtual content can include virtual objects, inanimate virtual content, animate virtual content configured to change over time or in response to a trigger, virtual two-dimensional content, virtual three-dimensional content, virtual overlays on a portion of a physical environment or on a physical object, virtual additions to a physical environment or physical object, virtual promotional content, virtual representations of physical objects, virtual representations of physical environments, virtual documents, virtual characters or personas, virtual computer screens, virtual widgets, or any other format for presenting information virtually. Consistent with the present disclosure, virtual content can include any visual presentation rendered by a computer or processing device. In one embodiment, the virtual content includes virtual objects that are visual presentations rendered by a computer within a limited area and configured to represent a particular type of object (e.g., an inanimate virtual object, an animate virtual object, virtual furniture, virtual decorative objects, virtual widgets, or other virtual representations, etc.). The rendered visual presentation can be changed, for example, to reflect a change to a state object or a change in the viewing angle of the object, so as to mimic a change in the appearance of a physical object. In another embodiment, the virtual content can include a virtual display (also referred to herein as a "virtual display screen" or "virtual screen"), such as a virtual computer screen, a virtual tablet screen, or a virtual smartphone screen, configured to display information generated by an operating system, and the operating system can be configured to receive text data from a physical keyboard and / or a virtual keyboard and cause the display of text content on the virtual display screen. In one example, as shown in FIG. 1, the virtual content may include a virtual environment that includes a virtual computer screen and a plurality of virtual objects.In some examples, the virtual display may be a virtual object that mimics and / or extends the functionality of a physical display screen. For example, the virtual display may be presented in an extended reality environment (e.g., a mixed reality environment, an augmented reality environment, a virtual reality environment, etc.) using an extended reality device. In one example, the virtual display can present content generated by a normal operating system that can be equally presented on a physical display screen. In one example, text content input using a keyboard (e.g., use of a physical keyboard, use of a virtual keyboard, etc.) can be presented on the virtual display in real-time as the text content is typed. In one example, a virtual cursor may be presented on the virtual display, and the virtual cursor may be controlled by a pointing device (physical pointing device, virtual pointing device, computer mouse, joystick, touchpad, physical touch controller, etc.). In one example, one or more windows of a graphical user interface operating system can be presented on the virtual display. In another example, the content presented on the virtual display may be interactive, i.e., the reaction to the user's action may change. In yet another example, the presentation of the virtual display may or may not include the presentation of a screen frame.

[0102] Some of the disclosed embodiments include a data structure or database and / or can access a data structure or database. The terms data structure and database consistent with this disclosure can include any collection of data values and the relationships between them. The data can be stored linearly, horizontally, hierarchically, relationally, non-relationally, unidimensionally, multidimensionally, operationally, in an ordered manner, in an unordered manner, object-oriented, centrally, non-centrally, distributively, customarily, or in any way that enables data access. By way of non-limiting example, data structures can include arrays, associative arrays, linked lists, binary trees, balanced trees, heaps, stacks, queues, sets, hash tables, records, tagged unions, entity-relationship models, graphs, hypergraphs, matrices, tensors, and the like. For example, data structures can include XML databases, RDBMS databases, SQL databases, or NoSQL alternatives for data storage / search, such as MongoDB, Redis, Couchbase, Datastax Enterprise Graph, Elastic Search, Splunk, Solr, Cassandra, Amazon DynamoDB, Scylla, HBase, and Neo4J. A data structure can be a component of the disclosed system or a remote computing component (e.g., a cloud-based data structure). The data within a data structure may be stored in contiguous or non-contiguous memory. Further, a data structure does not require that information be located in the same place. This can be distributed, for example, across multiple servers that may be owned or operated by the same or different entities. Thus, the term data structure in the singular includes multiple data structures.

[0103] In some embodiments, the system can determine a reliability level of the received input or any determined value. The term reliability level refers to a numerical value or any other indication of a level (e.g., within a predetermined range) that indicates the amount of reliability the system has in the determined data. For example, the confidence level may have a value between 1 and 10. Alternatively, the confidence level may be expressed as a percentage or any other numerical or non-numerical indication. In some cases, the system can compare the reliability level to a threshold. The term threshold can indicate a reference value, level, point, or range of values. During operation, if the reliability level of the determined data exceeds (or, depending on the particular use case, is below) the threshold, the system can follow a first course of action, and if the reliability level is below (or, depending on the particular use case, exceeds) it, the system can follow a second course of action. The value of the threshold may be predetermined for each type of object being investigated or may be dynamically selected based on different considerations.

[0104] Referring now to FIG. 1, a user is shown using an exemplary extended reality system consistent with an embodiment of the present disclosure. FIG. 1 is a representative illustration of only one embodiment, and it should be understood that some of the illustrated elements are omitted and other elements may be added within the scope of the present disclosure. As shown, user 100 is sitting behind table 102, supporting keyboard 104 and mouse 106. Keyboard 104 is connected by wire 108 to a wearable extended reality device 110 that displays virtual content to user 100. Instead of or in addition to wire 108, keyboard 104 can be wirelessly connected to wearable extended reality device 110. For illustrative purposes, the wearable extended reality device is shown as a pair of smart glasses, but as described above, wearable extended reality device 110 can be any type of head-mounted device used to present extended reality to user 100. The virtual content displayed by wearable extended reality device 110 includes a virtual screen 112 (also referred to herein as a "virtual display screen" or "virtual display") and a plurality of virtual widgets 114. Virtual widgets 114A-114D are displayed next to virtual screen 112, and virtual widget 114E is displayed on table 102. User 100 can use keyboard 104 to input text into document 116 displayed on virtual screen 112 and can use mouse 106 to control virtual cursor 118. In one example, virtual cursor 118 can move anywhere within virtual screen 112. In another example, virtual cursor 118 can move anywhere within virtual screen 112 and can also move to any of virtual widgets 114A-114D, but cannot move to virtual widget 114E. In yet another example, virtual cursor 118 can move anywhere within virtual screen 112 and can also move to any of virtual widgets 114A-114E. In an additional example, virtual cursor 118 can move anywhere within the extended reality environment that includes virtual screen 112 and virtual widgets 114A-114E.In yet another example, the virtual cursor can move only on all available surfaces (i.e., virtual surfaces or physical surfaces), or on selected surfaces within the extended reality environment. Alternatively or additionally, the user 100 can use hand gestures recognized by the wearable extended reality device 110 to interact with any of the virtual widgets 114A - 114E, or with a selected virtual widget. For example, the virtual widget 114E can be an interactive widget (e.g., a virtual slider controller) that can be manipulated with hand gestures.

[0105] FIG. 2 shows an example of a system 200 that provides an extended reality (XR) experience to a user such as user 100. It should be understood that FIG. 2 is a typical representation of only one embodiment, and within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added. The system 200 may be computer - based and may include computer system components, wearable devices, workstations, tablets, handheld computing devices, memory devices, and / or an internal network connecting the components. The system 200 may include or be connected to various network - computing resources (e.g., servers, routers, switches, network connections, storage devices, etc.) to support the services provided by the system 200. Consistent with the present disclosure, the system 200 can include an input unit 202, an XR unit 204, a mobile communication device 206, and a remote processing unit 208. The remote processing unit 208 can include a server 210 coupled to one or more physical or virtual storage devices such as data structure 212. The system 200 may also include or be connected to a communication network 214 that facilitates communication and data exchange between different system components and different entities associated with the system 200.

[0106] In accordance with the present disclosure, the input unit 202 can include one or more devices capable of receiving input from the user 100. In one embodiment, the input unit 202 can include a text input device such as a keyboard 104. The text input device can include all possible types of devices and mechanisms for inputting text information into the system 200. Examples of text input devices can include mechanical keyboards, membrane keyboards, flexible keyboards, QWERTY keyboards, Dvorak keyboards, Colemak keyboards, coded keyboards, wireless keyboards, keypads, key-based control panels, or other arrangements of control keys, visual input devices, or any other mechanism for inputting text (regardless of whether the mechanism is provided in a physical form or presented virtually). In one embodiment, the input unit 202 can also include a pointing input device such as a mouse 106. The pointing input device can include all possible types of devices and mechanisms for inputting two-dimensional or three-dimensional information into the system 200. In one example, two-dimensional input from the pointing input device may be used to interact with virtual content presented via the XR unit 204. Examples of pointing input devices can include computer mice, trackballs, touchpads, trackpads, touchscreens, joysticks, pointing sticks, styli, light pens, or any other physical or virtual input mechanism. In one embodiment, the input unit 202 can also include a graphical input device such as a touchscreen configured to detect contact, movement, or interruption of movement. The graphical input device can use any of a plurality of touch sensitivity technologies including, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more contact points. In one embodiment, the input unit 202 may also include one or more voice input devices such as a microphone.The voice input device can include all possible types of devices and mechanisms for inputting voice data to facilitate voice-enabled functions such as voice recognition, voice replication, digital recording, and telephone functions. In one embodiment, the input unit 202 may also include one or more image input devices, such as an image sensor, configured to capture image data. In one embodiment, the input unit 202 may also include one or more tactile gloves configured to capture hand movement and pose data. In one embodiment, the input unit 202 can also include one or more proximity sensors configured to detect the presence and / or movement of objects within a selected area near the sensor.

[0107] According to some embodiments, the system can include at least one sensor configured to detect and / or measure characteristics associated with a user, the user's actions, or the user's environment. An example of the at least one sensor is sensor 216 included in input unit 202. Sensor 216 can be a motion sensor, a touch sensor, an optical sensor, an infrared sensor, an audio sensor, an image sensor, a proximity sensor, a positioning sensor, a gyroscope, a temperature sensor, a biometric sensor, or any other sensing device to facilitate related functions. Sensor 216 can be integrated with the input device, connected to the input device, or separated from the input device. In one example, a thermometer can be included in mouse 106 to determine the body temperature of user 100. In another example, a positioning sensor can be integrated with keyboard 104 to determine the movement of user 100 relative to keyboard 104. Such a positioning sensor can be implemented using one of Global Positioning System (GPS), GLObal NAvigation Satellite System (GLONASS), Galileo Global Navigation System, BeiDou Navigation System, other Global Navigation Satellite Systems (GNSS), Indian Regional Navigation Satellite System (IRNSS), Local Positioning System (LPS), Real-Time Location System (RTLS), Indoor Positioning System (IPS), Wi-Fi-based positioning system, cellular triangulation, image-based positioning technology, indoor positioning technology, outdoor positioning technology, or any other positioning technology.

[0108] According to some embodiments, the system can include one or more sensors for identifying the position and / or movement of physical devices (such as physical input devices, physical computing devices, keyboard 104, mouse 106, wearable extended reality device 110, etc.). The one or more sensors may be included in the physical device or may be external to the physical device. In some examples, an image sensor external to the physical device (e.g., an image sensor included in another physical device) can be used to capture image data of the physical device, and the image data can be analyzed to identify the position and / or movement of the physical device. For example, the image data may be analyzed using a visual object tracking algorithm to identify the movement of the physical device, or may be analyzed using a visual object detection algorithm to identify the position of the physical device (e.g., relative to the image sensor in a global coordinate system, etc.). In some examples, an image sensor included in the physical device can be used to capture image data, and the image data can be analyzed to identify the position and / or movement of the physical device. For example, the image data may be analyzed using a visual odometry algorithm to identify the position of the physical device, or may be analyzed using an egomotion algorithm to identify the movement of the physical device, etc. In some examples, a positioning sensor such as an indoor positioning sensor or an outdoor positioning sensor may be included in the physical device and may be used to determine the position of the physical device. In some examples, a motion sensor such as an accelerometer or a gyroscope may be included in the physical device and may be used to determine the movement of the physical device. In some examples, a physical device such as a keyboard or a mouse may be configured to be placed on a physical surface. Such a physical device can include an optical mouse sensor (also known as a non-mechanical tracking engine) directed towards the physical surface, and the output of the optical mouse sensor can be analyzed to determine the movement of the physical device relative to the physical surface.

[0109] In accordance with the present disclosure, the XR unit 204 may include a wearable extended reality device configured to present virtual content to the user 100. An example of a wearable extended reality device is the wearable extended reality device 110. Further examples of wearable extended reality devices can include virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices, or any other device capable of generating extended reality content. Some non-limiting examples of such devices can include Nreal Light, Magic Leap One, Varjo, Quest 1 / 2, Vive, and the like. In some embodiments, the XR unit 204 can present virtual content to the user 100. Generally, extended reality devices can include all real and virtual composite environments generated by computer technology and wearability, as well as human-machine interactions. As described above, the term "extended reality" (XR) refers to a superset that includes the entire spectrum from "complete reality" to "complete virtuality". It includes typical forms such as augmented reality (AR), mixed reality (MR), virtual reality (VR), and the areas interpolated between them. Therefore, it should be noted that the terms "XR device", "AR device", "VR device", and "MR device" can be used interchangeably herein and can refer to any of the various devices listed above.

[0110] In accordance with the present disclosure, the system can exchange data with various communication devices associated with a user, such as mobile communication device 206. The term "communication device" is intended to include all possible types of devices that can exchange data using a digital communication network, an analog communication network, or any other communication network configured to transmit data. In some examples, the communication device may include smartphones, tablets, smartwatches, personal digital assistants, desktop computers, laptop computers, IoT devices, dedicated terminals, wearable communication devices, and any other device that enables data communication. In some cases, mobile communication device 206 can supplement or replace input unit 202. Specifically, mobile communication device 206 may be associated with a physical touch controller that can function as a pointing input device. Additionally, mobile communication device 206 may also be used, for example, to implement a virtual keyboard and replace a text input device. For example, if user 100 leaves table 102 and walks into the break room with their smart glasses, they can receive an email that requires a quick response. In this case, the user can choose to use their smartwatch as an input device and type a response to the email while it is being virtually presented by the smart glasses.

[0111] In accordance with the present disclosure, embodiments of the system can include the use of a cloud server. The term "cloud server" refers to a computer platform that provides services over a network such as the Internet. In the exemplary embodiment shown in FIG. 2, server 210 can use virtual machines that may not correspond to individual hardware. For example, computing and / or memory capabilities may be implemented by allocating an appropriate portion of the desired computing / memory power from a scalable repository such as a data center or a distributed computing environment. Specifically, in one embodiment, remote processing unit 208 can be used with XR unit 204 to provide virtual content to user 100. In one configuration example, server 210 may be a cloud server that functions as an operating system (OS) for wearable extended reality devices. In one example, server 210 can implement the methods described herein using custom hardwired logic, one or more application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), firmware, and / or program logic that combines with a computer system to make server 210 a dedicated machine.

[0112] In some embodiments, server 210 can access data structure 212 to determine, for example, virtual content for displaying user 100. Data structure 212 can utilize volatile or non-volatile, magnetic, semiconductor, tape, optical, removable, non-removable, other types of storage devices or tangible or non-transitory computer-readable media, or any medium or mechanism for storing information. Data structure 212 can be part of server 210 as shown, or can be separate from server 210. If data structure 212 is not part of server 210, server 210 can exchange data with data structure 212 via a communication link. Data structure 212 can include one or more memory devices that store data and instructions used to execute one or more features of the disclosed methods. In one embodiment, data structure 212 can include any of a plurality of suitable data structures, from small data structures hosted on a workstation to large data structures distributed among data centers. Data structure 212 can also include any combination of one or more data structures controlled by a memory controller device (e.g., a server) or software.

[0113] In accordance with the present disclosure, a communication network can be any type of network (including infrastructure) that supports communication, exchanges information, and / or facilitates the exchange of information between components of a system. For example, communication network 214 within system 200 can include, for example, a telephone network, an extranet, an intranet, the Internet, satellite communication, offline communication, wireless communication, transponder communication, a local area network (LAN), a wireless network (e.g., a Wi-Fi / 802.11 network), a wide area network (WAN), a virtual private network (VPN), a digital communication network, an analog communication network, or any other mechanism or combination of mechanisms that enables data transmission.

[0114] The components and configurations of system 200 shown in FIG. 2 are for illustrative purposes only, as the system components used to implement the disclosed processes and features may vary, and are not intended to limit the disclosed embodiments.

[0115] FIG. 3 is a block diagram showing a configuration example of input unit 202. It should be understood that FIG. 3 is a typical representation of only one embodiment, and within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added. In the embodiment of FIG. 3, input unit 202 can directly or indirectly access a bus 300 (or other communication mechanism) that interconnects subsystems and components for transferring information within input unit 202. For example, bus 300 can interconnect a memory interface 310, a network interface 320, an input interface 330, a power supply 340, an output interface 350, a processing device 360, a sensor interface 370, and a database 380.

[0116] The memory interface 310 shown in FIG. 3 can be used to access software products and / or data stored in a non-transitory computer-readable medium. Generally, a non-transitory computer-readable storage medium refers to any type of physical memory that can store information or data readable by at least one processor. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, any other optical data storage medium, any physical medium with a pattern of holes, PROM, EPROM, flash-EPROM or any other flash memory, NVRAM, cache, registers, any other memory chip or cartridge, and network versions thereof. The terms "memory" and "computer-readable storage medium" can refer to multiple structures such as multiple memories or computer-readable storage media located within the input unit or remotely. Further, one or more computer-readable storage media can be utilized when implementing a method implemented by a computer. Thus, the term computer-readable storage medium should be understood to include tangible items and exclude carrier waves and transient signals. In the specific embodiment shown in FIG. 3, the memory interface 310 can be used to access software products and / or data stored in a memory device such as the memory device 311. The memory device 311 may include high-speed random access memory and / or non-volatile memory, such as one or more magnetic disk storage devices, one or more optical storage devices, and / or flash memory (e.g., NAND, NOR). Consistent with the present disclosure, the components of the memory device 311 may be distributed across two or more units of the system 200 and / or two or more memory devices.

[0117] The memory device 311 shown in FIG. 3 can include software modules for executing processes consistent with the present disclosure. In particular, the memory device 311 can include an input determination module 312, an output determination module 313, a sensor communication module 314, a virtual content determination module 315, a virtual content communication module 316, and a database access module 317. Modules 312-317 can include software instructions for execution by at least one processor (e.g., processing device 360) associated with the input unit 202. The input determination module 312, the output determination module 313, the sensor communication module 314, the virtual content determination module 315, the virtual content communication module 316, and the database access module 317 can cooperate to perform various operations. For example, the input determination module 312 may determine text using data received from, for example, the keyboard 104. Thereafter, the output determination module 313 can cause the most recently input text to be presented, for example, on a dedicated display 352 physically or wirelessly coupled to the keyboard 104. In this way, when the user 100 inputs, a preview of the typed text can be viewed without constantly moving the head up and down to view the virtual screen 112. The sensor communication module 314 can receive data from different sensors to determine the state of the user 100. Thereafter, the virtual content determination module 315 can determine the virtual content to be displayed based on the received input and the determined status of the user 100. For example, the determined virtual content may be a virtual presentation of the most recently input text on a virtual screen disposed virtually adjacent to the keyboard 104. The virtual content communication module 316 may obtain virtual content not determined by the virtual content determination module 315 (e.g., another user's avatar). The search for virtual content may be from the database 380, from the remote processing unit 208, or from any other source.

[0118] In some embodiments, the input determination module 312 can adjust the operation of the input interface 330 to receive pointer input 331, text input 332, voice input 333, and XR-related input 334. Details of pointer input, text input, and voice input have been described above. The term "XR-related input" can include any type of data that may cause a change in the virtual content displayed to the user 100. In one embodiment, the XR-related input 334 may include image data of the user 100, a wearable extended reality device (e.g., a detected hand gesture of the user 100). In another embodiment, the XR-related input 334 may include wireless communication indicating the presence of another user in proximity to the user 100. Consistent with the present disclosure, the input determination module 312 can receive different types of input data simultaneously. Thereafter, the input determination module 312 can further apply different rules based on the type of input detected. For example, pointer input may be prioritized over voice input.

[0119] In some embodiments, the output determination module 313 can adjust the operation of the output interface 350 to generate an output using the optical indicator 351, the display 352, and / or the speaker 353. Generally, the output generated by the output determination module 313 does not include virtual content presented by the wearable extended reality device. Instead, the output generated by the output determination module 313 includes various outputs regarding the operation of the input unit 202 and / or the XR unit 204. In one embodiment, the optical indicator 351 can include an optical indicator indicating the state of the wearable extended reality device. For example, the optical indicator can display green light when the wearable extended reality device 110 is connected to the keyboard 104, and can blink when the battery of the wearable extended reality device 110 is low. In another embodiment, the display 352 can be used to display operation information. For example, the display can present an error message when the wearable extended reality device is inoperable. In another embodiment, the speaker 353 can be used to output sound, for example, when the user 100 wants to play music for other users.

[0120] In some embodiments, the sensor communication module 314 can adjust the operation of the sensor interface 370 to receive sensor data from one or more sensors integrated with or connected to the input device. The one or more sensors can include an audio sensor 371, an image sensor 372, a motion sensor 373, an environmental sensor 374 (e.g., a temperature sensor, an ambient light detector, etc.), and other sensors 375. In one embodiment, the data received from the sensor communication module 314 can be used to determine the physical orientation of the input device. The physical orientation of the input device can indicate the user's state and can be determined based on a combination of tilt motion, roll motion, and lateral motion. Thereafter, the physical orientation of the input device can be used by the virtual content determination module 315 to modify the display parameters of the virtual content to match the user's state (e.g., attention, sleepy, active, sitting, standing, leaning back, leaning forward, walking, moving, riding, etc.).

[0121] In some embodiments, the virtual content determination module 315 can determine the virtual content to be displayed by the wearable extended reality device. The virtual content can be determined based on data from the input determination module 312, the sensor communication module 314, and other sources (e.g., the database 380). In some embodiments, determining the virtual content can include determining the distance, size, and orientation of virtual objects. The determination of the position of a virtual object may be determined based on the type of the virtual object. Specifically, with respect to the example shown in FIG. 1, since the virtual widget 114E is a virtual controller (e.g., a volume bar), the virtual content determination module 315 can determine to arrange four virtual widgets 114A-114D on both sides of the virtual screen 112 and arrange the virtual widget 114E on the table 102. The determination of the position of a virtual object may be further determined based on the user's preference. For example, for a left-handed user, the virtual content determination module 315 can determine to arrange the virtual volume bar to the left of the keyboard 104, and for a right-handed user, the virtual content determination module 315 can determine to arrange the virtual volume bar to the right of the keyboard 104.

[0122] In some embodiments, the virtual content communication module 316 may adjust the operation of the network interface 320 to obtain data from one or more sources to be presented to the user 100 as virtual content. The one or more sources may include other XR units 204, the user's mobile communication device 206, a remote processing unit 208, publicly available information, and the like. In one embodiment, the virtual content communication module 316 can communicate with the mobile communication device 206 to provide a virtual representation of the mobile communication device 206. For example, the virtual representation can enable the user 100 to read messages and interact with applications installed on the mobile communication device 206. The virtual content communication module 316 can also adjust the operation of the network interface 320 to share virtual content with other users. In one example, the virtual content communication module 316 uses data from the input determination module to identify a trigger (e.g., the trigger may include a user gesture) and transfer the content from the virtual display to a physical display (e.g., a TV) or a virtual display of a different user.

[0123] In some embodiments, the database access module 317 can cooperate with the database 380 to retrieve stored data. The retrieved data can include, for example, privacy levels associated with different virtual objects, relationships between virtual objects and physical objects, user preferences, user past behaviors, and the like. As described above, the virtual content determination module 315 can use the data stored in the database 380 to determine virtual content. The database 380 can include separate databases, such as, for example, a vector database, a raster database, a tile database, a viewport database, and / or a user input database. The data stored in the database 380 may be received from modules 314-317 or other components of the system 200. Further, the data stored in the database 380 may be provided as input using data input, data transfer, or data upload.

[0124] Modules 312 - 317 may be implemented in software, hardware, firmware, or any combination thereof. In some embodiments, any one or more of Modules 312 - 317 and the data associated with Database 380 may be stored in XR Unit 204, Mobile Communication Device 206, or Remote Processing Unit 208. The processing device of System 200 may be configured to execute the instructions of Modules 312 - 317. In some embodiments, aspects of Modules 312 - 317 may be executable by one or more processors, alone or in various combinations with each other, in hardware, software (including one or more signal processing and / or application specific integrated circuits), firmware, or any combination thereof. Specifically, Modules 312 - 317 may be configured to interact with each other and / or with other modules of System 200 to perform functions consistent with the disclosed embodiments. For example, Input Unit 202 may execute instructions including an image processing algorithm on data from XR Unit 204 to determine the movement of User 100's head. Further, throughout this specification, each function described with respect to Input Unit 202, or with respect to a component of Input Unit 202, may correspond to a set of instructions for performing the function. These instructions need not be implemented as separate software programs, procedures, or modules. Memory Device 311 may include additional modules and instructions or fewer modules and instructions. For example, Memory Device 311 may store an operating system such as ANDROID, iOS, UNIX, OSX, WINDOWS, DARWIN, RTXC, LINUX, or an embedded operating system such as VXWorkS. The operating system may handle basic system services and include instructions for performing hardware - dependent tasks.

[0125] The network interface 320 shown in FIG. 3 can provide bidirectional data communication to a network such as the communication network 214. In one embodiment, the network interface 320 can include an Integrated Services Digital Network (ISDN) card, a cellular modem, a satellite modem, or a modem to provide a data communication connection via the Internet. As another example, the network interface 320 can include a Wireless Local Area Network (WLAN) card. In another embodiment, the network interface 320 can include an Ethernet port connected to a radio frequency receiver and transmitter and / or an optical (e.g., infrared) receiver and transmitter. The specific design and implementation of the network interface 320 can depend on the communication network on which the input unit 202 is intended to operate. For example, in some embodiments, the input unit 202 can include a network interface 320 designed to operate via a GSM network, a GPRS network, an EDGE network, a Wi-Fi or WiMax network, and a Bluetooth network. In any such embodiment, the network interface 320 may be configured to transmit and receive electrical, electromagnetic, or optical signals that carry digital data streams or digital signals representing various types of information.

[0126] The input interface 330 shown in FIG. 3 can receive inputs from various input devices, such as a keyboard, a mouse, a touchpad, a touch screen, one or more buttons, a joystick, a microphone, an image sensor, and any other device configured to detect physical or virtual inputs. The received input may be in at least one form of text, voice, sound, hand gesture, body gesture, tactile information, and any other type of physical or virtual input generated by the user. In the illustrated embodiment, the input interface 330 can receive a pointer input 331, a text input 332, a voice input 333, and an XR-related input 334. In an additional embodiment, the input interface 330 may be an integrated circuit that can function as a bridge between the processing device 360 and any of the above input devices.

[0127] The power supply 340 shown in FIG. 3 can supply electrical energy to the power input unit 202 and optionally can also supply power to the XR unit 204. Generally, the power supply included in any device or system of the present disclosure can be any device that can repeatedly store, distribute, or transmit power, including but not limited to one or more batteries (e.g., lead-acid batteries, lithium-ion batteries, nickel-metal hydride batteries, nickel-cadmium batteries), one or more capacitors, one or more connections to an external power source, one or more power converters, or any combination thereof. Referring to the example shown in FIG. 3, the power supply may be portable, which means that the input unit 202 can be easily carried by hand (e.g., the total weight of the power supply 340 can be less than 1 pound). The portability of the power supply enables the user 100 to use the input unit 202 in various situations. In other embodiments, the power supply 340 may be associated with a connection to an external power source (such as a power grid) that can be used to charge the power supply 340. Further, the power supply 340 may be configured to charge one or more batteries included in the XR unit 204. For example, a pair of extended reality glasses (e.g., wearable extended reality device 110) can be charged (e.g., wirelessly or non-wirelessly) when they are placed on or near the input unit 202.

[0128] The output interface 350 shown in FIG. 3 can generate outputs from various output devices, for example, using an optical indicator 351, a display 352, and / or a speaker 353. In one embodiment, the output interface 350 may be an integrated circuit that can function as a bridge between the processing device 360 and at least one of the above output devices. The optical indicator 351 can include one or more light sources, such as, for example, an LED array associated with different colors. The display 352 can include a screen (e.g., an LCD or dot matrix screen) or a touch screen. The speaker 353 can include audio headphones, a hearing aid type device, a speaker, bone conduction headphones, an interface providing tactile cues, a vibration stimulation device, and the like.

[0129] The processing device 360 shown in FIG. 3 can include at least one processor configured to execute a computer program, application, method, process, or other software to implement the embodiments described in this disclosure. Generally, a processing device included in any device or system of this disclosure can include one or more integrated circuits, microchips, microcontrollers, microprocessors, central processing devices (CPUs), graphics processing devices (GPUs), digital signal processors (DSPs), all or part of a field programmable gate array (FPGA), or other circuits suitable for executing instructions or performing logical operations. The processing device can include at least one processor configured to implement the functions of the disclosed methods, such as a microprocessor manufactured by Intel (trademark). The processing device can include a single core or a multi-core processor that simultaneously executes parallel processing. In one example, the processing device can be a single-core processor configured with virtual processing technology. The processing device can implement virtual machine technology or other technologies to provide the ability to execute, control, execute, operate, store, etc. multiple software processes, applications, programs, etc. In another example, the processing device can include a multi-core processor configuration (e.g., dual, quad-core, etc.) configured to provide a parallel processing function that enables devices associated with the processing device to execute multiple processes simultaneously. It should be understood that other types of processor configurations can be implemented to provide the functions disclosed herein.

[0130] The sensor interface 370 shown in FIG. 3 can obtain sensor data from various sensors, such as a voice sensor 371, an image sensor 372, a motion sensor 373, an environmental sensor 374, and other sensors 375. In one embodiment, the sensor interface 370 can be an integrated circuit that can function as a bridge between the processing device 360 and at least one of the above sensors.

[0131] The audio sensor 371 may include one or more audio sensors configured to capture audio by converting the audio into digital information. Some examples of audio sensors can include microphones, unidirectional microphones, bidirectional microphones, cardioid microphones, omnidirectional microphones, on-board microphones, wired microphones, wireless microphones, or any combination of the above. Consistent with the present disclosure, the processing device 360 can modify the presentation of virtual content based on data received from the audio sensor 371 (e.g., an audio command).

[0132] The image sensor 372 can include one or more image sensors configured to capture visual information by converting light into image data. Consistent with the present disclosure, the image sensor may be included in any device or system in the present disclosure and can be any device that can detect optical signals in the near-infrared, infrared, visible, and ultraviolet spectra and convert them into electrical signals. Examples of image sensors can include digital cameras, phone cameras, charge-coupled devices (CCDs), complementary metal-oxide-semiconductor (CMOS), or N-type metal-oxide-semiconductor (NMOS, live MOS) active pixel sensors. The electrical signals can be used to generate image data. Consistent with the present disclosure, the image data can include pixel data streams, digital images, digital video streams, data derived from captured images, and data that can be used to construct one or more 3D images, sequences of 3D images, 3D videos, or virtual 3D representations. The image data acquired by the image sensor 372 can be transmitted to any processing device of the system 200 by wired or wireless transmission. For example, the image data can be processed for object detection, event detection, action detection, face detection, person detection, recognition of known persons, or any other information that can be used by the system 200. Consistent with the present disclosure, the processing device 360 can modify the presentation of virtual content based on the image data received from the image sensor 372.

[0133] The motion sensor 373 can include one or more motion sensors configured to measure the movement of the input unit 202 or the movement of an object within the environment of the input unit 202. Specifically, the motion sensor can detect the movement of an object within the environment of the input unit 202, measure the speed of an object within the environment of the input unit 202, measure the acceleration of an object within the environment of the input unit 202, detect the movement of the input unit 202, measure the speed of the input unit 202, measure the acceleration of the input unit 202, etc., and can perform at least one of these. In some embodiments, the motion sensor 373 can include one or more accelerometers configured to detect a change in the appropriate acceleration of the input unit 202 and / or measure the appropriate acceleration. In other embodiments, the motion sensor 373 can include one or more gyroscopes configured to detect a change in the orientation of the input unit 202 and / or measure information regarding the orientation of the input unit 202. In other embodiments, the motion sensor 373 can include one or more that use an image sensor, a lidar sensor, a radar sensor, or a proximity sensor. For example, by analyzing the captured image, the processing device can determine the movement of the input unit 202, for example, using an egomotion algorithm. Further, the processing device can determine the movement of an object within the environment of the input unit 202, for example, using an object tracking algorithm. Consistent with the present disclosure, the processing device 360 can modify the presentation of the virtual content based on the determined movement of the input unit 202 or the determined movement of an object within the environment of the input unit 202. For example, cause the virtual display to follow the movement of the input unit 202.

[0134] The environmental sensor 374 can include one or more sensors of different types configured to capture data reflecting the environment of the input unit 202. In some embodiments, the environmental sensor 374 is configured to perform at least one of measuring chemical properties in the environment of the input unit 202, measuring changes in chemical properties in the environment of the input unit 202, detecting the presence of chemical substances in the environment of the input unit 202, and measuring the concentration of chemical substances in the environment of the input unit 202, and can include one or more chemical sensors. Examples of such chemical properties can include pH level, toxicity, and temperature. Examples of such chemical substances can include electrolytes, specific enzymes, specific hormones, specific proteins, smoke, carbon dioxide, carbon monoxide, oxygen, ozone, hydrogen, and hydrogen sulfide. In other embodiments, the environmental sensor 374 can include one or more temperature sensors configured to detect changes in the temperature of the environment of the input unit 202 and / or measure the temperature of the environment of the input unit 202. In other embodiments, the environmental sensor 374 can include one or more barometers configured to detect changes in the air pressure in the environment of the input unit 202 and / or measure the air pressure in the environment of the input unit 202. In other embodiments, the environmental sensor 374 can include one or more light sensors configured to detect changes in ambient light in the environment of the input unit 202. Consistent with the present disclosure, the processing device 360 can modify the presentation of virtual content based on the input from the environmental sensor 374. For example, automatically reducing the brightness of virtual content when the environment of user 100 becomes dark.

[0135] The other sensor 375 can include a weight sensor, a light sensor, a resistance sensor, an ultrasonic sensor, a proximity sensor, a biometric sensor, or other detection devices to facilitate related functions. In certain embodiments, the other sensor 375 can include one or more positioning sensors configured to obtain positioning information of the input unit 202, detect a change in the position of the input unit 202, and / or measure the position of the input unit 202. Alternatively, the GPS software can enable the input unit 202 to access an external GPS receiver (e.g., connected via a serial port or Bluetooth). Consistent with the present disclosure, the processing device 360 can modify the presentation of the virtual content based on the input from the other sensor 375. For example, presenting personal information only after identifying the user 100 using data from a biometric sensor.

[0136] The components and arrangements shown in FIG. 3 are not intended to limit the disclosed embodiments. As will be understood by those having the benefit of this disclosure, numerous variations and / or modifications can be made to the illustrated configuration of the input unit 202. For example, not all components are essential for the operation of the input unit. Any component can be placed in any suitable part of the input unit, and the components can be rearranged in various configurations while providing the functions of the disclosed embodiments. For example, some input units may not include all of the elements as shown in the input unit 202.

[0137] FIG. 4 is a block diagram showing a configuration example of the XR unit 204. It should be understood that FIG. 4 is a typical representation of just one embodiment, and within the scope of the present disclosure, some of the illustrated elements can be omitted and other elements can be added. In the embodiment of FIG. 4, the XR unit 204 can directly or indirectly access a bus 400 (or other communication mechanism) that interconnects subsystems and components for transferring information within the XR unit 204. For example, the bus 400 can interconnect a memory interface 410, a network interface 420, an input interface 430, a power supply 440, an output interface 450, a processing device 460, a sensor interface 470, and a database 480.

[0138] The memory interface 410 shown in FIG. 4 is assumed to have the same functions as those of the memory interface 310 described in detail above. The memory interface 410 can be used to access software products and / or data stored in a non-transitory computer-readable medium or a memory device such as the memory device 411. The memory device 411 can include software modules for executing processes consistent with the present disclosure. In particular, the memory device 411 can include an input determination module 412, an output determination module 413, a sensor communication module 414, a virtual content determination module 415, a virtual content communication module 416, and a database access module 417. The modules 412 to 417 may include software instructions for execution by at least one processor (e.g., the processing device 460) associated with the XR unit 204. The input determination module 412, the output determination module 413, the sensor communication module 414, the virtual content determination module 415, the virtual content communication module 416, and the database access module 417 can cooperate to perform various operations. For example, the input determination module 412 can determine a user interface (UI) input received from the input unit 202. At the same time, the sensor communication module 414 can receive data from different sensors to determine the state of the user 100. The virtual content determination module 415 can determine virtual content to be displayed based on the received input and the determined state of the user 100. The virtual content communication module 416 can search for virtual content not determined by the virtual content determination module 415. The search for virtual content may be from the database 380, the database 480, the mobile communication device 206, or the remote processing unit 208. Based on the output of the virtual content determination module 415, the output determination module 413 can cause a change in the virtual content displayed to the user 100 by the projector 454.

[0139] In some embodiments, the input determination module 412 can adjust the operation of the input interface 430 to receive gesture input 431, virtual input 432, voice input 433, and UI input 434. Consistent with the present disclosure, the input determination module 412 can receive different types of input data simultaneously. In one embodiment, the input determination module 412 can apply different rules based on the type of detected input. For example, gesture input can be prioritized over virtual input. In some embodiments, the output determination module 413 can adjust the operation of the output interface 450 to generate an output using the optical indicator 451, display 452, speaker 453, and projector 454. In one embodiment, the optical indicator 451 can include an optical indicator indicating the state of the wearable extended reality device. For example, the optical indicator can display a green light when the wearable extended reality device 110 is connected to the input unit 202 and can blink when the battery of the wearable extended reality device 110 is low. In another embodiment, the display 452 can be used to display operation information. In another embodiment, the speaker 453 can include bone conduction headphones used to output sound to the user 100. In another embodiment, the projector 454 can present virtual content to the user 100.

[0140] The operations of the sensor communication module, virtual content determination module, virtual content communication module, and database access module have been described above with reference to FIG. 3, and the details thereof will not be repeated here. Modules 412-417 may be implemented in software, hardware, firmware, a combination thereof, or the like.

[0141] The network interface 420 shown in FIG. 4 is assumed to have the same functions as the network interface 320 described in detail above. The specific design and implementation of the network interface 420 may depend on the communication network in which the XR unit 204 is intended to operate. For example, in some embodiments, the XR unit 204 is configured to be selectively connectable to the input unit 202 by wire. When connected by wire, the network interface 420 can enable communication with the input unit 202, and when not connected by wire, the network interface 420 can enable communication with the mobile communication device 206.

[0142] The input interface 430 shown in FIG. 4 is assumed to have the same functions as the input interface 330 described in detail above. In this case, the input interface 430 may communicate with an image sensor to obtain a gesture input 431 (e.g., the finger of user 100 pointing to a virtual object), communicate with other XR units 204 to obtain a virtual input 432 (e.g., a virtual object shared with the XR unit 204, or a gesture of an avatar detected in the virtual environment), communicate with a microphone to obtain an audio input 433 (e.g., an audio command), and communicate with the input unit 202 to obtain a UI input 434 (e.g., virtual content determined by the virtual content determination module 315).

[0143] The power supply 440 shown in FIG. 4 is assumed to have the same functions as the power supply 340 described above, and supplies electrical energy to power the XR unit 204. In some embodiments, the power supply 440 may be charged by the power supply 340. For example, the power supply 440 may be wirelessly changed when the XR unit 204 is placed on or near the input unit 202.

[0144] The output interface 450 shown in FIG. 4 is assumed to have the same functions as the output interface 350 described in detail above. In this case, the output interface 450 can produce outputs from the optical indicator 451, the display 452, the speaker 453, and the projector 454. The projector 454 may be any device, apparatus, instrument, etc. that can project (or direct) light to display virtual content on a surface. The surface may be part of the XR unit 204, part of the user 100's eye, or part of an object proximate to the user 100. In one embodiment, the projector 454 can include an illumination unit that focuses light within a limited solid angle by one or more mirrors and lenses and provides a high value of luminous intensity in a defined direction.

[0145] The processing device 460 shown in FIG. 4 is assumed to have the same functions as the processing device 360 described in detail above. When the XR unit 204 is connected to the input unit 202, the processing device 460 may cooperate with the processing device 360. Specifically, the processing device 460 can implement virtual machine technology or other technologies to provide the ability to execute, control, execute, operate, store, etc. a plurality of software processes, applications, programs, etc. It should be understood that other types of processor configurations can be implemented to provide the capabilities disclosed herein.

[0146] The sensor interface 470 shown in FIG. 4 is assumed to have the same functions as the sensor interface 370 described in detail above. Specifically, the sensor interface 470 can communicate with the voice sensor 471, the image sensor 472, the motion sensor 473, the environmental sensor 474, and other sensors 475. The operations of the voice sensor, the image sensor, the motion sensor, the environmental sensor, and other sensors have been described above with reference to FIG. 3, and the details are not repeated here. It is understood that other types and combinations of sensors may be used to provide the capabilities disclosed herein.

[0147] The components and arrangements shown in FIG. 4 are not intended to limit the disclosed embodiments. As will be understood by those of ordinary skill in the art having the benefit of this disclosure, numerous variations and / or modifications can be made to the configuration of the illustrated XR unit 204. For example, not all components may be essential for the operation of the XR unit 204 in all cases. Any component may be disposed in any suitable part of the system 200, and the components may be rearranged in various configurations while providing the functions of the disclosed embodiments. For example, some XR units may not include all of the elements within the XR unit 204 (e.g., the wearable extended reality device 110 may not have the optical indicator 451).

[0148] FIG. 5 is a block diagram showing a configuration example of the remote processing unit 208. It should be understood that FIG. 5 is a typical representation of only one embodiment, and within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added. In the embodiment of FIG. 5, the remote processing unit 208 can include a server 210 that directly or indirectly accesses a bus 500 (or other communication mechanism) that interconnects subsystems and components for transferring information within the server 210. For example, the bus 500 can interconnect a memory interface 510, a network interface 520, a power supply 540, a processing device 560, and a database 580. The remote processing unit 208 can also include one or more data structures. For example, data structures 212A, 212B, and 212C.

[0149] The memory interface 510 shown in FIG. 5 is assumed to have the same functions as the memory interface 310 described in detail above. The memory interface 510 can be used to access software products and / or data stored in a non-transitory computer-readable medium or in other memory devices such as the memory devices 311, 411, 511, or the data structures 212A, 212B, and 212C. The memory device 511 can include software modules for executing processes consistent with the present disclosure. In particular, the memory device 511 can include a shared memory module 512, a node registration module 513, a load distribution module 514, one or more computing nodes 515, an internal communication module 516, an external communication module 517, and a database access module (not shown). The modules 512-517 can include software instructions for execution by at least one processor (e.g., the processing device 560) associated with the remote processing unit 208. The shared memory module 512, the node registration module 513, the load distribution module 514, the computing module 515, and the external communication module 517 can cooperate to perform various operations.

[0150] The shared memory module 512 can enable information sharing between the remote processing unit 208 and other components of the system 200. In some embodiments, the shared memory module 512 may be configured to allow the processing device 560 (and other processing devices within the system 200) to access, search, and store data. For example, using the shared memory module 512, the processing device 560 can perform at least one of the steps of executing a software program stored in the memory device 511, the database 580, or the data structures 212A-C, storing information in the memory device 511, the database 580, or the data structures 212A-C, or searching for information from the memory device 511, the database 580, or the data structures 212A-C.

[0151] The node registration module 513 can be configured to track the availability of one or more computing nodes 515. In some examples, the node registration module 513 may be implemented as a software program, such as a software program executed by one or more computing nodes 515, a hardware solution, or a combined software and hardware solution. In some embodiments, the node registration module 513 can communicate with one or more computing nodes 515 using, for example, the internal communication module 516. In some examples, one or more computing nodes 515 can notify the node registration module 513 of their status by, for example, sending a message at startup, shutdown, at regular intervals, at a selected time, in response to a query received from the node registration module 513, or at any other determined time. In some examples, the node registration module 513 can query the status of one or more computing nodes 515 by, for example, sending a message at startup, at regular intervals, at a selected time, or at any other determined time.

[0152] The load distribution module 514 can be configured to divide the workload among one or more computing nodes 515. In some examples, the load distribution module 514 may be implemented as a software program such as a software program executed by one or more of the computing nodes 515, a hardware solution, or a combined software and hardware solution. In some embodiments, the load distribution module 514 can interact with the node registration module 513 to obtain information regarding the availability of one or more computing nodes 515. In some embodiments, the load distribution module 514 can communicate with one or more computing nodes 515 using, for example, the internal communication module 516. In some examples, one or more computing nodes 515 can notify the load distribution module 514 of their status by responding to queries received from the load distribution module 514, or by sending messages at startup, shutdown, at regular intervals, at selected times, or at any other determined time. In some examples, the load distribution module 514 can query the status of one or more computing nodes 515 by sending messages at startup, at regular intervals, at preselected times, or at any other determined time.

[0153] The internal communication module 516 may be configured to receive and / or transmit information from one or more components of the remote processing unit 208. For example, control signals and / or synchronization signals can be transmitted and / or received via the internal communication module 516. In one embodiment, input information of a computer program, output information of a computer program, and / or intermediate information of a computer program can be transmitted and / or received via the internal communication module 516. In another embodiment, information received via the internal communication module 516 may be stored in the memory device 511, the database 580, the data structures 212A - C, or other memory devices within the system 200. For example, information retrieved from the data structure 212A may be transmitted using the internal communication module 516. In another example, input data can be received using the internal communication module 516 and stored in the data structure 212B.

[0154] The external communication module 517 may be configured to receive and / or transmit information from one or more components of the system 200. For example, control signals can be transmitted and / or received via the external communication module 517. In one embodiment, information received via the external communication module 517 can be stored in the memory device 511, the database 580, the data structures 212A - C, and / or any memory device within the system 200. In another embodiment, information retrieved from any of the data structures 212A - C may be transmitted to the XR unit 204 using the external communication module 517. In another embodiment, input data can be transmitted and / or received using the external communication module 517. Examples of such input data can include data received from the input unit 202, information captured from the environment of the user 100 using one or more sensors (e.g., audio sensor 471, image sensor 472, motion sensor 473, environmental sensor 474, other sensors 475), etc.

[0155] In some embodiments, the aspects of modules 512-517 may be implemented in hardware, software (including one or more signal processing and / or application specific integrated circuits), firmware, or any combination thereof, and be executable by one or more processors, either alone or in various combinations with each other. Specifically, modules 512-517 may be configured to interact with each other and / or with other modules of system 200 to perform functions consistent with the disclosed embodiments. Memory device 511 may include additional modules and instructions or fewer modules and instructions.

[0156] The network interface 520, power supply 540, processing device 560, and database 580 shown in FIG. 5 are assumed to have functions similar to those of the similar elements described above with reference to FIGS. 4 and 5. The specific design and implementation of the above-described components may vary based on the embodiments of system 200. Further, remote processing unit 208 may include more or fewer components. For example, remote processing unit 208 may include an input interface configured to receive input directly from one or more input devices.

[0157] In accordance with the present disclosure, a processing device of system 200 (e.g., a processor within mobile communication device 206, a processor within server 210, a processor within a wearable extended reality device such as wearable extended reality device 110, and / or a processor within an input device associated with wearable extended reality device 110 such as keyboard 104) can use a machine learning algorithm to implement any of the methods disclosed herein. In some embodiments, a machine learning algorithm (also referred to as a machine learning model in the present disclosure) can be trained using training examples, as described, for example, below. Some non-limiting examples of such machine learning algorithms include classification algorithms, data regression algorithms, image segmentation algorithms, visual detection algorithms (such as object detectors, face detectors, person detectors, motion detectors, edge detectors, etc.), visual recognition algorithms (such as face recognition, person recognition, object recognition, etc.), speech recognition algorithms, mathematical embedding algorithms, natural language processing algorithms, support vector machines, random forests, nearest neighbor algorithms, deep learning algorithms, artificial neural network algorithms, convolutional neural network algorithms, recurrent neural network algorithms, linear machine learning models, non-linear machine learning models, ensemble algorithms, and the like. For example, a trained machine learning algorithm may include an inference model such as a prediction model, a classification model, a data regression model, a clustering model, a segmentation model, an artificial neural network (e.g., a deep neural network, a convolutional neural network, a recurrent neural network, etc.), a random forest, a support vector machine, and the like. In some examples, a training example can include an input of the example and a desired output corresponding to the input of the example. Further, in some examples, a machine learning algorithm trained using training examples can generate a trained machine learning algorithm, and the trained machine learning algorithm can be used to estimate an output for an input not included in the training examples.In some examples, the engineers, scientists, processes, and machines that train machine learning algorithms can further use validation examples and / or test examples. For example, the validation examples and / or test examples can include exemplary inputs along with the desired outputs corresponding to the exemplary inputs, and the trained machine learning algorithm and / or the intermediate trained machine learning algorithm can be used to estimate the outputs of the exemplary inputs of the validation examples and / or test examples, the estimated outputs can be compared with the corresponding desired outputs, and the trained machine learning algorithm and / or the intermediate trained machine learning algorithm can be evaluated based on the results of the comparison. In some examples, the machine learning algorithm can have parameters and hyperparameters, the hyperparameters can be set manually by a person or automatically by a process external to the machine learning algorithm (such as a hyperparameter search algorithm), and the parameters of the machine learning algorithm can be set by the machine learning algorithm based on the training examples. In some embodiments, the hyperparameters may be set based on the training examples and the validation examples, and the parameters may be set based on the training examples and the selected hyperparameters. For example, given the hyperparameters, the parameters may be conditionally independent from the validation examples.

[0158] In some embodiments, a trained machine learning algorithm (also referred to herein as a machine learning model and a trained machine learning model) can be used to analyze an input and generate an output, for example, as described below. In some examples, the trained machine learning algorithm can be used as an inference model that generates an inferred output when an input is provided. For example, the trained machine learning algorithm can include a classification algorithm, the input can include samples, and the inferred output can include the classification of the samples (e.g., a predicted label, a predicted tag, etc.). In another example, the trained machine learning algorithm can include a regression model, the input can include samples, and the inferred output can include an inferred value corresponding to the samples. In yet another example, the trained machine learning algorithm can include a clustering model, the input can include samples, and the inferred output can include the assignment of the samples to at least one cluster. In an additional example, the trained machine learning algorithm can include a classification algorithm, the input can include an image, and the inferred output can include the classification of the item depicted in the image. In yet another example, the trained machine learning algorithm can include a regression model, the input can include an image, and the inferred output can include an inferred value corresponding to the item depicted in the image (e.g., an estimated characteristic of the item such as the size, volume, age of the person depicted in the image, the distance from the item depicted in the image, etc.). In an additional example, the trained machine learning algorithm can include an image segmentation model, the input can include an image, and the inferred output can include the segmentation of the image. In yet another example, the trained machine learning algorithm can include an object detector, the input can include an image, and the inferred output can include one or more detected objects within the image and / or one or more positions of the objects within the image.In some examples, the trained machine learning algorithm can include one or more formulas and / or one or more functions and / or one or more rules and / or one or more procedures, the input can be used as input to the formula and / or function and / or rule and / or procedure, and the inferred output can be based on the output of the formula and / or function and / or rule and / or procedure (e.g., selecting one of the outputs of the formula and / or function and / or rule and / or procedure, using a statistical measure of the output of the formula and / or function and / or rule and / or procedure, etc.).

[0159] In accordance with the present disclosure, the processing device of system 200 can analyze image data captured by an image sensor (e.g., image sensor 372, image sensor 472, or any other image sensor) to implement any of the methods disclosed herein. In some embodiments, analyzing the image data can include analyzing the image data to obtain preprocessed image data and then analyzing the image data and / or the preprocessed image data to obtain a desired result. Those skilled in the art will recognize that the following are examples, and that the image data can be preprocessed using other types of preprocessing methods. In some examples, the image data can be preprocessed by using a conversion function to convert the image data to obtain converted image data, and the preprocessed image data can include the converted image data. For example, the converted image data can include one or more convolutions of the image data. For example, the conversion function can include one or more image filters such as a low-pass filter, a high-pass filter, a band-pass filter, an all-pass filter, etc. In some examples, the conversion function can include a non-linear function. In some examples, the image data can be preprocessed by smoothing at least a portion of the image data, such as using a Gaussian convolution or using a median filter. In some examples, the image data can be preprocessed to obtain different representations of the image data. For example, the preprocessed image data can include a representation of at least a portion of the image data in the frequency domain, a discrete Fourier transform of at least a portion of the image data, a discrete wavelet transform of at least a portion of the image data, a time / frequency representation of at least a portion of the image data, a low-dimensional representation of at least a portion of the image data, an irreversible representation of at least a portion of the image data, a reversible representation of at least a portion of the image data, any of the above chronological sequences, and any combination of the above. In some examples, the image data can be preprocessed to extract edges, and the preprocessed image data can include information based on and / or related to the extracted edges. In some examples, the image data can be preprocessed to extract image features from the image data.Some non-limiting examples of such image features can include and / or be related to information based on edges, corners, blobs, ridges, scale-invariant feature transform (SIFT) features, temporal features, etc. In some examples, analyzing the image data can include calculating at least one convolution of at least a portion of the image data and using the at least one calculated convolution to calculate at least one resultant value and / or perform a determination, identification, recognition, classification, etc.

[0160] In accordance with other aspects of the present disclosure, the processing device of system 200 can analyze image data to implement any of the methods disclosed herein. In some embodiments, analyzing the image can include analyzing the image data and / or pre-processed image data using one or more rules, functions, procedures, artificial neural networks, object detection algorithms, face detection algorithms, visual event detection algorithms, action detection algorithms, motion detection algorithms, background subtraction algorithms, inference models, etc. Some non-limiting examples of such inference models can include results for training examples of training algorithms such as manually pre-programmed inference models, classification models, regression models, machine learning algorithms, and / or deep learning algorithms, where the training examples can include examples of data instances, and in some cases, the data instances may be labeled with corresponding desired labels and / or results, etc. In some embodiments, analyzing the image data (e.g., by the methods, steps, and modules described herein) can include analyzing pixels, voxels, point clouds, distance data, etc. included in the image data.

[0161] Convolution may include convolution in any dimension. One-dimensional convolution is a function that transforms an original sequence of numbers into a transformed sequence of numbers. One-dimensional convolution can be defined by a series of scalars. Each specific value within the transformed sequence can be determined by calculating a linear combination of the values within the subsequence of the original sequence that corresponds to the specific value. The resulting value of the calculated convolution can include any value within the transformed sequence. Similarly, n-dimensional convolution is a function that transforms an original n-dimensional array into a transformed array. N-dimensional convolution can be defined by an n-dimensional array of scalars (known as the kernel of the n-dimensional convolution). Each specific value within the transformed array can be determined by calculating a linear combination of the values within the n-dimensional region of the original array that corresponds to the specific value. The resulting value of the calculated convolution can include any value within the transformed array. In some examples, an image can include one or more components (e.g., color components, depth components, etc.), and each component can include a two-dimensional array of pixel values. In one example, calculating the convolution of an image can include calculating a two-dimensional convolution on one or more components of the image. In another example, calculating the convolution of an image can include stacking arrays from different components to create a three-dimensional array and calculating a three-dimensional convolution on the resulting three-dimensional array. In some examples, a video can include one or more components (e.g., color components, depth components, etc.), and each component can include a three-dimensional array of pixel values (having two spatial axes and one temporal axis). In one example, calculating the convolution of a video can include calculating a three-dimensional convolution on one or more components of the video. In another example, calculating the convolution of a video can include stacking arrays from different components to create a four-dimensional array and calculating a four-dimensional convolution on the resulting four-dimensional array.

[0162] In some embodiments, the integrated computing interface device can include a portable housing having a key area and a non-key area. The housing of the integrated computing interface device can include an outer cover or shell that can include one or more components associated with the integrated computing interface device. The disclosed exemplary housing can surround the components of the integrated computing interface device and can cover some or all of the components of the integrated computing interface device. The disclosed exemplary housing can be considered to have one or more openings that can expose certain components of the integrated computing interface device (e.g., a USB or other port) or can project certain components (e.g., keys of a keyboard) out of the housing. The housing can support certain components of the integrated computing interface device (e.g., a circuit board) in an inner portion of the housing. The housing can include additional structural features that can enable one or more components of the integrated computing interface device to be attached to the housing. The housing can be square, rectangular, or other shapes sized to fit on a user's desk, lap, or other suitable work surface. The housing can be made of plastic, metal, a combination of plastic and metal, or other suitable materials.

[0163] The housing of an all-in-one computing device may include a key area and a non-key area different from the key area. The key area of the all-in-one computing interface device can include one or more keys that enable a user to input alphanumeric or other characters as input. For example, in some embodiments, a keyboard may be associated with the key area of the housing. The keyboard may be a standard typewriter-style keyboard (e.g., a QWERTY-style keyboard), or another suitable keyboard layout such as a Dvorak layout or a coded layout. The keyboard can include any suitable number of keys. For example, a "full-size" keyboard may include up to 104 or 105 keys. In some embodiments, the keyboard may include at least 30 keys. In other embodiments, the keyboard may include less than 10 keys, at least 10 keys, at least 20 keys, at least 50 keys, at least 80 keys, at least 100 keys, etc.

[0164] The key area may include alphanumeric keys, function keys, modifier keys, cursor keys, system keys, multimedia control keys, or other physical keys that perform computer-specific functions when pressed. The key area may also include virtual keys or programmable keys such that the function of the keys changes depending on the function or application being executed on the all-in-one computing interface device.

[0165] In some embodiments, the keyboard may include dedicated input keys for performing actions by the wearable extended reality device. The dedicated input keys can enable the user to interact with virtual widgets viewed via the wearable extended reality device. The dedicated input keys can take a photo of one or more virtual widgets displayed via the wearable extended reality device (i.e., a "screenshot"). When the wearable extended reality device includes a camera, the dedicated input keys can take a photo with the camera. In one embodiment, the image can include what the user views through the wearable extended reality device using any virtual widgets included in the image (i.e., an image with a virtual overlay). The keyboard can include multiple dedicated input keys for performing actions, and each key is configured to perform a different action. In one embodiment, the dedicated input keys can be programmable by the user to perform one or more actions (e.g., a "macro"). Note that the examples of actions performed by the dedicated input keys above are not limiting, and other actions may be performed by the dedicated input keys.

[0166] In some embodiments, the keyboard may include dedicated input keys for changing the illumination of a virtual display projected by the wearable extended reality device. The illumination of the virtual display can be adjusted by turning the virtual display on or off, or by increasing or decreasing the brightness, contrast, or other color settings of the virtual display. In one embodiment, the keyboard can include multiple dedicated input keys for adjusting different settings of the virtual display. The settings of the virtual display that can be adjusted can include, for example, image settings such as brightness, contrast, sharpness, or display mode (e.g., game mode according to a predetermined setting), color settings such as color component levels or other color adjustment settings, the position of the virtual display relative to the position of the user's head, or other settings that can improve the user's field of view of the virtual display.

[0167] The non-key area of the portable housing may be an area of the housing that does not contain a key. The non-key area may be an area of the housing that does not contain a key, or may exist to complete a desired shape of the housing that extends beyond the key area of the housing in any direction. The non-key area may be an area that may include input devices such as a trackpad, a touch screen, a touch bar, or other forms of cursor control for an integrated computing interface device. The non-key area may be subdivided into multiple different non-key areas, such as a trackpad or other cursor control unit, an extension of the housing, or a cover or grill for one or more speakers or other audio output devices included within the housing. The non-key area can include a display for presenting information to the user, can include one or more openings that allow air to circulate through the housing (e.g., to cool components housed within the housing), or can include one or more doors or access ports that allow access to an interior portion of the housing (e.g., a battery compartment configured to hold a removable battery internally or to allow installation and / or removal of certain components from within the housing).

[0168] In some embodiments, the holder may be associated with the non-key area of the housing. The holder may be a recess that extends beneath the surface of the housing, or may be completely seated on top of the surface of the housing. The holder may be disposed at any portion of the non-key area of the housing. In some embodiments, the holder may be disposed in the non-key area of the housing adjacent to the key area of the housing. The holder can include one or more structural features for selectively engaging one or more items, such as a writing instrument, a wire or cable, a dongle, paper (i.e., to enable the holder to function like a copy stand), or other items that the user may desire to easily access or store.

[0169] In some embodiments, the holder may be configured for selective engagement and disengagement with a wearable extended reality device such that when the wearable extended reality device selectively engages with the housing via the holder, the wearable extended reality device is transportable with the housing. In some embodiments, the structural features of the holder may be configured to selectively engage with the wearable extended reality device such that the wearable extended reality device can snap fit, press fit, or pressure fit into at least a portion of the holder. When the wearable extended reality device selectively engages with the housing via the holder, the wearable extended reality device may be securely connected to the housing via the holder and thus be transportable with the housing.

[0170] FIG. 6 is a top view of an exemplary embodiment of an integrated computing interface device 610 having a wearable extended reality device 612 that selectively engages with the integrated computing interface device 610. It should be understood that FIG. 6 is only an exemplary representation of one embodiment and that within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added.

[0171] The integrated computing interface device 610 may include a housing 614. The housing 614 may include a key region 616 and non-key regions 618a, 618b. As shown in FIG. 6, the non-key region 618a may include a region located above the key region 616, and the non-key region 618b may include a region located below the key region 616. The housing 614 may include only one non-key region (e.g., only one of the non-key regions 618a or 618b), the non-key regions 618a and 618b may be adjacent to each other, or it may have two or more non-key regions.

[0172] The key area 616 may include a keyboard 620. The key area 616 may be subdivided into a plurality of key areas, and the key areas may be continuous or separated from each other.

[0173] The integrated computing interface device 610 may include a holder 622 configured to selectively engage with the wearable extended reality device 612. In some embodiments, the wearable extended reality device may include a pair of smart glasses. As shown in FIG. 6, the wearable extended reality device 612 may be a pair of smart glasses. The smart glasses may look like conventional glasses and may include "smart" features such as a camera arranged to take a photo of what the user is currently looking at and one or more displays configured to project an image onto the lenses of the smart glasses. In some embodiments, the wearable extended reality device 612 may take other forms including one or more lenses, such as goggles or other forms of wearable devices.

[0174] In some embodiments, the integrated computing interface device may include a trackpad associated with the housing. The trackpad may enable a user of the integrated computing interface device to control a cursor, select items, or activate items on the integrated computing interface device. The trackpad may include a single surface or a segmented surface, such as a cursor control portion and one or more button portions.

[0175] In an embodiment where the wearable extended reality device is a pair of smart glasses, the integrated computing interface device is configured such that when the pair of smart glasses selectively engages with the housing via the holder, the temple of the smart glasses contacts the track pad. Thus, for example, the holder may be disposed on a first portion of the housing, the track pad may be disposed on a second portion of the housing, and the first portion of the housing is spaced apart from the second portion of the housing. The holder may be spaced apart from the track pad by a distance approximately equal to the length of the temple portion of the pair of smart glasses.

[0176] The temples of the smart glasses can each include an elastic track pad protector at their distal ends. The temples may extend parallel to each other in one direction from the lenses of the smart glasses to enable the smart glasses to be worn by the user. In some examples, the pair of smart glasses may include at least two temples and at least one lens. Each temple can include a temple tip portion, and each temple tip portion can include an outer portion of the elastic track pad protector. In some examples, the pair of smart glasses can include at least two temples and at least one lens, and each temple can include an elastic track pad protector. The track pad protector can protect the distal ends of the temples of the smart glasses from damage when the smart glasses selectively engage with the holder and the distal ends of the temples are close to the surface of the housing (e.g., the track pad). The track pad protector may include a sleeve that slides over the distal ends of the temples or may be integrally formed with the distal ends of the temples. The track pad protector may be made of a soft or flexible material so that when the pair of smart glasses selectively engages with the housing via the holder and the distal ends of the temples of the pair of smart glasses contact the track pad, the distal ends of the temples of the pair of smart glasses do not scratch or damage the track pad.

[0177] FIG. 7A is a top view of a second exemplary embodiment of an integrated computing interface device 710 having a wearable extended reality device in the form of a pair of smart glasses 712 that selectively engage with the integrated computing interface device 710, and FIG. 7B is a left side view. FIGS. 7A and 7B are exemplary representations of only one embodiment, and it should be understood that within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added.

[0178] The integrated computing interface device 710 can include a housing 714. The housing 714 may include a key region 716 and non-key regions 718a, 718b. The key region 716 can include a keyboard 720. The integrated computing interface device 710 can include a holder 722 configured to selectively engage with a pair of smart glasses 712. The non-key region 718b may be disposed under the keyboard 720 and may include a trackpad 724. The integrated computing interface device 710 shown in FIGS. 7A and 7B can have structural and functional characteristics similar to those of the integrated computing interface device 610 shown in FIG. 6. For example, the integrated computing interface device 710 can include all or part of the elements of the integrated computing interface device 610.

[0179] A pair of smart glasses 712 can include a lens portion 726 and two temples 728, one temple at each end of the lens portion 726. Each temple 728 can include a proximal end 730 that connects the temple 728 to the lens portion 726 and a distal end 732 located at the opposite end of the temple 728. The distal end 732 can include an elastic track pad protector 734. As described above, the elastic track pad protector 734 may be disposed on the distal end 732 or may be integrally formed with the distal end 732.

[0180] In some embodiments, the holder of the integrated computing interface device can include at least two gripping elements configured to selectively engage with a template of the smart glasses. In some embodiments, the gripping elements may be configured to selectively engage with different portions of the smart glasses, such as one or both lenses, the bridge between the lenses, or other portions of the smart glasses. The gripping elements may be integrally formed with the holder, removably individual from the holder, or removably together from the holder. For example, the gripping elements may be integrally formed with each other and removably as a unit from the holder. The gripping elements may be spring-biased towards each other by their shape or by using a spring or spring-like member.

[0181] Each gripping element can include a protrusion from the surface of the integrated computing interface device. The protrusion may extend perpendicularly away from the surface of the integrated computing interface device. A recess may be formed within the protrusion and configured to hold the template. The recess may be on a side of the protrusion opposite to the surface of the integrated computing interface device such that the template is disposed on the upper surface of the protrusion. The recess may be on a side of the protrusion parallel to the surface of the integrated computing interface device such that the template is disposed on the side of the protrusion. The holder may be made of a flexible material, a rigid or semi-rigid material, or a rigid or semi-rigid material with flexible recesses. The flexible recesses may be integrally formed with the protrusion or may be a flexible material covering the recess.

[0182] FIG. 8A is a front perspective view of a wearable extended reality device selectively engaged with a first embodiment of a holder. FIG. 8B is a rear perspective view of the wearable extended reality device selectively disengaged from the first embodiment of the holder shown in FIG. 8A. It should be understood that FIGS. 8A and 8B are merely representative of one embodiment and that within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added.

[0183] The holder 810 can include two gripping elements 812 configured to selectively engage a template 814 of a wearable extended reality device 816 shown as a pair of smart glasses in FIGS. 8A and 8B and spaced apart from each other. In one example, the holder 810 may be part of the holder 622 and / or the holder 722 and / or the holder 1218a. In some embodiments, other configurations of the gripping elements 812 may be possible, such as only one gripping element 812 or two gripping elements 812 may be disposed on the same side of the holder 810.

[0184] In some embodiments, each gripping element 812 can include a recess 818 on an upper surface for engaging the template 814. In some embodiments, each gripping element 812 can have a flat upper surface for engaging the template 814. As shown in FIG. 8B, the recess 818 may be U-shaped to partially surround the template 814. In some embodiments, the recess 818 can be of a different shape to engage the template 814.

[0185] In some embodiments, the holder of the integrated computing interface device can include a clip for selectively connecting the wearable extended reality device to the housing. Selectively connecting the wearable extended reality device to the housing is an example of selectively engaging the wearable extended reality device with the housing. The clip can be disposed at any portion of the holder and can include a protrusion from the surface of the integrated computing interface device. The clip can selectively engage with any portion of the wearable extended reality device to connect the wearable extended reality device to the holder. In embodiments where the wearable extended reality device is a pair of smart glasses, the clip can selectively engage with a temple, a portion of the lens, a portion of the rim surrounding the lens, a bridge, or a nose pad. The holder can include additional mechanisms for selectively engaging with other portions of the wearable extended reality device that are not selectively engaged by the clip. The additional features can include one or more additional protrusions extending from the surface of the holder. The holder can be made of a flexible material, a rigid or semi-rigid material, or a rigid or semi-rigid material having a flexible clip or flexible protrusions. The flexible clip or flexible protrusions can be integrally formed with the holder, removable from the holder, or a flexible material covering the clip or protrusion.

[0186] FIG. 9A is a front perspective view of a wearable extended reality device selectively engaged with a second exemplary embodiment of a holder. FIG. 9B is a rear perspective view of the wearable extended reality device selectively disengaged from the second embodiment of the holder shown in FIG. 9A. It is to be understood that FIGS. 9A and 9B are exemplary representations of only one embodiment and that within the scope of the present disclosure, some of the illustrated elements can be omitted and other elements can be added.

[0187] The holder 910 can include a clip 912 configured to selectively engage with a bridge 914 of the wearable extended reality device 916. As shown in the figures, as shown in FIGS. 9A and 9B, the wearable extended reality device 916 can include a pair of smart glasses. In one example, the holder 910 may be part of the holder 622 and / or the holder 722 and / or the holder 1218a. In a typical embodiment as shown in FIG. 9B, the clip 912 can include a post 918 configured to fit between nose pads 920 of the wearable extended reality device 916. As shown in FIG. 9B, the post 918 may have a circular or cylindrical shape. The post 918 may have other shapes (e.g., having a square, rectangular, elliptical, or polygonal cross-section) such that the nose pads 920 of the wearable extended reality device 916 fit around the post 918 and selectively engage with the post 918.

[0188] The clip 912 can include a bridge protrusion 922 configured to contact a front portion of the bridge 914. The bridge protrusion 922 may be spaced apart from the post 918 such that when the wearable extended reality device 916 selectively engages with the holder 910, a portion of the bridge 914 is positioned between the bridge protrusion 922 and the post 918.

[0189] The holder 910 can include at least two lens protrusions 924. The lens protrusions 924 are spaced apart from the clip 912 such that each lens protrusion 924 selectively engages with an outer portion of a lens 926 of the wearable extended reality device 916. As shown in FIG. 9A, the bridge protrusion 922 may be shaped such that an inner portion of the lens 926 of the wearable extended reality device 916 selectively engages with the bridge protrusion 922.

[0190] In some embodiments, clip 912 may include only post 918. In some embodiments, holder 910 may include only clip 912 and may not include lens protrusion 924. In some embodiments, holder 910 may include only lens protrusion 924 and may not include clip 912.

[0191] In some embodiments, the holder of the integrated computing interface device can include a compartment for selectively surrounding at least a portion of the wearable extended reality device when the wearable extended reality device selectively engages with the holder. The compartment can include a recess or a sleeve within the housing for receiving one or more portions of the wearable extended reality device. The compartment can include a protrusion from the surface of the holder and can be shaped such that when the wearable extended reality device selectively engages with the holder, the wearable extended reality device does not slide back and forth or side to side within the compartment. The protrusion can include one or more walls extending above the surface of the holder such that the walls surround a portion of the wearable extended reality device. The walls may be configured to accommodate different shapes of the wearable extended reality device. For example, if the wearable extended reality device is a pair of smart glasses, the walls can include cutout portions such that the nose pads of the smart glasses do not contact the walls. The walls may also taper towards the surface of the holder to accommodate the lenses of the smart glasses such that the bottom portion of the lens contacts the surface of the holder. The holder can be made of a flexible material, a rigid or semi-rigid material, or a rigid or semi-rigid material having a flexible compartment. The flexible compartment may be integrally formed with the holder or may be a flexible material covering the compartment.

[0192] FIG. 10A is a front perspective view of a wearable extended reality device that selectively engages with a third exemplary embodiment of a holder. FIG. 10B is a rear perspective view of the wearable extended reality device selectively disengaged from the third exemplary embodiment of the holder shown in FIG. 10A. FIGS. 10A and 10B are merely exemplary representations of one embodiment, and it should be understood that within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added.

[0193] The holder 1010 can include a compartment 1012 configured to selectively surround at least a portion of the wearable extended reality device 1014. In one example, the holder 1010 may be part of the holder 622 and / or the holder 722 and / or the holder 1218a. The compartment 1012 can include a wall 1016 that extends along a portion of the surface of the holder 1010 such that the wall 1016 defines the outer perimeter of the compartment 1012. As shown in FIG. 10A, the wall 1016 may taper towards the center and the surface of the holder 1010 such that when the wearable extended reality device 1014 selectively engages with the holder 1010, the lens 1018 of the wearable extended reality device 1014 contacts the surface of the holder 1010. Based on this exemplary shape, the wall 1016 can contact the side of the lens 1018 facing the user rather than the side facing outside the lens 1018. In some embodiments, the wall 1016 may not taper such that the wall 1016 is of substantially uniform height on all sides to define a slot configured to receive the lens 1018. As shown in FIG. 10B, the wall 1016 can include a cutout portion 1020 for accommodating the nose pad 1022 of the wearable extended reality device 1014 such that the nose pad 1022 of the wearable extended reality device 1014 does not contact the wall 1016.

[0194] In some embodiments, the holder of the integrated computing interface device can include at least one recess corresponding to the shape of a portion of the wearable extended reality device. In embodiments where the wearable extended reality device is a pair of smart glasses, the holder may include one or more recesses shaped to correspond to the shape of the lenses of the smart glasses. In embodiments where the wearable extended reality device is goggles, the one or more recesses may be shaped to correspond to the shape of the lenses of the goggles. The one or more recesses may extend under the surface of the holder such that the one or more recesses extend into a portion of the housing. The bottom of the one or more recesses may contact the surface of the housing such that at least a portion of the holder extends above the surface of the housing. The holder may be made of a flexible material, a rigid or semi-rigid material, or a rigid or semi-rigid material with flexible recesses. The flexible recesses may be integrally formed with the holder or may be a flexible material covering the recesses.

[0195] FIG. 11A is a front perspective view of a wearable extended reality device selectively engaged with a fourth exemplary embodiment of a holder. FIG. 11B is a rear perspective view of the wearable extended reality device selectively disengaged from the fourth exemplary embodiment of the holder shown in FIG. 11A. FIGS. 11A and 11B are exemplary representations of only one embodiment, and it should be understood that within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added.

[0196] The holder 1110 can include a recess 1112 corresponding to a part of the shape of the wearable extended reality device 1114. In one example, the holder 1110 may be part of the holder 622 and / or the holder 722 and / or the holder 1218a. As shown in FIGS. 11A and 11B, the wearable extended reality device 1114 can include a pair of smart glasses, and there may be two recesses 1112 spaced apart from each other within the holder 1110, and each recess 1112 corresponds to the shape of the lens 1116 of the smart glasses.

[0197] In some embodiments, the holder can also include a nose bridge protrusion. In an embodiment where the wearable extended reality device includes a pair of extended reality glasses, at least one recess may include two recesses on both sides of the nose bridge projection for receiving the lenses of the extended reality glasses. Note that the terms "extended reality glasses" and "smart glasses" may be used interchangeably herein. The nose bridge protrusion may be a protrusion extending away from the surface of the holder and may be configured to support a nose pad or the bridge of the smart glasses. Each of the two recesses may be shaped to receive a part of one lens, a rim surrounding the lens, or a frame surrounding the lens. The holder can be made of a flexible material, a rigid or semi-rigid material, or a rigid or semi-rigid material with a flexible nose bridge protrusion. The flexible nose bridge protrusion may be integrally formed with the holder or may be a flexible material covering the nose bridge protrusion.

[0198] Referring back to FIGS. 11A and 11B, the wearable extended reality device 1114 is a pair of smart glasses. The holder 1110 can include a nose bridge protrusion 1118 configured to support the nose pad 1120 or the bridge 1122 of the smart glasses.

[0199] In an embodiment where the wearable extended reality device includes a pair of smart glasses, the holder may be configured such that when the lens of the smart glasses is disposed on one side of the keyboard, the temple of the smart glasses extends over the keyboard and the distal end of the smart glasses is disposed on the side of the keyboard opposite the lens. The holder can include a mechanism that helps to place the temple of the smart glasses on the surface of the housing. The feature can include a protrusion extending upward from the housing for engaging the temple of the smart glasses. The protrusion may be disposed in the key region or the non-key region of the housing.

[0200] In some embodiments, the holder can include a protrusion disposed near the keyboard (e.g., between the holder and the key region), and when the smart glasses selectively engage with the holder, the temple of the smart glasses extends over the keyboard and a gap is created between the temple of the smart glasses and the keyboard such that the distal end of the temple is located on the side surface of the keyboard opposite the lens. The distal end of the temple may not contact the housing because the protrusion can lift the distal end above the surface of the housing. The protrusion may be made of an elastic or other compressible material so that the temple is not damaged or scratched when the temple of the smart glasses contacts the protrusion.

[0201] FIG. 12A is a top view of a third embodiment of an integrated computing interface device 1210 having a wearable extended reality device in the form of a pair of smart glasses 1212 that selectively engages with the integrated computing interface device 1210, and FIG. 12B is a left side view. FIGS. 12A and 12B are exemplary representations of only one embodiment, and it should be understood that within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added. The integrated computing interface device 1210 shown in FIGS. 12A and 12B can have structural and functional characteristics similar to the integrated computing interface device 610 shown in FIG. 6 and / or the integrated computing interface device 710 shown in FIGS. 7A and 7B. For example, the integrated computing interface device 1210 can include all or part of the elements of the integrated computing interface device 610. In another example, the integrated computing interface device 1210 may include all or part of the elements of the integrated computing interface device 710.

[0202] The integrated computing interface device 1210 can include a housing 1214. The housing 1214 may include a keyboard 1216 and a holder 1218a configured to selectively engage with the smart glasses 1212. The holder 1218a can include a protrusion 1220 disposed near the keyboard 1216. When the lens 1222 of the smart glasses 1212 is selectively engaged with the holder 1218a, the temple 1224 contacts the protrusion 1220, thereby creating a gap 1226 between the temple 1224 and the keyboard 1216. When the lens 1222 of the smart glasses 1212 is selectively engaged with the holder 1218a, the temple 1224 can extend over the keyboard 1216 such that the distal end 1228 of the temple 1224 is located on the side opposite the lens 1222 of the keyboard 1216. The distal end 1228 may be spaced apart from the housing 1214 such that the distal end 1228 does not contact the housing 1214 when the smart glasses 1212 are selectively engaged with the holder 1218a.

[0203] In some embodiments, the integrated computing interface device may further include a charger associated with the housing. The charger may be configured to charge the wearable extended reality device when the wearable extended reality device is selectively engaged with the holder. In such embodiments, the wearable extended reality device can include a battery or other power source to be charged. The charger can supply a DC voltage or current to charge the battery of the wearable extended reality device. The battery may be charged by a wired connection or a wireless connection. The housing and the wearable extended reality device may be configured for wireless charging of the wearable extended reality device by the charger. The charger may be disposed at any suitable portion of the housing such that the charger can supply power to the wearable extended reality device when the wearable extended reality device is selectively engaged with the holder.

[0204] The wearable extended reality device can include one or more electrical contacts, and the housing can include one or more corresponding electrical contacts that engage to charge the wearable extended reality device when engaged with the holder. In some examples, one or more of the electrical contacts included in the wearable extended reality device may be disposed on one or both of the lenses, one or both of the temples, or a portion of the frame near where the lens and temple connect. In some examples, one or more of the corresponding electrical contacts included in the housing may be disposed within the holder or within the housing adjacent to the holder such that when the wearable extended reality device selectively engages with the holder, one or more of the electrical contacts of the wearable extended reality device are in sufficient proximity for wireless charging to occur. The wireless charging can be performed according to a wireless charging standard such as Qi, AirFuel Resonant, near-field magnetic coupling (NFMC), radio frequency (RF), or other suitable wireless charging protocol. In some embodiments, the number of electrical contacts included in the housing need not match the number of electrical contacts included in the wearable extended reality device.

[0205] In an embodiment where the wearable extended reality device is a pair of smart glasses, each lens can include an electrical contact, and the housing can include one or more corresponding electrical contacts. When the smart glasses are selectively engaged with the housing, the electrical contacts within the lenses can be disposed in sufficient proximity to one or more of the corresponding electrical contacts within the housing to complete a wireless charging circuit.

[0206] For example, in the embodiments shown in FIGS. 8A and 8B, the electrical contacts may be disposed within the temple 814 of the smart glasses 816. The corresponding electrical contacts may be disposed within the gripping element 812 of the holder 810.

[0207] As another example, in the embodiments shown in FIGS. 9A and 9B, the electrical contacts may be disposed on the bridge 914 and / or the nose pads 920 of the wearable extended reality device 916. The corresponding electrical contacts may be disposed within the clip 912 and / or the post 918 of the holder 910. In another embodiment of the embodiments shown in FIGS. 9A and 9B, the electrical contacts may be disposed within or around the lens 926 of the wearable extended reality device 916. The corresponding electrical contacts may be disposed on the lens protrusion 924 and / or a portion of the holder 910 under the lens 926.

[0208] As another example, in the embodiments shown in FIGS. 10A and 10B, the electrical contacts may be disposed within or around the lens 1018 of the wearable extended reality device 1014. The corresponding electrical contacts may be disposed on the wall 1016.

[0209] As another example, in the embodiments shown in FIGS. 11A and 11B, the electrical contacts may be disposed within or around the lens 1116 of the wearable extended reality device 1114. The corresponding electrical contacts may be disposed in the recess 1112 of the holder 1110. In another embodiment of the embodiments shown in FIGS. 11A and 11B, the electrical contacts may be disposed within the nose pad 1120 or the bridge 1122 of the wearable extended reality device 1114. The corresponding electrical contacts may be disposed within the nose bridge protrusion 1118 of the holder 1110.

[0210] In some embodiments, the housing may also include a wire port configured to receive a wire extending from a wearable extended reality device. The wire may be any type of wire suitable for providing power and / or data between the integrated computing interface device and the wearable extended reality device. For example, if the wire is removable from the wearable extended reality device, the wire may be a universal serial bus (USB) type wire with connectors appropriate for the wire port and the wearable extended reality device. The wire port may be located at any portion of the housing that is easily accessible to a user of the integrated computing interface device. The wire may extend from any portion of the wearable extended reality device. The wire may be fixedly attached to the wearable extended reality device or may be removable from the wearable extended reality device. In embodiments where the wearable extended reality device is a pair of smart glasses, the wire may extend from the temple. The wire may be located at any point along the length of the temple such that the wire does not obstruct the user's vision or affect the user's ability to wear the smart glasses.

[0211] In some embodiments, the wire port may be located on the front side of the integrated computing interface device configured to face the user while the user is typing on a keyboard. In one example, the wire port may be located substantially at the center of the front side of the integrated computing interface device (e.g., less than 1 cm from the center, less than 2 cm from the center, less than 4 cm from the center, less than 8 cm from the center, etc.). In another example, the wire port may be located away from the center. In another example, the wire port may be located on a side surface (e.g., left side, right side, etc.) of the front side of the integrated computing interface device, e.g., less than 1 cm from the side edge, less than 2 cm from the side edge, less than 4 cm from the side edge, less than 8 cm from the side edge, etc.

[0212] For example, in the embodiment shown in FIG. 6, the wire port 624 may be disposed within the housing 614. While the user is typing on the keyboard 620, the wire port 624 may be disposed in a non-key area 618b near the user of the integrated computing interface device 610. The wire 626 can be connected to the wearable extended reality device 612 and received by the wire port 624. The wire 626 may be optional and is shown by a dashed outline in FIG. 6. In some examples, the wire 626 may be selectively detachable from the wearable extended reality device 612. In other examples, the wire 626 may be permanently connected to the wearable extended reality device 612. In some examples, the wire 626 may be selectively detachable from the wire port 624. In other examples, the wire 626 may be permanently connected to the wire port 624. In some examples, the wire 626 may be fully or partially retracted within the housing 610 and / or within a compartment formed by the housing 610. When the wire 626 is in a fully or partially retracted state, the wire 626 may be pulled out of the housing 610 and / or the compartment, for example, by the user.

[0213] As another example, in the embodiments shown in FIGS. 7A and 7B, the wire port 736 may be disposed within the housing 714. While the user is typing on the keyboard 720, the wire port 736 may be disposed in a non-key area 718b near the user of the integrated computing interface device 710. The wire 738 may be connected to the smart glasses 712 and may be received by the wire port 736. By disposing the wire port 736 in the non-key area 718b, it is possible to allow the user to type on the keyboard 720 while the wire 738 is connected to the wire port 736. The wire 738 is optional and is shown with a dashed outline in FIGS. 7A and 7B. In some examples, the wire 738 may be selectively detachable from the wearable extended reality device 712. In other examples, the wire 738 may be permanently connected to the wearable extended reality device 712. In some examples, the wire 738 may be selectively detachable from the wire port 736. In other examples, the wire 738 may be permanently connected to the wire port 736. In some examples, the wire 738 may be fully or partially retracted within the housing 710 and / or within a compartment formed by the housing 710. When the wire 738 is in a fully or partially retracted state, the wire 738 may be pulled out of the housing 710 and / or the compartment, for example, by the user.

[0214] In some embodiments, the wire may be configured to electrically charge the wearable extended reality device when the wire is connected to the wire port. The wire may be any type of wire suitable for supplying power to the wearable extended reality device. For example, the wire may be a universal serial bus (USB) type wire having appropriate connectors for the wire port and the wearable extended reality device.

[0215] In some embodiments, the integrated computing interface device may also include at least one processor disposed within the housing. The wire may be configured to enable digital data communication between the wearable extended reality device and the at least one processor. The processor can include any processing device suitable for digital data communication. In addition to enabling digital data communication, the processing device may be configured to execute a computer program on the integrated computing interface device. The wire may be any type of wire suitable for enabling digital data communication between the wearable extended reality device and the at least one processor. For example, the wire may be a universal serial bus (USB) type wire with a wire port and a connector suitable for the wearable extended reality device.

[0216] In some embodiments, the integrated computing interface device may also include a processor disposed within the housing. The processor may be configured to wirelessly pair with a wearable extended reality device. Wireless pairing is a process of wirelessly linking the integrated computing interface device and the wearable extended reality device to enable wireless data communication therebetween. The processor can include any processing device suitable for implementing a wireless pairing protocol between the integrated computing interface device and the wearable extended reality device. The wireless pairing protocol may be WiFi (IEEE 802.11-based), radio frequency (RF such as ZigBee or ZWave), radio frequency identification (RFID, e.g., active reader passive tag or active reader active tag), Bluetooth, near field communication (NFC), or any other wireless pairing protocol available for short-range communication. The integrated computing interface device can include a visual marker adjacent to the keyboard to facilitate wireless pairing with the wearable extended reality device. The visual marker can help ensure that the wearable extended reality device is within the wireless communication range of the integrated computing interface device. In some embodiments, the keyboard can include a dedicated function key for initiating the wireless pairing process.

[0217] In some embodiments, the integrated computing interface device may also include at least one motion sensor disposed within the housing and at least one processor operatively connected thereto. The at least one processor can be programmed to implement an operating mode based on inputs received from the at least one motion sensor. In some embodiments, the motion sensor can determine whether the integrated computing interface device is in motion and adjust the operating mode of the integrated computing interface device based on the motion. In some embodiments, the motion sensor can determine whether the wearable extended reality device is moving relative to or with the integrated computing interface device and adjust the operating mode of the integrated computing interface device or the wearable extended reality device based on the motion. For example, when the user of the wearable extended reality device is walking, the number of items displayed to the user can be limited to prevent distraction of the user.

[0218] The at least one motion sensor can include an accelerometer, a gyroscope, a magnetometer, an image sensor, a motion sensor implemented by analyzing an image captured using the image sensor using an egomotion algorithm, or other types of sensors configured to measure the movement of objects within the environment of the integrated computing interface device. For example, the at least one motion sensor can be the motion sensor 373 described above in connection with FIG. 3. The at least one processor can include any processing device configured to receive inputs from the at least one motion sensor and programmed to implement an operating mode based on the inputs.

[0219] In some embodiments, at least one processor can be further programmed to automatically adjust the settings of a virtual display presented by a wearable extended reality device based on input received from at least one motion sensor. At least one processor can include any processing device configured to be programmed to automatically adjust one or more settings of a virtual display presented by a wearable extended reality device. The settings of the virtual display can be adjusted based on the environment in which the user is located (e.g., moving from a low-light indoor environment to a bright-light outdoor environment). The settings of the virtual display that can be adjusted can include image settings such as brightness, contrast, sharpness, or display mode (e.g., a game mode having a predetermined setting), color settings such as color component levels or other color adjustment settings, the position of the virtual display relative to the position of the user's head, or other settings that can improve the user's field of view of the virtual display. In the embodiment shown in FIG. 1, the settings of the virtual screen 112 may be automatically adjusted by at least one processor.

[0220] In some embodiments, at least one processor may be further programmed to output a notification when the integrated computing interface device has moved beyond a threshold distance while the wearable extended reality device is disengaged from the holder. The at least one processor can include any processing device configured to be programmed to output a notification when the integrated computing interface device has moved beyond a threshold distance while the wearable extended reality device is disengaged from the holder. For example, a notification may be provided to the user to alert the user that there is a possibility that the user is moving the wearable extended reality device out of the wireless communication range so that the interaction with the integrated computing interface device is interrupted unless the user moves closer than the threshold distance. As another example, if the wearable extended reality device is connected to the integrated computing interface device by a wire, a notification may be provided if the user attempts to move further away from the integrated computing interface device than the length of the wire, as there is a possibility that the wire may be disconnected, the wearable extended reality device may be accidentally removed from the user's head, or the integrated computing interface device may collide with the surface on which the integrated computing interface device is placed.

[0221] The notification can include a warning, an alarm, or other audible and / or visual indicators. The notification may be output via either a device (e.g., an optical indicator 351, a display 352, and / or a speaker 353) connected to the output interface 350 shown in FIG. 3 or an output interface 450 (e.g., an optical indicator 451, a display 452, a speaker 453, and / or a projector 454) shown in FIG. 4. The threshold distance can be a percentage of the length of the wire connected between the integrated computing interface device and the wearable extended reality device, a fixed distance with respect to the length of the wire, a fixed distance from the integrated computing interface device, a percentage of the distance range of the wireless communication protocol between the integrated computing interface device and the wearable extended reality device, a fixed distance with respect to the distance range of the wireless communication protocol, or any other arbitrary distance that separates the wearable extended reality device from the integrated computing interface device.

[0222] In some embodiments, the integrated computing interface device may further include at least one sensor within the housing and at least one processor operatively connected thereto. The at least one sensor may be configured to provide an input indicating whether the wearable extended reality device is engaged with the holder. The at least one processor may be programmed to use the input received to implement an operating mode based on whether the wearable extended reality device is engaged with the holder. For example, in response to an input indicating that the wearable extended reality device is engaged with the holder, the at least one processor may implement a first operating mode, and in response to an input indicating that the wearable extended reality device is not engaged with the holder, the at least one processor may implement a second operating mode, which may be different from the first operating mode. In some examples, the at least one processor may be programmed to automatically adjust the settings of the virtual display presented by the wearable extended reality device based on whether the wearable extended reality device is engaged with the holder, as described herein, for example, with respect to adjustment of settings based on input from at least one motion sensor. In some examples, the at least one processor may be programmed to output an audible indication when at least one of the wearable extended reality devices engages with the holder or when the wearable extended reality device disengages from the holder. In some examples, the operating mode may be, or may include, at least one power mode of the at least one processor, a communication device included in the integrated computing interface device, or the wearable extended reality device. In one example, when the wearable extended reality device is engaged with the holder, the power mode may be a turn-off mode, a sleep mode, a hibernation mode, etc.In another example, the power mode when the wearable extended reality device is engaged with the holder can be associated with lower power consumption than the power mode when the wearable extended reality device is not engaged with the holder (e.g., using fewer hardware components when the wearable extended reality device is engaged with the holder, using a lower clock speed when the wearable extended reality device is engaged with the holder, etc.).

[0223] In some examples, the operating mode may include a display mode for presenting virtual content via the wearable extended reality device. In one example, in one operating mode, virtual content cannot be presented via the wearable extended reality device (e.g., when the wearable extended reality device is engaged with the holder), and in another operating mode, selected virtual content can be presented via the wearable extended reality device (e.g., when the wearable extended reality device is not engaged with the holder). In another example, in one operating mode, the virtual content is presented via the wearable extended reality device in a smaller size (e.g., when the wearable extended reality device is engaged with the holder), and in another operating mode, the selected virtual content can be presented via the wearable extended reality device in a larger size (e.g., when the wearable extended reality device is not engaged with the holder). In another example, in one operating mode, the virtual content is presented via the wearable extended reality device with a lower opacity (e.g., when the wearable extended reality device is engaged with the holder), and in another operating mode, the selected virtual content can be presented via the wearable extended reality device with a higher opacity (e.g., when the wearable extended reality device is not engaged with the holder). In another example, in one operating mode, the virtual content is presented via the wearable extended reality device with a lower luminance (e.g., when the wearable extended reality device is engaged with the holder), and in another operating mode, the selected virtual content can be presented via the wearable extended reality device with a higher luminance (e.g., when the wearable extended reality device is not engaged with the holder).

[0224] In some examples, when the wearable extended reality device is engaged with the holder, the operating mode may be selected based on virtual content presented via the wearable extended reality device before the wearable extended reality device engages with the holder. For example, in response to first virtual content (e.g., virtual content related to hardware maintenance of virtual content, virtual content related to high-priority tasks, etc.), a first operating mode may be selected, and in response to second virtual content (e.g., virtual content that requires user involvement, virtual content related to low-priority tasks, etc.), a second operating mode may be selected, and the second mode may be different from the first mode. In some examples, when the wearable extended reality device engages with the holder, the operating mode can be selected based on an analysis of image data captured using at least one image sensor (e.g., at least one image sensor included in the wearable extended reality device, included in the integrated computing interface device, etc.). For example, the image data may be analyzed using a visual classification algorithm to classify the physical environment of the integrated computing interface into a specific class from a plurality of alternative classes, and the operating mode may be selected based on the specific class. Some non-limiting examples of such classes can include "outdoor", "indoor", "office", "home", "meeting room", "at least one person in the environment", "at least two people in the environment", "no person in the environment", etc. In another example, the image data may be analyzed using a visual motion recognition algorithm to detect motion in the physical environment of the integrated computing interface, and the operating mode may be selected based on whether motion towards the integrated computing interface is identified.

[0225] In some embodiments, the integrated computing interface device may also include a protective cover. The cover can protect a portion of the housing, such as a key area, a non-key area, and a holder, from damage during transportation. The protective cover may be completely removable from the housing or may be attached to the housing by one or more of its sides. One side of the protective cover may be fixedly attached to the housing. The protective cover may include two layers of a soft material (such as a non-woven fabric) that wraps a second material (such as silicon). The protective cover can include a first layer of a soft material (such as a non-woven fabric) and a second layer of a second material (such as silicon). The protective cover may be made of any number of layers or different types of materials to provide bump, shock, or impact protection to the keyboard and / or the housing and / or the wearable extended reality device when the wearable extended reality device selectively engages with the holder.

[0226] The protective cover may be operable in two storage modes. In a first storage mode, the protective cover may be configured to overlap the wearable extended reality device within the housing. For example, in the first storage mode, the protective cover can provide bump, shock, or impact protection to the keyboard and / or the housing and / or the wearable extended reality device when the wearable extended reality device selectively engages with the holder. The protective cover can include one or more mechanisms, such as one or more protrusions, configured to hold the wearable extended reality device in the first storage mode. For example, the protective cover can include one or more fold lines that allow the protective cover to bend over the wearable extended reality device when the wearable extended reality device selectively engages with the holder and to bend differently when the wearable extended reality device is selectively disengaged from the holder.

[0227] In the second storage mode, the protective cover may be configured to lift the housing. For example, in the second storage mode, the protective cover can enable the housing to be raised relative to the surface on which the housing is placed, such as a table, a desk, or the user's knee. To raise the housing, the protective cover can be segmented such that the protective cover is folded into different positions, thereby enabling the housing to be raised to one or more distances above the surface on which the housing is placed. In the second storage mode, the protective cover may not overlap the wearable extended reality device.

[0228] FIG. 13A is a right perspective view of an integrated computing interface device 1310 having a protective cover 1312 in a first storage mode. It should be understood that FIG. 13A is a typical representation of just one embodiment and that within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added. The protective cover 1312 can cover the upper portion of the housing 1314, and the bottom surface of the housing 1314 contacts the surface 1316 on which the integrated computing interface device 1310 can be placed. In some non-limiting examples, the integrated computing interface device 1310 can include all or part of at least one element of the integrated computing interface device 610, the integrated computing interface device 710, or the integrated computing interface device 1210.

[0229] FIG. 13B is a left perspective view of the integrated computing interface device 1310 of FIG. 13A having the protective cover 1312 in the second storage mode. FIG. 13B is a representative illustration of merely one embodiment, and it should be understood that within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added. When the protective cover 1312 is in the second storage mode, the protective cover 1312 may be at least partially separated from the housing 1314 to allow access to the keyboard 1318, the wearable extended reality device 1320, and the holder 1322 for the wearable extended reality device 1320. The protective cover 1312 may be segmented to create individual segments (such as segments 1324, 1326, 1328, and 1330) such that the folded protective cover 1312 can lift at least a portion of the housing 1314 above the surface 1316. The housing can be lifted by placing the folded protective cover 1312 under one end of the housing and lifting that end of the housing. Note that the number of segments 1324 through 1330 shown in FIG. 13B is exemplary, and the protective cover 1312 can have fewer or more segments.

[0230] The protective cover can also include at least one camera associated with the protective cover. For example, the protective cover can include one or more cameras or other types of imaging devices configured to capture images of the keyboard, the user, or the environment around the user. The one or more cameras can include one or more selfie cameras, rear cameras, or other cameras associated with the protective cover such that the cameras can be usable when the protective cover is in the second storage mode.

[0231] The protective cover can also include at least one protrusion on at least two sides of at least one camera. For example, the protective cover can include two protrusions from two sides of at least one camera, one protrusion surrounding at least one camera from at least two sides, or another number of protrusions, and at least one protrusion is configured to prevent at least one camera from contacting the plane when the cover is disposed on the plane such that the camera faces the plane.

[0232] FIG. 14 is a right side perspective view of another embodiment of an integrated computing interface device 1410 having a protective cover 1412 in a second storage mode. It should be understood that FIG. 14 is a typical representation of just one embodiment and that within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added.

[0233] When the protective cover 1412 is in the second storage mode, the protective cover 1412 is at least partially separated from the housing 1414 to enable access to the keyboard 1416, the wearable extended reality device 1418, and the holder 1420 for the wearable extended reality device 1418. The protective cover 1412 can be segmented to create individual segments (such as segments 1422, 1424, and 1426) such that the protective cover 1412 can be folded to lift at least a portion of the housing 1414 above the surface 1428. Note that the number of segments 1422 to 1426 shown in FIG. 14 is typical and the protective cover 1412 can have fewer or more segments.

[0234] Camera 1430 may be disposed within protective cover 1412 within segment 1422. Note that camera 1430 may be disposed at any portion of protective cover 1412. Three protrusions 1432 extend outwardly from protective cover 1412 so as to form a triangular shape around camera 1430. Protrusions 1432 are configured to prevent camera 1430 from contacting surface 1428 when protective cover 1412 is disposed on surface 1428 such that camera 1430 faces surface 1428.

[0235] Conventionally, a camera has been disposed at an upper frame of a laptop screen. This position enables the camera to face the user when the laptop is used from a position elevated relative to the surface on which the laptop is disposed. This elevated position enables a desired viewing angle for the user. Some of the disclosed embodiments enable a laptop without a large physical screen using an extended reality sensing device. However, this creates a problem with the camera position. Placing the camera on the keyboard results in an undesired viewing angle for the user that the user is not accustomed to. Therefore, it is desirable to place the camera in an elevated position relative to the keyboard. One solution provided by some of the disclosed embodiments includes disposing the camera within a foldable cover of the keyboard that is configured to fold such that the camera is lifted above the keyboard and faces the user.

[0236] In some embodiments, the integrated computing interface device can include a housing having a key area and a non-key area. The integrated computing interface can include any device having several functional components. The device can function as an interface that enables human interaction with a machine such as a computer. In one example, the integrated computing interface device can include a computing device configured to operate with a wearable extended reality device external to the integrated computing interface device to enable the presentation of an extended reality environment, for example, via the wearable extended reality device. In one example, the integrated computing interface device can include, for example, within a single housing, a computing device (such as a processor, a CPU, etc.) integrated with an input device (such as a keyboard, a touch screen, a touch pad, etc.), a digital communication device configured to connect the integrated computing interface device to a digital communication network (such as Ethernet, a cellular network, the Internet, etc.), and a communication port configured to connect a wearable extended reality device external to the integrated computing interface device.

[0237] A housing can include any physical structure in which one or more components are housed or contained. Such a housing can have key regions such as a general area where keys are located and a non-key area that generally has no keys. A key can include any button, switch, trigger, toggle, or any other element that can be actuated via a physical operation. In some embodiments, the keys can be essentially mechanical (e.g., mechanical push buttons such as those found on a typical computer keyboard). In other examples, the keys can be soft (e.g., a touch display where an image of a key is simulated). By way of example, a key region can cover an area of an alphanumeric or numeric keyboard, and the non-key area can be another area of the interface that has no keys. In some examples, the key region and / or the non-key area can be an area of the outer surface (or upper outer surface) of an integrated computing interface device. In some embodiments, the integrated computing interface device can include a housing having an input area and a non-input area. In one example, the input area can be an outer surface area (or upper outer surface area) of an integrated computing interface device that includes a touch screen. In another example, the input area can be an outer surface area (or upper outer surface area) of an integrated computing interface device that includes a touch pad. In yet another example, the input area can be a key region as described above and / or can include a key region. In one example, the non-input area can be an outer surface area (or upper outer surface area) of the interface that has no input device. In another example, the input area can be an outer surface area (or upper outer surface area) of an integrated computing interface device that includes a particular type of input device, and the non-input area can be another outer surface area (or another upper outer surface area) of the integrated computing interface device that lacks a particular type of input device.

[0238] The housing can include an outer cover or shell. The housing can surround the components of the integrated computing interface device and can cover some or all of the components of the integrated computing interface device. The housing can be considered to have one or more openings that can expose certain components of the integrated computing interface device (e.g., a USB or other port or an image sensor) or can project certain components (e.g., keys of a keyboard) out of the housing. The housing can support certain components of the integrated computing interface device (e.g., a circuit board) in an inner portion of the housing.

[0239] The housing of an all-in-one computing device may include a key area and a non-key area different from the key area. As described in more detail above, the key area can include one or more keys that enable a user to input alphanumeric or other characters. For example, in some embodiments, a keyboard may be associated with the key area of the housing. The keyboard may be a standard typewriter-style keyboard (e.g., a QWERTY-style keyboard), or another suitable keyboard layout such as a Dvorak layout or a coded layout. The keyboard can include any suitable number of keys. For example, a "full-size" keyboard may include up to 104 or 105 keys. In one example, the keyboard may include at least 10 keys, at least 30 keys, at least 80 keys, at least 100 keys, etc. In one example, the keyboard may be included in an all-in-one computing interface device, an outer surface area of the all-in-one computing interface device, an upper outer surface area of the all-in-one computing interface device, a housing, an outer surface of the housing, an upper outer surface of the housing, a key area, etc. In some embodiments, an input device may be associated with an input area of the housing. Some non-limiting examples of such input devices can include a touch screen, a touch pad, the keyboard described above, etc. For example, the input device may be a touch screen, and the touch screen may have a diagonal length of at least 1 inch, at least 5 inches, at least 10 inches, etc. In another example, the input device may be a touch pad, and the touch pad may have a diagonal length of at least 1 inch, at least 5 inches, at least 10 inches, etc. In one example, the input device may be included in an all-in-one computing interface device, an outer surface area of the all-in-one computing interface device, a housing, an outer surface of the housing, an upper outer surface area of the all-in-one computing interface device, a housing, an upper outer surface of the housing, a key area, etc.

[0240] As described in more detail above, the non-key area may be an area of the housing that does not contain keys and may exist to complete the desired shape of the housing extending beyond the key area of the housing in any direction. The non-key area can be an area that may include input elements such as a trackpad, a trackball, a touch screen, a touch bar, or other forms of cursor control for an integrated computing interface device. The non-key area may be subdivided into a plurality of different non-key areas, such as a trackpad or other cursor control section, an extension of the housing, or a cover or grill for one or more speakers or other audio output devices included within the housing. In some embodiments, the housing may include two or more non-key areas. For example, a first non-key area may be disposed at the upper edge of the key area and a second non-key area may be disposed at the lower edge of the key area. In some embodiments, the non-key area may simply be a part of the housing and have no function other than functioning as a part of the housing. In some examples, as described in more detail above, the non-input area may be an area of the housing that does not contain an input device or does not contain a particular type of input device and may exist to complete the desired shape of the housing extending beyond the input area of the housing in any direction. In some embodiments, the housing may include two or more non-input areas.

[0241] In some embodiments, the integrated computing interface device can include at least one image sensor. The image sensor can include a device that converts photons (i.e., light) into an electrical signal for interpretation. For example, the image sensor can incorporate a charge-coupled device (CCD) or an active pixel sensor (CMOS sensor) manufactured using complementary MOS (CMOS) or N-type MOS (NMOS or live MOS) technology. The at least one image sensor may be configured to capture an image and / or video of a user of the integrated computing interface device, or alternatively may be disposed in the user's physical environment. As described above, the image sensor may be configured to capture visual information by converting light into image data. In some embodiments, the at least one image sensor may be at least one of a color image sensor, a monochrome image sensor, a stereo image sensor, an infrared image sensor, or a depth image sensor.

[0242] In some examples, image data captured using at least one image sensor included in an integrated computing interface device can be analyzed to determine whether a user of a wearable extended reality device is approaching the integrated computing interface device. For example, the image data can be analyzed using a visual classification algorithm to determine whether a user of the wearable extended reality device is approaching the integrated computing interface device, whether a person approaching the integrated computing interface device is the user of the wearable extended reality device, and so on. In some examples, the image data can be analyzed to identify the wearable extended reality device used by a person approaching the integrated computing interface device. For example, a unique visual code can be presented on the wearable extended reality device (e.g., on a sticker or on a display screen outside the wearable extended reality device), the image data can be analyzed to detect and recognize the unique visual code, and the wearable extended reality device can be identified by accessing a data structure that associates the wearable extended reality device with the visual code based on the identified visual code. Further, in some examples, the pairing of the integrated computing inte...

Claims

1. A non - transitory computer - readable medium including instructions that, when executed by at least one processor, cause the at least one processor to perform operations for implementing a hybrid virtual key in an extended reality environment, the operations including: During a first period, receiving a first signal corresponding to positions on the touch - sensing surface of a plurality of virtual activatable elements virtually projected by a wearable extended reality device on the touch - sensing surface; Determining, from the first signal, the positions of the plurality of virtual activatable elements on the touch - sensing surface; Receiving a touch input from a user via the touch - sensing surface, the touch input including a second signal generated as a result of an interaction with at least one sensor within the touch - sensing surface; Determining a coordinate position associated with the touch input based on the second signal generated as a result of the interaction with the at least one sensor within the touch - sensing surface; Comparing the coordinate position of the touch input with at least one of the determined positions to identify one of the plurality of virtual activatable elements corresponding to the touch input; Causing a change in virtual content associated with the wearable extended reality device, the change corresponding to the identified one of the plurality of virtual activatable elements.

2. The non - transitory computer - readable medium according to claim 1, wherein the plurality of virtual activatable elements virtually projected on the touch - sensing surface is a suitable subset of a group of virtual activatable elements, and the subset is determined based on the user's action.

3. The non - transitory computer - readable medium according to claim 1, wherein the plurality of virtual activatable elements virtually projected on the touch - sensing surface is a suitable subset of a group of virtual activatable elements, and the subset is determined based on the user's physical position.

4. The plurality of virtual activatable elements virtually projected on the touch sensing surface are an appropriate subset of a group of virtual activatable elements, and the subset is determined based on an event in the user's environment. The non - transient computer - readable medium according to claim 1.

5. The positions of the plurality of virtual activatable elements on the touch sensing surface are determined based on at least one of the user's action, the user's physical location, the physical location of the wearable extended reality device, the physical location of the touch sensing surface, or an event in the user's environment. The non - transient computer - readable medium according to claim 1.

6. The operation further includes, at the time of detecting the touch input, opening an application, and the step of causing the change of the virtual content is based on the opening of the application. The non - transient computer - readable medium according to claim 1.

7. The operation further includes, at the time of detecting the touch input, changing an output parameter, and the step of causing the change of the virtual content is based on the change of the output parameter. The non - transient computer - readable medium according to claim 1.

8. The operation further includes arranging the plurality of virtual activatable elements on the touch sensing surface based on a default arrangement previously selected by the user. The non - transient computer - readable medium according to claim 1.

9. The virtual content includes a virtual display, and the operation further includes enabling the touch sensing surface to navigate a cursor within the virtual display. The non - transient computer - readable medium according to claim 1.

10. The operation further includes determining the type of the touch input based on the second signal, and the change of the virtual content corresponds to the identified one of the plurality of virtual activatable elements and the determined type of the touch input. The non - transient computer - readable medium according to claim 1.

11. The operation is Receiving an additional signal corresponding to a position on the keyboard adjacent to the touch-sensing surface of the additional virtual activatable element virtually projected onto a key of the keyboard by the wearable extended reality device; Determining the position of the additional virtual activatable element on the key of the keyboard from the additional signal; Receiving a key press input via at least one key of the keyboard; Identifying one of the additional virtual activatable elements corresponding to the key press input; Causing a second change to the virtual content associated with the wearable extended reality device, the second change corresponding to the identified one of the additional virtual activatable elements; and further comprising the steps of, the non-transitory computer-readable medium according to claim 1.

12. The operation further includes receiving a keyboard configuration selection and causing the wearable extended reality device to virtually project the additional virtual activatable element corresponding to the selected keyboard configuration. The non-transitory computer-readable medium according to claim 11.

13. The operation further includes selecting the additional virtual activatable element based on at least one of a user action, a physical user position, a physical position of the wearable extended reality device, a physical position of the keyboard, or an event in the user environment. The non-transitory computer-readable medium according to claim 11.

14. The operation is Determining whether the user is a wearer of the wearable extended reality device; In response to determining that the user is a wearer of the wearable extended reality device, causing the change to the virtual content associated with the wearable extended reality device; In response to determining that the user is not a wearer of the wearable extended reality device, canceling the step of causing the change to the virtual content associated with the wearable extended reality device. The non-transitory computer-readable medium according to claim 1 further comprising the steps of.

15. where the user is a wearer of a second wearable extended reality device, and when the second wearable extended reality device projects a second plurality of virtual activatable elements onto the touch sensing surface, the operation comprises determining, based on the coordinate position of the touch input, that the touch input corresponds to a particular one of the second plurality of virtual activatable elements; causing a second change to the virtual content associated with the wearable extended reality device, the second change corresponding to the particular one of the second plurality of virtual activatable elements, the non-transitory computer-readable medium of claim 14, further comprising.

16. The non-transitory computer-readable medium of claim 1, wherein the operation further comprises deactivating at least one function of at least a portion of the touch sensing surface during a second period in which the plurality of virtual activatable elements are not projected onto the touch sensing surface.

17. The non-transitory computer-readable medium of claim 1, wherein the operation further comprises deactivating at least one function of at least a portion of the touch sensing surface during a second period in which different pluralities of virtual activatable elements are projected onto the touch sensing surface by the wearable extended reality device.

18. The non-transitory computer-readable medium of claim 17, wherein the operation further comprises maintaining at least one function of at least a portion of the touch sensing surface during a third period after the first period and before the different pluralities of virtual activatable elements are projected onto the touch sensing surface, during which third period the touch sensitive surface is outside the field of view of the wearable extended reality device and thus the pluralities of virtual activatable elements are not projected onto the touch sensing surface.

19. A method for implementing hybrid virtual keys in an extended reality environment, the operation comprising During a first period, receiving a first signal corresponding to a position on the touch sensing surface of a plurality of virtual activatable elements virtually projected by a wearable extended reality device on the touch sensing surface; Determining, from the first signal, the position of the plurality of virtual activatable elements on the touch sensing surface; Receiving a touch input from a user via the touch sensing surface, the touch input including a second signal generated as a result of interaction with at least one sensor within the touch sensing surface; Determining a coordinate position associated with the touch input based on the second signal generated as a result of the interaction with the at least one sensor within the touch sensing surface; Comparing the coordinate position of the touch input with at least one of the determined positions to identify one of the plurality of virtual activatable elements corresponding to the touch input; Causing a change in virtual content associated with the wearable extended reality device, the change corresponding to the identified one of the plurality of virtual activatable elements. A method comprising the steps. [

20. ] A system for implementing hybrid virtual keys in an extended reality environment, At least one processor, During a first period, receiving a first signal corresponding to a position on the touch sensing surface of a plurality of virtual activatable elements virtually projected by a wearable extended reality device on the touch sensing surface, Determining, from the first signal, the position of the plurality of virtual activatable elements on the touch sensing surface, Receiving a touch input from a user via the touch sensing surface, the touch input including a second signal generated as a result of interaction with at least one sensor within the touch sensing surface, Determining a coordinate position associated with the touch input based on the second signal generated as a result of the interaction with the at least one sensor within the touch sensing surface, Comparing the coordinate position of the touch input with at least one of the determined positions to identify one of the plurality of virtual activatable elements corresponding to the touch input, A system including at least one processor configured to cause a change in virtual content associated with the wearable extended reality device, the change corresponding to the identified one of the plurality of virtual activatable elements.

Images