Extended Reality for Productivity

Abstract

The integrated computing interface device may include a portable housing having a key region and a non-key region, a keyboard associated with the key region of the housing, and a holder associated with the non-key region of the housing. The holder may be configured to selectively engage and disengage the wearable extended reality device such that the wearable extended reality device is transportable with the housing when the wearable extended reality device selectively engages the housing via the holder.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Provisional Patent Application No. 63 / 147,051 filed February 8, 2021, U.S. Provisional Patent Application No. 63 / 157,768 filed March 7, 2021, U.S. Provisional Patent Application No. 63 / 173,095 filed April 9, 2021, U.S. Provisional Patent Application No. 63 / 213,019 filed June 21, 2021, U.S. Provisional Patent Application No. 63 / 215,500 filed June 27, 2021, U.S. Provisional Patent Application No. 63 / 216,335 filed June 29, 2021, and U.S. Provisional Patent Application No. 63 / 217,336 filed June 30, 2021. This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 226,977, filed July 29, 2021, U.S. Provisional Patent Application No. 63 / 300,005, filed January 16, 2022, U.S. Provisional Patent Application No. 63 / 307,207, filed February 7, 2022, U.S. Provisional Patent Application No. 63 / 307,203, filed February 7, 2022, and U.S. Provisional Patent Application No. 63 / 307,217, filed February 7, 2022, all of which are incorporated by reference in their entireties.

[0002] Technical Field The present disclosure relates generally to the field of extended reality. More specifically, the present disclosure relates to systems, methods, and devices for providing productivity applications using extended reality environments. [Background technology]

[0003] For many years, PC users have faced the productivity dilemma of either limiting mobility (when choosing a desktop computer) or limiting screen size (when choosing a laptop computer). One partial solution to this problem is the use of a docking station. A docking station is an interface device for connecting a laptop computer with other devices. By plugging a laptop computer into a docking station, laptop users can enjoy the improved visibility provided by a larger monitor. However, because the large monitor is stationary, user mobility, while improved, is still limited. For example, even laptop users with docking stations lack the flexibility to use two 32-inch screens wherever they want.

[0004] Some of the disclosed embodiments are directed to providing a new approach to solving productivity dilemmas by using Extended Reality (XR) to provide a mobile environment that allows users to experience the comfort of a stationary workspace wherever they want by providing a virtual desktop-like screen. Summary of the Invention

[0005] Embodiments consistent with the present disclosure provide systems, methods, and devices for providing and supporting productivity applications using extended reality environments.

[0006] Some disclosed embodiments may include an integrated computing interface device, which may include a portable housing having a key region and a non-key region, a keyboard associated with the key region of the housing, and a holder associated with the non-key region of the housing. The holder may be configured to selectively engage and disengage with a wearable extended reality device such that the wearable extended reality device is transportable with the housing when the wearable extended reality device selectively engages with the housing via the holder.

[0007] Some disclosed embodiments may include an integrated computing interface device including a housing, at least one image sensor, and a foldable protective cover. The housing may have a key area and a non-key area, and a keyboard associated with the key area. The foldable protective cover incorporates the at least one image sensor. The protective cover may be configured to be manipulated into a plurality of folding configurations, including a first folding configuration and a second folding configuration, where in the first folding configuration, the protective cover may be configured to accommodate at least a portion of the key area and the non-key area, and in the second folding configuration, the optical axis of the at least one image sensor may be configured to stand substantially facing a user of the integrated computing interface device while the user types on the keyboard.

[0008] Some disclosed embodiments may include a case for an integrated computing interface device including at least one image sensor and a foldable protective cover incorporating the at least one image sensor. The protective cover may be configured to be manipulated into multiple folding configurations. In a first folding configuration, the protective cover may be configured to accommodate a housing of the integrated computing interface device having a key region and a non-key region. In a second folding configuration, the protective cover may be configured to stand with an optical axis of the at least one image sensor generally facing a user of the integrated computing interface device while the user types on a keyboard associated with the key region.

[0009] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for modifying the display of virtual content based on temperature. Some of these embodiments may include displaying virtual content via a wearable extended reality device, where heat is generated by at least one component of the wearable extended reality device during the display of the virtual content; receiving information indicating a temperature associated with the wearable extended reality device; determining, based on the received information, whether a change in display settings of the virtual content is necessary; and, based on the determination, modifying the display settings of the virtual content to achieve a target temperature.

[0010] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for implementing a hybrid virtual key in an extended reality environment. Some of these embodiments may include receiving, during a first time period, a first signal corresponding to positions on the touch-sensitive surface of a plurality of virtual activatable elements virtually projected by a wearable extended reality device onto the touch-sensitive surface, determining positions of the plurality of virtual activatable elements on the touch-sensitive surface from the first signal, receiving a user touch input via the touch-sensitive surface, the touch input including a second signal generated as a result of interaction with at least one sensor in the touch-sensitive surface, determining a coordinate location associated with the touch input based on the second signal generated as a result of interaction with the at least one sensor in the touch-sensitive surface, comparing the coordinate location of the touch input with at least one of the determined locations to identify one of the plurality of virtual activatable elements corresponding to the touch input, and causing a change in virtual content associated with the wearable extended reality device, the change corresponding to the identified one of the plurality of virtual activatable elements.

[0011] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for controlling a virtual display using a combination of a keyboard and a wearable extended reality device. Some of these embodiments may include receiving a first signal representing a first hand movement from a first hand position sensor associated with the wearable extended reality device, receiving a second signal representing a second hand movement from a second hand position sensor associated with the keyboard, the second hand movement including an action other than interacting with a feedback component, and controlling the virtual display based on the first signal and the second signal.

[0012] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for integrating a movable input device with a virtual display projected through a wearable extended reality device. Some of these embodiments may include receiving a movement signal associated with the movable input device, the movement signal reflecting a physical movement of the movable input device; outputting a first display signal to the wearable extended reality device during a first time period, the first display signal configured to cause the wearable extended reality device to virtually present content in a first orientation; outputting a second display signal to the wearable extended reality device during a second time period different from the first time period, the second display signal configured to cause the wearable extended reality device to virtually present content in a second orientation different from the first orientation; and switching between outputting the first display signal and the second display signal based on the received movement signal of the movable input device.

[0013] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for virtually augmenting a physical keyboard. Some of these embodiments may include receiving image data from an image sensor associated with a wearable extended reality device, where the image data represents a keyboard placed on a surface; determining that the keyboard is paired with the wearable extended reality device; receiving input to cause a display of a virtual controller in conjunction with the keyboard; displaying, via the wearable extended reality device, the virtual controller at a first location on the surface, where the virtual controller has an original spatial orientation relative to the keyboard at the first location; detecting movement of the keyboard to a different location on the surface; and presenting the virtual controller at a second location on the surface in response to the detected movement of the keyboard, where a subsequent spatial orientation of the virtual controller relative to the keyboard at the second location corresponds to the original spatial orientation.

[0014] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for coordinating virtual content display with a mobility status, including: accessing rules associating a plurality of user mobility statuses with a plurality of display modes for presenting virtual content via a wearable extended reality device; receiving first sensor data from at least one sensor associated with the wearable extended reality device, the first sensor data reflecting a mobility status of a user of the wearable extended reality device during a first time period; determining, based on the first sensor data, that a user of the wearable extended reality device is associated with the first mobility status during the first time period; and displaying the virtual content via the wearable extended reality device associated with the first mobility status. The method may include implementing at least the first accessed rule to generate a first display of the virtual content; receiving second sensor data from the at least one sensor, the second sensor data reflecting a mobility status of the user during a second time period; determining, based on the second sensor data, that a user of the wearable extended reality device is associated with the second mobility status during the second time period; and implementing the at least the second accessed rule to generate a second display of the virtual content via the wearable extended reality device associated with the second mobility status, wherein the second display of the virtual content differs from the first display of the virtual content.

[0015] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for modifying a display of a virtual object docked to a movable input device. Some of these embodiments may include receiving image data from an image sensor associated with a wearable extended reality device, where the image data represents the input device positioned at a first position on a support surface, causing the wearable extended reality device to generate a representation of at least one virtual object proximate to the first position, docking the at least one virtual object to the input device, determining that the input device is at a second position on the support surface, and updating the representation of the at least one virtual object so that the at least one virtual object appears proximate to the second position in response to determining that the input device is at the second position, determining that the input device is at a third position detached from the support surface, and modifying the representation of the at least one virtual object in response to determining that the input device has been detached from the support surface.

[0016] Some disclosed embodiments may include a system, method, and non-transitory computer-readable medium for docking a virtual object to a virtual display screen in an extended reality environment. Some of these embodiments may include generating virtual content for presentation via a wearable extended reality device, the virtual content including a virtual display and a plurality of virtual objects located outside the virtual display, receiving a selection of at least one virtual object from the plurality of virtual objects, docking the at least one virtual object to the virtual display, receiving an input indicating an intention to reposition the virtual display without expressing an intention to move the at least one virtual object after docking the at least one virtual object to the virtual display, and repositioning the virtual display in response to the input, wherein repositioning the virtual display causes the at least one virtual object to move with the virtual display as a result of docking the at least one virtual object to the virtual display.

[0017] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for implementing selective virtual object display modifications. Some of these embodiments may include generating an extended reality environment via a wearable extended reality device, the extended reality environment including a first virtual plane associated with a physical object and a second virtual plane associated with an item, the second virtual plane extending transversely to the first virtual plane; accessing first instructions for docking a first set of virtual objects to a first position associated with the first virtual plane; accessing second instructions for docking a second set of virtual objects to a second position associated with the second virtual plane; receiving a first input associated with movement of the physical object; and in response to receiving the first input, causing a change in display of the first set of virtual objects in a manner corresponding to the movement of the physical object while maintaining the second set of virtual objects at their second positions; receiving a second input associated with movement of the item; and in response to receiving the second input, causing a change in display of the second set of virtual objects in a manner corresponding to the movement of the item while maintaining the first set of virtual objects at their first positions.

[0018] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for determining a display configuration for presenting virtual content. Some of these embodiments may include receiving image data from an image sensor associated with a wearable extended reality device, where the wearable extended reality device is configured to be paired with multiple input devices, each input device associated with a default display configuration; analyzing the image data to detect a particular input device placed on a surface; determining a value of at least one usage parameter for the particular input device; retrieving from memory the default display configuration associated with the particular input device; determining a display configuration for presenting the virtual content based on the value of the at least one usage parameter and the retrieved default display configuration; and causing presentation of the virtual content via the wearable extended reality device according to the determined display configuration.

[0019] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for augmenting a physical display with a virtual display. Some of these embodiments may include receiving a first signal representing a first object entirely presented on the physical display, receiving a second signal representing a second object having a first portion presented on the physical display and a second portion extending beyond a boundary of the physical display, receiving a third signal representing a third object initially presented on the physical display and subsequently moved entirely beyond a boundary of the physical display, causing, in response to receiving the second signal, a second portion of the second object to be presented via a wearable extended reality device in a virtual space while the first portion of the second object is presented on the physical display, and causing, in response to receiving the third signal, the third object to be presented entirely via the wearable extended reality device in the virtual space subsequent to the third object being presented entirely on the physical display.

[0020] Consistent with other disclosed embodiments, a non-transitory computer-readable storage medium can store program instructions that, when executed by at least one processing device, perform any of the methods described herein.

[0021] The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the scope of the claims.

[0022] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 is a schematic diagram of a user using an exemplary extended reality system consistent with some embodiments of the present disclosure. [Figure 2]FIG. 2 is a schematic diagram of the major components of the exemplary extended reality system of FIG. 1, consistent with some embodiments of the present disclosure. [Figure 3] FIG. 3 is a block diagram illustrating some of the components of an input unit, consistent with some embodiments of the present disclosure. [Figure 4] FIG. 4 is a block diagram illustrating some of the components of an extended reality unit consistent with some embodiments of the present disclosure. [Figure 5] FIG. 5 is a block diagram illustrating some of the components of a remote processing unit, consistent with some embodiments of the present disclosure. [Figure 6] FIG. 6 is a top view of a first exemplary embodiment of an integrated computing interface device having a wearable extended reality device selectively engaging the integrated computing interface device. [Figure 7] 7A and 7B are top and left side views, respectively, of a second exemplary embodiment of an integrated computing interface device having a wearable extended reality device selectively engaging the integrated computing interface device. [Figure 8] 8A is a front perspective view of a wearable extended reality device selectively engaged with a first exemplary embodiment of a holder. FIG. 8B is a rear perspective view of the wearable extended reality device selectively disengaged from the first exemplary embodiment of the holder shown in FIG. 8A. [Figure 9] 9A and 9B are front and rear perspective views of a wearable extended reality device selectively engaged with and disengaged from a second exemplary embodiment of a holder shown in FIG. [Figure 10]Figure 10A is a front perspective view of a wearable extended reality device selectively engaged with a third exemplary embodiment of a holder. Figure 10B is a rear perspective view of the wearable extended reality device selectively disengaged from the third exemplary embodiment of the holder shown in Figure 10A. [Figure 11] 11A and 11B are front and rear perspective views of a wearable extended reality device selectively engaged with and selectively disengaged from a fourth exemplary embodiment of a holder shown in FIG. 11A . [Figure 12] 12A and 12B are top and left side views, respectively, of a third exemplary embodiment of an integrated computing interface device having a wearable extended reality instrument selectively engaged with the integrated computing interface device. [Figure 13] 13A and 13B are side and left perspective views of an exemplary integrated computing interface device having a protective cover in a first storage mode and a second storage mode, respectively, consistent with certain embodiments of the present disclosure. [Figure 14] FIG. 14 is a side perspective view of a second exemplary embodiment of an integrated computing interface device having a protective cover in a second storage mode. [Figure 15] FIG. 15 is a front perspective view of a first exemplary embodiment of an integrated computing interface device. [Figure 16] FIG. 16 is a front perspective view of a second exemplary embodiment of an integrated computing interface device. [Figure 17] FIG. 17 is a top view of a third exemplary embodiment of an integrated computing interface device. [Figure 18]FIG. 18 is a front perspective view of a fourth exemplary embodiment of an integrated computing interface device having a foldable protective cover in a first folded configuration. [Figure 19] FIG. 19 is an exploded view of a portion of an exemplary embodiment of a foldable protective cover. [Figure 20] FIG. 20 is a side view of a fifth exemplary embodiment of an integrated computing interface device. [Figure 21] FIG. 21 is a side view of a sixth exemplary embodiment of an integrated computing interface device. [Figure 22] FIG. 22 is a front perspective view of a seventh exemplary embodiment of an integrated computing interface device. [Figure 23] FIG. 23 is a block diagram illustrating exemplary operating parameters of a wearable extended reality device, consistent with certain disclosed embodiments. [Figure 24] FIG. 24 is an exemplary chart illustrating changes in display settings over time based on heat-generating light source temperature, consistent with certain disclosed embodiments. [Figure 25] FIG. 25 is a diagram illustrating an example of reducing the display size of a portion of virtual content based on received temperature information, consistent with certain disclosed embodiments. [Figure 26] FIG. 26 is a flowchart illustrating an exemplary method for changing display settings based on temperature in a wearable extended reality device, consistent with certain embodiments of the present disclosure. [Figure 27] FIG. 27 illustrates an example of a wearable extended reality device that virtually projects onto a touch-sensitive surface, consistent with some embodiments of the present disclosure. [Figure 28] FIG. 28 is a diagram illustrating an example of a keyboard and touch-sensitive surface consistent with some embodiments of the present disclosure. [Figure 29] FIG. 29 is a diagram illustrating an example of a user interacting with a touch-sensitive surface, consistent with certain embodiments of the present disclosure. [Figure 30]FIG. 30 is a diagram illustrating an example of a user interacting with a touch-sensitive surface to navigate a cursor, consistent with certain embodiments of the present disclosure. [Figure 31] FIG. 31 illustrates a flowchart of an exemplary method for implementing a hybrid virtual key in an extended reality environment, consistent with certain embodiments of the present disclosure. [Figure 32] FIG. 32 illustrates an example keyboard with additional virtual activatable elements virtually projected onto the keyboard keys, consistent with certain embodiments of the present disclosure. [Figure 33] FIG. 33 illustrates an example of a combination keyboard and wearable extended reality device for controlling a virtual display, consistent with some embodiments of the present disclosure. [Figure 34] FIG. 34 is a diagram illustrating an example of a first hand position sensor associated with a wearable extended reality device, consistent with some embodiments of the present disclosure. [Figure 35] FIG. 35 is a diagram illustrating an example of a second hand position sensor associated with a keyboard, consistent with certain embodiments of the present disclosure. [Figure 36] FIG. 36 is a diagram illustrating an example of different types of first and second hand position sensors, consistent with certain embodiments of the present disclosure. [Figure 37] FIG. 37 illustrates an example of a keyboard including a touchpad and associated input area including keys, consistent with certain embodiments of the present disclosure. [Figure 38] FIG. 38 illustrates an example of a wearable extended reality device selectively connectable to a keyboard via a connector, consistent with some embodiments of the present disclosure. [Figure 39] FIG. 39 is a diagram illustrating an exemplary virtual display with a movable input device during a first time period, consistent with certain embodiments of the present disclosure. [Figure 40]FIG. 40 is a diagram illustrating an exemplary virtual display with a movable input device during a second time period, consistent with certain embodiments of the present disclosure. [Figure 41] FIG. 41 is a diagram illustrating exemplary virtual display and movable input device movement types consistent with certain embodiments of the present disclosure. [Figure 42] FIG. 42 is a diagram illustrating an exemplary virtual display of a first orientation for a movable input device before a first period of time, consistent with certain embodiments of the present disclosure. [Figure 43] FIG. 43 is a diagram illustrating resizing of an exemplary virtual display based on movement signals associated with a movable input device, consistent with certain embodiments of the present disclosure. [Figure 44] FIG. 44 is a diagram illustrating an exemplary virtual display configured to enable visual presentation of text input entered using a movable input device, consistent with certain embodiments of the present disclosure. [Figure 45A] FIG. 45A illustrates an exemplary process for integrating a movable input device with a virtual display projected through a wearable extended reality device, consistent with certain embodiments of the present disclosure. [Figure 45B] FIG. 45B illustrates another exemplary process for integrating a movable input device with a virtual display projected through a wearable extended reality device, consistent with certain embodiments of the present disclosure. [Figure 46] FIG. 46 is a diagram illustrating an example of a keyboard and virtual controller, consistent with certain embodiments of the present disclosure. [Figure 47] FIG. 47 illustrates an example of a keyboard and virtual controller moved from one location to another, consistent with certain embodiments of the present disclosure. [Figure 48] FIG. 48 illustrates another example of a keyboard and virtual controller moving from one location to another, consistent with certain embodiments of the present disclosure. [Figure 49]FIG. 49 is a block diagram of an exemplary process for virtually extending a physical keyboard, consistent with certain embodiments of the present disclosure. [Figure 50] Figure 50A is a diagram illustrating an example of various virtual content displays coordinated with different mobility statuses, consistent with certain embodiments of the present disclosure; Figure 50B is a diagram illustrating an example of various virtual content displays coordinated with different mobility statuses, consistent with certain embodiments of the present disclosure; Figure 50C is a diagram illustrating an example of various virtual content displays coordinated with different mobility statuses, consistent with certain embodiments of the present disclosure; and Figure 50D is a diagram illustrating an example of various virtual content displays coordinated with different mobility statuses, consistent with certain embodiments of the present disclosure. [Figure 51] 51A and 51B are diagrams illustrating examples of different display modes associated with different types of virtual objects with different mobility statuses, consistent with certain embodiments of the present disclosure. [Figure 52] 52A and 52B are diagrams illustrating examples of different display modes associated with different mobility statuses based on environmental context, consistent with certain embodiments of the present disclosure. [Figure 53] FIG. 53 is a flowchart of an exemplary method for coordinating virtual content display with mobility status, consistent with certain embodiments of the present disclosure. [Figure 54] FIG. 54 is a diagram generally illustrating a docking concept consistent with certain disclosed embodiments. [Figure 55]55A and 55B are exemplary illustrations of a keyboard docked with a virtual object in a first position on a support surface, consistent with certain disclosed embodiments. [Figure 56] 56A and 56B are exemplary illustrations of a keyboard moving from a position on a support surface to a position off the support surface, where one or more presented virtual objects are changed and disappear, consistent with certain disclosed embodiments. [Figure 57] FIG. 57 is a flowchart illustrating an exemplary method for developing docking based on detected keyboard position, consistent with certain disclosed embodiments. [Figure 58] FIG. 58 is a diagram illustrating an example of a virtual display and docked virtual object representing a user's phone, consistent with certain embodiments of the present disclosure. [Figure 59A] FIG. 59A is a diagram illustrating an example of a virtual display and multiple virtual objects located outside the virtual display before and after the virtual display changes position, consistent with certain embodiments of the present disclosure. [Figure 59B] FIG. 59B is a diagram illustrating an example of a virtual display and multiple virtual objects located outside the virtual display before and after the virtual display changes position, consistent with certain embodiments of the present disclosure. [Figure 60A] FIG. 60A illustrates an example of a virtual display and multiple virtual objects docked to the virtual display and other virtual objects before and after the virtual display changes position, consistent with certain embodiments of the present disclosure. [Figure 60B]FIG. 60B illustrates an example of a virtual display and multiple virtual objects docked to the virtual display and other virtual objects before and after the virtual display changes position, consistent with certain embodiments of the present disclosure. [Figure 61A] FIG. 61A is a diagram illustrating an example of a virtual display and a physical object, consistent with certain embodiments of the present disclosure. [Figure 61B] FIG. 61B is a diagram illustrating an example of a virtual display and a physical object, consistent with certain embodiments of the present disclosure. [Figure 62A] FIG. 62A is a diagram illustrating an example of a virtual display and multiple virtual objects before and after the virtual display changes position, consistent with certain embodiments of the present disclosure. [Figure 62B] FIG. 62B illustrates an example of a virtual display and multiple virtual objects before and after the virtual display changes position, consistent with certain embodiments of the present disclosure. [Figure 63] FIG. 63 illustrates a flowchart of an exemplary method for docking a virtual object to a virtual display screen, consistent with certain embodiments of the present disclosure. [Figure 64] FIG. 64 is a diagram illustrating an example of physical objects in a first plane and items in a second plane, consistent with certain embodiments of the present disclosure. [Figure 65] FIG. 65 is a diagram illustrating an example of a virtual object docked to a position in a virtual plane prior to a movement of a physical object, consistent with some embodiments of the present disclosure. [Figure 66] FIG. 66 is a diagram illustrating an example of the movement of physical and virtual objects, consistent with certain embodiments of the present disclosure. [Figure 67] FIG. 67 is a diagram illustrating an example of movement of an item and a virtual object, consistent with some embodiments of the present disclosure. [Figure 68]FIG. 68 is a flowchart of an exemplary method that may be executed by a processor to perform operations for implementing selective virtual object display modifications, consistent with certain embodiments of the present disclosure. [Figure 69] FIG. 69 is a schematic diagram illustrating an exemplary wearable extended reality device system consistent with some embodiments of the present disclosure. [Figure 70] FIG. 70 is a schematic diagram illustrating an exemplary display configuration, consistent with certain embodiments of the present disclosure. [Figure 71] FIG. 71 is a schematic diagram illustrating another exemplary display configuration, consistent with certain embodiments of the present disclosure. [Figure 72] FIG. 72 is a flowchart illustrating an exemplary process for determining a display configuration for presenting virtual content, consistent with certain embodiments of the present disclosure. [Figure 73] FIG. 73 is a diagram illustrating an example of virtual content displayed both on and off a computer screen, consistent with certain embodiments of the present disclosure. [Figure 74] FIG. 74 illustrates an example of virtual content displayed outside a smartwatch, consistent with certain embodiments of the present disclosure. [Figure 75] 75A-D are diagrams illustrating examples of virtual content movement between computer screens, consistent with certain embodiments of the present disclosure. [Figure 76] FIG. 76 is a flowchart illustrating an exemplary process for augmenting a working display, consistent with certain embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0024] The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numerals are used in the drawings and the following description to refer to the same or similar parts. While several exemplary embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, substitutions, additions, or modifications may be made to the components illustrated in the drawings, and the exemplary methods described herein may be modified by substituting, rearranging, deleting, or adding steps to the disclosed methods. Therefore, the following detailed description is not limited to the particular embodiments and examples, but includes the general principles described herein and illustrated in the drawings, in addition to the general principles encompassed by the appended claims.

[0025] This disclosure relates to systems and methods for providing users with extended reality environments. The term "extended reality environment," which may also be referred to as "extended reality space," "extended reality," or "augmented environment," refers to all types of real-virtual composite environments and human-machine interactions that are generated at least in part by computer technology. An extended reality environment may be a fully simulated virtual environment or a combination of real and virtual environments that a user can perceive from different perspectives. In some examples, a user can interact with elements of the extended reality environment. One non-limiting example of an extended reality environment may be a virtual reality environment, also known as "virtual reality" or "virtual environment." An immersive virtual reality environment may be a simulated non-physical environment that provides a user with the perception of being in a virtual environment. Another non-limiting example of an extended reality environment may be an augmented reality environment, also known as "augmented reality" or "augmented environment." An augmented reality environment may include a live direct or indirect view of a physical real-world environment augmented with virtual computer-generated perceptual information, such as virtual objects, with which the user can interact. Another non-limiting example of an extended reality environment is a mixed reality environment, also known as "mixed reality" or "composite environment." A mixed reality environment may be a hybrid of a physical, real-world environment and a virtual environment, where physical and virtual objects coexist and can interact in real time. In some examples, both augmented reality and mixed reality environments may include a combination of real and virtual worlds, real-time interaction, and precise 3D registration of virtual and real objects. In some examples, both augmented reality and mixed reality environments may include constructive overlaid sensory information that can be added to the physical environment. In other examples, both augmented reality and mixed reality environments may include disruptive virtual content that can mask at least a portion of the physical environment.

[0026] In some embodiments, systems and methods can provide an extended reality environment using an extended reality device. The term extended reality device can include any type of device or system that allows a user to perceive and / or interact with an extended reality environment. An extended reality device can allow a user to perceive and / or interact with the extended reality environment through one or more sensory modalities. Some non-limiting examples of such sensory modalities may include vision, hearing, touch, somatosensation, and olfaction. One example of an extended reality device is a virtual reality device that allows a user to perceive and / or interact with a virtual reality environment. Another example of an extended reality device is an augmented reality device that allows a user to perceive and / or interact with an augmented reality environment. Yet another example of an extended reality device is a mixed reality device that allows a user to perceive and / or interact with a mixed reality environment.

[0027] According to one aspect of the present disclosure, the extended reality device may be a wearable device such as a head-mounted device, e.g., smart glasses, smart contact lenses, a headset, or any other device worn by a human to present an extended reality to the human. Other extended reality devices may include a holographic projector, or any other device or system capable of providing augmented reality (AR), virtual reality (VR), mixed reality (MR), or any immersive experience. Typical components of a wearable extended reality device may include a stereoscopic head-mounted display, a stereoscopic head-mounted sound system, a head motion tracking sensor (e.g., a gyroscope, accelerometer, magnetometer, image sensor, structured light sensor, etc.), a head-mounted projector, an eye-tracking sensor, and at least one of the additional components described below. According to another aspect of the present disclosure, the extended reality device may be a non-wearable extended reality device. Specifically, a non-wearable extended reality device may include a multi-projection environment device. In some embodiments, the extended reality device may be configured to change the perspective of the field of view of the extended reality environment in response to user movement, particularly in response to movement of the user's head. In one example, a wearable extended reality device may change the field of view of the extended reality environment in response to a change in the user's head pose, such as by changing the spatial orientation without changing the user's spatial position in the extended reality environment. In another example, a non-wearable extended reality device may change the user's spatial position in the extended reality environment in response to a change in the user's position in the real world, such as by changing the user's spatial position in the extended reality environment without changing the orientation of the field of view relative to the spatial position.

[0028] According to some embodiments, the extended reality device may include a digital communications device configured to at least one of receive virtual content data configured to enable presentation of virtual content, transmit virtual content for sharing with at least one external device, receive contextual data from at least one external device, transmit contextual data to at least one external device, transmit usage data indicative of use of the extended reality device, and transmit data based on information captured using at least one sensor included in the extended reality device. In additional embodiments, the extended reality device may include a memory for storing at least one of virtual data configured to enable presentation of virtual content, contextual data, usage data indicative of use of the extended reality device, sensor data based on information captured using at least one sensor included in the extended reality device, software instructions configured to cause a processing device to present the virtual content, software instructions configured to cause the processing device to collect and analyze the contextual data, software instructions configured to cause the processing device to collect and analyze the usage data, and software instructions configured to cause the processing device to collect and analyze the sensor data. In additional embodiments, the extended reality device may include a processing device configured to perform at least one of rendering virtual content, collecting and analyzing contextual data, collecting and analyzing usage data, and collecting and analyzing sensor data. In additional embodiments, the extended reality device may include one or more sensors.The one or more sensors may include one or more image sensors (e.g., configured to capture images and / or video of a user of the device or the user's environment), one or more motion sensors (e.g., accelerometers, gyroscopes, magnetometers, etc.), one or more positioning sensors (e.g., GPS, outdoor positioning sensors, indoor positioning sensors, etc.), one or more temperature sensors (e.g., configured to measure the temperature of at least a portion of the device and / or environment), one or more contact sensors, one or more proximity sensors (e.g., configured to detect whether the device is currently being worn), one or more electrical impedance sensors (e.g., configured to measure the user's electrical impedance), one or more eye-tracking sensors, such as a gaze detector, an optical tracker, an electric potential tracker (e.g., an electro-oculogram (EOG) sensor), a video-based gaze tracker, an infrared / near-infrared sensor, a passive optical sensor, or any other technology capable of determining where a human is looking or gazing, etc.

[0029] In some embodiments, the systems and methods may use input devices to interact with the extended reality device. The term input device may include any physical device configured to receive input from a user or the user's environment and provide data to a computing device. The data provided to the computing device may be in digital and / or analog format. In one embodiment, the input device may store input received from a user in a memory device accessible by a processing device, which may access the stored data for analysis. In another embodiment, the input device may provide data directly to the processing device, for example, via a bus or another communication system configured to transfer data from the input device to the processing device. In some examples, input received by the input device may include key presses, tactile input data, motion data, positional data, gesture-based input data, directional data, or any other data intended for use in computation. Some examples of input devices may include buttons, keys, keyboards, computer mice, touchpads, touchscreens, joysticks, or other mechanisms capable of receiving input. Another example of an input device may include an integrated computing interface device that includes at least one physical component for receiving input from a user. The integrated computing interface device may include at least a memory, a processing device, and at least one physical component for receiving input from a user. In one example, the integrated computing interface device may further include a digital network interface that enables digital communication with other computing devices. In one example, the integrated computing interface device may further include a physical component for outputting information to a user. In some examples, all components of the integrated computing interface device may be contained in a single housing, while in other examples, the components may be distributed across two or more housings.Some non-limiting examples of physical components for receiving input from a user that may be included in an integrated computing interface device may include at least one of buttons, keys, a keyboard, a touchpad, a touchscreen, a joystick, or any other mechanism or sensor capable of receiving computational information. Some non-limiting examples of physical components for outputting information to a user may include at least one of a light indicator (such as an LED indicator), a screen, a touchscreen, a buzzer, an audio speaker, or any other audio, video, or tactile device that provides a human-perceptible output.

[0030] In some embodiments, the image data may be captured using one or more image sensors. In some examples, the image sensors may be included in an extended reality device, a wearable device, a wearable extended reality device, an input device, the user's environment, etc. In some examples, the image data may be read from memory, received from an external device, generated (e.g., using a generative model), etc. Some non-limiting examples of image data may include images, grayscale images, color images, 2D images, 3D images, video, 2D video, 3D video, frames, images, data derived from other image data, etc. In some examples, the image data may be encoded in any analog or digital format. Some non-limiting examples of such formats may include raw formats, compressed formats, uncompressed formats, lossy formats, lossless formats, JPEG, GIF, PNG, TIFF, BMP, NTSC, PAL, SECAM, MPEG, MPEG-4 Part 14, MOV, WMV, FLV, AVI, AVCHD, WebM, MKV, etc.

[0031] In some embodiments, the extended reality device can receive a digital signal, for example, from an input device. The term digital signal refers to a time-discrete series of digital values. The digital signal can represent, for example, sensor data, text data, audio data, video data, virtual data, or any other form of data that provides perceptible information. Consistent with the present disclosure, the digital signal can be configured to cause the extended reality device to present virtual content. In one embodiment, the virtual content may be presented in a selected orientation. In this embodiment, the digital signal can indicate the position and angle of a viewpoint in an environment, such as an extended reality environment. Specifically, the digital signal can include encoding of position and angle in six degrees of freedom (e.g., forward / backward, up / down, left / right, yaw, pitch, and roll). In another embodiment, the digital signal can include encoding of position as three-dimensional coordinates (e.g., x, y, and z) and encoding of angle as a vector resulting from the encoded position. Specifically, the digital signal may indicate the orientation and angle of the presented virtual content in absolute coordinates of the environment, for example, by encoding the yaw, pitch, and roll of the virtual content relative to a standard default angle. In another embodiment, the digital signal may indicate the orientation and angle of the presented virtual content relative to the viewpoint of another object (e.g., a virtual object, a physical object, etc.), for example, by encoding the yaw, pitch, and roll of the virtual content relative to a direction corresponding to the viewpoint or a direction corresponding to another object. In another embodiment, such a digital signal may include one or more projections of the virtual content, for example, in a presentation-ready format (e.g., an image, a video, etc.). For example, each such projection may correspond to a particular orientation or a particular angle. In another embodiment, the digital signal may include a representation of the virtual content, for example, by encoding the object in a three-dimensional array of voxels, a polygonal mesh, or any other format in which the virtual content can be presented.

[0032] In some embodiments, the digital signal may be configured to cause the extended reality device to present virtual content. The term virtual content may include any type of data representation that can be displayed to a user by the extended reality device. Virtual content may include virtual objects, inanimate virtual content, animate virtual content configured to change over time or in response to a trigger, virtual two-dimensional content, virtual three-dimensional content, virtual overlays on portions of or physical objects in a physical environment, virtual additions to a physical environment or object, virtual promotional content, virtual representations of physical objects, virtual representations of a physical environment, virtual documents, virtual characters or personas, virtual computer screens, virtual widgets, or any other format for virtually displaying information. Consistent with this disclosure, virtual content may include any visual presentation rendered by a computer or processing device. In one embodiment, virtual content may include virtual objects, which are visual presentations rendered by a computer within a limited area and configured to represent a particular type of object (e.g., an inanimate virtual object, an animate virtual object, virtual furniture, virtual decorative object, virtual widget, or other virtual representation, etc.). The rendered visual presentation may change to reflect changes to a state object or a change in the viewing angle of the object, for example, to mimic a change in the appearance of a physical object. In another embodiment, the virtual content may include a virtual display (also referred to herein as a "virtual display screen" or "virtual screen"), such as a virtual computer screen, a virtual tablet screen, or a virtual smartphone screen, configured to display information generated by an operating system, where the operating system may be configured to receive text data from a physical keyboard and / or a virtual keyboard and cause the display of the text content on the virtual display screen. In one example, as shown in FIG. 1 , the virtual content may include a virtual environment including a virtual computer screen and a plurality of virtual objects.In some examples, the virtual display may be a virtual object that mimics and / or extends the functionality of a physical display screen. For example, the virtual display may be presented in an extended reality environment (e.g., a mixed reality environment, an augmented reality environment, a virtual reality environment, etc.) using extended reality equipment. In one example, the virtual display may present content generated by a conventional operating system that could equally be presented on a physical display screen. In one example, text content entered using a keyboard (e.g., using a physical keyboard, using a virtual keyboard, etc.) may be presented on the virtual display in real time as the text content is typed. In one example, a virtual cursor may be presented on the virtual display, and the virtual cursor may be controlled by a pointing device (e.g., a physical pointing device, a virtual pointing device, a computer mouse, a joystick, a touchpad, a physical touch controller, etc.). In one example, one or more windows of a graphical user interface operating system may be presented on the virtual display. In another example, the content presented on the virtual display may be interactive, i.e., it may change in response to user actions. In yet another example, the presentation of the virtual display may or may not include the presentation of a screen frame.

[0033] Some disclosed embodiments may include and / or access a data structure or database. Consistent with this disclosure, the terms data structure and database may include any collection of data values ​​and the relationships between them. Data may be stored linearly, horizontally, hierarchically, relationally, non-relationally, unidimensionally, multidimensionally, operationally, orderedly, unorderedly, object-orientedly, centralizedly, decentralizedly, distributedly, customly, or in any manner that enables data access. By way of non-limiting example, data structures may include arrays, associative arrays, linked lists, binary trees, balanced trees, mountains, stacks, queues, sets, hash tables, records, tagged unions, entity-relationship models, graphs, hypergraphs, matrices, tensors, etc. For example, the data structure may include an XML database, an RDBMS database, an SQL database, or a NoSQL alternative for data storage / retrieval, such as, for example, MongoDB, Redis, Couchbase, Datastax Enterprise Graph, Elastic Search, Splunk, Solr, Cassandra, Amazon DynamoDB, Scylla, HBase, and Neo4J. The data structure may be a component of the disclosed system or a remote computing component (e.g., a cloud-based data structure). Data in the data structure may be stored in contiguous or discontiguous memory. Furthermore, the data structure does not require the information to be co-located. It may be distributed, for example, across multiple servers that may be owned or operated by the same or different entities. Thus, the singular term "data structure" includes multiple data structures.

[0034] In some embodiments, the system can determine a confidence level of a received input or any determined value. The term confidence level refers to a numerical value or any other indication of a level (e.g., within a predetermined range) that indicates the amount of confidence the system has in the determined data. For example, the confidence level may have a value between 1 and 10. Alternatively, the confidence level may be expressed as a percentage or any other numerical or non-numerical indication. In some cases, the system can compare the confidence level to a threshold. The term threshold can refer to a reference value, level, point, or range of values. During operation, if the determined confidence level of the data exceeds (or is below, depending on the particular use case), the system can take a first course of action; if the confidence level is below (or above, depending on the particular use case), the system can take a second course of action. The value of the threshold may be predetermined for each type of object being investigated or may be dynamically selected based on different considerations.

[0035] System Overview Reference is now made to FIG. 1 , which illustrates a user using an exemplary extended reality system consistent with various embodiments of the present disclosure. It should be understood that FIG. 1 is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure. As illustrated, a user 100 is seated behind a table 102 and supports a keyboard 104 and a mouse 106. The keyboard 104 is connected by wires 108 to a wearable extended reality device 110 that displays virtual content to the user 100. Alternatively, or in addition to the wires 108, the keyboard 104 may be wirelessly connected to the wearable extended reality device 110. For purposes of illustration, the wearable extended reality device is shown as a pair of smart glasses, but as noted above, the wearable extended reality device 110 may be any type of head-mounted device used to present extended reality to the user 100. The virtual content displayed by the wearable extended reality device 110 includes a virtual screen 112 (also referred to herein as a “virtual display screen” or “virtual display”) and multiple virtual widgets 114. Virtual widgets 114A-114D are displayed adjacent to the virtual screen 112, and virtual widget 114E is displayed on the table 102. The user 100 can use the keyboard 104 to enter text into a document 116 displayed on the virtual screen 112 and can use the mouse 106 to control a virtual cursor 118. In one example, the virtual cursor 118 can move anywhere within the virtual screen 112. In another example, the virtual cursor 118 can move anywhere within the virtual screen 112 and can move to any of the virtual widgets 114A-114D, but not to the virtual widget 114E. In yet another example, the virtual cursor 118 can move anywhere within the virtual screen 112 and can move to any of the virtual widgets 114A-114E.In an additional example, virtual cursor 118 can move anywhere in the extended reality environment, including virtual screen 112 and virtual widgets 114A-114E. In yet another example, virtual cursor 118 can move over all available surfaces (i.e., virtual surfaces or physical surfaces) or only over selected surfaces in the extended reality environment. Alternatively or additionally, user 100 can interact with any of virtual widgets 114A-114E, or a selected virtual widget, using hand gestures recognized by wearable extended reality device 110. For example, virtual widget 114E can be an interactive widget (e.g., a virtual slider controller) that can be manipulated with hand gestures.

[0036] FIG. 2 illustrates an example of a system 200 for providing an extended reality (XR) experience to a user, such as user 100. It should be understood that FIG. 2 is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure. System 200 may be computer-based and may include computer system components, wearable devices, workstations, tablets, handheld computing devices, memory devices, and / or an internal network connecting the components. System 200 may include or be connected to various network computing resources (e.g., servers, routers, switches, network connections, storage devices, etc.) to support the services provided by system 200. Consistent with the present disclosure, system 200 may include an input unit 202, an XR unit 204, a mobile communication device 206, and a remote processing unit 208. Remote processing unit 208 may include a server 210 coupled to one or more physical or virtual storage devices, such as a data structure 212. The system 200 may also include or be connected to a communication network 214 that facilitates communication and data exchange between different system components and different entities associated with the system 200 .

[0037] Consistent with the present disclosure, the input unit 202 may include one or more devices capable of receiving input from the user 100. In one embodiment, the input unit 202 may include a text input device such as the keyboard 104. The text input device may include all possible types of devices and mechanisms for inputting text information into the system 200. Examples of text input devices may include a mechanical keyboard, a membrane keyboard, a flexible keyboard, a QWERTY keyboard, a Dvorak keyboard, a Colemak keyboard, a corded keyboard, a wireless keyboard, a keypad, a key-based control panel, or other arrangement of control keys, a visual input device, or any other mechanism for inputting text (whether the mechanism is provided in physical form or presented virtually). In one embodiment, the input unit 202 may also include a pointing input device such as the mouse 106. The pointing input device may include all possible types of devices and mechanisms for inputting two-dimensional or three-dimensional information into the system 200. In one example, two-dimensional input from the pointing input device may be used to interact with virtual content presented via the XR unit 204. Examples of pointing input devices include a computer mouse, trackball, touchpad, trackpad, touchscreen, joystick, pointing stick, stylus, light pen, or any other physical or virtual input mechanism. In one embodiment, the input unit 202 may also include a graphical input device, such as a touchscreen, configured to detect contact, movement, or cessation of movement. The graphical input device may use any of a number of touch-sensitivity technologies, including, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact. In one embodiment, the input unit 202 may also include one or more audio input devices, such as a microphone.The audio input device may include all possible types of devices and mechanisms for inputting audio data to facilitate voice-enabled functions such as voice recognition, audio replication, digital recording, and telephone functions. In one embodiment, the input unit 202 may also include one or more image input devices, such as an image sensor, configured to capture image data. In one embodiment, the input unit 202 may also include one or more haptic gloves configured to capture hand movement and posture data. In one embodiment, the input unit 202 may also include one or more proximity sensors configured to detect the presence and / or movement of objects within a selected area near the sensor.

[0038] According to some embodiments, the system may include at least one sensor configured to detect and / or measure characteristics associated with a user, the user's actions, or the user's environment. One example of the at least one sensor is the sensor 216 included in the input unit 202. The sensor 216 may be a motion sensor, a touch sensor, a light sensor, an infrared sensor, an audio sensor, an image sensor, a proximity sensor, a positioning sensor, a gyroscope, a temperature sensor, a biometric sensor, or any other sensing device for facilitating related functions. The sensor 216 may be integrated with, connected to, or separate from the input device. In one example, the mouse 106 may include a thermometer to determine the user's 100 body temperature. In another example, a positioning sensor may be integrated with the keyboard 104 to determine the user's 100 movement relative to the keyboard 104. Such a positioning sensor may be implemented using one of the following positioning technologies: Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), Galileo Global Navigation System, BeiDou Navigation System, other Global Navigation Satellite System (GNSS), Indian Regional Navigation Satellite System (IRNSS), Local Positioning System (LPS), Real-Time Location System (RTLS), Indoor Positioning System (IPS), Wi-Fi based positioning system, cellular triangulation, image-based positioning technology, indoor positioning technology, outdoor positioning technology, or any other positioning technology.

[0039] According to some embodiments, the system may include one or more sensors for identifying the position and / or movement of a physical device (e.g., a physical input device, a physical computing device, a keyboard 104, a mouse 106, a wearable extended reality device 110, etc.). The one or more sensors may be included in the physical device or may be external to the physical device. In some examples, an image sensor external to the physical device (e.g., an image sensor included in another physical device) may be used to capture image data of the physical device, and the image data may be analyzed to identify the position and / or movement of the physical device. For example, the image data may be analyzed using a visual object tracking algorithm to identify movement of the physical device, a visual object detection algorithm to identify the position of the physical device (e.g., relative to the image sensor, in a global coordinate system, etc.), etc. In some examples, an image sensor included in the physical device may be used to capture image data, and the image data may be analyzed to identify the position and / or movement of the physical device. For example, the image data may be analyzed using a visual odometry algorithm to identify the position of the physical device, an egomotion algorithm to identify movement of the physical device, etc. In some examples, a positioning sensor, such as an indoor positioning sensor or an outdoor positioning sensor, may be included in the physical device and used to determine the position of the physical device. In some examples, a motion sensor, such as an accelerometer or gyroscope, may be included in the physical device and used to determine the movement of the physical device. In some examples, a physical device, such as a keyboard or mouse, may be configured to be placed on a physical surface. Such a physical device may include an optical mouse sensor (also known as a non-mechanical tracking engine) pointed at the physical surface, and the output of the optical mouse sensor may be analyzed to determine the movement of the physical device relative to the physical surface.

[0040] Consistent with this disclosure, the XR unit 204 may include a wearable extended reality device configured to present virtual content to the user 100. An example of a wearable extended reality device is the wearable extended reality device 110. Further examples of wearable extended reality devices may include virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices, or any other device capable of generating extended reality content. Some non-limiting examples of such devices include Nreal Light, Magic Leap One, Varjo, Quest 1 / 2, Vive, etc. In some embodiments, the XR unit 204 may present virtual content to the user 100. In general, extended reality devices may include all real and virtual composite environments and human-machine interactions generated by computer technology and wearables. As discussed above, the term “extended reality” (XR) refers to a superset that includes the entire spectrum from “fully real” to “fully virtual.” It includes representative forms such as augmented reality (AR), mixed reality (MR), virtual reality (VR), and regions interpolated therebetween. Accordingly, it should be noted that the terms "XR device," "AR device," "VR device," and "MR device" can be used interchangeably herein and can refer to any device among the various devices listed above.

[0041] Consistent with the present disclosure, the system can exchange data with various communication devices associated with the user, such as a mobile communication device 206. The term “communication device” is intended to include all possible types of devices capable of exchanging data using a digital communication network, an analog communication network, or any other communication network configured to transmit data. In some examples, the communication device may include a smartphone, a tablet, a smartwatch, a personal digital assistant, a desktop computer, a laptop computer, an IoT device, a dedicated terminal, a wearable communication device, and any other device enabling data communication. In some cases, the mobile communication device 206 can supplement or replace the input unit 202. Specifically, the mobile communication device 206 may be associated with a physical touch controller that can function as a pointing input device. Furthermore, the mobile communication device 206 may also be used, for example, to implement a virtual keyboard and replace a text input device. For example, the user 100 may walk away from the table 102 to the break room with their smart glasses and receive an email requiring a quick response. In this case, the user may choose to use their smartwatch as an input device and type a response to the email while it is virtually presented by the smartglasses.

[0042] Consistent with the present disclosure, system embodiments may include the use of a cloud server. The term "cloud server" refers to a computer platform that provides services over a network, such as the Internet. In the exemplary embodiment shown in FIG. 2, the server 210 may use a virtual machine that may not correspond to individual pieces of hardware. For example, computing and / or storage capabilities may be implemented by allocating an appropriate portion of the desired computing / storage power from a scalable repository, such as a data center or a distributed computing environment. Specifically, in one embodiment, the remote processing unit 208 may be used in conjunction with the XR unit 204 to provide virtual content to the user 100. In one configuration, the server 210 may be a cloud server that functions as an operating system (OS) for the wearable extended reality device. In one example, the server 210 may implement the methods described herein using customized hardwired logic, one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), firmware, and / or program logic that, in combination with a computer system, makes the server 210 a dedicated machine.

[0043] In some embodiments, server 210 can access data structure 212 to, for example, determine virtual content for display to user 100. Data structure 212 can utilize volatile or non-volatile, magnetic, semiconductor, tape, optical, removable, non-removable, or other types of storage devices or tangible or non-transitory computer-readable media, or any medium or mechanism for storing information. Data structure 212 can be part of server 210, as shown, or can be separate from server 210. If data structure 212 is not part of server 210, server 210 can exchange data with data structure 212 via a communications link. Data structure 212 can include one or more memory devices that store data and instructions used to execute one or more features of the disclosed methods. In one embodiment, data structure 212 can include any of several suitable data structures, from a small data structure hosted on a workstation to a large data structure distributed across a data center. Data structure 212 can also include any combination of one or more data structures controlled by a memory controller device (e.g., a server) or software.

[0044] Consistent with this disclosure, a communications network may be any type of network (including infrastructure) that supports communications, exchanges information, and / or facilitates the exchange of information between components of a system. For example, communications network 214 in system 200 may include, for example, a telephone network, an extranet, an intranet, the Internet, satellite communications, offline communications, radio communications, transponder communications, a local area network (LAN), a wireless network (e.g., a Wi-Fi / 302.11 network), a wide area network (WAN), a virtual private network (VPN), a digital communications network, an analog communications network, or any other mechanism or combination of mechanisms that enables data transmission.

[0045] The components and arrangement of system 200 shown in FIG. 2 are intended to be illustrative only and are not intended to limit any embodiment, as the system components used to implement the disclosed processes and features may vary.

[0046] FIG. 3 is a block diagram illustrating an example configuration of input unit 202. It should be understood that FIG. 3 is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure. In the embodiment of FIG. 3, input unit 202 may directly or indirectly access bus 300 (or other communication mechanism) that interconnects subsystems and components for transferring information within input unit 202. For example, bus 300 may interconnect memory interface 310, network interface 320, input interface 330, power supply 340, output interface 350, processing device 360, sensor interface 370, and database 380.

[0047] The memory interface 310 shown in FIG. 3 can be used to access software products and / or data stored on a non-transitory computer-readable medium. Generally, a non-transitory computer-readable storage medium refers to any type of physical memory capable of storing information or data readable by at least one processor. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, CD-ROMs, DVDs, flash drives, disks, any other optical data storage media, any physical media with a pattern of holes, PROMs, EPROMs, FLASH-EPROMs or any other flash memory, NVRAMs, caches, registers, any other memory chips or cartridges, and networked versions thereof. The terms "memory" and "computer-readable storage medium" can refer to multiple structures, such as multiple memories or computer-readable storage media located within an input unit or at a remote location. Furthermore, one or more computer-readable storage media can be utilized in performing a computer-implemented method. Thus, the term computer-readable storage medium should be understood to include tangible items and exclude carrier waves and transient signals. 3, memory interface 310 may be used to access software products and / or data stored on a memory device, such as memory device 311. Memory device 311 may include high-speed random access memory and / or non-volatile memory, such as one or more magnetic disk storage devices, one or more optical storage devices, and / or flash memory (e.g., NAND, NOR). Consistent with the present disclosure, components of memory device 311 may be distributed across two or more units and / or two or more memory devices of system 200.

[0048] The memory device 311 shown in FIG. 3 may include software modules for executing processes consistent with the present disclosure. In particular, the memory device 311 may include an input determination module 312, an output determination module 313, a sensor communication module 314, a virtual content determination module 315, a virtual content communication module 316, and a database access module 317. Modules 312-317 may include software instructions for execution by at least one processor (e.g., processing device 360) associated with the input unit 202. The input determination module 312, the output determination module 313, the sensor communication module 314, the virtual content determination module 315, the virtual content communication module 316, and the database access module 317 may cooperate to perform various operations. For example, the input determination module 312 may determine text using data received, for example, from the keyboard 104. The output determination module 313 may then cause the presentation of the recently entered text on a dedicated display 352, for example, physically or wirelessly coupled to the keyboard 104. In this way, as the user 100 types, they can see a preview of the typed text without having to constantly move their head up and down to look at the virtual screen 112. The sensor communication module 314 can receive data from different sensors to determine the state of the user 100. The virtual content determination module 315 can then determine virtual content to display based on the received input and the determined status of the user 100. For example, the determined virtual content may be a virtual presentation of recently entered text on a virtual screen located virtually adjacent to the keyboard 104. The virtual content communication module 316 may retrieve virtual content not determined by the virtual content determination module 315 (e.g., another user's avatar). The retrieval of virtual content may be from the database 380, from the remote processing unit 208, or from any other source.

[0049] In some embodiments, the input determination module 312 can adjust the operation of the input interface 330 to receive pointer input 331, text input 332, voice input 333, and XR-related input 334. Details of pointer input, text input, and voice input are described above. The term “XR-related input” can include any type of data that may cause a change in virtual content displayed to the user 100. In one embodiment, the XR-related input 334 may include image data of the user 100, the wearable extended reality device (e.g., a detected hand gesture of the user 100), or other data. In another embodiment, the XR-related input 334 may include wireless communication indicating the presence of another user in proximity to the user 100. Consistent with the present disclosure, the input determination module 312 can simultaneously receive different types of input data. The input determination module 312 can then further apply different rules based on the type of input detected. For example, pointer input may be prioritized over voice input.

[0050] In some embodiments, the output determination module 313 can coordinate operation of the output interface 350 to generate output using light indicators 351, display 352, and / or speaker 353. Generally, the output generated by the output determination module 313 does not include virtual content presented by the wearable extended reality device. Instead, the output generated by the output determination module 313 includes various outputs related to the operation of the input unit 202 and / or the operation of the XR unit 204. In one embodiment, the light indicator 351 can include a light indicator that indicates the status of the wearable extended reality device. For example, the light indicator can display a green light when the wearable extended reality device 110 is connected to the keyboard 104 and flash when the battery of the wearable extended reality device 110 is low. In another embodiment, the display 352 can be used to display operational information. For example, the display can present an error message when the wearable extended reality device is inoperable. In another embodiment, the speaker 353 may be used to output sound, for example, if the user 100 wants to play music for other users.

[0051] In some embodiments, the sensor communications module 314 can coordinate the operation of the sensor interface 370 to receive sensor data from one or more sensors integrated with or connected to the input device. The one or more sensors can include an audio sensor 371, an image sensor 372, a motion sensor 373, an environmental sensor 374 (e.g., a temperature sensor, an ambient light detector, etc.), and other sensors 375. In one embodiment, the data received from the sensor communications module 314 can be used to determine the physical orientation of the input device. The physical orientation of the input device can indicate the user's state and can be determined based on a combination of tilt, roll, and lateral motion. The physical orientation of the input device can then be used by the virtual content determination module 315 to modify display parameters of the virtual content to match the user's state (e.g., alert, sleepy, active, sitting, standing, leaning back, leaning forward, walking, moving, riding, etc.).

[0052] In some embodiments, the virtual content determination module 315 can determine virtual content to be displayed by the wearable extended reality device. The virtual content can be determined based on data from the input determination module 312, the sensor communication module 314, and other sources (e.g., database 380). In some embodiments, determining the virtual content can include determining the distance, size, and orientation of a virtual object. The location of the virtual object can be determined based on the type of virtual object. Specifically, with respect to the example shown in FIG. 1 , the virtual content determination module 315 can determine to place four virtual widgets 114A-114D on both sides of the virtual screen 112 and place the virtual widget 114E on the table 102 because the virtual widget 114E is a virtual controller (e.g., a volume bar). The location of the virtual object can be further determined based on a user's preferences. For example, for a left-handed user, the virtual content determination module 315 may determine to place the virtual volume bar to the left of the keyboard 104, and for a right-handed user, the virtual content determination module 315 may determine to place the virtual volume bar to the right of the keyboard 104.

[0053] In some embodiments, the virtual content communication module 316 may coordinate the operation of the network interface 320 to obtain data from one or more sources to be presented as virtual content to the user 100. The one or more sources may include other XR units 204, the user's mobile communication device 206, the remote processing unit 208, publicly available information, etc. In one embodiment, the virtual content communication module 316 can communicate with the mobile communication device 206 to provide a virtual representation of the mobile communication device 206. For example, the virtual representation may allow the user 100 to read messages and interact with applications installed on the mobile communication device 206. The virtual content communication module 316 can also coordinate the operation of the network interface 320 to share virtual content with other users. In one example, the virtual content communication module 316 can use data from the input determination module to identify triggers (e.g., the triggers may include a user gesture) and transfer content from the virtual display to a physical display (e.g., a TV) or a different user's virtual display.

[0054] In some embodiments, database access module 317 can cooperate with database 380 to retrieve stored data. The retrieved data can include, for example, privacy levels associated with different virtual objects, relationships between virtual and physical objects, user preferences, user past behavior, etc. As described above, virtual content determination module 315 can determine virtual content using data stored in database 380. Database 380 can include separate databases, including, for example, a vector database, a raster database, a tile database, a viewport database, and / or a user input database. Data stored in database 380 can be received from modules 314-317 or other components of system 200. Additionally, data stored in database 380 can be provided as input using data entry, data transfer, or data upload.

[0055] Modules 312-317 may be implemented in software, hardware, firmware, any combination thereof, etc. In some embodiments, any one or more of modules 312-317 and data associated with database 380 may be stored in XR unit 204, mobile communication device 206, or remote processing unit 208. A processing device of system 200 may be configured to execute instructions of modules 312-317. In some embodiments, aspects of modules 312-317 may be implemented in hardware, software (including one or more signal processing and / or application specific integrated circuits), firmware, or any combination thereof, executable by one or more processors, alone or in various combinations with each other. In particular, modules 312-317 may be configured to interact with each other and / or other modules of system 200 to perform functions consistent with some disclosed embodiments. For example, input unit 202 may execute instructions including image processing algorithms on data from XR unit 204 to determine head movement of user 100. Additionally, throughout this specification, each function described with respect to input unit 202 or with respect to components of input unit 202 may correspond to a set of instructions for performing that function. These instructions need not be implemented as separate software programs, procedures, or modules. Memory device 311 may include additional or fewer modules and instructions. For example, memory device 311 may store an operating system such as ANDROID, iOS, UNIX, OSX, WINDOWS, DARWIN, RTXC, LINUX, or an embedded operating system such as VXWorkS. The operating system may include instructions for handling basic system services and performing hardware-dependent tasks.

[0056] The network interface 320 shown in FIG. 3 can provide bidirectional data communication to a network, such as the communications network 214. In one embodiment, the network interface 320 can include an Integrated Services Digital Network (ISDN) card, a cellular modem, a satellite modem, or a modem to provide a data communication connection over the Internet. As another example, the network interface 320 can include a wireless local area network (WLAN) card. In another embodiment, the network interface 320 can include an Ethernet port connected to a radio frequency receiver and transmitter and / or an optical (e.g., infrared) receiver and transmitter. The specific design and implementation of the network interface 320 can depend on the communications network over which the input unit 202 is intended to operate. For example, in some embodiments, the input unit 202 can include a network interface 320 designed to operate over a GSM network, a GPRS network, an EDGE network, a Wi-Fi or WiMax network, and a Bluetooth network. In any such implementation, the network interface 320 can be configured to send and receive electrical, electromagnetic, or optical signals that carry digital data streams or signals representing various types of information.

[0057] The input interface 330 shown in FIG. 3 can receive input from a variety of input devices, such as a keyboard, a mouse, a touchpad, a touchscreen, one or more buttons, a joystick, a microphone, an image sensor, and any other device configured to detect physical or virtual input. The received input may be in the form of at least one of text, voice, speech, hand gestures, body gestures, tactile information, and any other type of physical or virtual input generated by a user. In the illustrated embodiment, the input interface 330 can receive pointer input 331, text input 332, voice input 333, and XR-related input 334. In additional embodiments, the input interface 330 may be an integrated circuit that can function as a bridge between the processing device 360 ​​and any of the above input devices.

[0058] The power source 340 shown in FIG. 3 can provide electrical energy to the power input unit 202 and, optionally, to the XR unit 204. In general, the power source included in any device or system of the present disclosure may be any device capable of repeatedly storing, distributing, or transmitting electrical power, including, but not limited to, one or more batteries (e.g., lead-acid, lithium-ion, nickel-metal hydride, nickel-cadmium batteries), one or more capacitors, one or more connections to external power sources, one or more power converters, or any combination thereof. Referring to the example shown in FIG. 3 , the power source may be mobile, meaning that the input unit 202 can be easily carried by hand (e.g., the total weight of the power source 340 may be less than 1 pound). The mobility of the power source allows the user 100 to use the input unit 202 in a variety of situations. In other embodiments, the power source 340 may be associated with a connection to an external power source (such as a power grid) that can be used to charge the power source 340. Additionally, the power source 340 may be configured to charge one or more batteries included in the XR unit 204, such that a pair of extended reality glasses (e.g., wearable extended reality devices 110) may be charged (e.g., wirelessly or non-wirelessly) when they are placed on or near the input unit 202.

[0059] The output interface 350 shown in FIG. 3 can produce output from various output devices using, for example, light indicators 351, a display 352, and / or a speaker 353. In one embodiment, the output interface 350 can be an integrated circuit that can act as a bridge between the processing device 360 ​​and at least one of the above output devices. The light indicator 351 can include one or more light sources, such as, for example, an LED array associated with different colors. The display 352 can include a screen (e.g., an LCD or dot-matrix screen) or a touchscreen. The speaker 353 can include audio headphones, a hearing aid-type device, a speaker, bone conduction headphones, an interface that provides tactile cues, a vibration stimulation device, etc.

[0060] The processing device 360 ​​shown in FIG. 3 may include at least one processor configured to execute computer programs, applications, methods, processes, or other software to implement the embodiments described herein. Generally, the processing device included in any device or system of the present disclosure may include all or part of one or more integrated circuits, microchips, microcontrollers, microprocessors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), or other circuitry suitable for executing instructions or performing logical operations. The processing device may include at least one processor configured to perform the functions of the disclosed methods, such as a microprocessor manufactured by Intel™. The processing device may include a single-core or multiple-core processor that simultaneously performs parallel processing. In one example, the processing device may be a single-core processor configured with virtual processing technology. The processing device may implement virtual machine technology or other technologies to provide the ability to run, control, execute, manipulate, store, etc., multiple software processes, applications, programs, etc. In another example, the processing device may include a multi-core processor configuration (e.g., dual, quad-core, etc.) configured to provide parallel processing capabilities that allow devices associated with the processing device to execute multiple processes simultaneously. It should be understood that other types of processor configurations may be implemented to provide the functionality disclosed herein.

[0061] 3 can acquire sensor data from various sensors, such as an audio sensor 371, an image sensor 372, a motion sensor 373, an environmental sensor 374, and other sensors 375. In one embodiment, the sensor interface 370 can be an integrated circuit that can act as a bridge between the processing device 360 ​​and at least one of the above sensors.

[0062] Audio sensor 371 may include one or more audio sensors configured to capture audio by converting the audio into digital information. Some examples of audio sensors may include a microphone, a unidirectional microphone, a bidirectional microphone, a cardioid microphone, an omnidirectional microphone, an on-board microphone, a wired microphone, a wireless microphone, or any combination of the above. Consistent with the present disclosure, processing device 360 ​​can modify the presentation of virtual content based on data received from audio sensor 371 (e.g., voice commands).

[0063] Image sensor 372 may include one or more image sensors configured to capture visual information by converting light into image data. Consistent with this disclosure, the image sensor may be included in any device or system of this disclosure and may be any device capable of detecting and converting optical signals in the near-infrared, infrared, visible, and ultraviolet spectrums into electrical signals. Examples of image sensors may include digital cameras, phone cameras, semiconductor charge-coupled devices (CCDs), and complementary metal-oxide semiconductor (CMOS) or N-type metal-oxide semiconductor (NMOS, live MOS) active pixel sensors. The electrical signals may be used to generate image data. Consistent with this disclosure, image data may include pixel data streams, digital images, digital video streams, data derived from captured images, and data that may be used to construct one or more 3D images, a sequence of 3D images, a 3D video, or a virtual 3D representation. Image data acquired by image sensor 372 may be transmitted to any processing device of system 200 via wired or wireless transmission. For example, the image data may be processed for object detection, event detection, action detection, face detection, person detection, known person recognition, or any other information that may be used by system 200. Consistent with the present disclosure, processing device 360 ​​may modify the presentation of virtual content based on the image data received from image sensor 372.

[0064] The motion sensor 373 may include one or more motion sensors configured to measure movement of the input unit 202 or movement of objects in the environment of the input unit 202. Specifically, the motion sensor may perform at least one of: detecting movement of objects in the environment of the input unit 202, measuring the velocity of objects in the environment of the input unit 202, measuring the acceleration of objects in the environment of the input unit 202, detecting movement of the input unit 202, measuring the velocity of the input unit 202, measuring the acceleration of the input unit 202, etc. In some embodiments, the motion sensor 373 may include one or more accelerometers configured to detect changes in and / or measure the appropriate acceleration of the input unit 202. In other embodiments, the motion sensor 373 may include one or more gyroscopes configured to detect changes in the orientation of the input unit 202 and / or measure information related to the orientation of the input unit 202. In other embodiments, the motion sensor 373 may include one or more using image sensors, lidar sensors, radar sensors, or proximity sensors. For example, by analyzing the captured images, the processing device can determine the movement of the input unit 202, e.g., using an egomotion algorithm. Additionally, the processing device can determine the movement of objects in the environment of the input unit 202, e.g., using an object tracking algorithm. Consistent with this disclosure, the processing device 360 ​​can modify the presentation of virtual content based on the determined movement of the input unit 202 or the determined movement of objects in the environment of the input unit 202, e.g., by causing a virtual display to track the movement of the input unit 202.

[0065] The environmental sensors 374 may include one or more sensors of different types configured to capture data reflective of the input unit 202's environment. In some embodiments, the environmental sensors 374 may include one or more chemical sensors configured to measure at least one of: a chemical property in the input unit 202's environment; a change in a chemical property in the input unit 202's environment; a presence of a chemical in the input unit 202's environment; or a concentration of a chemical in the input unit 202's environment. Examples of such chemical properties may include pH levels, toxicity, and temperature. Examples of such chemicals may include electrolytes, certain enzymes, certain hormones, certain proteins, smoke, carbon dioxide, carbon monoxide, oxygen, ozone, hydrogen, and hydrogen sulfide. In other embodiments, the environmental sensors 374 may include one or more temperature sensors configured to detect a change in temperature of the input unit 202's environment and / or measure the temperature of the input unit 202's environment. In other embodiments, the environmental sensors 374 may include one or more barometers configured to detect a change in air pressure in the input unit 202's environment and / or measure the air pressure in the input unit 202's environment. In other embodiments, environmental sensor 374 may include one or more light sensors configured to detect changes in ambient light in the environment of input unit 202. Consistent with this disclosure, processing device 360 ​​may modify the presentation of virtual content based on input from environmental sensor 374. For example, automatically reducing the brightness of virtual content when the environment of user 100 becomes dark.

[0066] The other sensors 375 may include weight sensors, optical sensors, resistive sensors, ultrasonic sensors, proximity sensors, biometric sensors, or other sensing devices to facilitate related functions. In particular embodiments, the other sensors 375 may include one or more positioning sensors configured to obtain positioning information for the input unit 202, detect changes in the position of the input unit 202, and / or measure the position of the input unit 202. Alternatively, the GPS software may enable the input unit 202 to access an external GPS receiver (e.g., connecting via a serial port or Bluetooth). Consistent with the present disclosure, the processing device 360 ​​may modify the presentation of virtual content based on input from the other sensors 375. For example, presenting personal information only after identifying the user 100 using data from a biometric sensor.

[0067] The components and arrangement shown in FIG. 3 are not intended to limit any embodiment. Numerous variations and / or modifications can be made to the illustrated configuration of input unit 202, as will be understood by those skilled in the art having the benefit of this disclosure. For example, not all components are essential for the operation of the input unit. Any component may be located in any suitable portion of the input unit, and components may be rearranged in various configurations while still providing the functionality of various embodiments. For example, some input units may not include all of the elements shown in input unit 202.

[0068] FIG. 4 is a block diagram illustrating an example configuration of the XR unit 204. It should be understood that FIG. 4 is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure. In the embodiment of FIG. 4, the XR unit 204 may directly or indirectly access a bus 400 (or other communication mechanism) that interconnects subsystems and components for transferring information within the XR unit 204. For example, the bus 400 may interconnect a memory interface 410, a network interface 420, an input interface 430, a power supply 440, an output interface 450, a processing device 460, a sensor interface 470, and a database 480.

[0069] The memory interface 410 shown in FIG. 4 is assumed to have functionality similar to that of the memory interface 310 described in detail above. The memory interface 410 can be used to access software products and / or data stored on a non-transitory computer-readable medium or memory device, such as the memory device 411. The memory device 411 can include software modules for performing processes consistent with the present disclosure. In particular, the memory device 411 can include an input determination module 412, an output determination module 413, a sensor communication module 414, a virtual content determination module 415, a virtual content communication module 416, and a database access module 417. The modules 412-417 can include software instructions for execution by at least one processor (e.g., processing device 460) associated with the XR unit 204. The input determination module 412, the output determination module 413, the sensor communication module 414, the virtual content determination module 415, the virtual content communication module 416, and the database access module 417 can cooperate to perform various operations. For example, the input determination module 412 may determine user interface (UI) input received from the input unit 202. At the same time, the sensor communication module 414 may receive data from different sensors to determine the state of the user 100. The virtual content determination module 415 may determine virtual content to display based on the received input and the determined state of the user 100. The virtual content communication module 416 may search for virtual content not determined by the virtual content determination module 415. The search for virtual content may be from the database 380, the database 480, the mobile communication device 206, or the remote processing unit 208. Based on the output of the virtual content determination module 415, the output determination module 413 may cause a change in the virtual content displayed to the user 100 by the projector 454.

[0070] In some embodiments, the input determination module 412 can coordinate operation of the input interface 430 to receive gesture input 431, virtual input 432, voice input 433, and UI input 434. Consistent with the present disclosure, the input determination module 412 can simultaneously receive different types of input data. In one embodiment, the input determination module 412 can apply different rules based on the type of input detected. For example, gesture input may be prioritized over virtual input. In some embodiments, the output determination module 413 can coordinate operation of the output interface 450 to generate output using a light indicator 451, a display 452, a speaker 453, and a projector 454. In one embodiment, the light indicator 451 can include a light indicator that indicates the status of the wearable extended reality device. For example, the light indicator may display a green light when the wearable extended reality device 110 is connected to the input unit 202 and flash when the battery of the wearable extended reality device 110 is low. In another embodiment, the display 452 can be used to display operational information. In another embodiment, the speaker 453 may include bone conduction headphones used to output audio to the user 100. In another embodiment, the projector 454 may present virtual content to the user 100.

[0071] The operation of the sensor communication module, virtual content determination module, virtual content communication module, and database access module has been described above with reference to Figure 3, and the details thereof will not be repeated here. Modules 412-417 may be implemented in software, hardware, firmware, any combination thereof, etc.

[0072] 4 is assumed to have functionality similar to that of network interface 320 described in detail above. The specific design and implementation of network interface 420 may depend on the communication network in which XR unit 204 is intended to operate. For example, in some embodiments, XR unit 204 is configured to be selectively connectable by a wire to input unit 202. When connected by a wire, network interface 420 may enable communication with input unit 202, and when not connected by a wire, network interface 420 may enable communication with mobile communication device 206.

[0073] 4 is assumed to have functionality similar to that of the input interface 330 described in detail above. In this case, the input interface 430 may communicate with an image sensor to obtain gesture input 431 (e.g., a finger of the user 100 pointing at a virtual object), with other XR units 204 to obtain virtual input 432 (e.g., a virtual object shared with the XR unit 204 or a gesture of an avatar detected in the virtual environment), with a microphone to obtain voice input 433 (e.g., a voice command), and with the input unit 202 to obtain UI input 434 (e.g., virtual content determined by the virtual content determination module 315).

[0074] 4 is assumed to have functionality similar to that of the power source 340 described above, providing electrical energy to power the XR unit 204. In some embodiments, the power source 440 may be charged by the power source 340. For example, the power source 440 may be changed wirelessly when the XR unit 204 is placed on or near the input unit 202.

[0075] The output interface 450 shown in FIG. 4 is assumed to have functionality similar to that of the output interface 350 described in detail above. In this case, the output interface 450 can produce output from a light indicator 451, a display 452, a speaker 453, and a projector 454. The projector 454 can be any device, apparatus, equipment, etc. that can project (or direct) light to display virtual content onto a surface. The surface can be part of the XR unit 204, part of the eye of the user 100, or part of an object in proximity to the user 100. In one embodiment, the projector 454 can include an illumination unit that focuses light into a limited solid angle by one or more mirrors and lenses to provide a high value of luminous intensity in a defined direction.

[0076] 4 is assumed to have functionality similar to that of processing device 360 ​​described in detail above. When XR unit 204 is connected to input unit 202, processing device 460 may cooperate with processing device 360. In particular, processing device 460 may implement virtual machine technology or other technologies to provide the capability to run, control, execute, manipulate, store, etc., multiple software processes, applications, programs, etc. It should be understood that other types of processor configurations can be implemented to provide the capabilities disclosed herein.

[0077] The sensor interface 470 shown in Figure 4 is assumed to have functionality similar to that of the sensor interface 370 described in detail above. Specifically, the sensor interface 470 can communicate with an audio sensor 471, an image sensor 472, a motion sensor 473, an environmental sensor 474, and other sensors 475. The operation of the audio, image, motion, environmental, and other sensors is described above with reference to Figure 3, and the details will not be repeated here. It is understood that other types and combinations of sensors may be used to provide the capabilities disclosed herein.

[0078] The components and arrangement illustrated in FIG. 4 are not intended to limit any embodiment. As will be understood by one of ordinary skill in the art having the benefit of this disclosure, numerous variations and / or modifications may be made to the illustrated configuration of the XR unit 204. For example, not all components are essential to the operation of the XR unit 204 in all cases. Any component may be located in any suitable part of the system 200, and components may be rearranged in various configurations while still providing the functionality of various embodiments. For example, some XR units may not include all elements in the XR unit 204 (e.g., the wearable extended reality device 110 may not have a light indicator 451).

[0079] FIG. 5 is a block diagram illustrating an example configuration of a remote processing unit 208. It should be understood that FIG. 5 is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure. In the embodiment of FIG. 5, the remote processing unit 208 may include a server 210 that directly or indirectly accesses a bus 500 (or other communication mechanism) that interconnects subsystems and components for transferring information within the server 210. For example, the bus 500 may interconnect a memory interface 510, a network interface 520, a power supply 540, a processing device 560, and a database 580. The remote processing unit 208 may also include one or more data structures. For example, data structures 212A, 212B, and 212C.

[0080] The memory interface 510 shown in FIG. 5 is assumed to have functionality similar to that of the memory interface 310 described in detail above. The memory interface 510 can be used to access software products and / or data stored on a non-transitory computer-readable medium or in other memory devices, such as memory devices 311, 411, and 511, or data structures 212A, 212B, and 212C. The memory device 511 can include software modules for performing processes consistent with the present disclosure. In particular, the memory device 511 can include a shared memory module 512, a node registration module 513, a load balancing module 514, one or more compute nodes 515, an internal communications module 516, an external communications module 517, and a database access module (not shown). The modules 512-517 can include software instructions for execution by at least one processor (e.g., processing device 560) associated with the remote processing unit 208. The shared memory module 512, the node registration module 513, the load balancing module 514, the computation module 515, and the external communication module 517 may cooperate to perform various operations.

[0081] Shared memory module 512 may enable information sharing between remote processing unit 208 and other components of system 200. In some embodiments, shared memory module 512 may be configured to enable processing device 560 (and other processing devices in system 200) to access, retrieve, and store data. For example, using shared memory module 512, processing device 560 may perform at least one of the following steps: execute software programs stored in memory device 511, database 580, or data structures 212A-C; store information in memory device 511, database 580, or data structures 212A-C; or retrieve information from memory device 511, database 580, or data structures 212A-C.

[0082] The node registration module 513 can be configured to track the availability of one or more computing nodes 515. In some examples, the node registration module 513 may be implemented as a software program, such as a software program executed by one or more computing nodes 515, a hardware solution, or a combined software and hardware solution. In some implementations, the node registration module 513 can communicate with one or more computing nodes 515, for example, using the internal communication module 516. In some examples, one or more computing nodes 515 can inform the node registration module 513 of their status by sending a message, for example, upon startup, upon shutdown, at regular intervals, at selected times, in response to a query received from the node registration module 513, or at any other determined time. In some examples, the node registration module 513 can inquire about the status of one or more computing nodes 515 by sending a message, for example, upon startup, at regular intervals, at selected times, or at any other determined time.

[0083] The load balancing module 514 can be configured to divide the workload among one or more compute nodes 515. In some examples, the load balancing module 514 may be implemented as a software program, such as a software program executed by one or more of the compute nodes 515, a hardware solution, or a combined software and hardware solution. In some implementations, the load balancing module 514 can interact with the node registration module 513 to obtain information regarding the availability of one or more compute nodes 515. In some implementations, the load balancing module 514 can communicate with one or more compute nodes 515 using, for example, the internal communications module 516. In some examples, one or more compute nodes 515 can inform the load balancing module 514 of their status, for example, by sending a message upon startup, upon shutdown, at regular intervals, at selected times, in response to a query received from the load balancing module 514, or at any other determined time. In some examples, the load balancing module 514 can query the status of one or more compute nodes 515 by sending a message, for example, upon startup, at regular intervals, at preselected times, or at any other determined time.

[0084] Internal communications module 516 may be configured to receive and / or transmit information from one or more components of remote processing unit 208. For example, control and / or synchronization signals may be transmitted and / or received via internal communications module 516. In one embodiment, computer program input information, computer program output information, and / or computer program intermediate information may be transmitted and / or received via internal communications module 516. In another embodiment, information received via internal communications module 516 may be stored in memory device 511, database 580, data structures 212A-C, or other memory devices within system 200. For example, information retrieved from data structure 212A may be transmitted using internal communications module 516. In another example, input data may be received using internal communications module 516 and stored in data structure 212B.

[0085] The external communication module 517 may be configured to receive and / or transmit information from one or more components of the system 200. For example, control signals may be transmitted and / or received via the external communication module 517. In one embodiment, information received via the external communication module 517 may be stored in the memory device 511, the database 580, the data structures 212A-C, and / or any memory device within the system 200. In another embodiment, information retrieved from any of the data structures 212A-C may be transmitted to the XR unit 204 using the external communication module 517. In another embodiment, input data may be transmitted and / or received using the external communication module 517. Examples of such input data may include data received from the input unit 202, information captured from the environment of the user 100 using one or more sensors (e.g., the audio sensor 471, the image sensor 472, the motion sensor 473, the environmental sensor 474, the other sensors 475), etc.

[0086] In some embodiments, aspects of modules 512-517 may be implemented in hardware, software (including one or more signal processing and / or application specific integrated circuits), firmware, or any combination thereof, executable by one or more processors, alone or in various combinations with each other. In particular, modules 512-517 may be configured to interact with each other and / or other modules of system 200 to perform functions consistent with embodiments of the present disclosure. Memory device 511 may include additional or fewer modules and instructions.

[0087] 5 are assumed to have functionality similar to that of similar elements described above with reference to FIGS. 4 and 5. The specific design and implementation of the above-described components may vary based on the implementation of system 200. Additionally, remote processing unit 208 may include more or fewer components. For example, remote processing unit 208 may include an input interface configured to receive input directly from one or more input devices.

[0088] Consistent with the present disclosure, a processing device of system 200 (e.g., a processor in mobile communication device 206, a processor in server 210, a processor in a wearable extended reality device such as wearable extended reality device 110, and / or a processor in an input device associated with wearable extended reality device 110 such as keyboard 104) can use a machine learning algorithm to perform any of the methods disclosed herein. In some embodiments, the machine learning algorithm (also referred to in this disclosure as a machine learning model) can be trained using training examples, for example, as described below. Some non-limiting examples of such machine learning algorithms may include classification algorithms, data regression algorithms, image segmentation algorithms, visual detection algorithms (such as object detectors, face detectors, person detectors, motion detectors, edge detectors, etc.), visual recognition algorithms (such as face recognition, person recognition, object recognition, etc.), speech recognition algorithms, mathematical embedding algorithms, natural language processing algorithms, support vector machines, random forests, nearest neighbor algorithms, deep learning algorithms, artificial neural network algorithms, convolutional neural network algorithms, recurrent neural network algorithms, linear machine learning models, nonlinear machine learning models, ensemble algorithms, etc. For example, the trained machine learning algorithms may include predictive models, classification models, data regression models, clustering models, segmentation models, inference models such as artificial neural networks (e.g., deep neural networks, convolutional neural networks, recurrent neural networks, etc.), random forests, support vector machines, etc. In some examples, the training examples may include example inputs and desired outputs corresponding to the example inputs. Furthermore, in some examples, a machine learning algorithm trained using training examples can generate a trained machine learning algorithm, which can be used to estimate outputs for inputs not included in the training examples.In some examples, engineers, scientists, processes, and machines that train machine learning algorithms may further use validation examples and / or test examples. For example, the validation examples and / or test examples may include example inputs along with desired outputs corresponding to the example inputs, and the trained machine learning algorithm and / or an intermediately trained machine learning algorithm may be used to estimate outputs for the example inputs of the validation examples and / or test examples, and the estimated outputs may be compared with the corresponding desired outputs, and the trained machine learning algorithm and / or an intermediately trained machine learning algorithm may be evaluated based on the results of the comparison. In some examples, the machine learning algorithm may have parameters and hyperparameters, and the hyperparameters may be set manually by a person or automatically by a process external to the machine learning algorithm (such as a hyperparameter search algorithm), and the parameters of the machine learning algorithm may be set by the machine learning algorithm based on the training examples. In some implementations, the hyperparameters may be set based on the training examples and validation examples, and the parameters may be set based on the training examples and selected hyperparameters. For example, given the hyperparameters, the parameters may be conditionally independent of the validation examples.

[0089] In some embodiments, a trained machine learning algorithm (also referred to in this disclosure as a machine learning model and a trained machine learning model) can be used to analyze inputs and generate outputs, for example, as described below. In some examples, a trained machine learning algorithm can be used as an inference model that generates an inferred output when provided with an input. For example, the trained machine learning algorithm can include a classification algorithm, where the input can include samples and the inferred output can include a classification of the samples (e.g., an inferred label, an inferred tag, etc.). In another example, the trained machine learning algorithm can include a regression model, where the input can include samples and the inferred output can include an inferred value corresponding to the samples. In yet another example, the trained machine learning algorithm can include a clustering model, where the input can include samples and the inferred output can include an assignment of the samples to at least one cluster. In an additional example, the trained machine learning algorithm can include a classification algorithm, where the input can include images and the inferred output can include a classification of an item depicted in the images. In yet another example, the trained machine learning algorithm may include a regression model, the input may include an image, and the inferred output may include an inferred value corresponding to an item depicted in the image (e.g., an estimated characteristic of the item, such as the size, volume, age, or distance of a person depicted in the image from the item depicted in the image). In an additional example, the trained machine learning algorithm may include an image segmentation model, the input may include an image, and the inferred output may include a segmentation of the image. In yet another example, the trained machine learning algorithm may include an object detector, the input may include an image, and the inferred output may include one or more detected objects in the image and / or one or more locations of the objects in the image.In some examples, the trained machine learning algorithm may include one or more formulas and / or one or more functions and / or one or more rules and / or one or more procedures, the inputs may be used as inputs to the formulas and / or functions and / or rules and / or procedures, and the inferred output may be based on the output of the formulas and / or functions and / or rules and / or procedures (e.g., selecting one of the outputs of the formulas and / or functions and / or rules and / or procedures, using statistical measures of the outputs of the formulas and / or functions and / or rules and / or procedures, etc.).

[0090] Consistent with the present disclosure, the processing device of system 200 can analyze image data captured by an image sensor (e.g., image sensor 372, image sensor 472, or any other image sensor) to perform any of the methods disclosed herein. In some embodiments, analyzing the image data can include analyzing the image data to obtain preprocessed image data and then analyzing the image data and / or the preprocessed image data to obtain a desired result. Those skilled in the art will recognize that the following are examples and that the image data can be preprocessed using other types of preprocessing methods. In some examples, the image data can be preprocessed by transforming the image data using a transformation function to obtain transformed image data, where the preprocessed image data can include the transformed image data. For example, the transformed image data can include one or more convolutions of the image data. For example, the transformation function can include one or more image filters, such as a low-pass filter, a high-pass filter, a band-pass filter, an all-pass filter, etc. In some examples, the transformation function can include a nonlinear function. In some examples, the image data can be preprocessed by smoothing at least a portion of the image data, for example, using a Gaussian convolution, using a median filter, etc. In some examples, the image data can be preprocessed to obtain a different representation of the image data. For example, the preprocessed image data may include a representation of at least a portion of the image data in the frequency domain, a discrete Fourier transform of at least a portion of the image data, a discrete wavelet transform of at least a portion of the image data, a time / frequency representation of at least a portion of the image data, a reduced-dimensional representation of at least a portion of the image data, a lossy representation of at least a portion of the image data, a lossless representation of at least a portion of the image data, a time-ordered sequence of any of the above, any combination of the above, etc. In some examples, the image data may be preprocessed to extract edges, and the preprocessed image data may include information based on and / or related to the extracted edges. In some examples, the image data may be preprocessed to extract image features from the image data.Some non-limiting examples of such image features may include information based on and / or related to edges, corners, blobs, ridges, Scale Invariant Feature Transform (SIFT) features, temporal features, etc. In some examples, analyzing the image data may include computing at least one convolution of at least a portion of the image data and using the computed at least one convolution to compute at least one resultant value and / or to determine, identify, recognize, classify, etc.

[0091] Consistent with another aspect of the present disclosure, the processing device of system 200 can analyze image data to perform any of the methods disclosed herein. In some embodiments, analyzing the image can include analyzing the image data and / or preprocessed image data using one or more rules, functions, procedures, artificial neural networks, object detection algorithms, face detection algorithms, visual event detection algorithms, action detection algorithms, motion detection algorithms, background subtraction algorithms, inference models, etc. Some non-limiting examples of such inference models can include results on training examples of training algorithms, such as manually pre-programmed inference models, classification models, regression models, machine learning algorithms, and / or deep learning algorithms, where the training examples can include example data instances, and in some cases, the data instances may be labeled with corresponding desired labels and / or results, etc. In some embodiments, analyzing the image data (e.g., by methods, steps, and modules described herein) can include analyzing pixels, voxels, point clouds, range data, etc. included in the image data.

[0092] Convolution may include convolution of any dimension. A one-dimensional convolution is a function that transforms an original sequence into a transformed sequence. A one-dimensional convolution may be defined by a series of scalars. Each particular value in the transformed sequence may be determined by calculating a linear combination of values ​​in a subsequence of the original sequence that corresponds to a particular value. The resulting value of the calculated convolution can include any value in the transformed sequence. Similarly, an n-dimensional convolution is a function that transforms an original n-dimensional array into a transformed array. An n-dimensional convolution may be defined by an n-dimensional array of scalars (known as the kernel of the n-dimensional convolution). Each particular value in the transformed array may be determined by calculating a linear combination of values ​​in an n-dimensional region of the original array that corresponds to a particular value. The resulting value of the calculated convolution can include any value in the transformed array. In some examples, an image may include one or more components (e.g., color components, depth components, etc.), and each component may include a two-dimensional array of pixel values. In one example, calculating the convolution of an image may include calculating a two-dimensional convolution for one or more components of the image. In another example, computing the convolution of an image can include stacking arrays from different components to create a three-dimensional array and computing the three-dimensional convolution on the resulting three-dimensional array. In some examples, a video can include one or more components (e.g., color components, depth components, etc.), where each component can include a three-dimensional array of pixel values ​​(having two spatial axes and one temporal axis). In one example, computing the convolution of a video can include computing the three-dimensional convolution on one or more components of the video. In another example, computing the convolution of a video can include stacking arrays from different components to create a four-dimensional array and computing the four-dimensional convolution on the resulting four-dimensional array.

[0093] The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numerals are used in the drawings and the following description to refer to the same or similar parts. While several exemplary embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, substitutions, additions, or modifications may be made to the components illustrated in the drawings, and the exemplary methods described herein may be modified by substituting, rearranging, deleting, or adding steps to the disclosed methods. Therefore, the following detailed description is not limited to the disclosed embodiments and examples, but includes the general principles described herein and illustrated in the drawings, in addition to the general principles encompassed by the appended claims.

[0094] This disclosure relates to systems and methods for providing users with extended reality environments. The term "extended reality environment," which may also be referred to as "extended reality space," "extended reality," or "augmented environment," refers to all types of real-virtual composite environments and human-machine interactions that are generated at least in part by computer technology. An extended reality environment may be a fully simulated virtual environment or a combination of real and virtual environments that a user can perceive from different perspectives. In some examples, a user can interact with elements of the extended reality environment. One non-limiting example of an extended reality environment may be a virtual reality environment, also known as "virtual reality" or "virtual environment." An immersive virtual reality environment may be a simulated non-physical environment that provides a user with the perception of being in a virtual environment. Another non-limiting example of an extended reality environment may be an augmented reality environment, also known as "augmented reality" or "augmented environment." An augmented reality environment may include a live direct or indirect view of a physical real-world environment augmented with virtual computer-generated perceptual information, such as virtual objects, with which the user can interact. Another non-limiting example of an extended reality environment is a mixed reality environment, also known as "mixed reality" or "composite environment." A mixed reality environment may be a hybrid of a physical, real-world environment and a virtual environment, where physical and virtual objects coexist and can interact in real time. In some examples, both augmented reality and mixed reality environments may include a combination of real and virtual worlds, real-time interaction, and precise 3D registration of virtual and real objects. In some examples, both augmented reality and mixed reality environments may include constructive overlaid sensory information that can be added to the physical environment. In other examples, both augmented reality and mixed reality environments may include disruptive virtual content that can mask at least a portion of the physical environment.

[0095] In some embodiments, systems and methods can provide an extended reality environment using an extended reality device. The term extended reality device can include any type of device or system that allows a user to perceive and / or interact with an extended reality environment. An extended reality device can allow a user to perceive and / or interact with the extended reality environment through one or more sensory modalities. Some non-limiting examples of such sensory modalities may include vision, hearing, touch, somatosensation, and olfaction. One example of an extended reality device is a virtual reality device that allows a user to perceive and / or interact with a virtual reality environment. Another example of an extended reality device is an augmented reality device that allows a user to perceive and / or interact with an augmented reality environment. Yet another example of an extended reality device is a mixed reality device that allows a user to perceive and / or interact with a mixed reality environment.

[0096] According to one aspect of the present disclosure, the extended reality device may be a wearable device such as a head-mounted device, e.g., smart glasses, smart contact lenses, a headset, or any other device worn by a human to present an extended reality to the human. Other extended reality devices may include a holographic projector, or any other device or system capable of providing augmented reality (AR), virtual reality (VR), mixed reality (MR), or any immersive experience. Typical components of a wearable extended reality device may include a stereoscopic head-mounted display, a stereoscopic head-mounted sound system, a head motion tracking sensor (e.g., a gyroscope, accelerometer, magnetometer, image sensor, structured light sensor, etc.), a head-mounted projector, an eye-tracking sensor, and at least one of the additional components described below. According to another aspect of the present disclosure, the extended reality device may be a non-wearable extended reality device. Specifically, a non-wearable extended reality device may include a multi-projection environment device. In some embodiments, the extended reality device may be configured to change the perspective of the field of view of the extended reality environment in response to user movement, particularly in response to movement of the user's head. In one example, a wearable extended reality device may change the field of view of the extended reality environment in response to a change in the user's head pose, such as by changing the spatial orientation without changing the user's spatial position in the extended reality environment. In another example, a non-wearable extended reality device may change the user's spatial position in the extended reality environment in response to a change in the user's position in the real world, such as by changing the user's spatial position in the extended reality environment without changing the orientation of the field of view relative to the spatial position.

[0097] According to some embodiments, the extended reality device may include a digital communications device configured to at least one of receive virtual content data configured to enable presentation of virtual content, transmit virtual content for sharing with at least one external device, receive contextual data from at least one external device, transmit contextual data to at least one external device, transmit usage data indicative of use of the extended reality device, and transmit data based on information captured using at least one sensor included in the extended reality device. In additional embodiments, the extended reality device may include a memory for storing at least one of virtual data configured to enable presentation of virtual content, contextual data, usage data indicative of use of the extended reality device, sensor data based on information captured using at least one sensor included in the wearable extended reality device, software instructions configured to cause a processing device to present the virtual content, software instructions configured to cause the processing device to collect and analyze the contextual data, software instructions configured to cause the processing device to collect and analyze the usage data, and software instructions configured to cause the processing device to collect and analyze the sensor data. In additional embodiments, the extended reality device may include a processing device configured to perform at least one of rendering virtual content, collecting and analyzing contextual data, collecting and analyzing usage data, and collecting and analyzing sensor data. In additional embodiments, the extended reality device may include one or more sensors.The one or more sensors may include one or more image sensors (e.g., configured to capture images and / or video of a user of the device or the user's environment), one or more motion sensors (e.g., accelerometers, gyroscopes, magnetometers, etc.), one or more positioning sensors (e.g., GPS, outdoor positioning sensors, indoor positioning sensors, etc.), one or more temperature sensors (e.g., configured to measure the temperature of at least a portion of the device and / or environment), one or more contact sensors, one or more proximity sensors (e.g., configured to detect whether the device is currently being worn), one or more electrical impedance sensors (e.g., configured to measure the user's electrical impedance), one or more eye-tracking sensors, such as a gaze detector, an optical tracker, an electric potential tracker (e.g., an electro-oculogram (EOG) sensor), a video-based gaze tracker, an infrared / near-infrared sensor, a passive optical sensor, or any other technology capable of determining where a human is looking or gazing, etc.

[0098] In some embodiments, the systems and methods may use input devices to interact with the extended reality device. The term input device may include any physical device configured to receive input from a user or the user's environment and provide data to a computing device. The data provided to the computing device may be in digital and / or analog format. In one embodiment, the input device may store input received from a user in a memory device accessible by a processing device, which may access the stored data for analysis. In another embodiment, the input device may provide data directly to the processing device, for example, via a bus or another communication system configured to transfer data from the input device to the processing device. In some examples, input received by the input device may include key presses, tactile input data, motion data, positional data, gesture-based input data, directional data, or any other data intended for use in computation. Some examples of input devices may include buttons, keys, keyboards, computer mice, touchpads, touchscreens, joysticks, or other mechanisms capable of receiving input. Another example of an input device may include an integrated computing interface device that includes at least one physical component for receiving input from a user. The integrated computing interface device may include at least a memory, a processing device, and at least one physical component for receiving input from a user. In one example, the integrated computing interface device may further include a digital network interface that enables digital communication with other computing devices. In one example, the integrated computing interface device may further include a physical component for outputting information to a user. In some examples, all components of the integrated computing interface device may be contained in a single housing, while in other examples, the components may be distributed across two or more housings.Some non-limiting examples of physical components for receiving input from a user that may be included in an integrated computing interface device may include at least one of buttons, keys, a keyboard, a touchpad, a touchscreen, a joystick, or any other mechanism or sensor capable of receiving computational information. Some non-limiting examples of physical components for outputting information to a user may include at least one of a light indicator (such as an LED indicator), a screen, a touchscreen, a buzzer, an audio speaker, or any other audio, video, or tactile device that provides a human-perceptible output.

[0099] In some embodiments, the image data may be captured using one or more image sensors. In some examples, the image sensors may be included in an extended reality device, a wearable device, a wearable extended reality device, an input device, the user's environment, etc. In some examples, the image data may be read from memory, received from an external device, generated (e.g., using a generative model), etc. Some non-limiting examples of image data may include images, grayscale images, color images, 2D images, 3D images, video, 2D video, 3D video, frames, images, data derived from other image data, etc. In some examples, the image data may be encoded in any analog or digital format. Some non-limiting examples of such formats may include raw formats, compressed formats, uncompressed formats, lossy formats, lossless formats, JPEG, GIF, PNG, TIFF, BMP, NTSC, PAL, SECAM, MPEG, MPEG-4 Part 14, MOV, WMV, FLV, AVI, AVCHD, WebM, MKV, etc.

[0100] In some embodiments, the extended reality device may receive a digital signal, for example, from an input device. The term digital signal may refer to a time-discrete series of digital values. The digital signal may represent, for example, sensor data, text data, audio data, video data, virtual data, or any other form of data that provides perceptible information. Consistent with the present disclosure, the digital signal may be configured to cause the extended reality device to present virtual content. In one embodiment, the virtual content may be presented in a selected orientation. In this embodiment, the digital signal may indicate the position and angle of a viewpoint in an environment, such as an extended reality environment. Specifically, the digital signal may include encoding of position and angle in six degrees of freedom (e.g., forward / backward, up / down, left / right, yaw, pitch, and roll). In another embodiment, the digital signal may include encoding of position as three-dimensional coordinates (e.g., x, y, and z) and encoding of angle as a vector resulting from the encoded position. Specifically, the digital signal may indicate the orientation and angle of the presented virtual content in absolute coordinates of the environment, e.g., by encoding the yaw, pitch, and roll of the virtual content relative to a standard default angle. In another embodiment, the digital signal may indicate the orientation and angle of the presented virtual content relative to the viewpoint of another object (e.g., a virtual object, a physical object, etc.), e.g., by encoding the yaw, pitch, and roll of the virtual content relative to a direction corresponding to the viewpoint or a direction corresponding to the other object. In another embodiment, such a digital signal may include one or more projections of the virtual content, e.g., in a presentation-ready format (e.g., an image, a video, etc.). For example, each such projection may correspond to a particular orientation or a particular angle.In another embodiment, the digital signal may include a representation of the virtual content, for example, by encoding the object in a three-dimensional array of voxels, a polygonal mesh, or any other format in which the virtual content can be presented.

[0101] In some embodiments, the digital signal may be configured to cause the extended reality device to present virtual content. The term virtual content may include any type of data representation that can be displayed to a user by the extended reality device. Virtual content may include virtual objects, inanimate virtual content, animate virtual content configured to change over time or in response to a trigger, virtual two-dimensional content, virtual three-dimensional content, virtual overlays on portions of or physical objects in a physical environment, virtual additions to a physical environment or object, virtual promotional content, virtual representations of physical objects, virtual representations of a physical environment, virtual documents, virtual characters or personas, virtual computer screens, virtual widgets, or any other format for virtually displaying information. Consistent with this disclosure, virtual content may include any visual presentation rendered by a computer or processing device. In one embodiment, virtual content may include virtual objects, which are visual presentations rendered by a computer within a limited area and configured to represent a particular type of object (e.g., an inanimate virtual object, an animate virtual object, virtual furniture, virtual decorative object, virtual widget, or other virtual representation, etc.). The rendered visual presentation may change to reflect changes to a state object or a change in the viewing angle of the object, for example, to mimic a change in the appearance of a physical object. In another embodiment, the virtual content may include a virtual display (also referred to herein as a "virtual display screen" or "virtual screen"), such as a virtual computer screen, a virtual tablet screen, or a virtual smartphone screen, configured to display information generated by an operating system, where the operating system may be configured to receive text data from a physical keyboard and / or a virtual keyboard and cause the display of the text content on the virtual display screen. In one example, as shown in FIG. 1 , the virtual content may include a virtual environment including a virtual computer screen and a plurality of virtual objects.In some examples, the virtual display may be a virtual object that mimics and / or extends the functionality of a physical display screen. For example, the virtual display may be presented in an extended reality environment (e.g., a mixed reality environment, an augmented reality environment, a virtual reality environment, etc.) using extended reality equipment. In one example, the virtual display may present content generated by a conventional operating system that could equally be presented on a physical display screen. In one example, text content entered using a keyboard (e.g., using a physical keyboard, using a virtual keyboard, etc.) may be presented on the virtual display in real time as the text content is typed. In one example, a virtual cursor may be presented on the virtual display, and the virtual cursor may be controlled by a pointing device (e.g., a physical pointing device, a virtual pointing device, a computer mouse, a joystick, a touchpad, a physical touch controller, etc.). In one example, one or more windows of a graphical user interface operating system may be presented on the virtual display. In another example, the content presented on the virtual display may be interactive, i.e., it may change in response to user actions. In yet another example, the presentation of the virtual display may or may not include the presentation of a screen frame.

[0102] Some disclosed embodiments may include and / or access a data structure or database. Consistent with this disclosure, the terms data structure and database may include any collection of data values ​​and the relationships between them. Data may be stored linearly, horizontally, hierarchically, relationally, non-relationally, unidimensionally, multidimensionally, operationally, orderedly, unorderedly, object-orientedly, centralizedly, decentralizedly, distributedly, customly, or in any manner that enables data access. By way of non-limiting example, data structures may include arrays, associative arrays, linked lists, binary trees, balanced trees, mountains, stacks, queues, sets, hash tables, records, tagged unions, entity-relationship models, graphs, hypergraphs, matrices, tensors, etc. For example, the data structure may include an XML database, an RDBMS database, an SQL database, or a NoSQL alternative for data storage / retrieval, such as, for example, MongoDB, Redis, Couchbase, Datastax Enterprise Graph, Elastic Search, Splunk, Solr, Cassandra, Amazon DynamoDB, Scylla, HBase, and Neo4J. The data structure may be a component of the disclosed system or a remote computing component (e.g., a cloud-based data structure). Data in the data structure may be stored in contiguous or discontiguous memory. Furthermore, the data structure does not require the information to be co-located. It may be distributed, for example, across multiple servers that may be owned or operated by the same or different entities. Thus, the singular term "data structure" includes multiple data structures.

[0103] In some embodiments, the system can determine a confidence level of a received input or any determined value. The term confidence level refers to a numerical value or any other indication of a level (e.g., within a predetermined range) that indicates the amount of confidence the system has in the determined data. For example, the confidence level may have a value between 1 and 10. Alternatively, the confidence level may be expressed as a percentage or any other numerical or non-numerical indication. In some cases, the system can compare the confidence level to a threshold. The term threshold can refer to a reference value, level, point, or range of values. During operation, if the determined confidence level of the data exceeds (or is below, depending on the particular use case), the system can take a first course of action; if the confidence level is below (or above, depending on the particular use case), the system can take a second course of action. The value of the threshold may be predetermined for each type of object being investigated or may be dynamically selected based on different considerations.

[0104] Referring now to FIG. 1 , a user using an exemplary extended reality system consistent with embodiments of the present disclosure is shown. It should be understood that FIG. 1 is merely a representative representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure. As shown, a user 100 is seated behind a table 102 and supports a keyboard 104 and a mouse 106. The keyboard 104 is connected by wires 108 to a wearable extended reality device 110 that displays virtual content to the user 100. Alternatively, or in addition to the wires 108, the keyboard 104 may be wirelessly connected to the wearable extended reality device 110. For purposes of illustration, the wearable extended reality device is shown as a pair of smart glasses, but as noted above, the wearable extended reality device 110 may be any type of head-mounted device used to present extended reality to the user 100. The virtual content displayed by the wearable extended reality device 110 includes a virtual screen 112 (also referred to herein as a “virtual display screen” or “virtual display”) and multiple virtual widgets 114. Virtual widgets 114A-114D are displayed adjacent to the virtual screen 112, and virtual widget 114E is displayed on the table 102. The user 100 can use the keyboard 104 to enter text into a document 116 displayed on the virtual screen 112 and can use the mouse 106 to control a virtual cursor 118. In one example, the virtual cursor 118 can move anywhere within the virtual screen 112. In another example, the virtual cursor 118 can move anywhere within the virtual screen 112 and can move to any of the virtual widgets 114A-114D, but not to the virtual widget 114E. In yet another example, the virtual cursor 118 can move anywhere within the virtual screen 112 and can move to any of the virtual widgets 114A-114E. In a further example, the virtual cursor 118 can be moved anywhere in the extended reality environment, including the virtual screen 112 and the virtual widgets 114A-114E.In yet another example, the virtual cursor may move over all available surfaces (i.e., virtual or physical surfaces) or only over selected surfaces in the extended reality environment. Alternatively or additionally, user 100 may interact with any of virtual widgets 114A-114E, or a selected virtual widget, using hand gestures recognized by wearable extended reality device 110. For example, virtual widget 114E may be an interactive widget (e.g., a virtual slider controller) that can be manipulated with hand gestures.

[0105] FIG. 2 illustrates an example of a system 200 for providing an extended reality (XR) experience to a user, such as user 100. It should be understood that FIG. 2 is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure. System 200 may be computer-based and may include computer system components, wearable devices, workstations, tablets, handheld computing devices, memory devices, and / or an internal network connecting the components. System 200 may include or be connected to various network computing resources (e.g., servers, routers, switches, network connections, storage devices, etc.) to support the services provided by system 200. Consistent with the present disclosure, system 200 may include an input unit 202, an XR unit 204, a mobile communication device 206, and a remote processing unit 208. Remote processing unit 208 may include a server 210 coupled to one or more physical or virtual storage devices, such as a data structure 212. The system 200 may also include or be connected to a communication network 214 that facilitates communication and data exchange between different system components and different entities associated with the system 200 .

[0106] Consistent with the present disclosure, the input unit 202 may include one or more devices capable of receiving input from the user 100. In one embodiment, the input unit 202 may include a text input device such as the keyboard 104. The text input device may include all possible types of devices and mechanisms for inputting text information into the system 200. Examples of text input devices may include a mechanical keyboard, a membrane keyboard, a flexible keyboard, a QWERTY keyboard, a Dvorak keyboard, a Colemak keyboard, a corded keyboard, a wireless keyboard, a keypad, a key-based control panel, or other arrangement of control keys, a visual input device, or any other mechanism for inputting text (whether the mechanism is provided in physical form or presented virtually). In one embodiment, the input unit 202 may also include a pointing input device such as the mouse 106. The pointing input device may include all possible types of devices and mechanisms for inputting two-dimensional or three-dimensional information into the system 200. In one example, two-dimensional input from the pointing input device may be used to interact with virtual content presented via the XR unit 204. Examples of pointing input devices include a computer mouse, trackball, touchpad, trackpad, touchscreen, joystick, pointing stick, stylus, light pen, or any other physical or virtual input mechanism. In one embodiment, the input unit 202 may also include a graphical input device, such as a touchscreen, configured to detect contact, movement, or cessation of movement. The graphical input device may use any of a number of touch-sensitivity technologies, including, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact. In one embodiment, the input unit 202 may also include one or more audio input devices, such as a microphone.The audio input device may include all possible types of devices and mechanisms for inputting audio data to facilitate voice-enabled functions such as voice recognition, audio replication, digital recording, and telephone functions. In one embodiment, the input unit 202 may also include one or more image input devices, such as an image sensor, configured to capture image data. In one embodiment, the input unit 202 may also include one or more haptic gloves configured to capture hand movement and posture data. In one embodiment, the input unit 202 may also include one or more proximity sensors configured to detect the presence and / or movement of objects within a selected area near the sensor.

[0107] According to some embodiments, the system may include at least one sensor configured to detect and / or measure characteristics associated with a user, the user's actions, or the user's environment. One example of the at least one sensor is the sensor 216 included in the input unit 202. The sensor 216 may be a motion sensor, a touch sensor, a light sensor, an infrared sensor, an audio sensor, an image sensor, a proximity sensor, a positioning sensor, a gyroscope, a temperature sensor, a biometric sensor, or any other sensing device for facilitating related functions. The sensor 216 may be integrated with, connected to, or separate from the input device. In one example, the mouse 106 may include a thermometer to determine the user's 100 body temperature. In another example, a positioning sensor may be integrated with the keyboard 104 to determine the user's 100 movement relative to the keyboard 104. Such a positioning sensor may be implemented using one of the following positioning technologies: Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), Galileo Global Navigation System, BeiDou Navigation System, other Global Navigation Satellite System (GNSS), Indian Regional Navigation Satellite System (IRNSS), Local Positioning System (LPS), Real-Time Location System (RTLS), Indoor Positioning System (IPS), Wi-Fi based positioning system, cellular triangulation, image-based positioning technology, indoor positioning technology, outdoor positioning technology, or any other positioning technology.

[0108] According to some embodiments, the system may include one or more sensors for identifying the position and / or movement of a physical device (e.g., a physical input device, a physical computing device, a keyboard 104, a mouse 106, a wearable extended reality device 110, etc.). The one or more sensors may be included in the physical device or may be external to the physical device. In some examples, an image sensor external to the physical device (e.g., an image sensor included in another physical device) may be used to capture image data of the physical device, and the image data may be analyzed to identify the position and / or movement of the physical device. For example, the image data may be analyzed using a visual object tracking algorithm to identify movement of the physical device, a visual object detection algorithm to identify the position of the physical device (e.g., relative to the image sensor, in a global coordinate system, etc.), etc. In some examples, an image sensor included in the physical device may be used to capture image data, and the image data may be analyzed to identify the position and / or movement of the physical device. For example, the image data may be analyzed using a visual odometry algorithm to identify the position of the physical device, an egomotion algorithm to identify movement of the physical device, etc. In some examples, a positioning sensor, such as an indoor positioning sensor or an outdoor positioning sensor, may be included in the physical device and used to determine the position of the physical device. In some examples, a motion sensor, such as an accelerometer or gyroscope, may be included in the physical device and used to determine the movement of the physical device. In some examples, a physical device, such as a keyboard or mouse, may be configured to be placed on a physical surface. Such a physical device may include an optical mouse sensor (also known as a non-mechanical tracking engine) pointed at the physical surface, and the output of the optical mouse sensor may be analyzed to determine the movement of the physical device relative to the physical surface.

[0109] Consistent with this disclosure, the XR unit 204 may include a wearable extended reality device configured to present virtual content to the user 100. An example of a wearable extended reality device is the wearable extended reality device 110. Further examples of wearable extended reality devices may include virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices, or any other device capable of generating extended reality content. Some non-limiting examples of such devices include Nreal Light, Magic Leap One, Varjo, Quest 1 / 2, Vive, etc. In some embodiments, the XR unit 204 may present virtual content to the user 100. In general, extended reality devices may include all real and virtual composite environments and human-machine interactions generated by computer technology and wearables. As discussed above, the term “extended reality” (XR) refers to a superset that includes the entire spectrum from “fully real” to “fully virtual.” It includes representative forms such as augmented reality (AR), mixed reality (MR), virtual reality (VR), and regions interpolated therebetween. Accordingly, it should be noted that the terms "XR device," "AR device," "VR device," and "MR device" can be used interchangeably herein and can refer to any device among the various devices listed above.

[0110] Consistent with the present disclosure, the system can exchange data with various communication devices associated with the user, such as a mobile communication device 206. The term “communication device” is intended to include all possible types of devices capable of exchanging data using a digital communication network, an analog communication network, or any other communication network configured to transmit data. In some examples, the communication device may include a smartphone, a tablet, a smartwatch, a personal digital assistant, a desktop computer, a laptop computer, an IoT device, a dedicated terminal, a wearable communication device, and any other device enabling data communication. In some cases, the mobile communication device 206 can supplement or replace the input unit 202. Specifically, the mobile communication device 206 may be associated with a physical touch controller that can function as a pointing input device. Furthermore, the mobile communication device 206 may also be used, for example, to implement a virtual keyboard and replace a text input device. For example, the user 100 may walk away from the table 102 to the break room with their smart glasses and receive an email requiring a quick response. In this case, the user may choose to use their smartwatch as an input device and type a response to the email while it is virtually presented by the smartglasses.

[0111] Consistent with the present disclosure, system embodiments may include the use of a cloud server. The term "cloud server" refers to a computer platform that provides services over a network, such as the Internet. In the exemplary embodiment shown in FIG. 2, the server 210 may use a virtual machine that may not correspond to individual pieces of hardware. For example, computing and / or storage capabilities may be implemented by allocating an appropriate portion of the desired computing / storage power from a scalable repository, such as a data center or a distributed computing environment. Specifically, in one embodiment, the remote processing unit 208 may be used in conjunction with the XR unit 204 to provide virtual content to the user 100. In one configuration, the server 210 may be a cloud server that functions as an operating system (OS) for the wearable extended reality device. In one example, the server 210 may implement the methods described herein using customized hardwired logic, one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), firmware, and / or program logic that, in combination with a computer system, makes the server 210 a dedicated machine.

[0112] In some embodiments, server 210 can access data structure 212 to, for example, determine virtual content for display to user 100. Data structure 212 can utilize volatile or non-volatile, magnetic, semiconductor, tape, optical, removable, non-removable, or other types of storage devices or tangible or non-transitory computer-readable media, or any medium or mechanism for storing information. Data structure 212 can be part of server 210, as shown, or can be separate from server 210. If data structure 212 is not part of server 210, server 210 can exchange data with data structure 212 via a communications link. Data structure 212 can include one or more memory devices that store data and instructions used to execute one or more features of the disclosed methods. In one embodiment, data structure 212 can include any of several suitable data structures, from a small data structure hosted on a workstation to a large data structure distributed across a data center. Data structure 212 can also include any combination of one or more data structures controlled by a memory controller device (e.g., a server) or software.

[0113] Consistent with this disclosure, a communications network may be any type of network (including infrastructure) that supports communications, exchanges information, and / or facilitates the exchange of information between components of a system. For example, communications network 214 in system 200 may include, for example, a telephone network, an extranet, an intranet, the Internet, satellite communications, offline communications, radio communications, transponder communications, a local area network (LAN), a wireless network (e.g., a Wi-Fi / 302.11 network), a wide area network (WAN), a virtual private network (VPN), a digital communications network, an analog communications network, or any other mechanism or combination of mechanisms that enables data transmission.

[0114] The components and configuration of system 200 shown in FIG. 2 are intended to be illustrative only and not limiting of the disclosed embodiments, as the system components used to implement the disclosed processes and features may vary.

[0115] FIG. 3 is a block diagram illustrating an example configuration of input unit 202. It should be understood that FIG. 3 is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure. In the embodiment of FIG. 3, input unit 202 may directly or indirectly access bus 300 (or other communication mechanism) that interconnects subsystems and components for transferring information within input unit 202. For example, bus 300 may interconnect memory interface 310, network interface 320, input interface 330, power supply 340, output interface 350, processing device 360, sensor interface 370, and database 380.

[0116] The memory interface 310 shown in FIG. 3 can be used to access software products and / or data stored on a non-transitory computer-readable medium. Generally, a non-transitory computer-readable storage medium refers to any type of physical memory capable of storing information or data readable by at least one processor. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, CD-ROMs, DVDs, flash drives, disks, any other optical data storage media, any physical media with a pattern of holes, PROMs, EPROMs, FLASH-EPROMs or any other flash memory, NVRAMs, caches, registers, any other memory chips or cartridges, and networked versions thereof. The terms "memory" and "computer-readable storage medium" can refer to multiple structures, such as multiple memories or computer-readable storage media located within an input unit or at a remote location. Furthermore, one or more computer-readable storage media can be utilized in performing a computer-implemented method. Thus, the term computer-readable storage medium should be understood to include tangible items and exclude carrier waves and transient signals. 3, memory interface 310 may be used to access software products and / or data stored on a memory device, such as memory device 311. Memory device 311 may include high-speed random access memory and / or non-volatile memory, such as one or more magnetic disk storage devices, one or more optical storage devices, and / or flash memory (e.g., NAND, NOR). Consistent with the present disclosure, components of memory device 311 may be distributed across two or more units and / or two or more memory devices of system 200.

[0117] The memory device 311 shown in FIG. 3 may include software modules for executing processes consistent with the present disclosure. In particular, the memory device 311 may include an input determination module 312, an output determination module 313, a sensor communication module 314, a virtual content determination module 315, a virtual content communication module 316, and a database access module 317. Modules 312-317 may include software instructions for execution by at least one processor (e.g., processing device 360) associated with the input unit 202. The input determination module 312, the output determination module 313, the sensor communication module 314, the virtual content determination module 315, the virtual content communication module 316, and the database access module 317 may cooperate to perform various operations. For example, the input determination module 312 may determine text using data received, for example, from the keyboard 104. The output determination module 313 may then cause the presentation of the recently entered text on a dedicated display 352, for example, physically or wirelessly coupled to the keyboard 104. In this way, as the user 100 types, they can see a preview of the typed text without having to constantly move their head up and down to look at the virtual screen 112. The sensor communication module 314 can receive data from different sensors to determine the state of the user 100. The virtual content determination module 315 can then determine virtual content to display based on the received input and the determined status of the user 100. For example, the determined virtual content may be a virtual presentation of recently entered text on a virtual screen located virtually adjacent to the keyboard 104. The virtual content communication module 316 may retrieve virtual content not determined by the virtual content determination module 315 (e.g., another user's avatar). The retrieval of virtual content may be from the database 380, from the remote processing unit 208, or from any other source.

[0118] In some embodiments, the input determination module 312 can adjust the operation of the input interface 330 to receive pointer input 331, text input 332, voice input 333, and XR-related input 334. Details of pointer input, text input, and voice input are described above. The term “XR-related input” can include any type of data that may cause a change in virtual content displayed to the user 100. In one embodiment, the XR-related input 334 may include image data of the user 100, the wearable extended reality device (e.g., a detected hand gesture of the user 100), or other data. In another embodiment, the XR-related input 334 may include wireless communication indicating the presence of another user in proximity to the user 100. Consistent with the present disclosure, the input determination module 312 can simultaneously receive different types of input data. The input determination module 312 can then further apply different rules based on the type of input detected. For example, pointer input may be prioritized over voice input.

[0119] In some embodiments, the output determination module 313 can coordinate operation of the output interface 350 to generate output using light indicators 351, display 352, and / or speaker 353. Generally, the output generated by the output determination module 313 does not include virtual content presented by the wearable extended reality device. Instead, the output generated by the output determination module 313 includes various outputs related to the operation of the input unit 202 and / or the operation of the XR unit 204. In one embodiment, the light indicator 351 can include a light indicator that indicates the status of the wearable extended reality device. For example, the light indicator can display a green light when the wearable extended reality device 110 is connected to the keyboard 104 and flash when the battery of the wearable extended reality device 110 is low. In another embodiment, the display 352 can be used to display operational information. For example, the display can present an error message when the wearable extended reality device is inoperable. In another embodiment, the speaker 353 may be used to output sound, for example, if the user 100 wants to play music for other users.

[0120] In some embodiments, the sensor communications module 314 can coordinate the operation of the sensor interface 370 to receive sensor data from one or more sensors integrated with or connected to the input device. The one or more sensors can include an audio sensor 371, an image sensor 372, a motion sensor 373, an environmental sensor 374 (e.g., a temperature sensor, an ambient light detector, etc.), and other sensors 375. In one embodiment, the data received from the sensor communications module 314 can be used to determine the physical orientation of the input device. The physical orientation of the input device can indicate the user's state and can be determined based on a combination of tilt, roll, and lateral motion. The physical orientation of the input device can then be used by the virtual content determination module 315 to modify display parameters of the virtual content to match the user's state (e.g., alert, sleepy, active, sitting, standing, leaning back, leaning forward, walking, moving, riding, etc.).

[0121] In some embodiments, the virtual content determination module 315 can determine virtual content to be displayed by the wearable extended reality device. The virtual content can be determined based on data from the input determination module 312, the sensor communication module 314, and other sources (e.g., database 380). In some embodiments, determining the virtual content can include determining the distance, size, and orientation of a virtual object. The location of the virtual object can be determined based on the type of virtual object. Specifically, with respect to the example shown in FIG. 1 , the virtual content determination module 315 can determine to place four virtual widgets 114A-114D on both sides of the virtual screen 112 and place the virtual widget 114E on the table 102 because the virtual widget 114E is a virtual controller (e.g., a volume bar). The location of the virtual object can be further determined based on a user's preferences. For example, for a left-handed user, the virtual content determination module 315 may determine to place the virtual volume bar to the left of the keyboard 104, and for a right-handed user, the virtual content determination module 315 may determine to place the virtual volume bar to the right of the keyboard 104.

[0122] In some embodiments, the virtual content communication module 316 may coordinate the operation of the network interface 320 to obtain data from one or more sources to be presented as virtual content to the user 100. The one or more sources may include other XR units 204, the user's mobile communication device 206, the remote processing unit 208, publicly available information, etc. In one embodiment, the virtual content communication module 316 can communicate with the mobile communication device 206 to provide a virtual representation of the mobile communication device 206. For example, the virtual representation may allow the user 100 to read messages and interact with applications installed on the mobile communication device 206. The virtual content communication module 316 can also coordinate the operation of the network interface 320 to share virtual content with other users. In one example, the virtual content communication module 316 can use data from the input determination module to identify triggers (e.g., the triggers may include a user gesture) and transfer content from the virtual display to a physical display (e.g., a TV) or a different user's virtual display.

[0123] In some embodiments, database access module 317 can cooperate with database 380 to retrieve stored data. The retrieved data can include, for example, privacy levels associated with different virtual objects, relationships between virtual and physical objects, user preferences, user past behavior, etc. As described above, virtual content determination module 315 can determine virtual content using data stored in database 380. Database 380 can include separate databases, including, for example, a vector database, a raster database, a tile database, a viewport database, and / or a user input database. Data stored in database 380 can be received from modules 314-317 or other components of system 200. Additionally, data stored in database 380 can be provided as input using data entry, data transfer, or data upload.

[0124] Modules 312-317 may be implemented in software, hardware, firmware, any combination thereof, etc. In some embodiments, any one or more of modules 312-317 and data associated with database 380 may be stored in XR unit 204, mobile communication device 206, or remote processing unit 208. A processing device of system 200 may be configured to execute instructions of modules 312-317. In some embodiments, aspects of modules 312-317 may be implemented in hardware, software (including one or more signal processing and / or application specific integrated circuits), firmware, or any combination thereof, executable by one or more processors, alone or in various combinations with each other. In particular, modules 312-317 may be configured to interact with each other and / or other modules of system 200 to perform functions consistent with disclosed embodiments. For example, input unit 202 may execute instructions including image processing algorithms on data from XR unit 204 to determine head movement of user 100. Additionally, throughout this specification, each function described with respect to input unit 202 or with respect to components of input unit 202 may correspond to a set of instructions for performing that function. These instructions need not be implemented as separate software programs, procedures, or modules. Memory device 311 may include additional or fewer modules and instructions. For example, memory device 311 may store an operating system such as ANDROID, iOS, UNIX, OSX, WINDOWS, DARWIN, RTXC, LINUX, or an embedded operating system such as VXWorkS. The operating system may include instructions for handling basic system services and performing hardware-dependent tasks.

[0125] The network interface 320 shown in FIG. 3 can provide bidirectional data communication to a network, such as the communications network 214. In one embodiment, the network interface 320 can include an Integrated Services Digital Network (ISDN) card, a cellular modem, a satellite modem, or a modem to provide a data communication connection over the Internet. As another example, the network interface 320 can include a wireless local area network (WLAN) card. In another embodiment, the network interface 320 can include an Ethernet port connected to a radio frequency receiver and transmitter and / or an optical (e.g., infrared) receiver and transmitter. The specific design and implementation of the network interface 320 can depend on the communications network over which the input unit 202 is intended to operate. For example, in some embodiments, the input unit 202 can include a network interface 320 designed to operate over a GSM network, a GPRS network, an EDGE network, a Wi-Fi or WiMax network, and a Bluetooth network. In any such implementation, the network interface 320 can be configured to send and receive electrical, electromagnetic, or optical signals that carry digital data streams or signals representing various types of information.

[0126] The input interface 330 shown in FIG. 3 can receive input from a variety of input devices, such as a keyboard, a mouse, a touchpad, a touchscreen, one or more buttons, a joystick, a microphone, an image sensor, and any other device configured to detect physical or virtual input. The received input may be in the form of at least one of text, voice, speech, hand gestures, body gestures, tactile information, and any other type of physical or virtual input generated by a user. In the illustrated embodiment, the input interface 330 can receive pointer input 331, text input 332, voice input 333, and XR-related input 334. In additional embodiments, the input interface 330 may be an integrated circuit that can function as a bridge between the processing device 360 ​​and any of the above input devices.

[0127] The power source 340 shown in FIG. 3 can provide electrical energy to the power input unit 202 and, optionally, to the XR unit 204. In general, the power source included in any device or system of the present disclosure may be any device capable of repeatedly storing, distributing, or transmitting electrical power, including, but not limited to, one or more batteries (e.g., lead-acid, lithium-ion, nickel-metal hydride, nickel-cadmium batteries), one or more capacitors, one or more connections to external power sources, one or more power converters, or any combination thereof. Referring to the example shown in FIG. 3 , the power source may be mobile, meaning that the input unit 202 can be easily carried by hand (e.g., the total weight of the power source 340 may be less than 1 pound). The mobility of the power source allows the user 100 to use the input unit 202 in a variety of situations. In other embodiments, the power source 340 may be associated with a connection to an external power source (such as a power grid) that can be used to charge the power source 340. Additionally, the power source 340 may be configured to charge one or more batteries included in the XR unit 204, such that a pair of extended reality glasses (e.g., wearable extended reality devices 110) may be charged (e.g., wirelessly or non-wirelessly) when they are placed on or near the input unit 202.

[0128] The output interface 350 shown in FIG. 3 can produce output from various output devices using, for example, light indicators 351, a display 352, and / or a speaker 353. In one embodiment, the output interface 350 can be an integrated circuit that can act as a bridge between the processing device 360 ​​and at least one of the above output devices. The light indicator 351 can include one or more light sources, such as, for example, an LED array associated with different colors. The display 352 can include a screen (e.g., an LCD or dot-matrix screen) or a touchscreen. The speaker 353 can include audio headphones, a hearing aid-type device, a speaker, bone conduction headphones, an interface that provides tactile cues, a vibration stimulation device, etc.

[0129] The processing device 360 ​​shown in FIG. 3 may include at least one processor configured to execute computer programs, applications, methods, processes, or other software to implement the embodiments described herein. Generally, the processing device included in any device or system of the present disclosure may include all or part of one or more integrated circuits, microchips, microcontrollers, microprocessors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), or other circuitry suitable for executing instructions or performing logical operations. The processing device may include at least one processor configured to perform the functions of the disclosed methods, such as a microprocessor manufactured by Intel™. The processing device may include a single-core or multiple-core processor that simultaneously performs parallel processing. In one example, the processing device may be a single-core processor configured with virtual processing technology. The processing device may implement virtual machine technology or other technologies to provide the ability to run, control, execute, manipulate, store, etc., multiple software processes, applications, programs, etc. In another example, the processing device may include a multi-core processor configuration (e.g., dual, quad-core, etc.) configured to provide parallel processing capabilities that allow devices associated with the processing device to execute multiple processes simultaneously. It should be understood that other types of processor configurations may be implemented to provide the functionality disclosed herein.

[0130] 3 can acquire sensor data from various sensors, such as an audio sensor 371, an image sensor 372, a motion sensor 373, an environmental sensor 374, and other sensors 375. In one embodiment, the sensor interface 370 can be an integrated circuit that can act as a bridge between the processing device 360 ​​and at least one of the above sensors.

[0131] Audio sensor 371 may include one or more audio sensors configured to capture audio by converting the audio into digital information. Some examples of audio sensors may include a microphone, a unidirectional microphone, a bidirectional microphone, a cardioid microphone, an omnidirectional microphone, an on-board microphone, a wired microphone, a wireless microphone, or any combination of the above. Consistent with the present disclosure, processing device 360 ​​can modify the presentation of virtual content based on data received from audio sensor 371 (e.g., voice commands).

[0132] Image sensor 372 may include one or more image sensors configured to capture visual information by converting light into image data. Consistent with this disclosure, the image sensor may be included in any device or system of this disclosure and may be any device capable of detecting and converting optical signals in the near-infrared, infrared, visible, and ultraviolet spectrums into electrical signals. Examples of image sensors may include digital cameras, phone cameras, semiconductor charge-coupled devices (CCDs), and complementary metal-oxide semiconductor (CMOS) or N-type metal-oxide semiconductor (NMOS, live MOS) active pixel sensors. The electrical signals may be used to generate image data. Consistent with this disclosure, image data may include pixel data streams, digital images, digital video streams, data derived from captured images, and data that may be used to construct one or more 3D images, a sequence of 3D images, a 3D video, or a virtual 3D representation. Image data acquired by image sensor 372 may be transmitted to any processing device of system 200 via wired or wireless transmission. For example, the image data may be processed for object detection, event detection, action detection, face detection, person detection, known person recognition, or any other information that may be used by system 200. Consistent with the present disclosure, processing device 360 ​​may modify the presentation of virtual content based on the image data received from image sensor 372.

[0133] The motion sensor 373 may include one or more motion sensors configured to measure movement of the input unit 202 or movement of objects in the environment of the input unit 202. Specifically, the motion sensor may perform at least one of: detecting movement of objects in the environment of the input unit 202, measuring the velocity of objects in the environment of the input unit 202, measuring the acceleration of objects in the environment of the input unit 202, detecting movement of the input unit 202, measuring the velocity of the input unit 202, measuring the acceleration of the input unit 202, etc. In some embodiments, the motion sensor 373 may include one or more accelerometers configured to detect changes in and / or measure the appropriate acceleration of the input unit 202. In other embodiments, the motion sensor 373 may include one or more gyroscopes configured to detect changes in the orientation of the input unit 202 and / or measure information related to the orientation of the input unit 202. In other embodiments, the motion sensor 373 may include one or more using image sensors, lidar sensors, radar sensors, or proximity sensors. For example, by analyzing the captured images, the processing device can determine the movement of the input unit 202, e.g., using an egomotion algorithm. Additionally, the processing device can determine the movement of objects in the environment of the input unit 202, e.g., using an object tracking algorithm. Consistent with this disclosure, the processing device 360 ​​can modify the presentation of virtual content based on the determined movement of the input unit 202 or the determined movement of objects in the environment of the input unit 202, e.g., by causing a virtual display to track the movement of the input unit 202.

[0134] The environmental sensors 374 may include one or more sensors of different types configured to capture data reflective of the input unit 202's environment. In some embodiments, the environmental sensors 374 may include one or more chemical sensors configured to measure at least one of: a chemical property in the input unit 202's environment; a change in a chemical property in the input unit 202's environment; a presence of a chemical in the input unit 202's environment; or a concentration of a chemical in the input unit 202's environment. Examples of such chemical properties may include pH levels, toxicity, and temperature. Examples of such chemicals may include electrolytes, certain enzymes, certain hormones, certain proteins, smoke, carbon dioxide, carbon monoxide, oxygen, ozone, hydrogen, and hydrogen sulfide. In other embodiments, the environmental sensors 374 may include one or more temperature sensors configured to detect a change in temperature of the input unit 202's environment and / or measure the temperature of the input unit 202's environment. In other embodiments, the environmental sensors 374 may include one or more barometers configured to detect a change in air pressure in the input unit 202's environment and / or measure the air pressure in the input unit 202's environment. In other embodiments, environmental sensor 374 may include one or more light sensors configured to detect changes in ambient light in the environment of input unit 202. Consistent with this disclosure, processing device 360 ​​may modify the presentation of virtual content based on input from environmental sensor 374. For example, automatically reducing the brightness of virtual content when the environment of user 100 becomes dark.

[0135] The other sensors 375 may include weight sensors, optical sensors, resistive sensors, ultrasonic sensors, proximity sensors, biometric sensors, or other sensing devices to facilitate related functions. In particular embodiments, the other sensors 375 may include one or more positioning sensors configured to obtain positioning information for the input unit 202, detect changes in the position of the input unit 202, and / or measure the position of the input unit 202. Alternatively, the GPS software may enable the input unit 202 to access an external GPS receiver (e.g., connecting via a serial port or Bluetooth). Consistent with the present disclosure, the processing device 360 ​​may modify the presentation of virtual content based on input from the other sensors 375. For example, presenting personal information only after identifying the user 100 using data from a biometric sensor.

[0136] The components and arrangement shown in FIG. 3 are not intended to limit the disclosed embodiments. Numerous variations and / or modifications can be made to the illustrated configuration of input unit 202, as will be understood by those skilled in the art having the benefit of this disclosure. For example, not all components are essential for the operation of the input unit. Any component can be located in any suitable portion of the input unit, and the components can be rearranged in various configurations while still providing the functionality of the disclosed embodiments. For example, some input units may not include all of the elements shown in input unit 202.

[0137] FIG. 4 is a block diagram illustrating an example configuration of the XR unit 204. It should be understood that FIG. 4 is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure. In the embodiment of FIG. 4, the XR unit 204 may directly or indirectly access a bus 400 (or other communication mechanism) that interconnects subsystems and components for transferring information within the XR unit 204. For example, the bus 400 may interconnect a memory interface 410, a network interface 420, an input interface 430, a power supply 440, an output interface 450, a processing device 460, a sensor interface 470, and a database 480.

[0138] The memory interface 410 shown in FIG. 4 is assumed to have functionality similar to that of the memory interface 310 described in detail above. The memory interface 410 can be used to access software products and / or data stored on a non-transitory computer-readable medium or memory device, such as the memory device 411. The memory device 411 can include software modules for performing processes consistent with the present disclosure. In particular, the memory device 411 can include an input determination module 412, an output determination module 413, a sensor communication module 414, a virtual content determination module 415, a virtual content communication module 416, and a database access module 417. The modules 412-417 can include software instructions for execution by at least one processor (e.g., processing device 460) associated with the XR unit 204. The input determination module 412, the output determination module 413, the sensor communication module 414, the virtual content determination module 415, the virtual content communication module 416, and the database access module 417 can cooperate to perform various operations. For example, the input determination module 412 may determine user interface (UI) input received from the input unit 202. At the same time, the sensor communication module 414 may receive data from different sensors to determine the state of the user 100. The virtual content determination module 415 may determine virtual content to display based on the received input and the determined state of the user 100. The virtual content communication module 416 may search for virtual content not determined by the virtual content determination module 415. The search for virtual content may be from the database 380, the database 480, the mobile communication device 206, or the remote processing unit 208. Based on the output of the virtual content determination module 415, the output determination module 413 may cause a change in the virtual content displayed to the user 100 by the projector 454.

[0139] In some embodiments, the input determination module 412 can coordinate operation of the input interface 430 to receive gesture input 431, virtual input 432, voice input 433, and UI input 434. Consistent with the present disclosure, the input determination module 412 can simultaneously receive different types of input data. In one embodiment, the input determination module 412 can apply different rules based on the type of input detected. For example, gesture input may be prioritized over virtual input. In some embodiments, the output determination module 413 can coordinate operation of the output interface 450 to generate output using a light indicator 451, a display 452, a speaker 453, and a projector 454. In one embodiment, the light indicator 451 can include a light indicator that indicates the status of the wearable extended reality device. For example, the light indicator may display a green light when the wearable extended reality device 110 is connected to the input unit 202 and flash when the battery of the wearable extended reality device 110 is low. In another embodiment, the display 452 can be used to display operational information. In another embodiment, the speaker 453 may include bone conduction headphones used to output audio to the user 100. In another embodiment, the projector 454 may present virtual content to the user 100.

[0140] The operation of the sensor communication module, virtual content determination module, virtual content communication module, and database access module has been described above with reference to Figure 3, and the details thereof will not be repeated here. Modules 412-417 may be implemented in software, hardware, firmware, any combination thereof, etc.

[0141] 4 is assumed to have functionality similar to that of network interface 320 described in detail above. The specific design and implementation of network interface 420 may depend on the communication network in which XR unit 204 is intended to operate. For example, in some embodiments, XR unit 204 is configured to be selectively connectable by a wire to input unit 202. When connected by a wire, network interface 420 may enable communication with input unit 202, and when not connected by a wire, network interface 420 may enable communication with mobile communication device 206.

[0142] 4 is assumed to have functionality similar to that of the input interface 330 described in detail above. In this case, the input interface 430 may communicate with an image sensor to obtain gesture input 431 (e.g., a finger of the user 100 pointing at a virtual object), with other XR units 204 to obtain virtual input 432 (e.g., a virtual object shared with the XR unit 204 or a gesture of an avatar detected in the virtual environment), with a microphone to obtain voice input 433 (e.g., a voice command), and with the input unit 202 to obtain UI input 434 (e.g., virtual content determined by the virtual content determination module 315).

[0143] 4 is assumed to have functionality similar to that of the power source 340 described above, providing electrical energy to power the XR unit 204. In some embodiments, the power source 440 may be charged by the power source 340. For example, the power source 440 may be changed wirelessly when the XR unit 204 is placed on or near the input unit 202.

[0144] The output interface 450 shown in FIG. 4 is assumed to have functionality similar to that of the output interface 350 described in detail above. In this case, the output interface 450 can produce output from a light indicator 451, a display 452, a speaker 453, and a projector 454. The projector 454 can be any device, apparatus, equipment, etc. that can project (or direct) light to display virtual content onto a surface. The surface can be part of the XR unit 204, part of the eye of the user 100, or part of an object in proximity to the user 100. In one embodiment, the projector 454 can include an illumination unit that focuses light into a limited solid angle by one or more mirrors and lenses to provide a high value of luminous intensity in a defined direction.

[0145] 4 is assumed to have functionality similar to that of processing device 360 ​​described in detail above. When XR unit 204 is connected to input unit 202, processing device 460 may cooperate with processing device 360. In particular, processing device 460 may implement virtual machine technology or other technologies to provide the capability to run, control, execute, manipulate, store, etc., multiple software processes, applications, programs, etc. It should be understood that other types of processor configurations can be implemented to provide the capabilities disclosed herein.

[0146] The sensor interface 470 shown in Figure 4 is assumed to have functionality similar to that of the sensor interface 370 described in detail above. Specifically, the sensor interface 470 can communicate with an audio sensor 471, an image sensor 472, a motion sensor 473, an environmental sensor 474, and other sensors 475. The operation of the audio, image, motion, environmental, and other sensors is described above with reference to Figure 3, and the details will not be repeated here. It is understood that other types and combinations of sensors may be used to provide the capabilities disclosed herein.

[0147] The components and arrangement illustrated in FIG. 4 are not intended to limit the disclosed embodiments. As will be understood by one of ordinary skill in the art having the benefit of this disclosure, numerous variations and / or modifications may be made to the illustrated configuration of the XR unit 204. For example, not all components are essential to the operation of the XR unit 204 in all cases. Any component may be located in any suitable part of the system 200, and components may be rearranged in various configurations while still providing the functionality of the disclosed embodiments. For example, some XR units may not include all elements in the XR unit 204 (e.g., the wearable extended reality device 110 may not have a light indicator 451).

[0148] FIG. 5 is a block diagram illustrating an example configuration of a remote processing unit 208. It should be understood that FIG. 5 is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure. In the embodiment of FIG. 5, the remote processing unit 208 may include a server 210 that directly or indirectly accesses a bus 500 (or other communication mechanism) that interconnects subsystems and components for transferring information within the server 210. For example, the bus 500 may interconnect a memory interface 510, a network interface 520, a power supply 540, a processing device 560, and a database 580. The remote processing unit 208 may also include one or more data structures. For example, data structures 212A, 212B, and 212C.

[0149] The memory interface 510 shown in FIG. 5 is assumed to have functionality similar to that of the memory interface 310 described in detail above. The memory interface 510 can be used to access software products and / or data stored on a non-transitory computer-readable medium or in other memory devices, such as memory devices 311, 411, and 511, or data structures 212A, 212B, and 212C. The memory device 511 can include software modules for performing processes consistent with the present disclosure. In particular, the memory device 511 can include a shared memory module 512, a node registration module 513, a load balancing module 514, one or more compute nodes 515, an internal communications module 516, an external communications module 517, and a database access module (not shown). The modules 512-517 can include software instructions for execution by at least one processor (e.g., processing device 560) associated with the remote processing unit 208. The shared memory module 512, the node registration module 513, the load balancing module 514, the computation module 515, and the external communication module 517 may cooperate to perform various operations.

[0150] Shared memory module 512 may enable information sharing between remote processing unit 208 and other components of system 200. In some embodiments, shared memory module 512 may be configured to enable processing device 560 (and other processing devices in system 200) to access, retrieve, and store data. For example, using shared memory module 512, processing device 560 may perform at least one of the following steps: execute software programs stored in memory device 511, database 580, or data structures 212A-C; store information in memory device 511, database 580, or data structures 212A-C; or retrieve information from memory device 511, database 580, or data structures 212A-C.

[0151] The node registration module 513 can be configured to track the availability of one or more computing nodes 515. In some examples, the node registration module 513 may be implemented as a software program, such as a software program executed by one or more computing nodes 515, a hardware solution, or a combined software and hardware solution. In some implementations, the node registration module 513 can communicate with one or more computing nodes 515, for example, using the internal communication module 516. In some examples, one or more computing nodes 515 can inform the node registration module 513 of their status by sending a message, for example, upon startup, upon shutdown, at regular intervals, at selected times, in response to a query received from the node registration module 513, or at any other determined time. In some examples, the node registration module 513 can inquire about the status of one or more computing nodes 515 by sending a message, for example, upon startup, at regular intervals, at selected times, or at any other determined time.

[0152] The load balancing module 514 can be configured to divide the workload among one or more compute nodes 515. In some examples, the load balancing module 514 may be implemented as a software program, such as a software program executed by one or more of the compute nodes 515, a hardware solution, or a combined software and hardware solution. In some implementations, the load balancing module 514 can interact with the node registration module 513 to obtain information regarding the availability of one or more compute nodes 515. In some implementations, the load balancing module 514 can communicate with one or more compute nodes 515 using, for example, the internal communications module 516. In some examples, one or more compute nodes 515 can inform the load balancing module 514 of their status, for example, by sending a message upon startup, upon shutdown, at regular intervals, at selected times, in response to a query received from the load balancing module 514, or at any other determined time. In some examples, the load balancing module 514 can query the status of one or more compute nodes 515 by sending a message, for example, upon startup, at regular intervals, at preselected times, or at any other determined time.

[0153] Internal communications module 516 may be configured to receive and / or transmit information from one or more components of remote processing unit 208. For example, control and / or synchronization signals may be transmitted and / or received via internal communications module 516. In one embodiment, computer program input information, computer program output information, and / or computer program intermediate information may be transmitted and / or received via internal communications module 516. In another embodiment, information received via internal communications module 516 may be stored in memory device 511, database 580, data structures 212A-C, or other memory devices within system 200. For example, information retrieved from data structure 212A may be transmitted using internal communications module 516. In another example, input data may be received using internal communications module 516 and stored in data structure 212B.

[0154] The external communication module 517 may be configured to receive and / or transmit information from one or more components of the system 200. For example, control signals may be transmitted and / or received via the external communication module 517. In one embodiment, information received via the external communication module 517 may be stored in the memory device 511, the database 580, the data structures 212A-C, and / or any memory device within the system 200. In another embodiment, information retrieved from any of the data structures 212A-C may be transmitted to the XR unit 204 using the external communication module 517. In another embodiment, input data may be transmitted and / or received using the external communication module 517. Examples of such input data may include data received from the input unit 202, information captured from the environment of the user 100 using one or more sensors (e.g., the audio sensor 471, the image sensor 472, the motion sensor 473, the environmental sensor 474, the other sensors 475), etc.

[0155] In some embodiments, aspects of modules 512-517 may be implemented in hardware, software (including one or more signal processing and / or application specific integrated circuits), firmware, or any combination thereof, executable by one or more processors, alone or in various combinations with each other. In particular, modules 512-517 may be configured to interact with each other and / or other modules of system 200 to perform functions consistent with disclosed embodiments. Memory device 511 may include additional or fewer modules and instructions.

[0156] 5 are assumed to have functionality similar to that of similar elements described above with reference to FIGS. 4 and 5. The specific design and implementation of the above-described components may vary based on the implementation of system 200. Additionally, remote processing unit 208 may include more or fewer components. For example, remote processing unit 208 may include an input interface configured to receive input directly from one or more input devices.

[0157] Consistent with the present disclosure, a processing device of system 200 (e.g., a processor in mobile communication device 206, a processor in server 210, a processor in a wearable extended reality device such as wearable extended reality device 110, and / or a processor in an input device associated with wearable extended reality device 110 such as keyboard 104) can use a machine learning algorithm to perform any of the methods disclosed herein. In some embodiments, the machine learning algorithm (also referred to in this disclosure as a machine learning model) can be trained using training examples, for example, as described below. Some non-limiting examples of such machine learning algorithms may include classification algorithms, data regression algorithms, image segmentation algorithms, visual detection algorithms (such as object detectors, face detectors, person detectors, motion detectors, edge detectors, etc.), visual recognition algorithms (such as face recognition, person recognition, object recognition, etc.), speech recognition algorithms, mathematical embedding algorithms, natural language processing algorithms, support vector machines, random forests, nearest neighbor algorithms, deep learning algorithms, artificial neural network algorithms, convolutional neural network algorithms, recurrent neural network algorithms, linear machine learning models, nonlinear machine learning models, ensemble algorithms, etc. For example, the trained machine learning algorithms may include predictive models, classification models, data regression models, clustering models, segmentation models, inference models such as artificial neural networks (e.g., deep neural networks, convolutional neural networks, recurrent neural networks, etc.), random forests, support vector machines, etc. In some examples, the training examples may include example inputs and desired outputs corresponding to the example inputs. Furthermore, in some examples, a machine learning algorithm trained using training examples can generate a trained machine learning algorithm, which can be used to estimate outputs for inputs not included in the training examples.In some examples, engineers, scientists, processes, and machines that train machine learning algorithms may further use validation examples and / or test examples. For example, the validation examples and / or test examples may include example inputs along with desired outputs corresponding to the example inputs, and the trained machine learning algorithm and / or an intermediately trained machine learning algorithm may be used to estimate outputs for the example inputs of the validation examples and / or test examples, and the estimated outputs may be compared with the corresponding desired outputs, and the trained machine learning algorithm and / or an intermediately trained machine learning algorithm may be evaluated based on the results of the comparison. In some examples, the machine learning algorithm may have parameters and hyperparameters, and the hyperparameters may be set manually by a person or automatically by a process external to the machine learning algorithm (such as a hyperparameter search algorithm), and the parameters of the machine learning algorithm may be set by the machine learning algorithm based on the training examples. In some implementations, the hyperparameters may be set based on the training examples and validation examples, and the parameters may be set based on the training examples and selected hyperparameters. For example, given the hyperparameters, the parameters may be conditionally independent of the validation examples.

[0158] In some embodiments, a trained machine learning algorithm (also referred to in this disclosure as a machine learning model and a trained machine learning model) can be used to analyze inputs and generate outputs, for example, as described below. In some examples, a trained machine learning algorithm can be used as an inference model that generates an inferred output when provided with an input. For example, the trained machine learning algorithm can include a classification algorithm, where the input can include samples and the inferred output can include a classification of the samples (e.g., an inferred label, an inferred tag, etc.). In another example, the trained machine learning algorithm can include a regression model, where the input can include samples and the inferred output can include an inferred value corresponding to the samples. In yet another example, the trained machine learning algorithm can include a clustering model, where the input can include samples and the inferred output can include an assignment of the samples to at least one cluster. In an additional example, the trained machine learning algorithm can include a classification algorithm, where the input can include images and the inferred output can include a classification of an item depicted in the images. In yet another example, the trained machine learning algorithm may include a regression model, the input may include an image, and the inferred output may include an inferred value corresponding to an item depicted in the image (e.g., an estimated characteristic of the item, such as the size, volume, age, or distance of a person depicted in the image from the item depicted in the image). In an additional example, the trained machine learning algorithm may include an image segmentation model, the input may include an image, and the inferred output may include a segmentation of the image. In yet another example, the trained machine learning algorithm may include an object detector, the input may include an image, and the inferred output may include one or more detected objects in the image and / or one or more locations of the objects in the image.In some examples, the trained machine learning algorithm may include one or more formulas and / or one or more functions and / or one or more rules and / or one or more procedures, the inputs may be used as inputs to the formulas and / or functions and / or rules and / or procedures, and the inferred output may be based on the output of the formulas and / or functions and / or rules and / or procedures (e.g., selecting one of the outputs of the formulas and / or functions and / or rules and / or procedures, using statistical measures of the outputs of the formulas and / or functions and / or rules and / or procedures, etc.).

[0159] Consistent with the present disclosure, the processing device of system 200 can analyze image data captured by an image sensor (e.g., image sensor 372, image sensor 472, or any other image sensor) to perform any of the methods disclosed herein. In some embodiments, analyzing the image data can include analyzing the image data to obtain preprocessed image data and then analyzing the image data and / or the preprocessed image data to obtain a desired result. Those skilled in the art will recognize that the following are examples and that the image data can be preprocessed using other types of preprocessing methods. In some examples, the image data can be preprocessed by transforming the image data using a transformation function to obtain transformed image data, where the preprocessed image data can include the transformed image data. For example, the transformed image data can include one or more convolutions of the image data. For example, the transformation function can include one or more image filters, such as a low-pass filter, a high-pass filter, a band-pass filter, an all-pass filter, etc. In some examples, the transformation function can include a nonlinear function. In some examples, the image data can be preprocessed by smoothing at least a portion of the image data, for example, using a Gaussian convolution, using a median filter, etc. In some examples, the image data can be preprocessed to obtain a different representation of the image data. For example, the preprocessed image data may include a representation of at least a portion of the image data in the frequency domain, a discrete Fourier transform of at least a portion of the image data, a discrete wavelet transform of at least a portion of the image data, a time / frequency representation of at least a portion of the image data, a reduced-dimensional representation of at least a portion of the image data, a lossy representation of at least a portion of the image data, a lossless representation of at least a portion of the image data, a time-ordered sequence of any of the above, any combination of the above, etc. In some examples, the image data may be preprocessed to extract edges, and the preprocessed image data may include information based on and / or related to the extracted edges. In some examples, the image data may be preprocessed to extract image features from the image data.Some non-limiting examples of such image features may include information based on and / or related to edges, corners, blobs, ridges, Scale Invariant Feature Transform (SIFT) features, temporal features, etc. In some examples, analyzing the image data may include computing at least one convolution of at least a portion of the image data and using the computed at least one convolution to compute at least one resultant value and / or to determine, identify, recognize, classify, etc.

[0160] Consistent with other aspects of the present disclosure, the processing device of system 200 can analyze image data to perform any of the methods disclosed herein. In some embodiments, analyzing the image can include analyzing the image data and / or preprocessed image data using one or more rules, functions, procedures, artificial neural networks, object detection algorithms, face detection algorithms, visual event detection algorithms, action detection algorithms, motion detection algorithms, background subtraction algorithms, inference models, etc. Some non-limiting examples of such inference models can include results on training examples of training algorithms, such as manually pre-programmed inference models, classification models, regression models, machine learning algorithms, and / or deep learning algorithms, where the training examples can include examples of data instances, and in some cases, the data instances may be labeled with corresponding desired labels and / or results, etc. In some embodiments, analyzing the image data (e.g., by methods, steps, and modules described herein) can include analyzing pixels, voxels, point clouds, range data, etc. included in the image data.

[0161] Convolution may include convolution of any dimension. A one-dimensional convolution is a function that transforms an original sequence into a transformed sequence. A one-dimensional convolution may be defined by a series of scalars. Each particular value in the transformed sequence may be determined by calculating a linear combination of values ​​in a subsequence of the original sequence that corresponds to a particular value. The resulting value of the calculated convolution can include any value in the transformed sequence. Similarly, an n-dimensional convolution is a function that transforms an original n-dimensional array into a transformed array. An n-dimensional convolution may be defined by an n-dimensional array of scalars (known as the kernel of the n-dimensional convolution). Each particular value in the transformed array may be determined by calculating a linear combination of values ​​in an n-dimensional region of the original array that corresponds to a particular value. The resulting value of the calculated convolution can include any value in the transformed array. In some examples, an image may include one or more components (e.g., color components, depth components, etc.), and each component may include a two-dimensional array of pixel values. In one example, calculating the convolution of an image may include calculating a two-dimensional convolution for one or more components of the image. In another example, computing the convolution of an image can include stacking arrays from different components to create a three-dimensional array and computing the three-dimensional convolution on the resulting three-dimensional array. In some examples, a video can include one or more components (e.g., color components, depth components, etc.), where each component can include a three-dimensional array of pixel values ​​(having two spatial axes and one temporal axis). In one example, computing the convolution of a video can include computing the three-dimensional convolution on one or more components of the video. In another example, computing the convolution of a video can include stacking arrays from different components to create a four-dimensional array and computing the four-dimensional convolution on the resulting four-dimensional array.

[0162] In some embodiments, the integrated computing interface device can include a portable housing having a key region and a non-key region. The housing of the integrated computing interface device can include an outer cover or shell that can contain one or more components associated with the integrated computing interface device. The disclosed exemplary housing can enclose the components of the integrated computing interface device and can cover some or all of the components of the integrated computing interface device. It is contemplated that the disclosed exemplary housing can have one or more openings that can expose certain components of the integrated computing interface device (e.g., USB or other ports) or allow certain components (e.g., keyboard keys) to protrude from the housing. The housing can support certain components of the integrated computing interface device (e.g., a circuit board) on an interior portion of the housing. The housing can include additional structural features that can enable one or more components of the integrated computing interface device to be attached to the housing. The housing can be square, rectangular, or other shape sized to fit on a user's desk, laptop, or other suitable work surface. The housing can be made of plastic, metal, a combination of plastic and metal, or other suitable materials.

[0163] The housing of the integrated computing device may include a key region and a non-key region distinct from the key region. The key region of the integrated computing interface device may include one or more keys that can enable a user to enter alphanumeric or other characters as input. For example, in some embodiments, a keyboard may be associated with the key region of the housing. The keyboard may be a standard typewriter-style keyboard (e.g., a QWERTY-style keyboard) or another suitable keyboard layout, such as a Dvorak layout or a coded layout. The keyboard may include any suitable number of keys; for example, a "full-size" keyboard may include up to 104 or 105 keys. In some embodiments, the keyboard may include at least 30 keys. In other embodiments, the keyboard may include fewer than 10 keys, at least 10 keys, at least 20 keys, at least 50 keys, at least 80 keys, at least 100 keys, etc.

[0164] The key region may include alphanumeric keys, function keys, modifier keys, cursor keys, system keys, multimedia control keys, or other physical keys that perform computer-specific functions when pressed. The key region may also include virtual or programmable keys such that the function of the keys changes depending on the function or application being run on the integrated computing interface device.

[0165] In some embodiments, the keyboard may include dedicated input keys for performing actions through the wearable extended reality device. The dedicated input keys may enable a user to interact with virtual widgets viewed through the wearable extended reality device. The dedicated input keys may take a picture of one or more virtual widgets displayed through the wearable extended reality device (i.e., a "screenshot"). If the wearable extended reality device includes a camera, the dedicated input keys may take a picture with the camera. In one embodiment, the image may include what the user sees through the wearable extended reality device with any virtual widgets included in the image (i.e., the image with a virtual overlay). The keyboard may include multiple dedicated input keys for performing actions, each key configured to perform a different action. In one embodiment, the dedicated input keys may be programmable by a user to perform one or more actions (e.g., a "macro"). Note that the above examples of actions performed by the dedicated input keys are not limiting, and other actions may be performed by the dedicated input keys.

[0166] In some embodiments, the keyboard may include dedicated input keys for changing the lighting of the virtual display projected by the wearable extended reality device. The lighting of the virtual display may be adjusted by turning the virtual display on or off or by increasing or decreasing the brightness, contrast, or other color settings of the virtual display. In one embodiment, the keyboard may include multiple dedicated input keys for adjusting different settings of the virtual display. Settings of the virtual display that may be adjusted include, for example, image settings such as brightness, contrast, sharpness, or display mode (e.g., game mode with predetermined settings), color settings such as color component levels or other color adjustment settings, the position of the virtual display relative to the user's head position, or other settings that may improve the user's view of the virtual display.

[0167] A non-key area of ​​a portable housing may be an area of ​​the housing that does not contain keys. A non-key area may be an area of ​​the housing that does not contain keys, or it may exist to complete a desired shape of the housing that extends beyond the key area of ​​the housing in any direction. A non-key area may be an area that may contain an input device such as a trackpad, touchscreen, touch bar, or other form of cursor control for an integrated computing interface device. A non-key area may be subdivided into multiple different non-key areas, such as a trackpad or other cursor control, an extension of the housing, or a cover or grill for one or more speakers or other audio output devices included within the housing. A non-key area may include a display for displaying information to a user, may include one or more openings that allow air to circulate through the housing (e.g., to cool components housed within the housing), or may include one or more doors or access ports that allow access to an interior portion of the housing (e.g., a battery compartment configured to hold a removable battery therein or to allow certain components to be installed and / or removed from the interior of the housing).

[0168] In some embodiments, the holder may be associated with a non-key region of the housing. The holder may be a recess extending below the surface of the housing or may rest completely on the surface of the housing. The holder may be located in any portion of the non-key region of the housing. In some embodiments, the holder may be located in a non-key region of the housing adjacent to a key region of the housing. The holder may include one or more structural features for selectively engaging one or more items, such as a writing implement, a wire or cable, a dongle, paper (i.e., to allow the holder to function like a copy stand), or other items that a user may want to easily access or store.

[0169] In some embodiments, the holder may be configured for selective engagement and disengagement with the wearable extended reality device such that when the wearable extended reality device selectively engages with the housing via the holder, the wearable extended reality device is transportable with the housing. In some embodiments, structural features of the holder may be configured to selectively engage with the wearable extended reality device such that the wearable extended reality device can be snap-fit, press-fit, or pressure-fit into at least a portion of the holder. When the wearable extended reality device selectively engages with the housing via the holder, the wearable extended reality device may be transportable with the housing by being securely connected to the housing via the holder.

[0170] 6 is a top view of an exemplary embodiment of an integrated computing interface device 610 having a wearable extended reality device 612 selectively engaging with the integrated computing interface device 610. It should be understood that FIG. 6 is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure.

[0171] Integrated computing interface device 610 may include a housing 614. Housing 614 may include a key region 616 and non-key regions 618a and 618b. As shown in FIG. 6 , non-key region 618a may include a region located above key region 616, and non-key region 618b may include a region located below key region 616. Housing 614 may include only one non-key region (e.g., only either non-key region 618a or non-key region 618b), non-key region 618a and non-key region 618b may be adjacent to each other, or housing 614 may have two or more non-key regions.

[0172] Key area 616 may include a keyboard 620. Key area 616 may be subdivided into multiple key areas, which may be contiguous or separated from one another.

[0173] The integrated computing interface device 610 may include a holder 622 that can be configured to selectively engage with the wearable extended reality device 612. In some embodiments, the wearable extended reality device may include a pair of smart glasses. As shown in FIG. 6, the wearable extended reality device 612 may be a pair of smart glasses. The smart glasses may appear similar to traditional eyeglasses and may include "smart" features such as a camera positioned to take a picture of what the user is currently viewing and one or more displays configured to project images onto the lenses of the smart glasses. In some embodiments, the wearable extended reality device 612 may take other forms, such as goggles or other forms of wearable devices, that may include one or more lenses.

[0174] In some embodiments, the integrated computing interface device can include a trackpad associated with the housing. The trackpad can enable a user of the integrated computing interface device to control a cursor, select items, or activate items on the integrated computing interface device. The trackpad can include a single surface or a segmented surface, such as a cursor control portion and one or more button portions.

[0175] In an embodiment in which the wearable extended reality device is a pair of smart glasses, the integrated computing interface device is configured so that temples of the smart glasses contact the trackpad when the pair of smart glasses selectively engage with the housing via the holder. Thus, for example, the holder may be disposed on a first portion of the housing and the trackpad may be disposed on a second portion of the housing, with the first portion of the housing spaced apart from the second portion of the housing. The holder may be spaced apart from the trackpad by a distance approximately equal to the length of the temple portions of the pair of smart glasses.

[0176] Each temple of the smart glasses may include an elastic track pad protector at its distal end. The temples may extend parallel to each other in one direction from the lens of the smart glasses to enable the smart glasses to be worn by a user. In some examples, a pair of smart glasses may include at least two temples and at least one lens. Each temple may include a temple tip, and each temple tip may include an elastic track pad protector outer portion. In some examples, a pair of smart glasses may include at least two temples and at least one lens, and each temple may include an elastic track pad protector. The track pad protector may protect the distal ends of the temples of the smart glasses from damage when the smart glasses selectively engage with the holder and the distal ends of the temples are proximate to a surface of the housing (e.g., the track pad). The track pad protector may include a sleeve that slides over the distal ends of the temples or may be integrally formed with the distal ends of the temples. The track pad protector may be made of a soft or flexible material so that when the pair of smart glasses selectively engage with the housing via the holder and the distal ends of the temples of the pair of smart glasses contact the track pad, the distal ends of the temples of the pair of smart glasses do not scratch or damage the track pad.

[0177] Figure 7A is a top view and Figure 7B is a left side view of a second exemplary embodiment of an integrated computing interface device 710 having wearable extended reality equipment in the form of a pair of smart glasses 712 selectively engaging with the integrated computing interface device 710. It should be understood that Figures 7A and 7B are merely exemplary representations of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure.

[0178] The integrated computing interface device 710 may include a housing 714. The housing 714 may include a key area 716 and non-key areas 718a, 718b. The key area 716 may include a keyboard 720. The integrated computing interface device 710 may include a holder 722 configured to selectively engage with a pair of smart glasses 712. The non-key area 718b may be disposed below the keyboard 720 and may include a trackpad 724. The integrated computing interface device 710 shown in FIGS. 7A and 7B may have similar structural and functional characteristics to the integrated computing interface device 610 shown in FIG. 6. For example, the integrated computing interface device 710 may include all or some of the elements of the integrated computing interface device 610.

[0179] The pair of smart glasses 712 may include a lens portion 726 and two temples 728, one temple at each end of the lens portion 726. Each temple 728 may include a proximal end 730 connecting the temple 728 to the lens portion 726 and a distal end 732 located at the opposite end of the temple 728. The distal end 732 may include a resilient track pad protector 734. As described above, the resilient track pad protector 734 may be disposed on the distal end 732 or may be integrally formed with the distal end 732.

[0180] In some embodiments, the holder of the integrated computing interface device can include at least two gripping elements configured to selectively engage with temples of the smart glasses. In some embodiments, the gripping elements can be configured to selectively engage different portions of the smart glasses, such as one or both lenses, a bridge between the lenses, or other portions of the smart glasses. The gripping elements can be integrally formed with the holder, individually detachable from the holder, or detachable from the holder altogether. For example, the gripping elements can be integrally formed with each other and detachable as a unit from the holder. The gripping elements can be spring-biased toward each other by their shape or by using a spring or spring-like member.

[0181] Each grip element may include a protrusion from the surface of the integrated computing interface device. The protrusion may extend vertically away from the surface of the integrated computing interface device. A recess may be formed within the ridge and configured to hold the temple. The recess may be on a side of the ridge opposite the surface of the integrated computing interface device so that the temple is located on the top surface of the protrusion. The recess may be on a side of the ridge parallel to the surface of the integrated computing interface device so that the temple is located on the side of the ridge. The holder may be made of a flexible material, a rigid or semi-rigid material, or a rigid or semi-rigid material with a flexible recess. The flexible recess may be integrally formed with the ridge or may be a flexible material covering the recess.

[0182] Figure 8A is a front perspective view of a wearable extended reality device selectively engaging with a first embodiment of a holder. Figure 8B is a rear perspective view of the wearable extended reality device selectively disengaging from the first embodiment of the holder shown in Figure 8A. It should be understood that Figures 8A and 8B are merely exemplary representations of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure.

[0183] Holder 810 may include two gripping elements 812 spaced apart from one another and configured to selectively engage temples 814 of wearable extended reality device 816, shown as a pair of smart glasses in FIGS. 8A and 8B. In one example, holder 810 may be part of holder 622 and / or holder 722 and / or holder 1218a. In some embodiments, there may be other configurations of gripping elements 812, such as only one gripping element 812 or two gripping elements 812 disposed on the same side of holder 810.

[0184] In some embodiments, each gripping element 812 can include a recess 818 on its top surface for engaging with temple 814. In some embodiments, each gripping element 812 can have a flat top surface for engaging with temple 814. As shown in FIG. 8B, recess 818 can be U-shaped, partially surrounding temple 814. In some embodiments, recess 818 can be a different shape to engage with temple 814.

[0185] In some embodiments, the holder of the integrated computing interface device can include a clip for selectively connecting the wearable extended reality device to the housing. Selectively connecting the wearable extended reality device to the housing is one example of selectively engaging the wearable extended reality device to the housing. The clip can be located on any portion of the holder or can include a protrusion from the surface of the integrated computing interface device. The clip can selectively engage with any portion of the wearable extended reality device to connect the wearable extended reality device to the holder. In embodiments in which the wearable extended reality device is a pair of smart glasses, the clip can selectively engage with the temples, a portion of the lenses, a portion of the rim surrounding the lenses, the bridge, or the nose pads. The holder can include additional mechanisms for selectively engaging other portions of the wearable extended reality device not selectively engaged by the clip. Additional features can include one or more additional protrusions extending from the surface of the holder. The holder can be made of a flexible material, a rigid or semi-rigid material, or a rigid or semi-rigid material with a flexible clip or flexible protrusion that may be integrally formed with the holder, may be removable from the holder, or may be a flexible material covering the clip or protrusion.

[0186] Figure 9A is a front perspective view of a wearable extended reality device selectively engaged with a second exemplary embodiment of a holder. Figure 9B is a rear perspective view of the wearable extended reality device selectively disengaged from the second exemplary embodiment of the holder shown in Figure 9A. It should be understood that Figures 9A and 9B are merely exemplary representations of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure.

[0187] Holder 910 can include a clip 912 configured to selectively engage with a bridge 914 of a wearable extended reality device 916. As shown in the figures, as shown in FIGS. 9A and 9B, wearable extended reality device 916 can include a pair of smart glasses. In one example, holder 910 can be part of holder 622 and / or holder 722 and / or holder 1218a. In an exemplary embodiment, as shown in FIG. 9B, clip 912 can include a post 918 configured to fit between nose pads 920 of wearable extended reality device 916. As shown in FIG. 9B, post 918 can have a circular or cylindrical shape. Post 918 can also have other shapes (e.g., having a square, rectangular, oval, or polygonal cross section) such that nose pads 920 of wearable extended reality device 916 fit around and selectively engage post 918.

[0188] The clip 912 can include a bridge protrusion 922 configured to contact a front portion of the bridge 914. The bridge protrusion 922 can be spaced from the post 918 such that a portion of the bridge 914 is located between the bridge protrusion 922 and the post 918 when the wearable extended reality device 916 is selectively engaged with the holder 910.

[0189] The holder 910 can include at least two lens protrusions 924. The lens protrusions 924 are spaced apart from the clip 912 such that each lens protrusion 924 selectively engages an outer portion of a lens 926 of the wearable extended reality device 916. As shown in FIG. 9A , the bridge protrusions 922 can be shaped such that an inner portion of the lens 926 of the wearable extended reality device 916 selectively engages the bridge protrusions 922.

[0190] In some embodiments, the clip 912 may include only the post 918. In some embodiments, the holder 910 may include only the clip 912 and not the lens protrusion 924. In some embodiments, the holder 910 may include only the lens protrusion 924 and not the clip 912.

[0191] In some embodiments, the holder for the integrated computing interface device can include a compartment for selectively enclosing at least a portion of the wearable extended reality device when the wearable extended reality device selectively engages with the holder. The compartment can include a recess or sleeve within the housing to receive one or more portions of the wearable extended reality device. The compartment can include a protrusion from the surface of the holder and can be shaped to prevent the wearable extended reality device from sliding back and forth or side to side within the compartment when the wearable extended reality device selectively engages with the holder. The protrusion can include one or more walls extending above the surface of the holder such that the walls enclose a portion of the wearable extended reality device. The walls can be configured to accommodate different shapes of the wearable extended reality device. For example, if the wearable extended reality device is a pair of smart glasses, the walls can include cutouts to prevent the nose pads of the smart glasses from contacting the walls. The walls may also taper toward the surface of the holder to accommodate the lens of the smart glasses, such that a bottom portion of the lens contacts the surface of the holder. The holder may be made of a flexible material, a rigid or semi-rigid material, or a rigid or semi-rigid material with a flexible compartment. The flexible compartment may be integrally formed with the holder or may be a flexible material covering the compartment.

[0192] Figure 10A is a front perspective view of a wearable extended reality device selectively engaged with a third exemplary embodiment of a holder. Figure 10B is a rear perspective view of the wearable extended reality device selectively disengaged from the third exemplary embodiment of the holder shown in Figure 10A. It should be understood that Figures 10A and 10B are merely exemplary representations of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure.

[0193] The holder 1010 may include a compartment 1012 configured to selectively surround at least a portion of the wearable extended reality device 1014. In one example, the holder 1010 may be part of holder 622 and / or holder 722 and / or holder 1218a. The compartment 1012 may include a wall 1016 extending along a portion of a surface of the holder 1010 such that the wall 1016 defines the perimeter of the compartment 1012. As shown in FIG. 10A , the wall 1016 may taper toward the center and surface of the holder 1010 such that the lens 1018 of the wearable extended reality device 1014 contacts the surface of the holder 1010 when the wearable extended reality device 1014 is selectively engaged with the holder 1010. Based on this exemplary shape, the wall 1016 may contact the user-facing side of the lens 1018, but not the outward-facing side of the lens 1018. In some embodiments, the wall 1016 may not be tapered, such that the wall 1016 is of a generally uniform height on all sides, to define a slot configured to receive the lens 1018. As shown in FIG. 10B , the wall 1016 may include a cutout portion 1020 to accommodate a nose pad 1022 of the wearable extended reality device 1014, such that the nose pad 1022 does not contact the wall 1016.

[0194] In some embodiments, the holder of the integrated computing interface device can include at least one recess corresponding to the shape of a portion of the wearable extended reality device. In embodiments in which the wearable extended reality device is a pair of smart glasses, the holder can include one or more recesses that can be shaped to correspond to the shape of the lenses of the smart glasses. In embodiments in which the wearable extended reality device is goggles, the one or more recesses can be shaped to correspond to the shape of the lenses of the goggles. The one or more recesses can extend below the surface of the holder such that the one or more recesses extend into a portion of the housing. The bottom of the one or more recesses can contact the surface of the housing such that at least a portion of the holder extends above the surface of the housing. The holder can be made of a flexible material, a rigid or semi-rigid material, or a rigid or semi-rigid material with a flexible recess. The flexible recess can be integrally formed with the holder or can be a flexible material that covers the recess.

[0195] 11A is a front perspective view of a wearable extended reality device selectively engaging with a fourth exemplary embodiment of a holder. FIG. 11B is a rear perspective view of the wearable extended reality device selectively disengaging from the fourth exemplary embodiment of the holder shown in FIG. 11A. It should be understood that FIGS. 11A and 11B are merely exemplary representations of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure.

[0196] The holder 1110 can include a depression 1112 that corresponds to the shape of a portion of the wearable extended reality device 1114. In one example, the holder 1110 can be part of holder 622 and / or holder 722 and / or holder 1218a. As shown in FIGS. 11A and 11B, the wearable extended reality device 1114 can include a pair of smart glasses, and there can be two depressions 1112 spaced apart in the holder 1110, each depression 1112 corresponding to the shape of a lens 1116 of the smart glasses.

[0197] In some embodiments, the holder can also include a nose bridge protrusion. In embodiments in which the wearable extended reality device includes a pair of extended reality glasses, the at least one recess may include two recesses on either side of the nose bridge projection to receive the lenses of the extended reality glasses. Note that the terms "extended reality glasses" and "smart glasses" may be used interchangeably herein. The nose bridge protrusion may be a protrusion extending away from the surface of the holder and may be configured to support the nose pad or bridge of the smart glasses. Each of the two recesses may be shaped to receive one lens, a rim surrounding a lens, or a portion of a frame surrounding a lens. The holder can be made of a flexible material, a rigid or semi-rigid material, or a rigid or semi-rigid material with a flexible nose bridge protrusion. The flexible nose bridge protrusion may be integrally formed with the holder or may be a flexible material covering the nose bridge protrusion.

[0198] 11A and 11B, the wearable extended reality device 1114 is a pair of smart glasses. The holder 1110 can include a nose bridge protrusion 1118 configured to support nose pads 1120 or a bridge 1122 of the smart glasses.

[0199] In embodiments in which the wearable extended reality device includes a pair of smart glasses, the holder may be configured such that when the lenses of the smart glasses are positioned on one side of the keyboard, the temples of the smart glasses extend over the keyboard and the distal end of the smart glasses is positioned on the side of the keyboard opposite the lenses. The holder may include a mechanism to assist in positioning the temples of the smart glasses over a surface of the housing. The feature may include a protrusion extending upward from the housing to engage the temples of the smart glasses. The protrusion may be located in a keyed or non-keyed area of ​​the housing.

[0200] In some embodiments, the holder may include protrusions located near the keyboard (e.g., between the holder and the key area) and configured to create a gap between the temples of the smart glasses and the keyboard such that, when the smart glasses selectively engage with the holder, the temples of the smart glasses extend over the keyboard and distal ends of the temples are located on the sides of the keyboard opposite the lenses. The distal ends of the temples may not contact the housing because the protrusions can raise the distal ends above the surface of the housing. The protrusions may be made of an elastic or other compressible material to prevent the temples of the smart glasses from being damaged or scratched when they contact the protrusions.

[0201] FIG. 12A is a top view, and FIG. 12B is a left side view, of a third embodiment of an integrated computing interface device 1210 having wearable extended reality equipment in the form of a pair of smart glasses 1212 selectively engaging with the integrated computing interface device 1210. It should be understood that FIGS. 12A and 12B are merely exemplary representations of one embodiment, and that some illustrated elements may be omitted and other elements added within the scope of the present disclosure. The integrated computing interface device 1210 shown in FIGS. 12A and 12B may have similar structural and functional characteristics to the integrated computing interface device 610 shown in FIG. 6 and / or the integrated computing interface device 710 shown in FIGS. 7A and 7B. For example, the integrated computing interface device 1210 may include all or some of the elements of the integrated computing interface device 610. In another example, the integrated computing interface device 1210 may include all or some of the elements of the integrated computing interface device 710.

[0202] The integrated computing interface device 1210 may include a housing 1214. The housing 1214 may include a keyboard 1216 and a holder 1218a, which may be configured to selectively engage with the smart glasses 1212. The holder 1218a may include a protrusion 1220 disposed near the keyboard 1216. When the lenses 1222 of the smart glasses 1212 are selectively engaged with the holder 1218a, the temples 1224 may contact the protrusion 1220, thereby creating a gap 1226 between the temples 1224 and the keyboard 1216. When the lenses 1222 of the smart glasses 1212 are selectively engaged with the holder 1218a, the temples 1224 may extend over the keyboard 1216 such that distal ends 1228 of the temples 1224 are located opposite the lenses 1222 of the keyboard 1216. The distal end 1228 may be spaced apart from the housing 1214 such that the distal end 1228 does not contact the housing 1214 when the smart glasses 1212 are selectively engaged with the holder 1218a.

[0203] In some embodiments, the integrated computing interface device may further include a charger associated with the housing. The charger may be configured to charge the wearable extended reality device when the wearable extended reality device selectively engages with the holder. In such embodiments, the wearable extended reality device may include a battery or other power source to be charged. The charger may provide a DC voltage or current to charge the battery of the wearable extended reality device. The battery may be charged by a wired connection or a wireless connection. The housing and the wearable extended reality device may be configured for wireless charging of the wearable extended reality device by the charger. The charger may be located in any suitable portion of the housing such that the charger can provide power to the wearable extended reality device when the wearable extended reality device selectively engages with the holder.

[0204] The wearable extended reality device can include one or more electrical contacts, and the housing can include one or more corresponding electrical contacts that engage to charge the wearable extended reality device when engaged with the holder. In some examples, the one or more electrical contacts included in the wearable extended reality device can be located on one or both lenses, one or both temples, or a portion of the frame near where the lenses and temples connect. In some examples, the one or more corresponding electrical contacts included in the housing can be located within the holder or adjacent to the holder, such that when the wearable extended reality device selectively engages with the holder, the one or more electrical contacts of the wearable extended reality device are in sufficient proximity so that wireless charging can occur. Wireless charging can be accomplished via a wireless charging standard, such as Qi, AirFuel Resonant, Near-Field Magnetic Coupling (NFMC), Radio Frequency (RF), or other suitable wireless charging protocol. In some embodiments, the number of electrical contacts included in the housing need not match the number of electrical contacts included in the wearable extended reality device.

[0205] In embodiments in which the wearable extended reality device is a pair of smart glasses, each lens can include an electrical contact and the housing can include one or more corresponding electrical contacts. When the smart glasses are selectively engaged with the housing, the electrical contacts in the lenses can be positioned in sufficient proximity to one or more corresponding electrical contacts in the housing to complete a wireless charging circuit.

[0206] 8A and 8B, the electrical contacts may be located in the temples 814 of the smart glasses 816. Corresponding electrical contacts may be located in the gripping element 812 of the holder 810.

[0207] 9A and 9B, electrical contacts may be located on bridge 914 and / or nose pad 920 of wearable extended reality device 916. Corresponding electrical contacts may be located in clip 912 and / or post 918 of holder 910. In another implementation of the embodiment shown in FIGS. 9A and 9B, electrical contacts may be located in or around lens 926 of wearable extended reality device 916. Corresponding electrical contacts may be located on lens protrusion 924 and / or a portion of holder 910 below lens 926.

[0208] 10A and 10B, electrical contacts may be located in or around the lenses 1018 of the wearable extended reality device 1014. Corresponding electrical contacts may be located on the wall 1016.

[0209] 11A and 11B, the electrical contacts may be located in or around the lenses 1116 of the wearable extended reality device 1114. Corresponding electrical contacts may be located in the recesses 1112 of the holder 1110. In an alternative implementation of the embodiment shown in FIGS. 11A and 11B, the electrical contacts may be located in the nose pads 1120 or bridge 1122 of the wearable extended reality device 1114. Corresponding electrical contacts may be located in the nose bridge protrusions 1118 of the holder 1110.

[0210] In some embodiments, the housing may also include a wire port configured to accept a wire extending from the wearable extended reality device. The wire may be any type of wire suitable for providing power and / or data between the integrated computing interface device and the wearable extended reality device. For example, if the wire is detachable from the wearable extended reality device, the wire may be a Universal Serial Bus (USB) type wire with a wire port and an appropriate connector on the wearable extended reality device. The wire port may be located in any portion of the housing that is easily accessible to a user of the integrated computing interface device. The wire may extend from any portion of the wearable extended reality device. The wire may be fixedly attached to the wearable extended reality device or may be detachable from the wearable extended reality device. In embodiments in which the wearable extended reality device is a pair of smart glasses, the wire may extend from the temples. The wire may be located at any point along the length of the temples so that the wire does not obstruct the user's vision or affect the user's ability to wear the smart glasses.

[0211] In some embodiments, the wire port may be located on a front side of the integrated computing interface device configured to face the user while the user is typing on the keyboard. In one example, the wire port may be located substantially in the center of the front side of the integrated computing interface device (e.g., less than 1 cm from the center, less than 2 cm from the center, less than 4 cm from the center, less than 8 cm from the center, etc.). In another example, the wire port may be located off-center. In another example, the wire port may be located on a side of the front side (e.g., left side, right side, etc.) of the integrated computing interface device, e.g., less than 1 cm from a side edge of the front side, less than 2 cm from a side edge, less than 4 cm from a side edge, less than 8 cm from a side edge, etc.

[0212] For example, in the embodiment shown in FIG. 6 , wire port 624 may be located within housing 614. Wire port 624 may be located in a non-key area 618b of integrated computing interface device 610 near the user while the user types on keyboard 620. Wires 626 may be connected to wearable extended reality device 612 and received by wire port 624. Wires 626 may be optional and are shown in dashed outline in FIG. 6 . In some examples, wires 626 may be selectively detached from wearable extended reality device 612. In other examples, wires 626 may be permanently connected to wearable extended reality device 612. In some examples, wires 626 may be selectively detached from wire port 624. In other examples, wires 626 may be permanently connected to wire port 624. In some examples, wires 626 may be fully or partially retracted within housing 610 and / or a compartment formed by housing 610. When the wire 626 is in a fully or partially retracted state, the wire 626 may be pulled from the housing 610 and / or from the compartment, for example, by a user.

[0213] As another example, in the embodiment shown in FIGS. 7A and 7B , wire port 736 may be located within housing 714. Wire port 736 may be located in a non-key area 718 b of integrated computing interface device 710 near the user while the user types on keyboard 720. Wire 738 may be connected to smart glasses 712 and received by wire port 736. Locating wire port 736 in non-key area 718 b may allow the user to type on keyboard 720 while wire 738 is connected to wire port 736. Wire 738 is optional and is shown in dashed outline in FIGS. 7A and 7B . In some examples, wire 738 may be selectively detachable from wearable extended reality device 712. In other examples, wire 738 may be permanently connected to wearable extended reality device 712. In some examples, wire 738 may be selectively detachable from wire port 736. In other examples, wire 738 may be permanently connected to wire port 736. In some examples, wire 738 may be fully or partially retracted within housing 710 and / or a compartment formed by housing 710. When wire 738 is in a fully or partially retracted state, wire 738 may be pulled from housing 710 and / or from the compartment, for example, by a user.

[0214] In some embodiments, the wire may be configured to electrically charge the wearable extended reality device when the wire is connected to the wire port. The wire may be any type of wire suitable for providing power to the wearable extended reality device. For example, the wire may be a Universal Serial Bus (USB) type wire with a connector suitable for the wire port and the wearable extended reality device.

[0215] In some embodiments, the integrated computing interface device may also include at least one processor disposed within the housing. The wire may be configured to enable digital data communication between the wearable extended reality device and the at least one processor. The processor may include any processing device suitable for digital data communication. In addition to enabling digital data communication, the processing device may be configured to execute a computer program on the integrated computing interface device. The wire may be any type of wire suitable for enabling digital data communication between the wearable extended reality device and the at least one processor. For example, the wire may be a Universal Serial Bus (USB) type wire with a wire port and a connector suitable for the wearable extended reality device.

[0216] In some embodiments, the integrated computing interface device may also include a processor disposed within the housing. The processor may be configured to wirelessly pair with the wearable extended reality device. Wireless pairing is a process of wirelessly linking the integrated computing interface device and the wearable extended reality device to enable wireless data communication between the integrated computing interface device and the wearable extended reality device. The processor may include any processing device suitable for implementing a wireless pairing protocol between the integrated computing interface device and the wearable extended reality device. The wireless pairing protocol may be WiFi (IEEE 802.11-based), radio frequency (RF, such as ZigBee or ZWave), radio frequency identification (RFID, e.g., active reader passive tag or active reader active tag), Bluetooth, near field communication (NFC), or any other wireless pairing protocol available for short-range communication. The integrated computing interface device may include a visual marker adjacent to the keyboard to facilitate wireless pairing with the wearable extended reality device. The visual marker can help ensure that the wearable extended reality device is within wireless communication range of the integrated computing interface device. In some embodiments, the keyboard can include a dedicated function key for initiating the wireless pairing process.

[0217] In some embodiments, the integrated computing interface device may also include at least one motion sensor disposed within the housing and at least one processor operatively connected thereto. The at least one processor may be programmed to implement an operational mode based on input received from the at least one motion sensor. In some embodiments, the motion sensor may determine whether the integrated computing interface device is moving and adjust the operational mode of the integrated computing interface device based on the movement. In some embodiments, the motion sensor may determine whether the wearable extended reality device is moving relative to or with the integrated computing interface device and adjust the operational mode of the integrated computing interface device or the wearable extended reality device based on the movement. For example, if a user of the wearable extended reality device is walking, the number of items displayed to the user may be limited to prevent distraction.

[0218] The at least one motion sensor may include an accelerometer, a gyroscope, a magnetometer, a motion sensor implemented by using an image sensor and analyzing images captured using the image sensor with an egomotion algorithm, or other type of sensor configured to measure movement of objects within an environment of the integrated computing interface device. For example, the at least one motion sensor may be motion sensor 373 described above in connection with FIG. 3. The at least one processor may include any processing device configured to receive input from the at least one motion sensor and programmed to implement an operational mode based on the input.

[0219] In some embodiments, the at least one processor may be further programmed to automatically adjust settings of the virtual display presented by the wearable extended reality device based on input received from the at least one motion sensor. The at least one processor may include any processing device configured to be programmed to automatically adjust one or more settings of the virtual display presented by the wearable extended reality device. The settings of the virtual display may be adjusted based on the environment in which the user is located (e.g., moving from indoors in low light to outdoors in bright light). The settings of the virtual display that may be adjusted may include image settings such as brightness, contrast, sharpness, or a display mode (e.g., a game mode with predetermined settings), color settings such as color component levels or other color adjustment settings, the position of the virtual display relative to the user's head position, or other settings that may improve the user's view of the virtual display. In the embodiment shown in FIG. 1, the settings of the virtual screen 112 may be automatically adjusted by the at least one processor.

[0220] In some embodiments, the at least one processor may be further programmed to output a notification when the integrated computing interface device moves beyond a threshold distance while the wearable extended reality device is disengaged from the holder. The at least one processor may include any processing device configured to be programmed to output a notification when the integrated computing interface device moves beyond a threshold distance while the wearable extended reality device is disengaged from the holder. For example, a notification may be provided to a user to warn them that they may be moving the wearable extended reality device out of wireless communication range, such that interaction with the integrated computing interface device is discontinued unless the user moves closer than the threshold distance. As another example, if the wearable extended reality device is connected to the integrated computing interface device by a wire, a notification may be provided if the user attempts to move further away from the integrated computing interface device than the length of the wire, such as if the wire were to break, the wearable extended reality device were to be accidentally removed from the user's head, or the integrated computing interface device were to be knocked off the surface on which the integrated computing interface device is placed.

[0221] The notification can include an alert, alarm, or other audio and / or visual indicator. The notification can be output via either a device (e.g., light indicator 351, display 352, and / or speaker 353) connected to output interface 350 shown in FIG. 3 or output interface 450 (e.g., light indicator 451, display 452, speaker 453, and / or projector 454) shown in FIG. 4. The threshold distance can be a percentage of the length of the wire connected between the integrated computing interface device and the wearable extended reality device, a fixed distance relative to the length of the wire, a fixed distance from the integrated computing interface device, a percentage of the distance range of the wireless communication protocol between the integrated computing interface device and the wearable extended reality device, a fixed distance relative to the distance range of the wireless communication protocol, or any other distance that separates the wearable extended reality device from the integrated computing interface device.

[0222] In some embodiments, the integrated computing interface device may further include at least one sensor within the housing and at least one processor operatively connected thereto. The at least one sensor may be configured to provide input indicating whether the wearable extended reality device is engaged with the holder. The at least one processor may be programmed to use the received input to implement an operational mode based on whether the wearable extended reality device is engaged with the holder. For example, in response to the input indicating that the wearable extended reality device is engaged with the holder, the at least one processor may implement a first operational mode, and in response to the input indicating that the wearable extended reality device is not engaged with the holder, the at least one processor may implement a second operational mode, which may be different from the first operational mode. In some examples, the at least one processor may be programmed to automatically adjust settings of a virtual display presented by the wearable extended reality device based on whether the wearable extended reality device is engaged with the holder, e.g., as described herein with respect to adjusting settings based on input from the at least one motion sensor. In some examples, the at least one processor may be programmed to output an audible indication when at least one of the wearable extended reality devices engages with the holder or when the wearable extended reality device is removed from the holder. In some examples, the operational mode may be or include a power mode of at least one of the at least one processor, a communication device included in the integrated computing interface device, or the wearable extended reality device. In one example, when the wearable extended reality device is engaged with the holder, the power mode may be a turn-off mode, a sleep mode, a hibernation mode, etc.In another example, a power mode when the wearable extended reality device is engaged with a holder may be associated with lower power consumption than a power mode when the wearable extended reality device is not engaged with a holder (e.g., using fewer hardware components when the wearable extended reality device is engaged with a holder, using a lower clock speed when the wearable extended reality device is engaged with a holder, etc.).

[0223] In some examples, the operational modes may include a display mode for presenting virtual content through the wearable extended reality device. In one example, in one operational mode, no virtual content may be presented through the wearable extended reality device (e.g., when the wearable extended reality device is engaged with a holder), and in another operational mode, selected virtual content may be presented through the wearable extended reality device (e.g., when the wearable extended reality device is not engaged with a holder). In another example, in one operational mode, virtual content may be presented through the wearable extended reality device at a smaller size (e.g., when the wearable extended reality device is engaged with a holder), and in another operational mode, selected virtual content may be presented through the wearable extended reality device at a larger size (e.g., when the wearable extended reality device is not engaged with a holder). In another example, in one mode of operation, virtual content may be presented through the wearable extended reality device at a lower opacity (e.g., when the wearable extended reality device is engaged with a holder), and in another mode of operation, selected virtual content may be presented through the wearable extended reality device at a higher opacity (e.g., when the wearable extended reality device is not engaged with a holder). In another example, in one mode of operation, virtual content may be presented through the wearable extended reality device at a lower brightness (e.g., when the wearable extended reality device is engaged with a holder), and in another mode of operation, selected virtual content may be presented through the wearable extended reality device at a higher brightness (e.g., when the wearable extended reality device is not engaged with a holder).

[0224] In some examples, when the wearable extended reality device is engaged with the holder, an operational mode may be selected based on virtual content presented through the wearable extended reality device before the wearable extended reality device was engaged with the holder. For example, a first operational mode may be selected in response to first virtual content (e.g., virtual content related to hardware maintenance, virtual content associated with a high priority task, etc.), and a second operational mode may be selected in response to second virtual content (e.g., virtual content requiring user intervention, virtual content associated with a low priority task, etc.), where the second mode may be different from the first mode. In some examples, when the wearable extended reality device is engaged with the holder, an operational mode may be selected based on analysis of image data captured using at least one image sensor (e.g., at least one image sensor included in an integrated computing interface device included in the wearable extended reality device, etc.). For example, the image data may be analyzed using a visual classification algorithm to classify the physical environment of the integrated computing interface into a particular class from multiple alternative classes, and the operational mode may be selected based on the particular class. Some non-limiting examples of such classes may include "outdoors," "indoors," "office," "home," "conference room," "at least one person in the environment," "at least two people in the environment," "no people in the environment," etc. In another example, the image data may be analyzed using a visual motion recognition algorithm to detect movement in the physical environment of the integrated computing interface, and an operational mode may be selected based on whether movement towards the integrated computing interface is identified.

[0225] In some embodiments, the integrated computing interface device may also include a protective cover. The cover can protect portions of the housing, such as the key area, the non-key area, and the portion of the housing including the holder, from damage during transportation. The protective cover may be completely removable from the housing or may be attached to the housing by one or more of its sides. One side of the protective cover may be fixedly attached to the housing. The protective cover may include two layers of a soft material (such as a nonwoven fabric) encasing a second material (such as silicone). The protective cover may include a first layer of a soft material (such as a nonwoven fabric) and a second layer of a second material (such as silicone). The protective cover may be made of any number of layers or different types of materials to provide bump, shock, or impact protection to the keyboard and / or housing and / or wearable extended reality device when the wearable extended reality device selectively engages with the holder.

[0226] The protective cover may be operable in two storage modes. In a first storage mode, the protective cover may be configured to overlie the wearable extended reality device within the housing. For example, in the first storage mode, the protective cover can provide bump, shock, or impact protection to the keyboard and / or the housing and / or the wearable extended reality device when the wearable extended reality device is selectively engaged with the holder. The protective cover may include one or more features, such as one or more protrusions, configured to hold the wearable extended reality device in the first storage mode. For example, the protective cover may include one or more fold lines that allow the protective cover to flex over the wearable extended reality device when the wearable extended reality device is selectively engaged with the holder and to flex differently when the wearable extended reality device is selectively disengaged from the holder.

[0227] In the second stowing mode, the protective cover may be configured to elevate the housing. For example, in the second stowing mode, the protective cover may allow the housing to be elevated relative to a surface on which the housing is placed, such as a table, desk, or a user's lap. To elevate the housing, the protective cover may be segmented such that the protective cover can be folded into different positions, thereby elevating the housing to one or more distances above the surface on which the housing is placed. In the second stowing mode, the protective cover may not overlie the wearable extended reality device.

[0228] 13A is a right-side perspective view of integrated computing interface device 1310 having protective cover 1312 in a first storage mode. It should be understood that FIG. 13A is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of this disclosure. Protective cover 1312 may cover a top portion of housing 1314, and a bottom surface of housing 1314 contacts surface 1316 upon which integrated computing interface device 1310 may be placed. In some non-limiting examples, integrated computing interface device 1310 may include all or a portion of at least one element of integrated computing interface device 610, integrated computing interface device 710, or integrated computing interface device 1210.

[0229] FIG. 13B is a left perspective view of the integrated computing interface device 1310 of FIG. 13A with the protective cover 1312 in the second storage mode. It should be understood that FIG. 13B is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of this disclosure. When the protective cover 1312 is in the second storage mode, the protective cover 1312 may be at least partially separated from the housing 1314 to allow access to the keyboard 1318, the wearable extended reality device 1320, and the holder 1322 for the wearable extended reality device 1320. The protective cover 1312 may be segmented to create individual segments (e.g., segments 1324, 1326, 1328, and 1330) such that the protective cover 1312 can be folded to elevate at least a portion of the housing 1314 above the surface 1316. The housing can be lifted by placing the folded protective cover 1312 under one end of the housing and lifting the end of the housing. Note that the number of segments 1324-1330 shown in Figure 13B is exemplary, and the protective cover 1312 can have fewer or more segments.

[0230] The protective cover may also include at least one camera associated with the protective cover. For example, the protective cover may include one or more cameras or other types of imaging devices configured to capture images of the keyboard, the user, or the environment around the user. The one or more cameras may include one or more selfie cameras, back cameras, or other cameras associated with the protective cover such that the cameras may be usable when the protective cover is in the second storage mode.

[0231] The protective cover may also include at least one protrusion on at least two sides of the at least one camera. For example, the protective cover may include two protrusions from two sides of the at least one camera, one protrusion surrounding the at least one camera from at least two sides, or another number of protrusions, where the at least one protrusion is configured to prevent the at least one camera from contacting the flat surface when the cover is placed on the flat surface with the camera facing the flat surface.

[0232] 14 is a right perspective view of another embodiment of an integrated computing interface device 1410 having a protective cover 1412 in a second, storage mode. It should be understood that FIG. 14 is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure.

[0233] When protective cover 1412 is in the second, stowed mode, protective cover 1412 is at least partially separated from housing 1414 to allow access to keyboard 1416, wearable extended reality device 1418, and holder 1420 for wearable extended reality device 1418. Protective cover 1412 can be segmented to create individual segments (such as segments 1422, 1424, and 1426) such that protective cover 1412 can fold down to lift at least a portion of housing 1414 above surface 1428. It should be noted that the number of segments 1422 through 1426 shown in FIG. 14 are exemplary, and protective cover 1412 can have fewer or more segments.

[0234] Camera 1430 may be disposed within protective cover 1412 within segment 1422. Note that camera 1430 may be disposed on any portion of protective cover 1412. Three protrusions 1432 extend outward from protective cover 1412 to form a triangular shape around camera 1430. Protrusions 1432 are configured to prevent camera 1430 from contacting surface 1428 when protective cover 1412 is placed on surface 1428 with camera 1430 facing surface 1428.

[0235] Traditionally, cameras are located on the top frame of a laptop screen. This position allows the camera to face the user when the laptop is used from an elevated position relative to the surface on which the laptop is placed. This elevated position allows for a desirable viewing angle for the user. Some disclosed embodiments enable laptops without a large physical screen using extended reality devices. However, this creates a camera positioning problem. Placing the camera above the keyboard creates an undesirable viewing angle for the user that the user is not accustomed to. Therefore, it is desirable to place the camera in an elevated position relative to the keyboard. One solution provided by some disclosed embodiments includes placing the camera within a foldable cover of the keyboard, configured to be elevated above the keyboard and folded into a user-facing configuration.

[0236] In some embodiments, the integrated computing interface device can include a housing having a key region and a non-key region. The integrated computing interface can include any device having several functional components. The device can function as an interface that enables human interaction with a machine, such as a computer. In one example, the integrated computing interface device can include a computing device configured to operate with a wearable extended reality device external to the integrated computing interface device, for example, to enable the presentation of an extended reality environment via the wearable extended reality device. In one example, the integrated computing interface device can include, e.g., within a single housing, a computing device (e.g., a processor, a CPU, etc.) integrated with an input device (e.g., a keyboard, a touchscreen, a touchpad, etc.), a digital communication device configured to connect the integrated computing interface device to a digital communication network (e.g., Ethernet, a cellular network, the Internet, etc.), and a communication port configured to connect the wearable extended reality device external to the integrated computing interface device.

[0237] A housing can include any physical structure in which one or more components are housed or accommodated. Such a housing can have a key area, such as a general area where keys are located and a non-key area that is generally devoid of keys. Keys can include any button, switch, trigger, toggle, or any other element that can be activated via physical manipulation. In some embodiments, the keys can be mechanical in nature (e.g., mechanical push buttons such as those found on a typical computer keyboard). In other examples, the keys can be soft (e.g., a touch display on which images of keys are simulated). By way of example, the key area can cover an area of ​​an alphanumeric or numeric keyboard, and the non-key area can be another area of ​​the interface that is devoid of keys. In some examples, the key area and / or the non-key area can be an area on the exterior surface (or upper exterior surface) of the integrated computing interface device. In some embodiments, the integrated computing interface device can include a housing having an input area and a non-input area. In one example, the input area can be an exterior surface area (or upper exterior surface area) of an integrated computing interface device that includes a touchscreen. In another example, the input area can be an exterior surface area (or upper exterior surface area) of an integrated computing interface device that includes a touchpad. In yet another example, the input area may be and / or include a key area, as described above. In one example, the non-input area may be an exterior area (or an upper exterior area) of the interface that is devoid of input devices. In another example, the input area may be an exterior area (or an upper exterior area) of an integrated computing interface device that includes a particular type of input device, and the non-input area may be another exterior area (or another upper exterior area) of the integrated computing interface device that lacks a particular type of input device.

[0238] The housing can include an outer cover or shell. The housing can surround the components of the integrated computing interface device and can cover some or all of the components of the integrated computing interface device. It is contemplated that the housing can have one or more openings that can expose certain components of the integrated computing interface device (e.g., USB or other ports or an image sensor) or allow certain components (e.g., keyboard keys) to protrude from the housing. The housing can support certain components of the integrated computing interface device (e.g., a circuit board) in an interior portion of the housing.

[0239] The housing of the integrated computing device may include a key region and a non-key region distinct from the key region. As described in more detail above, the key region may include one or more keys that can enable a user to enter alphanumeric or other characters as input. For example, in some embodiments, a keyboard may be associated with the key region of the housing. The keyboard may be a standard typewriter-style keyboard (e.g., a QWERTY-style keyboard) or another suitable keyboard layout, such as a Dvorak layout or a coded layout. The keyboard may include any suitable number of keys; for example, a “full-size” keyboard may include up to 104 or 105 keys. In one example, the keyboard may include at least 10 keys, at least 30 keys, at least 80 keys, at least 100 keys, etc. In one example, the keyboard may be included in the integrated computing interface device, an exterior surface region of the integrated computing interface device, an upper exterior surface region of the integrated computing interface device, the housing, the exterior surface of the housing, the upper exterior surface of the housing, the key region, etc. In some embodiments, an input device may be associated with the input region of the housing. Some non-limiting examples of such input devices include a touchscreen, a touchpad, the keyboards mentioned above, etc. For example, the input device may be a touchscreen, and the touchscreen may have a diagonal length of at least 1 inch, at least 5 inches, at least 10 inches, etc. In another example, the input device may be a touchpad, and the touchpad may have a diagonal length of at least 1 inch, at least 5 inches, at least 10 inches, etc. In one example, the input device may be included in an integrated computing interface device, an exterior surface region of the integrated computing interface device, a housing, an exterior surface of the housing, an upper exterior surface region of the integrated computing interface device, a housing, an upper exterior surface of the housing, a key region, etc.

[0240] As described in more detail above, a non-key area may be an area of ​​the housing that does not include keys, or may be present to complete a desired shape of the housing that extends beyond the key area of ​​the housing in any direction. A non-key area may be an area that may include an input element, such as a trackpad, trackball, touchscreen, touchbar, or other form of cursor control for an integrated computing interface device. A non-key area may be subdivided into multiple different non-key areas, such as a trackpad or other cursor control, an extension of the housing, or a cover or grille for one or more speakers or other audio output devices included within the housing. In some embodiments, a housing may include two or more non-key areas. For example, a first non-key area may be located on the top edge of the key area, and a second non-key area may be located on the bottom edge of the key area. In some embodiments, a non-key area may simply be part of the housing and have no function other than to function as part of the housing. In some examples, as described in more detail above, a non-input area may be an area of ​​the housing that does not include an input device or does not include a particular type of input device, or may be present to complete a desired shape of the housing that extends beyond the input area of ​​the housing in any direction. In some embodiments, the housing may include two or more non-input areas.

[0241] In some embodiments, the integrated computing interface device may include at least one image sensor. The image sensor may include a device that converts photons (i.e., light) into electrical signals for interpretation. For example, the image sensor may incorporate a charge-coupled device (CCD) or active pixel sensor (CMOS sensor) fabricated with complementary metal-oxide semiconductor (CMOS) or negative-type metal-oxide semiconductor (NMOS or negative-type MOS) technology. The at least one image sensor may be configured to capture images and / or video of a user of the integrated computing interface device or may be otherwise disposed in the user's physical environment. As previously mentioned, the image sensor may be configured to capture visual information by converting light into image data. In some embodiments, the at least one image sensor may be at least one of a color image sensor, a monochrome image sensor, a stereo image sensor, an infrared image sensor, or a depth image sensor.

[0242] In some examples, image data captured using at least one image sensor included in the integrated computing interface device can be analyzed to determine whether a user of the wearable extended reality device is approaching the integrated computing interface device. For example, the image data can be analyzed using a visual classification algorithm to determine whether a user of the wearable extended reality device is approaching the integrated computing interface device, whether a person approaching the integrated computing interface device is a user of the wearable extended reality device, etc. In some examples, the image data can be analyzed to identify a wearable extended reality device used by a person approaching the integrated computing interface device. For example, a unique visual code can be presented on the wearable extended reality device (e.g., on a sticker or on a display screen outside the wearable extended reality device), the image data can be analyzed to detect and recognize the unique visual code, and a data structure associating the wearable extended reality device with the visual code can be accessed based on the identified visual code to identify the wearable extended reality device. Additionally, in some examples, pairing of the integrated computing interface device with the identified wearable extended reality device may be initiated upon identification of the wearable extended reality device.

[0243] In some embodiments, the integrated computing interface device can include a foldable protective cover incorporating at least one image sensor, which can be configured to be manipulated into multiple folded configurations. A cover can be considered foldable if it has a non-rigid structure that allows it to adjust or conform to at least some degree to a structure disposed beneath the cover. By way of example, a foldable cover can have distinct folds or creases, or can be generally flexible without specific creases. Collapsibility can be facilitated by the creases of the cover and / or based on its material composition. For example, the cover can be made of a flexible material that allows it to be folded.

[0244] In some folded configurations, the cover can protect portions of the housing from damage during shipping, such as portions of the housing including at least a portion of the key area and the non-key area (or at least a portion of the input area and the non-input area). In some embodiments, if the housing has multiple non-key areas, the cover may not cover all of the non-key areas. For example, in some embodiments, the cover may not extend over non-key areas located at the bottom edge of the key area, or may cover only a portion of the non-key area located at the bottom edge of the key area. In some embodiments, if the housing has multiple non-input areas, the cover may not cover all of the non-input areas. For example, in some embodiments, the cover may not extend over non-input areas located at the bottom edge of the input area, or may cover only a portion of the non-input area located at the bottom edge of the input area.

[0245] The cover can incorporate an image sensor. For example, an opening in the cover can expose a lens of the image sensor, the image sensor can be embedded within a layer of the cover (with an opening in at least one layer to expose the lens of the image sensor), the image sensor can be affixed to an exterior surface of the cover, or the image sensor can be otherwise configured to connect or associate with the cover.

[0246] In some embodiments, the protective cover may have a quadrilateral shape with one side connected to the housing. In some embodiments, the cover shape may not be a perfect quadrilateral. For example, the corners and / or edges of the cover may be rounded, chamfered, or beveled as a result of its manufacturing process. In some embodiments, the protective cover may have a shape similar to the top surface of the housing such that the protective cover completely covers the top surface of the housing, including covering at least a portion of the key area and non-key area (or including covering at least a portion of the input area and non-input area). In some embodiments, the cover may be configured to extend beyond one or more edges of the top surface of the housing such that the cover can at least partially wrap around one or more sides of the housing.

[0247] In some embodiments, the at least one image sensor may be positioned closer to a first side of the protective cover connected to the housing than to a second side of the protective cover opposite the first side. In some embodiments, the first side of the protective cover may be connected to the housing by being fixedly attached to the housing. For example, the first side of the protective cover may be attached or coupled to the housing by a hinge mechanism. A rotating hinge may be used, or the hinge may be made of a flexible plastic or fabric material. These are just a few examples.

[0248] Any type of hinge mechanism known in the art can be used. Although not required, in some embodiments, the hinge mechanism may be made of a flexible material, with a first edge of the flexible material fixedly attached to the housing and a second edge opposite the first edge fixedly attached to the protective cover. In some embodiments, the flexible material may include fabric, silicone, rubber, or other polymeric or elastomeric material. In some embodiments, the hinge mechanism may include one or more knuckles (e.g., rigid or semi-rigid loop structures) connected to and extending away from the protective cover, one or more knuckles connected to and extending away from the housing, and a pin inserted through all of the knuckles to form a hinge such that the protective cover can rotate or pivot about the pin. When assembled together before inserting the pin, the knuckles of the protective cover and the knuckles of the housing may be aligned or interleaved.

[0249] In some embodiments, the first side of the protective cover may be removably connected to the housing. For example, the first side of the protective cover may include one or more magnets, and the housing may include a ferromagnetic material that allows interaction with the magnets, such that the protective co...

Claims

1. 1. A non-transitory computer-readable medium for integrating a moveable input device with a virtual display projected through a wearable extended reality device, the computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to: receiving a movement signal associated with the moveable input device, the movement signal reflecting a physical movement of the moveable input device; outputting a first display signal to the wearable extended reality device during a first time period, the first display signal configured to cause the wearable extended reality device to virtually present content in a first orientation; outputting a second display signal to the wearable extended reality device during a second time period different from the first time period, the second display signal configured to cause the wearable extended reality device to virtually present the content in a second orientation different from the first orientation; and switching between the output of the first display signal and the output of the second display signal based on the received movement signal of the movable input device.

2. The non-transitory computer-readable medium of claim 1 , wherein the movement signal of the movable input device is determined based on an analysis of data captured with at least one sensor associated with the movable input device.

3. The non-transitory computer-readable medium of claim 1 , wherein the movement signal associated with the movable input device is determined based on an analysis of an image of the movable input device.

4. The non-transitory computer-readable medium of claim 1 , wherein the movement signal reflects a physical movement of the movable input device relative to a surface on which the movable input device is placed during the first period of time.

5. The non-transitory computer-readable medium of claim 1 , wherein the movement signal indicates at least one of tilt, roll, and lateral movement of the movable input device.

6. The non-transitory computer-readable medium of claim 1 , wherein the movement signal is received after the first period of time and before the second period of time.

7. 10. The non-transitory computer-readable medium of claim 1, wherein the instructions are configured to enable the wearable extended reality device to receive additional movement signals during the second period, thereby enabling the wearable extended reality device to continuously adjust the virtual presentation of the content.

8. The non-transitory computer-readable medium of claim 1 , wherein the steps further comprise determining the first orientation based on an orientation of the movable input device prior to the first period of time.

9. The non-transitory computer-readable medium of claim 1 , wherein the steps further comprise resizing the virtual display based on the received movement signals associated with the movable input device.

10. 2. The non-transitory computer-readable medium of claim 1, wherein the steps further comprise switching between the output of the first indication signal and the output of the second indication signal when the physical movement of the movable input device is greater than at least one threshold.

11. The non-transitory computer-readable medium of claim 10 , wherein the at least one threshold comprises a combination of a tilt threshold, a roll threshold, and a lateral motion threshold.

12. The non-transitory computer-readable medium of claim 10 , wherein the movable input device is configured to be placed on a surface, and wherein a value of the at least one threshold is based on a type of the surface.

13. 11. The non-transitory computer-readable medium of claim 10, wherein the at least one threshold is selected based on a distance of the virtual display from the movable input device during the first period of time.

14. 11. The non-transitory computer-readable medium of claim 10, wherein the at least one threshold is selected based on an orientation of the virtual display from the moveable input device during the first period of time.

15. The non-transitory computer-readable medium of claim 10 , wherein the at least one threshold is selected based on a type of the content.

16. 10. The non-transitory computer-readable medium of claim 1, wherein the wearable extended reality device is configured to be paired with a plurality of moveable input devices, and the first orientation is determined based on a default virtual display configuration associated with one of the plurality of moveable input devices paired with the wearable extended reality device.

17. The non-transitory computer-readable medium of claim 1 , wherein the content is a virtual display configured to enable a visual presentation of text input entered using the moveable input device.

18. 20. The non-transitory computer-readable medium of claim 17, further comprising providing a visual indication outside the virtual display of text input entered using the moveable input device when the virtual display is outside of a field of view of the wearable extended reality device.

19. 1. A method for integrating a moveable input device with a virtual display projected through a wearable extended reality device, comprising: receiving a movement signal associated with the moveable input device, the movement signal reflecting a physical movement of the moveable input device; outputting a first display signal to the wearable extended reality device during a first time period, the first display signal configured to cause the wearable extended reality device to virtually present content in a first orientation; outputting a second display signal to the wearable extended reality device during a second time period different from the first time period, the second display signal configured to cause the wearable extended reality device to virtually present the content in a second orientation different from the first orientation; and switching between the output of the first display signal and the output of the second display signal based on the received movement signal of the movable input device.

20. 1. A system for integrating a moveable input device with a virtual display projected through a wearable extended reality device, comprising: At least one processor, receiving a movement signal associated with the movable input device, the movement signal reflecting a physical movement of the movable input device; outputting a first display signal to the wearable extended reality device during a first time period, the first display signal configured to cause the wearable extended reality device to virtually present content in a first orientation; outputting a second display signal to the wearable extended reality device during a second time period different from the first time period, the second display signal configured to cause the wearable extended reality device to virtually present the content in a second orientation different from the first orientation; A system comprising at least one processor programmed to switch between the output of the first display signal and the output of the second display signal based on the received movement signal of the movable input device.

Images