User Interaction in Extended Reality

Abstract

A system, method, and non-transitory computer-readable medium for selectively controlling the display of a virtual object are provided. The virtual object may be virtually presented in an environment via a wearable extended reality device operable in a first and second display mode. In the first display mode, a position of the virtual object is maintained in the environment regardless of detected movement of the wearable extended reality device, and in the second display mode, the virtual object moves in the environment in response to detected movement of the wearable extended reality device. Movement of the wearable extended reality device may be detected. A selection of the first or second display mode may be received. A display signal configured to present the virtual object in a manner consistent with the selected display mode may be output for presentation via the wearable extended reality device in response to the selected display mode.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Provisional Patent Application No. 63 / 147,051 filed February 8, 2021, U.S. Provisional Patent Application No. 63 / 157,768 filed March 7, 2021, U.S. Provisional Patent Application No. 63 / 173,095 filed April 9, 2021, U.S. Provisional Patent Application No. 63 / 213,019 filed June 21, 2021, U.S. Provisional Patent Application No. 63 / 215,500 filed June 27, 2021, U.S. Provisional Patent Application No. 63 / 216,335 filed June 29, 2021, and U.S. Provisional Patent Application No. 63 / 216,335 filed June 29, 2021. This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 226,977, filed July 29, 2011, U.S. Provisional Patent Application No. 63 / 300,005, filed January 16, 2022, U.S. Provisional Patent Application No. 63 / 307,207, filed February 7, 2022, U.S. Provisional Patent Application No. 63,307,203, filed February 7, 2022, and U.S. Provisional Patent Application No. 63 / 307,217, filed February 7, 2022, all of which U.S. provisional patent applications are incorporated by reference in their entireties.

[0002] Technical Field The present disclosure relates generally to the field of extended reality, and more particularly to systems, methods, and devices for providing productivity applications using extended reality environments. [Background technology]

[0003] For many years, PC users have faced the productivity dilemma of either limiting mobility (when choosing a desktop computer) or limiting screen size (when choosing a laptop computer). One partial solution to this dilemma is the use of a docking station. A docking station is an interface device for connecting a laptop computer with other devices. By plugging a laptop computer into a docking station, a laptop user can enjoy the improved visibility provided by a larger monitor. However, because the large monitor is stationary, the user's mobility, while improved, is still limited. For example, even laptop users with docking stations do not have the flexibility to use two 32-inch screens wherever they want.

[0004] Some of the disclosed embodiments are directed to providing a new approach to solving productivity dilemmas, using Extended Reality (XR) to provide a mobile environment that allows users to experience the comfort of a stationary workspace wherever they want by providing a virtual desktop-like screen. Summary of the Invention

[0005] SUMMARY OF THE INVENTION Embodiments consistent with the present disclosure provide systems, methods, and devices for providing and supporting productivity applications using extended reality environments.

[0006] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for controlling a viewpoint in an extended reality environment using a physical touch controller. Some of these embodiments may involve outputting a first display signal reflecting a first viewpoint of a scene for presentation via a wearable extended reality device, receiving, via a touch sensor, a first input signal caused by a first multi-finger interaction with the touch sensor, and outputting, in response to the first input signal, a second display signal configured to cause a second viewpoint of the scene to be presented via the wearable extended reality device by modifying the first viewpoint of the scene, receiving, via the touch sensor, a second input signal caused by a second multi-finger interaction with the touch sensor, and outputting, in response to the second input signal, a third display signal configured to cause a third viewpoint of the scene to be presented via the wearable extended reality device by modifying the second viewpoint of the scene.

[0007] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for enabling gestural interaction with invisible virtual objects. Some of these embodiments may involve receiving image data captured by at least one image sensor of a wearable extended reality device, where the image data includes a representation of a plurality of physical objects within a field of view associated with the at least one image sensor of the wearable extended reality device; displaying a plurality of virtual objects in a portion of the field of view, where the portion of the field of view is associated with a display system of the wearable extended reality device; receiving a selection of a particular physical object from the plurality of physical objects; receiving a selection of a particular virtual object from the plurality of virtual objects for association with the particular physical object; docking the particular virtual object with the particular physical object; receiving a gesture input indicating that a hand is interacting with the particular virtual object when the particular physical object and the particular virtual object are outside a portion of the field of view associated with the display system of the wearable extended reality device such that the particular virtual object is invisible to a user of the wearable extended reality device; and triggering an output associated with the particular virtual object in response to the gesture input.

[0008] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for performing incremental convergence operations in an extended reality environment. Some of these embodiments may involve displaying a plurality of distributed virtual objects across a plurality of virtual regions, the plurality of virtual regions including at least a first virtual region and a second virtual region different from the first virtual region, receiving an initial motor input tending toward the first virtual region, highlighting a group of virtual objects in the first virtual region based on the initial motor input, receiving a refined motor input tending toward a particular virtual object from among the highlighted group of virtual objects, and triggering a function associated with the particular virtual object based on the refined motor input.

[0009] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for facilitating environmentally adaptive extended reality display in a physical environment. Some of these embodiments may involve virtually displaying content via a wearable extended reality device operating in the physical environment, where the displaying of the content via the wearable extended reality device is associated with at least one adjustable extended reality display parameter; acquiring image data from the wearable extended reality device; detecting in the image data certain environmental changes unrelated to the virtually displayed content; accessing a group of rules that associate the environmental changes with changes in the at least one adjustable extended reality display parameter; determining that the certain environmental change corresponds to a certain rule of the group of rules; and implementing the certain rule to adjust the at least one adjustable extended reality display parameter based on the certain environmental change.

[0010] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for selectively controlling the display of virtual objects. Some of these embodiments may involve virtually presenting a plurality of virtual objects in an environment via a wearable extended reality device operable in a first display mode and a second display mode, where in the first display mode, positions of the plurality of virtual objects are maintained in the environment regardless of detected movement of the wearable extended reality device, and in the second display mode, the plurality of virtual objects move in the environment in response to detected movement of the wearable extended reality device; detecting movement of the wearable extended reality device; receiving, while the wearable extended reality device is moving, a selection of the first display mode or the second display mode for use in virtually presenting the plurality of virtual objects; and outputting, in response to the selected display mode, a display signal configured to present the plurality of virtual objects for presentation via the wearable extended reality device in a manner consistent with the selected display mode.

[0011] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for moving a virtual cursor along two intersecting virtual planes. Some of these embodiments include generating a display via a wearable extended reality device, the display including a virtual cursor and a plurality of virtual objects located on a first virtual plane intersecting a second virtual plane above a physical surface; receiving a first two-dimensional input via a surface input device while the virtual cursor is displayed on the first virtual plane, the first two-dimensional input reflecting an intent to select the first virtual object located on the first virtual plane; and generating a second two-dimensional input directed toward the first virtual object in response to the first two-dimensional input. The method can include causing a first cursor movement, the first cursor movement along a first virtual plane; receiving a second two-dimensional input via the surface input device while the virtual cursor is displayed on the first virtual plane, the second two-dimensional input reflecting an intent to select a second virtual object that appears on the physical surface; and causing a second cursor movement toward the second virtual object in response to the second two-dimensional input, the second cursor movement including a partial movement along the first virtual plane and a partial movement along the second virtual plane.

[0012] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for enabling cursor control in an extended reality space. Some of these embodiments may involve receiving first image data from an image sensor reflecting a first focal region of a user of a wearable extended reality device, the first focal region being within an initial field of view in the extended reality space, causing a first presentation of a virtual cursor in the first focal region, receiving second image data from the image sensor reflecting a second focal region of the user outside the initial field of view in the extended reality space, receiving input data indicative of the user's desire to interact with the virtual cursor, and causing a second presentation of the virtual cursor in the second focal region in response to the input data.

[0013] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for moving virtual content between virtual planes in three-dimensional space. Some of these embodiments may involve using a wearable extended reality device to virtually present a plurality of virtual objects on a plurality of virtual planes, the plurality of virtual planes including a first virtual plane and a second virtual plane; outputting a first display signal for presentation via the wearable extended reality device reflecting the virtual object at a first position on the first virtual plane; receiving an in-plane input signal to move the virtual object to a second position on the first virtual plane; causing the wearable extended reality device to virtually display the in-plane movement of the virtual object from the first position to the second position in response to the in-plane input signal; receiving an inter-plane input signal to move the virtual object to a third position on the second virtual plane while the virtual object is at the second position; and causing the wearable extended reality device to virtually display the inter-plane movement of the virtual object from the second position to the third position in response to the inter-plane input signal.

[0014] Consistent with other disclosed embodiments, a non-transitory computer-readable storage medium can store program instructions that are executed by at least one processing device to perform any of the methods described herein.

[0015] The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the scope of the claims.

[0016] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a schematic diagram of a user using an example extended reality system consistent with the present disclosure. [Figure 2] 2 is a schematic diagram of major components of an example extended reality system of FIG. 1, consistent with some embodiments of the present disclosure. [Figure 3] FIG. 2 is a block diagram illustrating some of the components of an input unit, consistent with some embodiments of the present disclosure. [Figure 4] FIG. 2 is a block diagram illustrating some of the components of an extended reality unit, consistent with some embodiments of the present disclosure. [Figure 5] FIG. 2 is a block diagram illustrating some of the components of a remote processing unit, consistent with some embodiments of the present disclosure. [Figure 6] 10A-10C illustrate examples of various types of multi-finger interactions that may be used to change the viewpoint of a scene, consistent with some embodiments of the present disclosure. [Figure 7] 10 illustrates an exemplary set of changes in viewpoint of a scene consistent with some embodiments of the present disclosure. [Figure 8] 1 illustrates examples of different viewpoints of a scene, consistent with some embodiments of the present disclosure. [Figure 9]1 is a flowchart illustrating an exemplary process for changing the viewpoint of a scene, consistent with some embodiments of the present disclosure. [Figure 10] 1 is a flowchart illustrating an example process for enabling gestural interaction with invisible virtual objects, consistent with some embodiments of the present disclosure. [Figure 11] FIG. 1 is a schematic diagram showing a horizontal field of view, consistent with some embodiments of the present disclosure. [Figure 12] FIG. 1 is a schematic diagram showing a vertical field of view, consistent with some embodiments of the present disclosure. [Figure 13] 1 is a schematic diagram illustrating the use of an exemplary wearable extended reality device consistent with some embodiments of the present disclosure. [Figure 14] 1A-1C are schematic diagrams illustrating various usage snapshots of a system for enabling gestural interaction with invisible virtual objects, consistent with some embodiments of the present disclosure. [Figure 15] 1A-1C are schematic diagrams illustrating various usage snapshots of a system for enabling gestural interaction with invisible virtual objects, consistent with some embodiments of the present disclosure. [Figure 16] 1A-1C are schematic diagrams illustrating various usage snapshots of a system for enabling gestural interaction with invisible virtual objects, consistent with some embodiments of the present disclosure. [Figure 17] 1A-1C are schematic diagrams illustrating various usage snapshots of a system for enabling gestural interaction with invisible virtual objects, consistent with some embodiments of the present disclosure. [Figure 18] 1A-1C are schematic diagrams illustrating various usage snapshots of a system for enabling gestural interaction with invisible virtual objects, consistent with some embodiments of the present disclosure. [Figure 19] 1A-1C are schematic diagrams illustrating various usage snapshots of a system for enabling gestural interaction with invisible virtual objects, consistent with some embodiments of the present disclosure. [Figure 20]10 is a flowchart illustrating an example process for providing feedback based on incremental convergence in an extended reality environment, consistent with some embodiments of the present disclosure. [Figure 21] 1 illustrates an exemplary extended reality environment displaying multiple distributed virtual objects, consistent with some embodiments of the present disclosure. [Figure 22] 22 illustrates an example of a motion input provided by a user and the resulting feedback displayed within the extended reality environment of FIG. 21, consistent with some embodiments of the present disclosure. [Figure 23] 22 illustrates an example of a motion input provided by a user and the resulting feedback displayed within the extended reality environment of FIG. 21, consistent with some embodiments of the present disclosure. [Figure 24] 22 illustrates an example of a motion input provided by a user and the resulting movement of a virtual object within the extended reality environment of FIG. 21, consistent with some embodiments of the present disclosure. [Figure 25] 1 is a flowchart illustrating an example process for highlighting and de-highlighting a virtual object in response to a motion input, consistent with some embodiments of the present disclosure. [Figure 26] 1 is a flowchart illustrating an example process for determining a confidence level associated with a motor input, consistent with certain embodiments of the present disclosure. [Figure 27] 1 is a flowchart of an example method for facilitating an environmentally adaptive extended reality display in a physical environment, consistent with some embodiments of the present disclosure. [Figure 28] 1 is a flowchart of an exemplary method for facilitating an environmentally adaptive extended reality display in a physical environment in which a wearable extended reality device includes a distraction-free mode, consistent with some embodiments of the present disclosure. [Figure 29A]1 is an exemplary schematic diagram of a wearable extended reality device in use before an environmental change, consistent with some embodiments of the present disclosure. FIG. [Figure 29B] 1 is an exemplary schematic diagram of adjustment of at least one extended reality display parameter in response to environmental changes, consistent with some embodiments of the present disclosure. [Figure 29C] 10 is an exemplary schematic diagram of another adjustment of at least one extended reality display parameter in response to an environmental change, consistent with some embodiments of the present disclosure. [Figure 30] 1 illustrates an exemplary environment with multiple virtual objects and a wearable extended reality device, consistent with some embodiments of the present disclosure. [Figure 31A] 1 illustrates an exemplary environment with multiple virtual objects in a first display mode, consistent with some embodiments of the present disclosure. [Figure 31B] 1 illustrates an exemplary environment with multiple virtual objects in a second display mode, consistent with some embodiments of the present disclosure. [Figure 32] 1 illustrates an exemplary environment with multiple virtual objects, consistent with some embodiments of the present disclosure. [Figure 33] 1 illustrates an exemplary environment with multiple virtual objects, consistent with some embodiments of the present disclosure. [Figure 34] 1 illustrates an exemplary environment with multiple virtual objects and at least one virtual object docked to a physical object, consistent with some embodiments of the present disclosure. [Figure 35] 1 illustrates an exemplary environment with multiple virtual objects, consistent with some embodiments of the present disclosure. [Figure 36] 1 illustrates an exemplary environment with multiple virtual objects, consistent with some embodiments of the present disclosure. [Figure 37A] 1 illustrates an example method for selectively controlling the display of virtual objects consistent with certain embodiments of the present disclosure. [Figure 37B]1 illustrates an example method for selectively controlling the display of virtual objects consistent with certain embodiments of the present disclosure. [Figure 38] 1 illustrates an exemplary method for virtually presenting multiple virtual objects, consistent with some embodiments of the present disclosure. [Figure 39] 1 illustrates an example method in which a particular virtual object of a plurality of virtual objects presents the output of a software application, consistent with some embodiments of the present disclosure. [Figure 40] 1 illustrates an exemplary method in which image data is received from an image sensor and analyzed, consistent with some embodiments of the present disclosure. [Figure 41] 10 is a flowchart illustrating an example process for moving a virtual cursor along two intersecting virtual planes, consistent with certain embodiments of the present disclosure. [Figure 42] 1 is a schematic diagram illustrating the use of an exemplary wearable extended reality device consistent with some embodiments of the present disclosure. [Figure 43] 1A-1C are schematic diagrams illustrating various usage snapshots of an example system for moving a virtual cursor along two transverse virtual planes, consistent with some embodiments of the present disclosure. [Figure 44] 1A-1C are schematic diagrams illustrating various usage snapshots of an example system for moving a virtual cursor along two transverse virtual planes, consistent with some embodiments of the present disclosure. [Figure 45] 1A-1C are schematic diagrams illustrating various usage snapshots of an example system for moving a virtual cursor along two transverse virtual planes, consistent with some embodiments of the present disclosure. [Figure 46] 1A-1C are schematic diagrams illustrating various usage snapshots of an example system for moving a virtual cursor along two transverse virtual planes, consistent with some embodiments of the present disclosure. [Figure 47] 1A-1C are schematic diagrams illustrating various usage snapshots of an example system for moving a virtual cursor along two transverse virtual planes, consistent with some embodiments of the present disclosure. [Figure 48A] 1 is an illustrative schematic diagram of a user using an extended reality unit in a first configuration, consistent with some embodiments of the present disclosure. [Figure 48B] 10 is an illustrative schematic diagram of a user using an example extended reality unit in a second configuration, consistent with some embodiments of the present disclosure. [Figure 49] 1 is a flowchart illustrating an example method for enabling cursor control in an extended reality space, consistent with some embodiments of the present disclosure. [Figure 50A] 1 illustrates an exemplary extended reality display with a wearable extended reality device, multiple virtual objects, and multiple virtual planes, consistent with some embodiments of the present disclosure. [Figure 50B] 1 illustrates an exemplary extended reality display for moving virtual content between virtual planes as an in-plane movement, consistent with some embodiments of the present disclosure. [Figure 50C] 1 illustrates an exemplary extended reality display for moving virtual content between virtual planes as inter-plane movement, consistent with some embodiments of the present disclosure. [Figure 50D] 1 illustrates an exemplary extended reality display with multiple curved virtual planes, consistent with some embodiments of the present disclosure. [Figure 51] FIG. 1 is a block diagram illustrating an example method for moving virtual content between virtual planes in three-dimensional space, consistent with some embodiments of the present disclosure. [Figure 52] FIG. 10 is a block diagram illustrating an exemplary method in which a first virtual plane is associated with a first distance and a second virtual plane is associated with a second distance, consistent with some embodiments of the present disclosure. [Figure 53] 1 is a block diagram illustrating an example method consistent with some embodiments of the present disclosure. [Figure 54]FIG. 10 is a block diagram illustrating an exemplary method in which the first and second virtual planes are curved, consistent with some embodiments of the present disclosure. [Figure 55] 1 is a block diagram illustrating an example method in which at least one of an in-plane input signal and an inter-plane input signal is received from an input device, consistent with some embodiments of the present disclosure. [Figure 56] 1 is a block diagram illustrating an example method for receiving an auxiliary input signal, consistent with some embodiments of the present disclosure. [Figure 57] FIG. 10 is a block diagram illustrating an example method for determining whether a virtual object is docked to another virtual object, consistent with some embodiments of the present disclosure. [Figure 58] FIG. 10 is a block diagram illustrating an example method for receiving an input signal indicating a virtual object is to be docked to a physical object, consistent with some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0018] The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the following description to refer to the same or similar parts. While several exemplary embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, substitutions, additions, or changes may be made to the components illustrated in the drawings, and the exemplary methods described herein may be modified by substituting, rearranging, deleting, or adding steps to the disclosed methods. Therefore, the following detailed description is not limited to the particular embodiments and examples, but includes the general principles described herein and illustrated in the drawings, in addition to the general principles encompassed by the appended claims.

[0019] The present disclosure relates to systems and methods for providing users with extended reality environments. The term "extended reality environment," which may also be referred to as "extended reality," "extended reality space," or "extended environment," refers to all types of real-virtual composite environments and human-machine interactions that are generated at least in part by computer technology. An extended reality environment may be a fully simulated virtual environment or a combination of a real environment and a virtual environment that a user can perceive from different perspectives. In some examples, a user can interact with elements of the extended reality environment. One non-limiting example of an extended reality environment may be a virtual reality environment, also known as "virtual reality" or "virtual environment." An immersive virtual reality environment may be a simulated non-physical environment that provides a user with the perception of being in a virtual environment. Another non-limiting example of an extended reality environment may be an augmented reality environment, also known as "augmented reality" or "augmented environment." An augmented reality environment may include a live direct or indirect view of a physical real-world environment augmented with virtual computer-generated perceptual information, such as virtual objects, with which a user can interact. Another non-limiting example of an extended reality environment is a mixed reality environment, also known as "mixed reality" or "composite environment." A mixed reality environment may be a hybrid of a physical, real-world environment and a virtual environment, where physical and virtual objects can coexist and interact in real time. In some examples, both augmented reality and mixed reality environments can include a combination of real and virtual worlds, real-time interaction, and precise 3D registration of virtual and real objects. In some examples, both augmented reality and mixed reality environments can include constructive overlaid sensory information that can be added to the physical environment. In other examples, both augmented reality and mixed reality environments can include disruptive virtual content that can mask at least a portion of the physical environment.

[0020] In some embodiments, systems and methods can provide an extended reality environment using extended reality devices. The term extended reality device can include any type of device or system that allows a user to perceive and / or interact with an extended reality environment. An extended reality device can allow a user to perceive and / or interact with the extended reality environment through one or more sensory modalities. Some non-limiting examples of such sensory modalities can include vision, hearing, touch, somatosensation, and olfaction. One example of an extended reality device is a virtual reality device that allows a user to perceive and / or interact with a virtual reality environment. Another example of an extended reality device is an augmented reality device that allows a user to perceive and / or interact with an augmented reality environment. Yet another example of an extended reality device is a mixed reality device that allows a user to perceive and / or interact with a mixed reality environment.

[0021] According to one aspect of the present disclosure, the extended reality device may be a wearable device such as a head-mounted device, e.g., smart glasses, smart contact lenses, a headset, or any other device worn by a human to present an extended reality to the human. Other extended reality devices may include a holographic projector, or any other device or system capable of providing augmented reality (AR), virtual reality (VR), mixed reality (MR), or any immersive experience. Typical components of a wearable extended reality device may include a stereoscopic head-mounted display, a stereoscopic head-mounted sound system, a head motion tracking sensor (e.g., a gyroscope, an accelerometer, a magnetometer, an image sensor, a structured light sensor, etc.), a head-mounted projector, an eye-tracking sensor, and at least one of the additional components described below. According to another aspect of the present disclosure, the extended reality device may be a non-wearable extended reality device. Specifically, the non-wearable extended reality device may include a multi-projection environment device. In some embodiments, the extended reality device may be configured to change the perspective of the field of view of the extended reality environment in response to user movement, particularly in response to user head movement. In one example, a wearable extended reality device may change the field of view of the extended reality environment in response to changes in the user's head pose, such as by changing the spatial orientation without changing the user's spatial position in the extended reality environment. In another example, a non-wearable extended reality device may change the user's spatial position in the extended reality environment in response to changes in the user's position in the real world, for example, by changing the user's spatial position in the extended reality environment without changing the orientation of the field of view relative to the spatial position.

[0022] According to some embodiments, the extended reality device may include a digital communication device configured to at least one of receive virtual content data configured to enable presentation of virtual content, transmit virtual content for sharing with at least one external device, receive contextual data from at least one external device, transmit contextual data to at least one external device, transmit usage data indicative of use of the extended reality device, and transmit data based on information captured using at least one sensor included in the extended reality device. In further embodiments, the extended reality device may include a memory for storing at least one of virtual data configured to enable presentation of virtual content, the contextual data, usage data indicative of use of the extended reality device, sensor data based on information captured using at least one sensor included in the extended reality device, software instructions configured to cause a processing device to present the virtual content, software instructions configured to cause the processing device to collect and analyze the contextual data, software instructions configured to cause the processing device to collect and analyze the usage data, and software instructions configured to cause the processing device to collect and analyze the sensor data. In further embodiments, the extended reality device may include a processing device configured to perform at least one of rendering virtual content, collecting and analyzing contextual data, collecting and analyzing usage data, and collecting and analyzing sensor data. In further embodiments, the extended reality device may include one or more sensors.The one or more sensors may include one or more image sensors (e.g., configured to capture images and / or video of a user of the device or the user's environment), one or more motion sensors (e.g., accelerometers, gyroscopes, magnetometers, etc.), one or more positioning sensors (e.g., GPS, outdoor positioning sensors, indoor positioning sensors, etc.), one or more temperature sensors (e.g., configured to measure the temperature of at least a portion of the device and / or environment), one or more contact sensors, one or more proximity sensors (e.g., configured to detect whether the device is currently being worn), one or more electrical impedance sensors (e.g., configured to measure the electrical impedance of the user), one or more gaze tracking sensors such as a gaze detector, an optical tracker, an electric potential tracker (e.g., an electro-oculogram (EOG) sensor), a video-based gaze tracker, an infrared / near-infrared sensor, a passive optical sensor, or any other technology capable of determining where a human is looking or gazing.

[0023] In some embodiments, the systems and methods may use input devices to interact with the extended reality device. The term input device may include any physical device configured to receive input from a user or the user's environment and provide data to a computing device. The data provided to the computing device may be in digital and / or analog format. In one embodiment, the input device may store input received from a user in a memory device accessible by the processing device, and the processing device may access the stored data for analysis. In another embodiment, the input device may provide data directly to the processing device, for example, via a bus or another communication system configured to transfer data from the input device to the processing device. In some examples, input received by the input device may include key presses, tactile input data, movement data, position data, gesture-based input data, directional data, or any other data to feed into a computation. Some examples of input devices may include buttons, keys, keyboards, computer mice, touchpads, touchscreens, joysticks, or other mechanisms capable of receiving input. Another example of an input device may include an integrated computing interface device that includes at least one physical component for receiving input from a user. The integrated computing interface device may include at least a memory, a processing device, and at least one physical component for receiving input from a user. In one example, the integrated computing interface device may further include a digital network interface that enables digital communication with other computing devices. In one example, the integrated computing interface device may further include a physical component for outputting information to a user. In some examples, all components of the integrated computing interface device may be included in a single housing, while in other examples, the components may be distributed across two or more housings.Some non-limiting examples of physical components for receiving input from a user that may be included in an integrated computing interface device may include at least one of a button, a key, a keyboard, a touchpad, a touchscreen, a joystick, or any other mechanism or sensor capable of receiving computational information. Some non-limiting examples of physical components for outputting information to a user may include at least one of a light indicator (such as an LED indicator), a screen, a touchscreen, a buzzer, an audio speaker, or any other audio, video, or tactile device that provides a human-perceptible output.

[0024] In some embodiments, the image data may be captured using one or more image sensors. In some examples, the image sensors may be included in an extended reality device, a wearable device, a wearable extended reality device, an input device, the user's environment, etc. In some examples, the image data may be read from memory, received from an external device, generated (e.g., using a generative model), etc. Some non-limiting examples of image data may include an image, a grayscale image, a color image, a 2D image, a 3D image, a video, a 2D video, a 3D video, a frame, an image, data derived from other image data, etc. In some examples, the image data may be encoded in any analog or digital format. Some non-limiting examples of such formats may include raw formats, compressed formats, uncompressed formats, lossy formats, lossless formats, JPEG, GIF, PNG, TIFF, BMP, NTSC, PAL, SECAM, MPEG, MPEG-4 Part 14, MOV, WMV, FLV, AVI, AVCHD, WebM, MKV, etc.

[0025] In some embodiments, the extended reality device can receive a digital signal, for example, from an input device. The term digital signal refers to a time-discrete series of digital values. The digital signal may correspond to, for example, sensor data, text data, audio data, video data, virtual data, or any other form of data that provides perceptible information. Consistent with the present disclosure, the digital signal can be configured to cause the extended reality device to present virtual content. In one embodiment, the virtual content may be presented in a selected orientation. In this embodiment, the digital signal can indicate the position and angle of a viewpoint in an environment, such as an extended reality environment. Specifically, the digital signal can include encoding of a position and angle in six degrees of freedom (e.g., forward / back, up / down, left / right, yaw, pitch, and roll). In another embodiment, the digital signal can include encoding of a position as a three-dimensional coordinate (e.g., x, y, and z) and encoding of an angle as a vector resulting from the encoded position. Specifically, the digital signal may indicate the orientation and angle of the presented virtual content in absolute coordinates of the environment, for example, by encoding the yaw, pitch, and roll of the virtual content relative to a standard default angle. In another embodiment, the digital signal may indicate the orientation and angle of the presented virtual content relative to the viewpoint of another object (e.g., a virtual object, a physical object, etc.), for example, by encoding the yaw, pitch, and roll of the virtual content relative to a direction corresponding to the viewpoint or a direction corresponding to the other object. In another embodiment, such a digital signal may include one or more projections of the virtual content, for example, in a presentation-ready format (e.g., an image, a video, etc.). For example, each such projection may correspond to a particular orientation or a particular angle. In another embodiment, the digital signal may include a representation of the virtual content, for example, by encoding the object in a three-dimensional array of voxels, a polygonal mesh, or any other format in which the virtual content can be presented.

[0026] In some embodiments, the digital signal may be configured to cause the extended reality device to present virtual content. The term virtual content may include any type of data representation that may be displayed to a user by the extended reality device. Virtual content may include virtual objects, inanimate virtual content, animate virtual content configured to change over time or in response to a trigger, virtual two-dimensional content, virtual three-dimensional content, virtual overlays on portions of a physical environment or on physical objects, virtual additions to a physical environment or object, virtual promotional content, virtual representations of physical objects, virtual representations of a physical environment, virtual documents, virtual characters or personas, virtual computer screens, virtual widgets, or any other format for virtually displaying information. Consistent with this disclosure, virtual content may include any visual presentation rendered by a computer or processing device. In one embodiment, virtual content may include virtual objects, which are visual presentations rendered by a computer within a limited area and configured to represent a particular type of object (e.g., an inanimate virtual object, an animate virtual object, virtual furniture, virtual decorative object, virtual widget, or other virtual representation, etc.). The rendered visual presentation may change to reflect changes to a state object or a change in the viewing angle of the object, for example, to mimic a change in the appearance of a physical object. In another embodiment, the virtual content may include a virtual display (also referred to herein as a "virtual display screen" or "virtual screen"), such as a virtual computer screen, a virtual tablet screen, or a virtual smartphone screen, configured to display information generated by an operating system, where the operating system may be configured to receive text data from a physical keyboard and / or a virtual keyboard and cause the display of the text content on the virtual display screen. In one example, as shown in FIG. 1 , the virtual content may include a virtual environment including a virtual computer screen and a plurality of virtual objects.In some examples, the virtual display may be a virtual object that mimics and / or extends the functionality of a physical display screen. For example, the virtual display may be presented in an extended reality environment (e.g., a mixed reality environment, an augmented reality environment, a virtual reality environment, etc.) using an extended reality device. In one example, the virtual display may present content generated by a conventional operating system that can be equivalently presented on a physical display screen. In one example, text content entered using a keyboard (e.g., using a physical keyboard, using a virtual keyboard, etc.) may be presented on the virtual display in real time as the text content is entered. In one example, a virtual cursor may be presented on the virtual display, and the virtual cursor may be controlled by a pointing device (a physical pointing device, a virtual pointing device, a computer mouse, a joystick, a touchpad, a physical touch controller, etc.). In one example, one or more windows of a graphical user interface operating system may be presented on the virtual display. In another example, the content presented on the virtual display may be interactive, i.e., may change in response to a user's actions. In yet another example, the presentation of the virtual display may or may not include the presentation of a screen frame.

[0027] Some disclosed embodiments may include and / or access a data structure or database. Consistent with this disclosure, the terms data structure and database may include any collection of data values ​​and the relationships between them. Data may be stored linearly, horizontally, hierarchically, relationally, non-relationally, unidimensionally, multidimensionally, operationally, orderedly, unorderedly, object-orientedly, centrally, decentralized, distributedly, decentralized, custom, or in any manner that enables data access. By way of non-limiting example, data structures may include arrays, associative arrays, linked lists, binary trees, balanced trees, mountains, stacks, queues, sets, hash tables, records, tagged unions, entity-relationship models, graphs, hypergraphs, matrices, tensors, etc. For example, the data structure may include an XML database, an RDBMS database, an SQL database, or a NoSQL alternative for data storage / retrieval, such as, for example, MongoDB, Redis, Couchbase, Datastax Enterprise Graph, Elastic Search, Splunk, Solr, Cassandra, Amazon DynamoDB, Scylla, HBase, and Neo4J. The data structure may be a component of the disclosed system or a remote computing component (e.g., a cloud-based data structure). Data within the data structure may be stored in contiguous or non-contiguous memory. Furthermore, the data structure does not require that the information be co-located. The data structure may be distributed, for example, across multiple servers that may be owned or operated by the same or different entities. Thus, the singular term "data structure" includes multiple data structures.

[0028] In some embodiments, the system can determine a confidence level for a received input or any determined value. The term confidence level refers to a numerical value or any other indication of a level (e.g., within a predetermined range) that indicates the amount of confidence the system has in the determined data. For example, the confidence level may have a value between 1 and 10. Alternatively, the confidence level may be displayed as a percentage or any other numerical or non-numerical indication. In some cases, the system can compare the confidence level to a threshold. The term threshold can refer to a reference value, level, point, or range of values. During operation, if the confidence level of the determined data exceeds the threshold (either above or below, depending on the particular use case), the system can follow a first course of action; if the confidence level is below the threshold (either above or above, depending on the particular use case), the system can follow a second course of action. The value of the threshold may be predetermined for each type of test object or may be dynamically selected based on different considerations.

[0029] System Overview Reference is now made to FIG. 1 , which illustrates a user using an exemplary extended reality system consistent with various embodiments of the present disclosure. It should be understood that FIG. 1 is an exemplary representation of just one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure. As shown, a user 100 is seated behind a table 102 and supports a keyboard 104 and a mouse 106. The keyboard 104 is connected by wire 108 to a wearable extended reality device 110 that displays virtual content to the user 100. Alternatively, or in addition to wire 108, the keyboard 104 may be wirelessly connected to the wearable extended reality device 110. For purposes of illustration, the wearable extended reality device is shown as a pair of smart glasses, but as previously mentioned, the wearable extended reality device 110 may be any type of head-mounted device used to present extended reality to the user 100. The virtual content displayed by the wearable extended reality device 110 includes a virtual screen 112 (also referred to herein as a “virtual display screen” or “virtual display”) and multiple virtual widgets 114. Virtual widgets 114A-114D are displayed adjacent to the virtual screen 112, and virtual widget 114E is displayed on the table 102. The user 100 can use the keyboard 104 to enter text into a document 116 displayed on the virtual screen 112 and can use the mouse 106 to control a virtual cursor 118. In one example, the virtual cursor 118 can move anywhere within the virtual screen 112. In another example, the virtual cursor 118 can move anywhere within the virtual screen 112 and can move to any of the virtual widgets 114A-114D, but not to virtual widget 114E. In yet another example, the virtual cursor 118 can move anywhere within the virtual screen 112 and can move to any of the virtual widgets 114A-114E.In a further example, virtual cursor 118 can move anywhere in the extended reality environment, including virtual screen 112 and virtual widgets 114A-114E. In yet another example, virtual cursor 118 can move over all available surfaces (i.e., virtual or physical surfaces) or only over selected surfaces in the extended reality environment. Alternatively or additionally, user 100 can interact with any of virtual widgets 114A-114E, or with a selected virtual widget, using hand gestures recognized by wearable extended reality device 110. For example, virtual widget 114E can be an interactive widget (e.g., a virtual slider controller) that can be manipulated with hand gestures.

[0030] FIG. 2 illustrates an example of a system 200 for providing an extended reality (XR) experience to a user, such as user 100. It should be understood that FIG. 2 is an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of this disclosure. System 200 may be computer-based and may include computer system components, wearable devices, workstations, tablets, handheld computing devices, memory devices, and / or an internal network connecting the components. System 200 may include or be connected to various network computing resources (e.g., servers, routers, switches, network connections, storage devices, etc.) to support the services provided by system 200. Consistent with this disclosure, system 200 may include an input unit 202, an XR unit 204, a mobile communication device 206, and a remote processing unit 208. Remote processing unit 208 may include a server 210 coupled to one or more physical or virtual storage devices, such as data structure 212. The system 200 may also include or be connected to a communication network 214 that facilitates communication and data exchange between different system components and different entities associated with the system 200 .

[0031] Consistent with the present disclosure, the input unit 202 may include one or more devices capable of receiving input from the user 100. In one embodiment, the input unit 202 may include a text input device such as the keyboard 104. The text input device may include all possible types of devices and mechanisms for inputting text information into the system 200. Examples of text input devices may include a mechanical keyboard, a membrane keyboard, a flexible keyboard, a QWERTY keyboard, a Dvorak keyboard, a Colemak keyboard, a corded keyboard, a wireless keyboard, a keypad, a key-based control panel, or other arrangement of control keys, a visual input device, or any other mechanism for inputting text, regardless of whether the mechanism is provided in physical form or presented virtually. In one embodiment, the input unit 202 may also include a pointing input device such as the mouse 106. The pointing input device may include all possible types of devices and mechanisms for inputting two-dimensional or three-dimensional information into the system 200. In one example, two-dimensional input from the pointing input device may be used to interact with virtual content presented via the XR unit 204. Examples of pointing input devices include a computer mouse, trackball, touchpad, trackpad, touchscreen, joystick, pointing stick, stylus, light pen, or any other physical or virtual input mechanism. In one embodiment, input unit 202 may also include a graphical input device, such as a touchscreen, configured to detect contact, movement, or interruption of movement. The graphical input device may use any of a number of touch-sensitivity technologies, including, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact. In one embodiment, input unit 202 may also include one or more audio input devices, such as a microphone.The audio input device may include all possible types of devices and mechanisms for inputting audio data to facilitate voice-enabled functions such as voice recognition, audio replication, digital recording, and telephone functions. In one embodiment, the input unit 202 may also include one or more image input devices, such as an image sensor, configured to capture image data. In one embodiment, the input unit 202 may also include one or more haptic gloves configured to capture hand movement and posture data. In one embodiment, the input unit 202 may also include one or more proximity sensors configured to detect the presence and / or movement of objects within a selected area near the sensor.

[0032] According to some embodiments, the system may include at least one sensor configured to detect and / or measure characteristics related to a user, the user's behavior, or the user's environment. One example of the at least one sensor is the sensor 216 included in the input unit 202. The sensor 216 may be a motion sensor, a touch sensor, a light sensor, an infrared sensor, an audio sensor, an image sensor, a proximity sensor, a positioning sensor, a gyroscope, a temperature sensor, a biometric sensor, or any other sensing device for facilitating related functions. The sensor 216 may be integrated with, connected to, or separate from the input device. In one example, the mouse 106 may include a thermometer to determine the user's 100 body temperature. In another example, a positioning sensor may be integrated with the keyboard 104 to determine the user's 100 movement relative to the keyboard 104. Such a positioning sensor may be implemented using one of the following positioning technologies: Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), Galileo Global Navigation System, BeiDou Navigation System, other Global Navigation Satellite System (GNSS), Indian Regional Navigation Satellite System (IRNSS), Local Positioning System (LPS), Real-Time Location System (RTLS), Indoor Positioning System (IPS), Wi-Fi based positioning system, cellular triangulation, image-based positioning technology, indoor positioning technology, outdoor positioning technology, or any other positioning technology.

[0033] According to some embodiments, the system may include one or more sensors for determining the position and / or movement of a physical device (e.g., a physical input device, a physical computing device, a keyboard 104, a mouse 106, a wearable extended reality device 110, etc.). The one or more sensors may be included in the physical device or may be external to the physical device. In some examples, an image sensor external to the physical device (e.g., an image sensor included in another physical device) may be used to capture image data of the physical device, and the image data may be analyzed to determine the position and / or movement of the physical device. For example, the image data may be analyzed using a visual object tracking algorithm to determine the movement of the physical device, a visual object detection algorithm to determine the position of the physical device (e.g., relative to the image sensor, such as in a global coordinate system), etc. In some examples, an image sensor included in the physical device may be used to capture image data, and the image data may be analyzed to determine the position and / or movement of the physical device. For example, the image data may be analyzed using a visual odometry algorithm to determine the position of the physical device, an egomotion algorithm to determine the movement of the physical device, etc. In some examples, a positioning sensor, such as an indoor positioning sensor or an outdoor positioning sensor, may be included in the physical device and used to determine the position of the physical device. In some examples, a motion sensor, such as an accelerometer or gyroscope, may be included in the physical device and used to determine the movement of the physical device. In some examples, a physical device, such as a keyboard or mouse, may be configured to be placed on a physical surface. Such a physical device may include an optical mouse sensor (also known as a non-mechanical tracking engine) pointed at the physical surface, and the output of the optical mouse sensor may be analyzed to determine the movement of the physical device relative to the physical surface.

[0034] Consistent with this disclosure, the XR unit 204 may include a wearable extended reality device configured to present virtual content to the user 100. An example of a wearable extended reality device is the wearable extended reality device 110. Further examples of wearable extended reality devices may include virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices, or any other device capable of generating extended reality content. Some non-limiting examples of such devices may include Nreal Light, Magic Leap One, Varjo, Quest 1 / 2, Vive, etc. In some embodiments, the XR unit 204 may present virtual content to the user 100. In general, extended reality devices may include all real and virtual composite environments and human-machine interactions generated by computer technology and wearables. As previously mentioned, the term “extended reality” (XR) refers to a superset that includes the entire spectrum from “fully real” to “fully virtual.” It includes representative forms such as augmented reality (AR), mixed reality (MR), virtual reality (VR), and regions interpolated therebetween. Accordingly, it should be noted that the terms "XR device," "AR device," "VR device," and "MR device" can be used interchangeably herein and can refer to any of the various devices listed above.

[0035] Consistent with the present disclosure, the system can exchange data with various communication devices associated with the user, such as a mobile communication device 206. The term “communication device” is intended to include all possible types of devices capable of exchanging data using a digital communication network, an analog communication network, or any other communication network configured to transmit data. In some examples, the communication device may include a smartphone, a tablet, a smartwatch, a personal digital assistant, a desktop computer, a laptop computer, an IoT device, a dedicated terminal, a wearable communication device, and any other device capable of data communication. In some cases, the mobile communication device 206 can supplement or replace the input unit 202. Specifically, the mobile communication device 206 may be associated with a physical touch controller that can function as a pointing input device. Furthermore, the mobile communication device 206 may also be used, for example, to implement a virtual keyboard and replace a text input device. For example, the user 100 may walk away from the table 102 and into the break room with their smart glasses and receive an email that requires a quick response. In this case, the user may choose to use their smartwatch as an input device and enter a response to the email while it is virtually presented by the smartglasses.

[0036] Consistent with the present disclosure, system embodiments may include the use of a cloud server. The term “cloud server” refers to a computer platform that provides services over a network, such as the Internet. In the exemplary embodiment shown in FIG. 2 , server 210 may use a virtual machine that may not correspond to individual hardware. For example, computing and / or storage capabilities may be implemented by allocating an appropriate portion of desired computing / storage power from a scalable repository, such as a data center or a distributed computing environment. Specifically, in one embodiment, remote processing unit 208 may be used in conjunction with XR unit 204 to provide virtual content to user 100. In one configuration, server 210 may be a cloud server that functions as an operating system (OS) for a wearable extended reality device. In one example, server 210 may implement the methods described herein using customized hardwired logic, one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), firmware, and / or program logic that, in combination with a computer system, makes server 210 a dedicated machine.

[0037] In some embodiments, server 210 can access data structure 212 to determine, for example, virtual content for display to user 100. Data structure 212 can utilize volatile or non-volatile, magnetic, semiconductor, tape, optical, removable, non-removable, or other types of storage devices or tangible or non-transitory computer-readable media, or any medium or mechanism for storing information. Data structure 212 can be part of server 210, as shown, or can be separate from server 210. If data structure 212 is not part of server 210, server 210 can exchange data with data structure 212 via a communications link. Data structure 212 can include one or more memory devices that store data and instructions used to execute one or more features of the disclosed methods. In one embodiment, data structure 212 can include any of several suitable data structures, from a small data structure hosted on a workstation to a large data structure distributed across a data center. Data structure 212 can also include any combination of one or more data structures controlled by a memory controller device (e.g., a server) or software.

[0038] Consistent with this disclosure, a communications network may be any type of network (including infrastructure) that supports communications, exchanges information, and / or facilitates the exchange of information between components of a system. For example, communications network 214 in system 200 may include, for example, a telephone network, an extranet, an intranet, the Internet, satellite communications, offline communications, wireless communications, transponder communications, a local area network (LAN), a wireless network (e.g., a Wi-Fi / 302.11 network), a wide area network (WAN), a virtual private network (VPN), a digital communications network, an analog communications network, or any other mechanism or combination of mechanisms that enables data transmission.

[0039] The components and arrangement of system 200 shown in FIG. 2 are intended to be illustrative only and not limiting of any embodiment, as the system components used to implement the disclosed processes and features may vary.

[0040] FIG. 3 is a block diagram illustrating an example configuration of input unit 202. It should be understood that FIG. 3 is an exemplary representation of just one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure. In the embodiment of FIG. 3, input unit 202 may directly or indirectly access bus 300 (or other communication mechanism) that interconnects subsystems and components for transferring information within input unit 202. For example, bus 300 may interconnect memory interface 310, network interface 320, input interface 330, power supply 340, output interface 350, processing device 360, sensor interface 370, and database 380.

[0041] The memory interface 310 shown in FIG. 3 can be used to access software products and / or data stored on a non-transitory computer-readable medium. Generally, a non-transitory computer-readable storage medium refers to any type of physical memory capable of storing information or data readable by at least one processor. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, CD-ROMs, DVDs, flash drives, disks, any other optical data storage media, any physical media with a pattern of holes, PROMs, EPROMs, FLASH-EPROMs or any other flash memory, NVRAMs, caches, registers, any other memory chips or cartridges, and networked versions thereof. The terms "memory" and "computer-readable storage medium" can refer to multiple structures, such as multiple memories or computer-readable storage media located within an input unit or at a remote location. Furthermore, one or more computer-readable storage media can be utilized in practicing a computer-implemented method. Accordingly, the term computer-readable storage medium should be understood to include tangible items and exclude carrier waves and transient signals. 3, memory interface 310 may be used to access software products and / or data stored on a memory device, such as memory device 311. Memory device 311 may include high-speed random access memory and / or non-volatile memory, such as one or more magnetic disk storage devices, one or more optical storage devices, and / or flash memory (e.g., NAND, NOR). Consistent with the present disclosure, components of memory device 311 may be distributed across multiple units and / or multiple memory devices of system 200.

[0042] The memory device 311 shown in FIG. 3 may include software modules for executing processes consistent with the present disclosure. In particular, the memory device 311 may include an input determination module 312, an output determination module 313, a sensor communication module 314, a virtual content determination module 315, a virtual content communication module 316, and a database access module 317. Modules 312-317 may include software instructions for execution by at least one processor (e.g., processing device 360) associated with the input unit 202. The input determination module 312, the output determination module 313, the sensor communication module 314, the virtual content determination module 315, the virtual content communication module 316, and the database access module 317 may cooperate to perform various operations. For example, the input determination module 312 may determine text using data received from, for example, the keyboard 104. The output determination module 313 may then cause the presentation of the recently entered text on, for example, a dedicated display 352 physically or wirelessly coupled to the keyboard 104. In this way, as the user 100 types, they can see a preview of the entered text without having to constantly move their head up and down to view the virtual screen 112. The sensor communication module 314 can receive data from different sensors to determine the state of the user 100. The virtual content determination module 315 can then determine virtual content to display based on the received input and the determined status of the user 100. For example, the determined virtual content may be a virtual presentation of recently entered text on a virtual screen positioned virtually adjacent to the keyboard 104. The virtual content communication module 316 may retrieve virtual content not determined by the virtual content determination module 315 (e.g., another user's avatar). The retrieval of virtual content may be from the database 380, from the remote processing unit 208, or from any other source.

[0043] In some embodiments, the input determination module 312 can adjust the operation of the input interface 330 to receive pointer input 331, text input 332, voice input 333, and XR-related input 334. Details of pointer input, text input, and voice input are described above. The term “XR-related input” can include any type of data that can cause a change in virtual content displayed to the user 100. In one embodiment, the XR-related input 334 can include image data of the user 100, the wearable extended reality device (e.g., a detected hand gesture of the user 100), or other data. In another embodiment, the XR-related input 334 can include wireless communication indicating the presence of another user in proximity to the user 100. Consistent with the present disclosure, the input determination module 312 can simultaneously receive different types of input data. The input determination module 312 can then further apply different rules based on the type of input detected. For example, pointer input may be prioritized over voice input.

[0044] In some embodiments, the output determination module 313 can coordinate operation of the output interface 350 to generate output using light indicators 351, display 352, and / or speaker 353. Generally, the output generated by the output determination module 313 does not include virtual content presented by the wearable extended reality device. Instead, the output generated by the output determination module 313 includes various outputs related to the operation of the input unit 202 and / or the operation of the XR unit 204. In one embodiment, the light indicator 351 can include a light indicator that indicates the status of the wearable extended reality device. For example, the light indicator can display a green light when the wearable extended reality device 110 is connected to the keyboard 104 and blink when the battery of the wearable extended reality device 110 is low. In another embodiment, the display 352 can be used to display operational information. For example, the display can present an error message when the wearable extended reality device is inoperable. In another embodiment, speaker 353 may be used to output sound, for example, if user 100 wishes to play music for other users.

[0045] In some embodiments, the sensor communications module 314 can coordinate the operation of the sensor interface 370 to receive sensor data from one or more sensors integrated with or connected to the input device. The one or more sensors can include an audio sensor 371, an image sensor 372, a motion sensor 373, an environmental sensor 374 (e.g., a temperature sensor, an ambient light detector, etc.), and other sensors 375. In one embodiment, the data received from the sensor communications module 314 can be used to determine the physical orientation of the input device. The physical orientation of the input device can indicate the user's state and can be determined based on a combination of tilt, roll, and lateral movement. The physical orientation of the input device can then be used by the virtual content determination module 315 to modify display parameters of the virtual content to match the user's state (e.g., alert, sleepy, active, sitting, standing, leaning back, leaning forward, walking, moving, riding, etc.).

[0046] In some embodiments, the virtual content determination module 315 can determine virtual content to be displayed by the wearable extended reality device. The virtual content can be determined based on data from the input determination module 312, the sensor communication module 314, and other sources (e.g., database 380). In some embodiments, determining the virtual content can include determining the distance, size, and orientation of a virtual object. The position of the virtual object can be determined based on the type of the virtual object. Specifically, with respect to the example shown in FIG. 1 , the virtual content determination module 315 can determine to place four virtual widgets 114A-114D on both sides of the virtual screen 112 and place the virtual widget 114E on the table 102 because the virtual widget 114E is a virtual controller (e.g., a volume bar). The position of the virtual object can be further determined based on a user's preference. For example, for a left-handed user, the virtual content determination module 315 may determine to place the virtual volume bar to the left of the keyboard 104, and for a right-handed user, the virtual content determination module 315 may determine to place the virtual volume bar to the right of the keyboard 104.

[0047] In some embodiments, the virtual content communication module 316 may coordinate operation of the network interface 320 to obtain data from one or more sources to be presented as virtual content to the user 100. The one or more sources may include other XR units 204, the user's mobile communication device 206, the remote processing unit 208, publicly available information, etc. In one embodiment, the virtual content communication module 316 may communicate with the mobile communication device 206 to provide a virtual display of the mobile communication device 206. For example, the virtual display may allow the user 100 to read messages and interact with applications installed on the mobile communication device 206. The virtual content communication module 316 may also coordinate operation of the network interface 320 to share virtual content with other users. In one example, the virtual content communication module 316 can use data from the input determination module to identify triggers (e.g., triggers may include user gestures) and transfer content from the virtual display to a physical display (e.g., a TV) or to a different user's virtual display.

[0048] In some embodiments, database access module 317 can cooperate with database 380 to retrieve stored data. The retrieved data can include, for example, privacy levels associated with different virtual objects, relationships between virtual and physical objects, user preferences, user past behavior, etc. As previously described, virtual content determination module 315 can determine virtual content using data stored in database 380. Database 380 can include separate databases, including, for example, a vector database, a raster database, a tile database, a viewport database, and / or a user input database. Data stored in database 380 can be received from modules 314-317 or other components of system 200. Additionally, data stored in database 380 can be provided as input using data entry, data transfer, or data upload.

[0049] Modules 312-317 may be implemented in software, hardware, firmware, any combination thereof, etc. In some embodiments, any one or more of modules 312-317 and data associated with database 380 may be stored in XR unit 204, mobile communication device 206, or remote processing unit 208. A processing device of system 200 may be configured to execute instructions of modules 312-317. In some embodiments, aspects of modules 312-317 may be implemented in hardware, software (including one or more signal processing and / or application specific integrated circuits), firmware, or any combination thereof, executable by one or more processors, alone or in various combinations with each other. In particular, modules 312-317 may be configured to interact with each other and / or other modules of system 200 to perform functions consistent with some disclosed embodiments. For example, input unit 202 may execute instructions including image processing algorithms on data from XR unit 204 to determine head movement of user 100. Furthermore, throughout this specification, each function described with respect to input unit 202 or with respect to components of input unit 202 can correspond to a set of instructions for performing that function. These instructions need not be implemented as a separate software program, procedure, or module. Memory device 311 can include additional or fewer modules and instructions. For example, memory device 311 can store an operating system such as ANDROID, iOS, UNIX, OSX, WINDOWS, DARWIN, RTXC, LINUX, or an embedded operating system such as VXWorkS. The operating system can include instructions for handling basic system services and performing hardware-dependent tasks.

[0050] The network interface 320 shown in FIG. 3 can provide two-way data communication to a network, such as communications network 214. In one embodiment, network interface 320 can include an Integrated Services Digital Network (ISDN) card, a cellular modem, a satellite modem, or a modem to provide a data communication connection over the Internet. As another example, network interface 320 can include a wireless local area network (WLAN) card. In another embodiment, network interface 320 can include an Ethernet port connected to a radio frequency receiver and transmitter and / or an optical (e.g., infrared) receiver and transmitter. The specific design and implementation of network interface 320 can depend on the communications network over which input unit 202 is intended to operate. For example, in some embodiments, input unit 202 can include a network interface 320 designed to operate over a GSM network, a GPRS network, an EDGE network, a Wi-Fi or WiMax network, and a Bluetooth network. In any such implementation, network interface 320 can be configured to send and receive electrical, electromagnetic, or optical signals that carry digital data streams or signals representing various types of information.

[0051] 3 can receive input from a variety of input devices, such as a keyboard, a mouse, a touchpad, a touchscreen, one or more buttons, a joystick, a microphone, an image sensor, and any other device configured to detect physical or virtual input. The received input may be in the form of at least one of text, voice, speech, hand gestures, body gestures, tactile information, and any other type of physical or virtual input generated by a user. In the illustrated embodiment, input interface 330 can receive pointer input 331, text input 332, voice input 333, and XR-related input 334. In further embodiments, input interface 330 may be an integrated circuit that can function as a bridge between processing device 360 ​​and any of the above input devices.

[0052] The power source 340 shown in FIG. 3 can provide electrical energy to the power input unit 202 and, optionally, to the XR unit 204. In general, the power source included in any device or system of the present disclosure may be any device capable of repeatedly storing, distributing, or transmitting electrical power, including, but not limited to, one or more batteries (e.g., lead-acid, lithium-ion, nickel-metal hydride, nickel-cadmium batteries), one or more capacitors, one or more connections to external power sources, one or more power converters, or any combination thereof. Referring to the example shown in FIG. 3 , the power source may be mobile, meaning that the input unit 202 can be easily carried by hand (e.g., the total weight of the power source 340 is ). The mobility of the power source allows the user 100 to use the input unit 202 in a variety of situations. In other embodiments, the power source 340 may be associated with a connection to an external power source (such as a power grid) that can be used to charge the power source 340. Additionally, the power source 340 may be configured to charge one or more batteries included in the XR unit 204, such that a pair of extended reality glasses (e.g., wearable extended reality devices 110) can be charged (e.g., wirelessly or non-wirelessly) when they are placed on or near the input unit 202.

[0053] 3 can trigger output from various output devices using, for example, light indicators 351, a display 352, and / or a speaker 353. In one embodiment, output interface 350 can be an integrated circuit that can act as a bridge between processing device 360 ​​and at least one of the above output devices. Light indicator 351 can include one or more light sources, such as, for example, an array of LEDs associated with different colors. Display 352 can include a screen (e.g., an LCD or dot matrix screen) or a touchscreen. Speaker 353 can include audio headphones, a hearing aid-type device, a speaker, bone conduction headphones, an interface that provides tactile cues, a vibration stimulator, etc.

[0054] The processing device 360 ​​shown in FIG. 3 may include at least one processor configured to execute computer programs, applications, methods, processes, or other software to perform embodiments described herein. Generally, the processing device included in any device or system of the present disclosure may include all or part of one or more integrated circuits, microchips, microcontrollers, microprocessors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), or other circuitry suitable for executing instructions or performing logical operations. The processing device may include at least one processor configured to perform the functions of the disclosed methods, such as a microprocessor manufactured by Intel™. The processing device may include a single-core or multiple-core processor that simultaneously performs parallel processing. In one example, the processing device may be a single-core processor configured with virtual processing technology. The processing device may implement virtual machine technology or other technologies to provide the ability to run, control, execute, manipulate, store, etc., multiple software processes, applications, programs, etc. In another example, a processing device may include a multi-core processor configuration (e.g., dual, quad-core, etc.) configured to provide parallel processing capabilities that allow devices associated with the processing device to execute multiple processes simultaneously. It should be understood that other types of processor configurations can be implemented to provide the functionality disclosed herein.

[0055] 3 can acquire sensor data from various sensors, such as an audio sensor 371, an image sensor 372, a motion sensor 373, an environmental sensor 374, and other sensors 375. In one embodiment, the sensor interface 370 can be an integrated circuit that can act as a bridge between the processing device 360 ​​and at least one of the above sensors.

[0056] Audio sensor 371 may include one or more audio sensors configured to capture audio by converting the audio into digital information. Some examples of audio sensors include a microphone, a unidirectional microphone, a bidirectional microphone, a cardioid microphone, an omnidirectional microphone, an on-board microphone, a wired microphone, a wireless microphone, or any combination of the above. Consistent with the present disclosure, processing device 360 ​​can alter the presentation of virtual content based on data received from audio sensor 371 (e.g., voice commands).

[0057] Image sensor 372 may include one or more image sensors configured to capture visual information by converting light into image data. Consistent with this disclosure, the image sensor may be included in any device or system of this disclosure and may be any device capable of detecting and converting optical signals in the near-infrared, infrared, visible, and ultraviolet spectrums into electrical signals. Examples of image sensors may include digital cameras, phone cameras, semiconductor charge-coupled devices (CCDs), complementary metal-oxide semiconductor (CMOS) or N-type metal-oxide semiconductor (NMOS, live MOS) active pixel sensors. The electrical signals may be used to generate image data. Consistent with this disclosure, image data may include pixel data streams, digital images, digital video streams, data derived from captured images, and data that may be used to construct one or more 3D images, a sequence of 3D images, a 3D video, or a virtual 3D display. Image data acquired by image sensor 372 may be transmitted to any processing device of system 200 via wired or wireless transmission. For example, the image data may be processed to detect objects, detect events, detect actions, detect faces, detect people, recognize known people, or any other information that may be used by system 200. Consistent with this disclosure, processing device 360 ​​may alter the presentation of virtual content based on the image data received from image sensor 372.

[0058] The motion sensor 373 may include one or more motion sensors configured to measure the movement of the input unit 202 or the movement of objects in the environment of the input unit 202. Specifically, the motion sensor may perform at least one of: detecting the movement of objects in the environment of the input unit 202, measuring the velocity of objects in the environment of the input unit 202, measuring the acceleration of objects in the environment of the input unit 202, detecting the movement of the input unit 202, measuring the velocity of the input unit 202, measuring the acceleration of the input unit 202, etc. In some embodiments, the motion sensor 373 may include one or more accelerometers configured to detect changes in and / or measure the appropriate acceleration of the input unit 202. In other embodiments, the motion sensor 373 may include one or more gyroscopes configured to detect changes in the orientation of the input unit 202 and / or measure information related to the orientation of the input unit 202. In other embodiments, the motion sensor 373 may include one or more using an image sensor, a LIDAR sensor, a radar sensor, or a proximity sensor. For example, by analyzing the captured images, the processing device can determine the movement of the input unit 202, e.g., using an egomotion algorithm. Additionally, the processing device can determine the movement of objects in the environment of the input unit 202, e.g., using an object tracking algorithm. Consistent with this disclosure, the processing device 360 ​​can modify the presentation of virtual content based on the determined movement of the input unit 202 or the determined movement of objects in the environment of the input unit 202, e.g., causing a virtual display to follow the movement of the input unit 202.

[0059] The environmental sensors 374 may include one or more sensors of different types configured to capture data reflective of the input unit 202's environment. In some embodiments, the environmental sensors 374 may include one or more chemical sensors configured to at least one of: measure a chemical property in the input unit 202's environment; measure a change in a chemical property in the input unit 202's environment; detect the presence of a chemical in the input unit 202's environment; or measure a concentration of a chemical in the input unit 202's environment. Examples of such chemical properties may include pH level, toxicity, and temperature. Examples of such chemicals may include electrolytes, certain enzymes, certain hormones, certain proteins, smoke, carbon dioxide, carbon monoxide, oxygen, ozone, hydrogen, and hydrogen sulfide. In other embodiments, the environmental sensors 374 may include one or more temperature sensors configured to detect a change in temperature of the input unit 202's environment and / or measure the temperature of the input unit 202's environment. In other embodiments, the environmental sensors 374 may include one or more barometers configured to detect a change in air pressure in the input unit 202's environment and / or measure the air pressure in the input unit 202's environment. In other embodiments, environmental sensor 374 may include one or more light sensors configured to detect changes in ambient light in the environment of input unit 202. Consistent with this disclosure, processing device 360 ​​may modify the presentation of virtual content based on input from environmental sensor 374. For example, automatically reducing the brightness of virtual content when the environment of user 100 becomes dark.

[0060] The other sensors 375 may include weight sensors, optical sensors, resistive sensors, ultrasonic sensors, proximity sensors, biometric sensors, or other sensing devices to facilitate related functions. In particular embodiments, the other sensors 375 may include one or more positioning sensors configured to obtain positioning information for the input unit 202, detect changes in the position of the input unit 202, and / or measure the position of the input unit 202. Alternatively, the GPS software may enable the input unit 202 to access an external GPS receiver (e.g., connecting via a serial port or Bluetooth). Consistent with the present disclosure, the processing device 360 ​​may alter the presentation of virtual content based on input from the other sensors 375. For example, presenting personal information only after identifying the user 100 using data from a biometric sensor.

[0061] The components and arrangement shown in FIG. 3 are not intended to limit any embodiment. Numerous variations and / or modifications can be made to the illustrated configuration of input unit 202, as will be understood by those skilled in the art having the benefit of this disclosure. For example, not all components are essential for the operation of the input unit. Any component may be located in any suitable portion of the input unit, and components may be rearranged in various configurations while still providing the functionality of various embodiments. For example, some input units may not include all of the elements shown in input unit 202.

[0062] FIG. 4 is a block diagram illustrating an example configuration of the XR unit 204. It should be understood that FIG. 4 is an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of this disclosure. In the embodiment of FIG. 4, the XR unit 204 may directly or indirectly access a bus 400 (or other communication mechanism) that interconnects subsystems and components for transferring information within the XR unit 204. For example, the bus 400 may interconnect a memory interface 410, a network interface 420, an input interface 430, a power supply 440, an output interface 450, a processing device 460, a sensor interface 470, and a database 480.

[0063] The memory interface 410 shown in FIG. 4 is assumed to have functionality similar to that of the memory interface 310 described in detail above. The memory interface 410 can be used to access software products and / or data stored on a non-transitory computer-readable medium or memory device, such as the memory device 411. The memory device 411 can include software modules for performing processes consistent with the present disclosure. In particular, the memory device 411 can include an input determination module 412, an output determination module 413, a sensor communication module 414, a virtual content determination module 415, a virtual content communication module 416, and a database access module 417. The modules 412-417 can include software instructions for execution by at least one processor (e.g., processing device 460) associated with the XR unit 204. The input determination module 412, the output determination module 413, the sensor communication module 414, the virtual content determination module 415, the virtual content communication module 416, and the database access module 417 can cooperate to perform various operations. For example, the input determination module 412 can determine user interface (UI) input received from the input unit 202. At the same time, the sensor communication module 414 can receive data from different sensors to determine the state of the user 100. The virtual content determination module 415 can determine virtual content to display based on the received input and the determined state of the user 100. The virtual content communication module 416 can search for virtual content not determined by the virtual content determination module 415. The search for virtual content may be from the database 380, the database 480, the mobile communication device 206, or the remote processing unit 208. Based on the output of the virtual content determination module 415, the output determination module 413 can cause a change in the virtual content displayed to the user 100 by the projector 454.

[0064] In some embodiments, the input determination module 412 can coordinate operation of the input interface 430 to receive gesture input 431, virtual input 432, voice input 433, and UI input 434. Consistent with the present disclosure, the input determination module 412 can simultaneously receive different types of input data. In one embodiment, the input determination module 412 can apply different rules based on the type of input detected. For example, gesture input may be prioritized over virtual input. In some embodiments, the output determination module 413 can coordinate operation of the output interface 450 to generate output using a light indicator 451, a display 452, a speaker 453, and a projector 454. In one embodiment, the light indicator 451 can include a light indicator that indicates the status of the wearable extended reality device. For example, the light indicator may display a green light when the wearable extended reality device 110 is connected to the input unit 202 and flash when the battery of the wearable extended reality device 110 is low. In another embodiment, the display 452 can be used to display operational information. In another embodiment, the speaker 453 may include bone conduction headphones used to output audio to the user 100. In another embodiment, the projector 454 may present virtual content to the user 100.

[0065] The operation of the sensor communication module, virtual content determination module, virtual content communication module, and database access module has been described above with reference to Figure 3, and the details thereof will not be repeated here. Modules 412-417 may be implemented in software, hardware, firmware, any combination thereof, etc.

[0066] 4 is assumed to have functionality similar to that of network interface 320 described in detail above. The specific design and implementation of network interface 420 may depend on the communication network in which XR unit 204 is intended to operate. For example, in some embodiments, XR unit 204 is configured to be selectively connectable by wire to input unit 202. When connected by wire, network interface 420 may enable communication with input unit 202, and when not connected by wire, network interface 420 may enable communication with mobile communication device 206.

[0067] 4 is assumed to have functionality similar to that of the input interface 330 described in detail above. In this case, the input interface 430 may communicate with an image sensor to obtain gesture input 431 (e.g., a finger of the user 100 pointing at a virtual object), with another XR unit 204 to obtain virtual input 432 (e.g., a virtual object shared with the XR unit 204 or a gesture of an avatar detected in the virtual environment), with a microphone to obtain voice input 433 (e.g., voice commands), and with the input unit 202 to obtain UI input 434 (e.g., virtual content determined by the virtual content determination module 315).

[0068] 4 is assumed to have functionality similar to that of power supply 340 described above, providing electrical energy to power XR unit 204. In some embodiments, power supply 440 may be charged by power supply 340. For example, power supply 440 may be changed wirelessly when XR unit 204 is placed on or near input unit 202.

[0069] 4 is assumed to have functionality similar to that of output interface 350 described in detail above. In this case, output interface 450 can trigger output from light indicator 451, display 452, speaker 453, and projector 454. Projector 454 may be any device, apparatus, fixture, etc. that can project (or direct) light to display virtual content onto a surface. The surface may be part of XR unit 204, part of the eye of user 100, or part of an object in proximity to user 100. In one embodiment, projector 454 includes an illumination unit that focuses light into a limited solid angle by one or more mirrors and lenses to provide a high value of luminous intensity in a defined direction.

[0070] 4 is assumed to have functionality similar to that of processing device 360 ​​described in detail above. When XR unit 204 is connected to input unit 202, processing device 460 may cooperate with processing device 360. In particular, processing device 460 may implement virtual machine technology or other technology to provide the capability to run, control, execute, manipulate, store, etc., multiple software processes, applications, programs, etc. It should be understood that other types of processor configurations can be implemented to provide the capabilities disclosed herein.

[0071] The sensor interface 470 shown in Figure 4 is assumed to have functionality similar to that of the sensor interface 370 described in detail above. Specifically, the sensor interface 470 can communicate with an audio sensor 471, an image sensor 472, a motion sensor 473, an environmental sensor 474, and other sensors 475. The operation of the audio, image, motion, environmental, and other sensors was described above with reference to Figure 3, and the details will not be repeated here. It is understood that other types and combinations of sensors may be used to provide the capabilities disclosed herein.

[0072] The components and arrangement shown in FIG. 4 are not intended to limit any embodiment. As will be understood by one of ordinary skill in the art having the benefit of this disclosure, numerous variations and / or modifications may be made to the configuration of the XR unit 204 shown. For example, not all components are essential to the operation of the XR unit 204 in all cases. Any component may be located in any suitable part of the system 200, and components may be rearranged in various configurations while still providing the functionality of various embodiments. For example, some XR units may not include all elements in the XR unit 204 (e.g., the wearable extended reality device 110 may not have a light indicator 451).

[0073] FIG. 5 is a block diagram illustrating an example configuration of remote processing unit 208. It should be understood that FIG. 5 is an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of this disclosure. In the embodiment of FIG. 5, remote processing unit 208 may include server 210 that directly or indirectly accesses bus 500 (or other communication mechanism) that interconnects subsystems and components for transferring information within server 210. For example, bus 500 may interconnect memory interface 510, network interface 520, power supply 540, processing device 560, and database 580. Remote processing unit 208 may also include one or more data structures. For example, data structures 212A, 212B, and 212C.

[0074] The memory interface 510 shown in FIG. 5 is assumed to have functionality similar to that of memory interface 310, described in detail above. Memory interface 510 can be used to access software products and / or data stored on a non-transitory computer-readable medium or in other memory devices, such as memory devices 311, 411, 511, or data structures 212A, 212B, and 212C. Memory device 511 can include software modules for performing processes consistent with the present disclosure. In particular, memory device 511 can include a shared memory module 512, a node registration module 513, a load balancing module 514, one or more compute nodes 515, an internal communications module 516, an external communications module 517, and a database access module (not shown). Modules 512-517 can include software instructions for execution by at least one processor (e.g., processing device 560) associated with remote processing unit 208. The shared memory module 512, the node registration module 513, the load balancing module 514, the computation module 515, and the external communication module 517 may cooperate to perform various operations.

[0075] Shared memory module 512 may enable information sharing between remote processing unit 208 and other components of system 200. In some embodiments, shared memory module 512 may be configured to enable processing device 560 (and other processing devices in system 200) to access, retrieve, and store data. For example, using shared memory module 512, processing device 560 may perform at least one of the following steps: execute software programs stored in memory device 511, database 580, or data structures 212A-C; store information in memory device 511, database 580, or data structures 212A-C; or retrieve information from memory device 511, database 580, or data structures 212A-C.

[0076] The node registration module 513 can be configured to track the availability of one or more computing nodes 515. In some examples, the node registration module 513 may be implemented as a software program, such as a software program executed by one or more computing nodes 515, a hardware solution, or a combined software and hardware solution. In some implementations, the node registration module 513 can communicate with one or more computing nodes 515, for example, using the internal communication module 516. In some examples, one or more computing nodes 515 can inform the node registration module 513 of their status by sending a message, for example, upon startup, upon shutdown, at regular intervals, at selected times, in response to a query received from the node registration module 513, or at any other determined time. In some examples, the node registration module 513 can inquire about the status of one or more computing nodes 515 by sending a message, for example, upon startup, at regular intervals, at selected times, or at any other determined time.

[0077] The load balancing module 514 can be configured to divide the workload among one or more compute nodes 515. In some examples, the load balancing module 514 may be implemented as a software program, such as a software program executed by one or more of the compute nodes 515, a hardware solution, or a combined software and hardware solution. In some implementations, the load balancing module 514 can interact with the node registration module 513 to obtain information regarding the availability of one or more compute nodes 515. In some implementations, the load balancing module 514 can communicate with one or more compute nodes 515, for example, using the internal communications module 516. In some examples, one or more compute nodes 515 can inform the load balancing module 514 of their status, for example, by sending a message upon startup, upon shutdown, at regular intervals, at selected times, in response to a query received from the load balancing module 514, or at any other determined time. In some examples, the load balancing module 514 may inquire about the status of one or more compute nodes 515 by sending a message, for example, at startup, at regular intervals, at preselected times, or at any other determined time.

[0078] Internal communications module 516 may be configured to receive and / or transmit information from one or more components of remote processing unit 208. For example, control and / or synchronization signals may be transmitted and / or received via internal communications module 516. In one embodiment, computer program input information, computer program output information, and / or computer program intermediate information may be transmitted and / or received via internal communications module 516. In another embodiment, information received via internal communications module 516 may be stored in memory device 511, database 580, data structures 212A-C, or other memory devices within system 200. For example, information obtained from data structure 212A may be transmitted using internal communications module 516. In another example, input data may be received using internal communications module 516 and stored in data structure 212B.

[0079] The external communication module 517 may be configured to receive and / or transmit information from one or more components of the system 200. For example, control signals may be transmitted and / or received via the external communication module 517. In one embodiment, information received via the external communication module 517 may be stored in the memory device 511, the database 580, the data structures 212A-C, and / or any memory device within the system 200. In another embodiment, information retrieved from any of the data structures 212A-C may be transmitted to the XR unit 204 using the external communication module 517. In another embodiment, input data may be transmitted and / or received using the external communication module 517. Examples of such input data may include data received from the input unit 202, information captured from the environment of the user 100 using one or more sensors (e.g., the audio sensor 471, the image sensor 472, the motion sensor 473, the environmental sensor 474, the other sensors 475), etc.

[0080] In some embodiments, aspects of modules 512-517 may be implemented in hardware, software (including one or more signal processing and / or application specific integrated circuits), firmware, or any combination thereof, executable by one or more processors, alone or in various combinations with each other. In particular, modules 512-517 may be configured to interact with each other and / or with other modules of system 200 to perform functions consistent with embodiments of the present disclosure. Memory device 511 may include additional or fewer modules and instructions.

[0081] 5 are assumed to have functionality similar to that of similar elements described above with reference to FIGS. 4 and 5. The specific design and implementation of the aforementioned components may vary based on the implementation of system 200. Additionally, remote processing unit 208 may include more or fewer components. For example, remote processing unit 208 may include an input interface configured to receive input directly from one or more input devices.

[0082] Consistent with the present disclosure, a processing device of system 200 (e.g., a processor in mobile communication device 206, a processor in server 210, a processor in a wearable extended reality device such as wearable extended reality device 110, and / or a processor in an input device associated with wearable extended reality device 110 such as keyboard 104) can use machine learning algorithms to perform any of the methods disclosed herein. In some embodiments, the machine learning algorithms (also referred to in this disclosure as machine learning models) can be trained using training examples, for example, as described below. Some non-limiting examples of such machine learning algorithms may include classification algorithms, data regression algorithms, image segmentation algorithms, visual detection algorithms (e.g., object detectors, face detectors, person detectors, motion detectors, edge detectors, etc.), visual recognition algorithms (e.g., face recognition, person recognition, object recognition, etc.), speech recognition algorithms, mathematical embedding algorithms, natural language processing algorithms, support vector machines, random forests, nearest neighbor algorithms, deep learning algorithms, artificial neural network algorithms, convolutional neural network algorithms, recurrent neural network algorithms, linear machine learning models, nonlinear machine learning models, ensemble algorithms, etc. For example, the trained machine learning algorithms may include predictive models, classification models, data regression models, clustering models, segmentation models, inference models such as artificial neural networks (e.g., deep neural networks, convolutional neural networks, recurrent neural networks, etc.), random forests, support vector machines, etc. In some examples, the training examples may include example inputs and desired outputs corresponding to the example inputs. Furthermore, in some examples, a machine learning algorithm trained using training examples can generate a trained machine learning algorithm, which can be used to estimate outputs for inputs not included in the training examples.In some examples, engineers, scientists, processes, and machines that train machine learning algorithms can further use validation examples and / or test examples. For example, the validation examples and / or test examples can include example inputs along with desired outputs corresponding to the example inputs, and the trained machine learning algorithm and / or an intermediately trained machine learning algorithm can be used to estimate outputs for the example inputs of the validation examples and / or test examples, and the estimated outputs can be compared with the corresponding desired outputs, and the trained machine learning algorithm and / or an intermediately trained machine learning algorithm can be evaluated based on the results of the comparison. In some examples, the machine learning algorithm can have parameters and hyperparameters, and the hyperparameters can be set manually by a person or automatically by a process external to the machine learning algorithm (e.g., a hyperparameter search algorithm), and the parameters of the machine learning algorithm can be set by the machine learning algorithm based on the training examples. In some implementations, the hyperparameters can be set based on the training examples and validation examples, and the parameters can be set based on the training examples and selected hyperparameters. For example, given the hyperparameters, the parameters can be conditionally independent of the validation examples.

[0083] In some embodiments, a trained machine learning algorithm (also referred to in this disclosure as a machine learning model and a trained machine learning model) can be used to analyze inputs and generate outputs, for example, as described below. In some examples, a trained machine learning algorithm can be used as an inference model that generates an inferred output when provided with an input. For example, the trained machine learning algorithm can include a classification algorithm, where the input can include samples and the inferred output can include a classification of the samples (e.g., an inferred label, an inferred tag, etc.). In another example, the trained machine learning algorithm can include a regression model, where the input can include samples and the inferred output can include an inferred value corresponding to the samples. In yet another example, the trained machine learning algorithm can include a clustering model, where the input can include samples and the inferred output can include an assignment of the samples to at least one cluster. In a further example, the trained machine learning algorithm can include a classification algorithm, where the input can include images and the inferred output can include a classification of an item depicted in the images. In yet another example, the trained machine learning algorithm may include a regression model, the input may include an image, and the inferred output may include an inferred value corresponding to an item depicted in the image (e.g., an estimated characteristic of the item, such as the size, volume, age, or distance of a person depicted in the image from the item depicted in the image). In a further example, the trained machine learning algorithm may include an image segmentation model, the input may include an image, and the inferred output may include a segmentation of the image. In yet another example, the trained machine learning algorithm may include an object detector, the input may include an image, and the inferred output may include one or more detected objects in the image and / or one or more locations of the objects in the image.In some examples, the trained machine learning algorithm may include one or more formulas and / or one or more functions and / or one or more rules and / or one or more procedures, inputs may be used as inputs to the formulas and / or functions and / or rules and / or procedures, and the inferred output may be based on the output of the formulas and / or functions and / or rules and / or procedures (e.g., selecting one of the outputs of the formulas and / or functions and / or rules and / or procedures, using statistical measures of the outputs of the formulas and / or functions and / or rules and / or procedures, etc.).

[0084] Consistent with the present disclosure, the processing device of system 200 can analyze image data captured by an image sensor (e.g., image sensor 372, image sensor 472, or any other image sensor) to perform any of the methods disclosed herein. In some embodiments, analyzing the image data can include analyzing the image data to obtain preprocessed image data and then analyzing the image data and / or the preprocessed image data to obtain a desired result. As one skilled in the art will appreciate, the following are examples, and the image data can be preprocessed using other types of preprocessing methods. In some examples, the image data can be preprocessed by transforming the image data using a transformation function to obtain transformed image data, where the preprocessed image data can include the transformed image data. For example, the transformed image data can include one or more convolutions of the image data. For example, the transformation function can include one or more image filters, such as a low-pass filter, a high-pass filter, a band-pass filter, an all-pass filter, etc. In some examples, the transformation function can include a nonlinear function. In some examples, the image data may be preprocessed by smoothing at least a portion of the image data, for example, using a Gaussian convolution, using a median filter, etc. In some examples, the image data may be preprocessed to obtain different representations of the image data. For example, the preprocessed image data may include a representation of at least a portion of the image data in the frequency domain, a discrete Fourier transform of at least a portion of the image data, a discrete wavelet transform of at least a portion of the image data, a time / frequency representation of at least a portion of the image data, a reduced dimensional representation of at least a portion of the image data, a lossy representation of at least a portion of the image data, a lossless representation of at least a portion of the image data, a time series of any of the above, any combination of the above, etc. In some examples, the image data may be preprocessed to extract edges, and the preprocessed image data may include information based on and / or related to the extracted edges. In some examples, the image data may be preprocessed to extract image features from the image data.Some non-limiting examples of such image features may include information based on and / or related to edges, corners, blobs, ridges, Scale Invariant Feature Transform (SIFT) features, temporal features, etc. In some examples, analyzing the image data may include computing at least one convolution of at least a portion of the image data and using the computed at least one convolution to compute at least one resultant value and / or to determine, identify, recognize, classify, etc.

[0085] Consistent with another aspect of the present disclosure, the processing device of system 200 can analyze image data to perform any of the methods disclosed herein. In some embodiments, analyzing the image can include analyzing the image data and / or preprocessed image data using one or more rules, functions, procedures, artificial neural networks, object detection algorithms, face detection algorithms, visual event detection algorithms, action detection algorithms, motion detection algorithms, background subtraction algorithms, inference models, etc. Some non-limiting examples of such inference models can include results on training examples of training algorithms, such as manually pre-programmed inference models, classification models, regression models, machine learning algorithms, and / or deep learning algorithms, where the training examples can include example data instances, and in some cases, the data instances can be labeled with corresponding desired labels and / or results, etc. In some embodiments, analyzing the image data (e.g., by methods, steps, and modules described herein) can include analyzing pixels, voxels, point clouds, range data, etc. included in the image data.

[0086] Convolutions may include convolutions of any dimension. A one-dimensional convolution is a function that transforms an original sequence into a transformed sequence. A one-dimensional convolution may be defined by a series of scalars. Each particular value in the transformed sequence may be determined by calculating a linear combination of values ​​in a subsequence of the original sequence that corresponds to a particular value. The resulting value of the calculated convolution may include any value in the transformed sequence. Similarly, an n-dimensional convolution is a function that transforms an original n-dimensional array into a transformed array. An n-dimensional convolution may be defined by an n-dimensional array of scalars (known as the kernel of the n-dimensional convolution). Each particular value in the transformed array may be determined by calculating a linear combination of values ​​in an n-dimensional region of the original array that corresponds to a particular value. The resulting value of the calculated convolution may include any value in the transformed array. In some examples, an image may include one or more components (e.g., color components, depth components, etc.), and each component may include a two-dimensional array of pixel values. In one example, calculating the convolution of an image may include calculating a two-dimensional convolution for one or more components of the image. In another example, computing the convolution of an image can include stacking arrays from different components to create a three-dimensional array and computing the three-dimensional convolution on the resulting three-dimensional array. In some examples, a video can include one or more components (e.g., color components, depth components, etc.), where each component can include a three-dimensional array of pixel values ​​(having two spatial axes and one temporal axis). In one example, computing the convolution of a video can include computing the three-dimensional convolution on one or more components of the video. In another example, computing the convolution of a video can include stacking arrays from different components to create a four-dimensional array and computing the four-dimensional convolution on the resulting four-dimensional array.

[0087] When using a wearable extended reality device, there may be a desire to change the perspective, such as by zooming in on an object or by changing the virtual orientation of the object. It may be desirable to enable this type of control via a touch controller that interacts with the wearable extended reality device, such that pinching or rotating a finger on a touch sensor changes the perspective of the scene. The following disclosure describes various systems, methods, and non-transitory computer-readable media for controlling the viewpoint in an extended reality environment using a physical touch controller.

[0088] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for controlling a viewpoint in an extended reality environment using a physical touch controller. The term non-transitory is intended to describe computer-readable storage media other than propagating electromagnetic signals, but is not intended to limit the types of physical computer-readable storage devices encompassed by the phrase computer-readable media. For example, the term non-transitory computer-readable media is intended to encompass types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored on a tangible computer-accessible storage medium in non-transitory form may further be transmitted by a transmission medium or signal, such as an electrical, electromagnetic, or digital signal, that may be conveyed over a communication medium, such as a network and / or a wireless link. Non-transitory computer-readable media may include magnetic disks, cards, tapes, drums, punch cards, paper tape, optical disks, bar codes, and magnetic ink characters, or any other tangible medium capable of storing data in a format readable by a mechanical device.

[0089] Controlling the viewpoint as used in this disclosure can include, for example, changing the position of a scene at a fixed distance, changing the distance of a scene, changing the angle of a scene, changing the size of a scene, causing a rotation of a scene, or any other manipulation of a scene that can cause a change in how an observer can visualize the scene. In one example, controlling the viewpoint can include changing a spatial transformation of a scene. Some non-limiting examples of spatial transformations can include a translation transformation, a rotation transformation, a reflection transformation, an expansion transformation, an affine transformation, a projective transformation, etc. In one example, controlling the viewpoint can include changing a first spatial transformation of a first portion of a scene and changing a second spatial transformation of a second portion of the scene (wherein the second spatial transformation can be different from the first spatial transformation, the changes to the second spatial transformation can be different from the changes to the first spatial transformation, and / or the second portion of the scene can be different from the first portion of the scene). A scene may include, for example, a location, place, position, point, spot, location, arena, stage, set, trajectory, section, segment, part, clip, sequence, animate or inanimate object, person, or any other component of a field of view. Changing the position of a scene by a fixed distance may include, for example, translating the scene horizontally, vertically, diagonally, rotationally, or any other direction within the three-dimensional field. Changing the distance of a scene may include, for example, increasing or decreasing the amount of space between the scene and any reference point horizontally, vertically, diagonally, rotationally, or any other direction within the three-dimensional field.

[0090] Changing the angle of a scene can include, for example, increasing or decreasing the slope, tilt, incline, or any other inclination between two reference points. Changing the size of a scene can include, for example, enlarging, increasing, widening, expanding, elongating, increasing, inflating, stretching, making larger, swelling, widening, amplifying, spreading out, stretching, expanding, stretching, contracting, decreasing, making smaller, lessening, narrowing, lowering, shortening, compressing, or shrinking one or more dimensions of any component, components, or combination of components in the scene. Other scene manipulations can include any visual alteration of the scene relative to the original scene.

[0091] A physical touch controller may include a device that enables manipulation of information on a display through detection of finger contact with a surface or finger movement on a surface. Any reference herein to a finger may equally apply to any other finger, pen, road, or other object. Any reference herein to finger contact and / or movement may equally apply to contact and / or movement of any other object, such as any other finger, pen, road, or multiple such objects (e.g., in a multi-touch surface). A physical touch controller may be implemented via software, hardware, or any combination thereof. In some embodiments, a physical touch controller may include, for example, one or more computer chips, one or more circuit boards, one or more electronic circuits, or any combination thereof. Additionally or alternatively, a physical touch controller may include, for example, a storage medium having executable instructions adapted to communicate with touch sensors. In particular embodiments, a physical touch controller may be connected to other components, such as sensors and a processor, using wired or wireless systems. For example, a touch controller may be implemented through pressure sensing of a finger (or finger surrogate) on a touch-sensitive surface or through an optical sensor that detects touch or finger movement. In some examples, the physical touch controller may enable manipulation of information on the display via at least one of a touch sensor, a touchpad, a trackpad, a touchscreen, and the like.

[0092] Some disclosed embodiments of computer-readable media may include instructions that, when executed by at least one processor, cause the at least one processor to perform one or more functions. Instructions may include any sequence, command, instruction, program, code, or condition that indicates how something should be done, operated, or processed. The processor that executes the instructions may be a general-purpose computer, but may also utilize any of a wide variety of other technologies, including, for example, a special-purpose computer, a microcomputer, a minicomputer, a mainframe computer, a programmed microprocessor, a microcontroller, a peripheral integrated circuit element, a CSIC (Customer Specific Integrated Circuit), an ASIC (Application Specific Integrated Circuit), a logic circuit, a digital signal processor, a programmable logic device such as an FPGA (Field Programmable Gate Array), a PLD (Programmable Logic Device), a PLA (Programmable Logic Array), an RFID integrated circuit, a smart chip, or any other device or arrangement of devices capable of performing the steps or processes of the present disclosure. In some embodiments, the at least one processor may comprise multiple processors that need not be physically co-located. The processors may be in separate geographic locations and may be communicatively connected to each other in any suitable manner. In some embodiments, each of the multiple processors may be comprised of a different physical piece of equipment.

[0093] Instructions embodied in a non-transitory computer-readable medium may include various instructions that cause a processor to perform one or more specific tasks. Such a set of instructions for performing a specific task may be characterized as, for example, a program, software program, software, engine, module, component, mechanism, unit, or tool. Some embodiments may include multiple software processing modules stored in memory and executed on at least one processor. The program modules may be in the form of any suitable programming language that can be converted into machine code or object code to enable at least one processor to read the instructions. Programming languages ​​used may include assembly language, Ada, APL, Basic, C, C++, COBOL, dBase, Forth, FORTRAN, Java, Modula-2, Pascal, Prolog, REXX, Visual Basic, JavaScript, or any other set of characters that generates machine code output.

[0094] Consistent with certain disclosed embodiments, a non-transitory computer-readable medium may include instructions that, when executed by at least one processor, cause the at least one processor to output a first display signal reflecting a first perspective of a scene for presentation via a wearable extended reality device. The output may include one or more of generating, delivering, or providing data. The output may be presented in the form of a graphical representation, for example, including one or more still or moving images, text, icons, video, or any combination thereof. The graphical representation may be two-dimensional, three-dimensional, holographic, or may include various other types of visual characteristics. The at least one processor may generate and transmit one or more analog or digital signals to a display device for presenting the graphical representation for viewing by a user. In some embodiments, the display device may include a wearable extended reality device. As described elsewhere in this disclosure, the wearable extended reality device may implement, for example, augmented reality technology, virtual reality technology, mixed reality technology, any combination thereof, or any other technology capable of combining a real environment with a virtual environment. The wearable extended reality device may take the form of, for example, a hat, visor, helmet, goggles, glasses, or any other object that can be worn. A perspective of a scene, as used in this disclosure, may include, for example, an orientation, angle, size, direction, position, aspect, spatial transformation, or any other virtual characteristic of a scene or a portion of a scene. At least one processor may be configured to cause presentation of a graphical representation including a first perspective of the scene. By way of example, as shown in FIG. 7 , the processor may be configured to cause presentation of a first perspective of a scene 710.

[0095] Some disclosed embodiments may include receiving, via a touch sensor, a first input signal caused by a first multi-finger interaction with the touch sensor. The touch sensor may additionally or alternatively include, for example, a piezoresistive sensor, a piezoelectric sensor, an optical sensor, a capacitive sensor, an elasto-resistive sensor, or any other type of sensor that determines information based on physical interaction. In some embodiments, the touch sensor may be incorporated into a keyboard, a mouse, a control rod, an optical or trackball mouse, or any other type of tactical input device. In some embodiments, the touch sensor may include a touch screen used alone or in combination with other input devices. The touch screen may be used to detect touch operations, such as those performed by a user using a finger, palm, wrist, knuckles, or any other body part of the user. Additionally or alternatively, the touch operation may be performed by a user using, for example, a touch pen, stylus, or any other external input device by touching the touch screen or by moving a device near the touch screen. Regardless of the type of touch sensor used, the touch sensors disclosed herein may be configured to convert physical touch information into an input signal.

[0096] The first input signal may include a signal that provides information about a user's touch based on the user's interaction with the touch sensor. The first input signal may include information about one or more parameters that can be determined based on the touch interaction with the touch sensor. For example, the first input signal may include information about pressure, force, strain, position, movement, velocity, acceleration, temperature, occupancy, or any other physical or mechanical characteristic. The first multi-finger interaction may include any interaction with the touch sensor that involves a user contacting the touch sensor using two or more fingers (or using any type of two or more objects, such as fingers, pens, or sticks). In some embodiments, the first multi-finger interaction may include, for example, a user pinching two or more fingers together, pulling two or more fingers apart, or performing any other action with two or more fingers on the touch sensor. FIG. 6 shows some examples of various multi-finger interactions, such as pinching in, pinching out, rotating, and sliding, that may be used in connection with some embodiments of the present disclosure. The first multi-finger interaction 610 represents a pinch-in action in which a user brings their thumb and index finger closer together. A second multi-finger interaction 612 represents a user moving their thumb and index finger farther away from each other in a pinch-out motion. A third multi-finger interaction 614 represents a user moving their thumb and index finger together clockwise or counterclockwise in a rotational motion. A fourth multi-finger interaction 616 represents a user moving their thumb and index finger in opposite directions in a sliding motion. In another example of a multi-finger interaction, a user can move two or more fingers in the same direction. Other multi-finger interactions may also be used.

[0097] Some disclosed embodiments may include, in response to the first input signal, outputting a second display signal configured to cause a second perspective of the scene to be presented via the wearable extended reality device by modifying a first perspective of the scene. The second display signal may have similar characteristics to the first display signal as described above and may be configured to modify the first perspective of the scene. Modifying the perspective of the scene may include, for example, modifying one or more of the orientation, angle, size, direction, position, aspect, parameters of a spatial transformation, a spatial transformation, or any other visual characteristic of the scene or a portion of the scene. Thus, the presented second perspective of the scene may include, for example, modifying one or more of a second orientation, a second angle, a second size, a second direction, a second position, a second aspect, a second spatial transformation, or any other characteristic of the presentation of the scene (or portion of the scene) in the extended reality environment that differs from the first perspective of the scene.

[0098] In some examples, an indication that a physical object is located at a particular location within the environment of the wearable extended reality device may be received. In some examples, image data captured using an image sensor included in the first wearable extended reality device may be received. For example, the image data may be received from an image sensor, from the first wearable extended reality device, from an intermediate device external to the first wearable extended reality device, from a memory unit, etc. The image data may be analyzed to detect a physical object at a particular location within the environment. In another example, a radar, lidar, or sonar sensor may be used to detect the presence of a physical object at a particular location within the environment. In some examples, a second viewpoint of the scene may be selected based on a physical object located at a particular location. In one example, the second viewpoint of the scene may be selected so that virtual objects in the scene do not appear to collide with the physical object. In another example, the second viewpoint of the scene may be selected so that a particular virtual object is not obscured (fully or partially) by a physical object to a user of the wearable extended reality device (e.g., based on the position of the wearable extended reality device). In one example, a ray casting algorithm may be used to determine that a particular virtual object is obscured by a physical object to a user of the wearable extended reality device. Further, the second display signal may be based on a selection of a second viewpoint of the scene. In one example, if the physical object is not located at a particular location, a first version of the second display signal may be output for presentation via the wearable extended reality device in response to the first input signal, the first version of the second display signal being configured to modify the first viewpoint of the scene, thereby causing a second viewpoint of the scene to be presented via the wearable extended reality device, the second viewpoint including the presentation of the virtual object at the particular location.Further, when the physical object is located at the particular location, a second display signal of a second version can be output for presentation via the wearable extended reality device in response to the first input signal, and the second display signal of the second version can be configured to change the first perspective of the scene, thereby causing an alternative perspective of the scene to be presented via the wearable extended reality device, where the alternative perspective may not include presentation of the virtual object at the particular location and / or may include presentation of the virtual object at an alternative location different from the particular location.

[0099] Some disclosed embodiments may further include receiving, via the touch sensor, a second input signal caused by a second multi-finger interaction with the touch sensor. The second input signal may be similar to the first input signal described above and may include a signal providing information regarding one or more parameters that can be determined based on the touch interaction with the touch sensor. The second multi-finger interaction may include any interaction with the touch sensor involving a user contacting the touch sensor using two or more fingers of a hand. Thus, for example, the second multi-finger interaction may include movements of two or more fingers of the hand similar to the movements described above with respect to the first multi-finger interaction.

[0100] Consistent with certain disclosed embodiments, the first multi-finger interaction and the second multi-finger interaction may be of a general multi-finger interaction type. The general multi-finger interaction type may encompass interactions involving fingers of the same hand, fingers of the same type (e.g., index finger, thumb, middle finger, etc.), fingers of opposite hands, the same type of movement, any combination thereof, or any other type of interaction between two fingers that shares commonality with another type of interaction between two fingers. By way of example, the first multi-finger interaction and the second multi-finger interaction may both include a finger pinch.

[0101] Consistent with certain disclosed embodiments, the first multi-finger interaction may be of a different type than the second multi-finger interaction type. For example, the first multi-finger interaction of a different type from the second multi-finger interaction type may include an interaction involving a different type of movement, different fingers or hands, any combination thereof, or any other type of interaction between two fingers that is different from another type of interaction between two fingers. As an example, the first multi-finger interaction may be a user increasing the distance between their thumb and index finger on one hand, and the second multi-finger interaction may be a user decreasing the distance between their thumb and index finger on the other hand. In another example, the first multi-finger interaction may be a user rotating their thumb and index finger clockwise on one hand, and the second multi-finger interaction may be a user rotating their thumb and index finger counterclockwise on the other hand. In another example, the first multi-finger interaction may be a user increasing the distance between their thumb and index finger clockwise on one hand, and the second multi-finger interaction may be a user rotating their thumb and index finger clockwise on the other hand.

[0102] Some disclosed embodiments may further include outputting, in response to the second input signal, a third display signal configured to cause a third perspective of the scene to be presented via the wearable extended reality device by modifying a second perspective of the scene. The third display signal may have similar characteristics to the first display signal as described above and may be configured to modify the second perspective of the scene. Modifying the second perspective of the scene may include modifications similar to those described above with respect to modifying the first perspective of the scene. For example, modifying the second perspective of the scene may also include modifying one or more of the orientation, angle, size, direction, position, aspect, parameters of a spatial transformation, spatial transformation, or any other visual characteristic of the scene or portion of the scene. Thus, the presented third perspective of the scene may include modifications of one or more of a third orientation, a third angle, a third size, a third direction, a third position, a third aspect, a third spatial transformation, or any other characteristic of the presentation of the scene (or portion of the scene) in the extended reality environment that differs from the second perspective of the scene.

[0103] By way of example, FIG. 7 illustrates a set of exemplary changes in perspective of a scene consistent with some embodiments of the present disclosure. For example, a processor may be configured to initially output a first display signal reflecting a first perspective of a scene 710 for presentation via the wearable extended reality device. The processor may receive, via the touch sensor, a first input signal 712 caused by a first multi-finger interaction 620 (e.g., a pinch) with the touch sensor. In response to the first input signal 712, the processor may output a second display signal configured to cause a second perspective of a scene 714 to be presented via the wearable extended reality device by changing the first perspective of the scene. As shown in FIG. 7, the first perspective of the scene 710 has changed to a second perspective of the scene 714. The processor may receive, for example, a second input signal 716 caused by a second multi-finger interaction 622 (e.g., a slide) with the touch sensor after the first perspective of the scene 710 has changed to the second perspective of the scene 714 via the touch sensor. 7, the second multi-finger interaction may be an upward swipe or drag motion of the user's thumb and index finger. In response to the second input signal 716, the processor may output a third display signal configured to cause a third perspective of scene 718 to be presented via the wearable extended reality device by changing the second perspective of scene 714 for presentation via the wearable extended reality device. As shown in FIG. 7, the second perspective of scene 714 has been changed to the third perspective of scene 718.

[0104] Consistent with certain disclosed embodiments, a scene can include multiple content windows. A content window can be an area of ​​an extended reality environment configured to display text, still or moving images, or other visual content. For example, a content window can include a web browser window, a word processing window, an operating system window, or any other area on a screen that displays information for a particular program. In another example, a content window can include a virtual display (such as a virtual object mimicking a computer screen as described herein). In some embodiments, the content windows can be configured such that, in a first perspective of the scene, all of the multiple content windows can be displayed at the same virtual distance from the wearable extended reality device; in a second perspective of the scene, at least one of the multiple windows is displayed at a virtual distance that is different from the distances of other windows of the multiple windows; and in a third perspective of the scene, at least one window is displayed at an orientation that is different from the orientations of the other windows of the multiple windows. In one example, a content window that is displayed at a distance that is different from the distances of other windows can include a content window that is displayed closer to the wearable extended reality device than the other content windows. As another example, a content window displayed at a different distance from other content windows may include a content window displayed farther from the wearable extended reality device than the other content windows, or a content window displayed at a different distance from other content windows may include a content window displayed with a zoomed-in object, while the other content windows are displayed with a zoomed-out object.A further example of a content window displayed at a distance different from that of other content windows can include a content window displayed with a zoomed-out object, while the other content windows are displayed with a zoomed-in object. A content window displayed at an orientation different from that of other content windows can include any indication that changes any aspect of the angle, tilt, slant, direction, or relative physical position of the content window compared to the other content windows. For example, a content window displayed at an orientation different from that of other content windows can include a content window that is rotated 90 degrees while the other content windows remain at 0 degrees. As another example, a content window displayed at an orientation different from that of other content windows can include a content window that faces the left side of the wearable extended reality device, while the other content windows face the right side of the wearable extended reality device.

[0105] Further examples of scene perspectives are shown in FIG. 8. For example, one perspective can include two content windows presented side-by-side, as shown in 810. Another perspective can include two content windows tilted toward the same direction, as shown in 812. In another perspective, the two content windows may face inward toward each other, as shown in 814. Alternatively, example 816 shows two content windows facing outward toward each other and tilted upward. Another example, 818, shows two content windows curved toward each other. In yet another example, as shown in 820, the two content windows may curve in the same direction, but one content window may be presented on top of the content of another window. These are just a few examples of scene perspectives, but any variation in the visual presentation of a scene is contemplated for use with the embodiments disclosed herein.

[0106] Consistent with some disclosed embodiments, a scene can include multiple virtual objects. The virtual objects can include representations that do not exist in the real world, such as icons, images of things that may or may not exist in the real world, two-dimensional virtual objects, three-dimensional virtual objects, animate virtual objects, inanimate virtual objects, or any other representation that simulates a non-physical representation, either real or imagined, of a physical object. In some embodiments, the one or more virtual objects can be configured such that, in a first perspective of the scene, one or more of the virtual objects can be displayed at the same or a variable virtual distance from the wearable extended reality device; in a second perspective of the scene, at least one of the virtual objects is displayed at a different size than other virtual objects of the plurality of virtual objects; and in a third perspective of the scene, the size of the at least one virtual object reverts to the size of the at least one virtual object before it was presented in the second perspective of the scene. When the one or more virtual objects are displayed at the same distance from the wearable extended reality device, a user of the wearable extended reality device can view the one or more virtual objects with the same amount of space between the user and each of the one or more virtual objects. When the one or more virtual objects are displayed at variable virtual distances from the wearable extended reality device, a user of the wearable extended reality device can view each or a portion of the one or more virtual objects at different amounts of space away from the user. When at least one of the virtual objects is displayed at a different size than the others of the plurality of virtual objects, a user of the wearable extended reality device can view the at least one of the virtual objects as larger or smaller than the others of the plurality of virtual objects. When the size of the at least one virtual object returns to the size of the at least one virtual object before it was presented in the second perspective of the scene, a user of the wearable extended reality device can view the at least one virtual object at a size smaller or larger than the size of the second perspective of the scene.

[0107] Consistent with certain disclosed embodiments, input from the touch sensor can enable selective positional control of multiple virtual objects to direct the multiple virtual objects at a variable virtual distance from the wearable extended reality device. The variable virtual distance can refer to the perceived distance of the virtual object from the device wearer. Just as the virtual object is not physical, the distance is not real. Rather, the distance of the virtual object from the wearer is as virtual as the virtual object itself. In one example, distance can appear as the size of a rendering of the virtual object, as the position of the virtual object when rendered from different viewpoints (e.g., from the two viewpoints of the observer's eyes, from a moving viewpoint of the observer, etc.), as the virtual object obscuring or being obscured by other objects (physical or virtual) that overlap the virtual object at the viewer's viewing angle, etc. Input from the touch sensor can change such virtual distance (i.e., alter the presentation to cause the wearer to perceive a change in virtual distance). Such user input can be configured to occur before the first input signal, such that the presentation via the wearable extended reality device can be configured to allow the user to view the touch sensor. To allow a user to interact with a touch sensor, it may be desirable to allow the user to see the touch sensor when multiple virtual objects obstruct the user's field of view such that the user cannot see the touch sensor. Selective position control may include the ability to change one or more size, orientation, distance, or any other spatial attribute of a virtual object relative to another virtual object. Thus, in some embodiments, a user can selectively change the distance of one or more virtual objects such that the virtual objects are positioned at variable distances from the wearable extended reality device.

[0108] Consistent with some disclosed embodiments, selective position control may include docking multiple virtual objects into a physical space and enabling a wearer of a wearable extended reality device to walk through the physical space and adjust the position of each virtual object separately. Docking multiple virtual objects into a physical space may include maintaining the size, orientation, distance, or any other spatial attribute of the multiple virtual objects with reference to the physical space. This type of selective position control may enable a user to move within the physical space while maintaining the size, orientation, distance, or any other spatial attribute of the multiple virtual objects as the user moves. For example, a user may move toward and select a virtual object and adjust any of the size, orientation, distance, or any other spatial attribute of that virtual object, while other sizes, orientations, distances, or any other spatial attributes of other virtual objects may remain unchanged. A user may move to another virtual object in the physical space and adjust any of the size, orientation, distance, or any other spatial attribute of that virtual object, while other sizes, orientations, distances, or any other spatial attributes of the other virtual objects may remain unchanged. The user can subsequently do the same for some or all of the virtual objects as the user moves around the physical space: the user adjusts either the size, orientation, distance, or any other spatial attribute of each virtual object, while other unadjusted properties or other spatial attributes of other virtual objects may remain unchanged.

[0109] Consistent with some disclosed embodiments, the second viewpoint of the scene may be selected based on the first input signal and whether the housing including the touch sensor is placed on a physical surface during the first multi-finger interaction. For example, the position of the input device may help define the viewpoint of the presentation of the scene in the extended reality environment. Display characteristics may be different, for example, if the associated touch sensor is resting on a table rather than being held by the user. Alternatively, for example, the user's relative position (left or right) with respect to the touch-sensitive display may affect the preferred viewpoint of the presentation of the scene in the extended reality environment. In some examples, at least one parameter of the second viewpoint may be determined based on the physical surface. For example, a virtual plane may be determined based on the physical surface (e.g., parallel to the physical surface, overlapping the physical surface, perpendicular to the physical surface, a selected orientation relative to the virtual surface, etc.), and a virtual object may be moved on the virtual surface. In one example, the direction and / or distance of the movement of the virtual object on the virtual surface may be selected based on the first input signal. For example, in response to a first input signal, a virtual object may be moved in a first direction and a first distance on a virtual surface, and in response to a second input signal, the virtual object may be moved in a second direction and a second distance on the virtual surface. In another example, a new position on the virtual surface for the virtual object may be selected based on the first input signal.

[0110] Viewpoint selection based in part on whether the housing is currently positioned on a physical surface may also be desirable to enable certain types of views depending on whether the user of the wearable extended reality device is mobile or stationary at a given time. For example, if the user is mobile, certain types of scene viewpoints may be undesirable because they may interfere with the user's safe movement, for example, by obstructing the user's field of view. When the user is moving, the user is likely holding or otherwise attached to the housing rather than placing it on the physical surface during the initial multi-finger interaction. When the user is stationary, the user may place the housing on the physical surface, thereby allowing the user to select other viewpoints that do not require the user to see a portion of the field of view in front of the user. When the user is stationary, the user is likely placing the housing on the physical surface during the initial multi-finger interaction rather than holding or otherwise attached to the housing. In one example, the selection of the second viewpoint of the scene can be further based on the type of surface, for example, whether the surface is a tabletop. In another example, the selection of a third viewpoint of the scene may be selected based on the second input signal and whether the housing is placed on the physical surface during the second multi-finger interaction.

[0111] In certain embodiments, the first input signal from the touch sensor can reflect a two-dimensional coordinate input, and the second display signal is configured to modify a first perspective of the scene to introduce a three-dimensional change to the scene. The two-dimensional coordinate input can include an input that affects a geometric configuration that lacks depth but has only two measurements, such as length and width. The three-dimensional change to the scene can include a change in the scene that has three measurements, such as length, width, and depth. For example, the first perspective of the scene can include a cube. In this example, the first input signal from the touch sensor can reflect an input that includes a user moving their thumb and index finger in opposite directions in a sliding motion. The second display signal in this example can be configured to modify the cube by increasing or decreasing its depth.

[0112] In some embodiments, the first input signal from the touch sensor can reflect a Cartesian coordinate input, and the second display signal configured to change the viewpoint of the scene is in a spherical coordinate system. The Cartesian coordinate input can include an input that affects a geometric configuration defined by three types of points, each type of point being on a separate orthogonal axis. The spherical coordinate system can include a three-dimensional space defined by three types of points, each type of point including two angles on the orthogonal axes and a distance from the origin of the coordinate system to the type of point created by the two angles on the orthogonal axes. For example, the first viewpoint of the scene can include a cube. In this example, the first input signal from the touch sensor can reflect an input including a user moving their thumb and index finger in opposite directions in a sliding motion. The second display signal in this example can be configured to change the cube by increasing or decreasing its depth. The Cartesian input can be used to change the viewpoint of the scene in spherical coordinates by converting the input's Cartesian coordinates to spherical coordinates for the change using any mathematical technique for converting between coordinate systems.

[0113] In some embodiments, each type of multi-finger interaction can control one of a number of actions for changing the viewpoint of a scene presented via the wearable extended reality device. For example, the one or more actions can include zooming in, zooming out, rotating, and sliding, and a multi-finger interaction representing a user pinching their thumb and index finger closer together can control a zooming in action. As another example, a multi-finger interaction representing a user moving their thumb and index finger farther apart in a pinching out action can control a zooming out action. As another example, a multi-finger interaction representing a user moving their thumb and index finger together clockwise or counterclockwise in a rotating action can control a rotating action. As another example, a multi-finger interaction representing a user moving their thumb and index finger in opposite directions in a sliding action can control a sliding action.

[0114] In some embodiments, the actions for changing the viewpoint of the scene may include at least two of changing the viewpoint position relative to the scene, changing the distance from the scene, changing the angle of the scene, and changing the size of the scene. However, these changes are not limited to the entire scene; the change of viewpoint position, change of distance, change of angle, and change of size relative to the scene may also be applied to one or more selected objects in the scene.

[0115] In some embodiments, the first input signal and the second input signal may be associated with a first operational mode of a touch sensor for controlling a viewpoint. The operational mode of a touch sensor may include, for example, a means, method, approach, method of operation, procedure, process, system, technology, or any other functional state of the touch sensor. The first operational mode may differ from the second operational mode with respect to the means, method, approach, method of operation, procedure, process, system, technology, or any other functional state of the touch sensor. For example, the first operational mode may include a touch sensor generating an input signal representative of a user increasing or decreasing the distance between the thumb and index finger, while the second operational mode may include a touch sensor generating an input signal representative of a user moving the thumb and index finger together in a clockwise or counterclockwise direction. As another example, the first operational mode may include a touch sensor generating an input signal representative of a user performing a multi-finger interaction within a defined area of ​​the touch sensor, while the second operational mode may include a touch sensor generating an input signal representative of a user performing a multi-finger interaction anywhere on the touch sensor.

[0116] In some embodiments, the at least one processor may be configured to receive, prior to receiving the first input signal, an initial input signal via the touch sensor associated with a second operational mode of the touch sensor for controlling a pointer presented via the wearable extended reality device. The initial input signal used in this disclosure may include, for example, a signal that provides information about a user's touch based on the user's interaction with the touch sensor and controls a position of the pointer in the extended reality environment presented via the wearable extended reality device. The initial input signal may include information about pressure, force, strain, position, movement, acceleration, occupancy, and any other parameter that can be determined based on a touch interaction with the touch sensor. The pointer may include, for example, an arrow, dial, director, guide, indicator, mark, signal, or any other visual representation of content or a position in the extended reality environment presented via the wearable extended reality device.

[0117] Some embodiments may include selecting a virtual object in a scene in response to the initial input signal. Selecting a virtual object in a scene may include selecting the virtual object in the scene for further manipulation. Selecting the virtual object in the scene may be achieved by using the initial input signal to control a pointer to click, select, highlight, pin, draw a box around, or perform any other type of marking on the virtual object. For example, the virtual object may be a tree in a forest scene. A user may select a tree in the forest by dragging a finger over the touch sensor and guiding the pointer position to the tree by clicking the tree to mark it as selected. In some embodiments, the at least one processor may be configured to cause a change in the display of the selected virtual object to be presented via the wearable extended reality device in response to the first input signal and the second input signal. The change in the display of the selected virtual object may include a change in viewpoint position relative to the object, a change in distance from the object, a change in angle of the object, and a change in size of the object. For example, the virtual object may be a tree in a forest scene. A user may increase the size of the tree by inputting a first input signal via the touch sensor by increasing the distance between their thumb and index finger. The user can then rotate the tree by inputting a second input signal via the touch sensor by moving the thumb and index finger together in a clockwise or counterclockwise direction.

[0118] In some embodiments, the instructions are configured to transmit the first display signal, the second display signal, and the third display signal to the wearable extended reality device via a first communication channel, and to receive the first input signal and the second input signal from a computing device other than the wearable extended reality device via a second communication channel. As used in this disclosure, a communication channel may include either a wired communication channel or a wireless communication channel. A wired communication channel may utilize a network cable, a coaxial cable, an Ethernet cable, or any other channel that transmits information via a wired connection. A wireless communication channel may utilize WiFi, Bluetooth™, NFC, or any other channel that transmits information without a wired connection. A computing device may be any device other than the wearable extended reality device, including a general computer such as a personal computer (PC), a UNIX workstation, a server, a mainframe computer, a personal digital assistant (PDA), a mobile phone, a smartphone, a tablet, any combination thereof, or any other device for automatically performing calculations.

[0119] In some embodiments, the computing device may be a smartphone connectable to the wearable extended reality device, and the touch sensor may be a touchscreen of the smartphone. As another example, the computing device may be a smartphone or a smartwatch connectable to the wearable extended reality device, and the touch sensor may be a touchscreen of the smartwatch. In such embodiments, the first and second multi-finger interactions may occur on the touchscreen of the smartphone or smartwatch.

[0120] In some embodiments, the computing device can include a keyboard connectable to the wearable extended reality device, and a touch sensor is associated with the computing device. The keyboard can be a standard typewriter-style keyboard (e.g., a QWERTY-style keyboard) or another suitable keyboard layout, such as a Dvorak layout or a coded layout. In one embodiment, the keyboard can include at least 30 keys. The keyboard can include alphanumeric keys, function keys, modifier keys, cursor keys, system keys, multimedia control keys, or other physical keys that perform computer-specific functions when pressed. The keyboard can also include virtual keys or programmable keys, the functions of which change depending on the function or application being executed on the integrated computing interface device. In some embodiments, the keyboard can include dedicated input keys for performing operations associated with the wearable extended reality device. In some embodiments, the keyboard can include dedicated input keys for changing the lighting of content (e.g., an extended reality environment, virtual content, etc.) projected through the wearable extended reality device. The touch sensor can be integrated into the keyboard via a touchpad that is part of the keyboard. The touch sensor may be incorporated into the keyboard as one or more touch sensors located on one or more keys of the keyboard. In such a configuration, the touch sensor is not wearable and is not connected to or configured to move with the wearable extended reality device.

[0121] Some embodiments may include obtaining default settings for displaying content using the wearable extended reality device. The default settings may be stored in memory or may be pre-set by a manufacturer, distributor, or user. The settings may include color, size, resolution, orientation, or any other conditions related to the display. The default settings may include any pre-selected options for such conditions that are adopted by a computer program or other mechanism if no alternative is specified at the time of use by a user or programmer. The default settings may be received from a user or may be determined based on the type of wearable extended reality device used, time, location, or any other type of heuristic data. Further embodiments of the non-transitory computer-readable medium include selecting a first viewpoint of a scene based on the default settings.

[0122] In some embodiments, the default setting is determined based on a previous viewpoint of the scene. The previous viewpoint of the scene can include any viewpoint prior to a given time for presenting the scene, such as a viewpoint from a previous use of the wearable extended reality device. For example, if the viewpoint of the scene was zoomed in a certain amount in the previous use, the default setting can be set so that the default display of the scene is that viewpoint zoomed in a certain amount. In other embodiments, the instructions can direct the processor to select a first viewpoint of the scene based on user preferences. The user preferences can include settings customized by a user of the wearable extended reality device, such as color, size, resolution, orientation, or any other condition related to the display that can be changed. For example, a user of the wearable extended reality device can set a user preference that directs the first viewpoint to be displayed only in black and white. In this example, the instructions can direct the processor to select a first viewpoint of the scene to be displayed only in black and white based on these user preferences.

[0123] In some embodiments, the default settings are associated with a keyboard paired with the wearable extended reality device. The paired keyboard may be the user's work keyboard, home keyboard, office keyboard, field keyboard, or any other type of peripheral device that allows the user to input text into any electronic device. The pairing may include either wired pairing or wireless pairing. Wired pairing may utilize coaxial cable, Ethernet, or any other channel that transmits information over a wired connection. Wireless pairing may utilize WiFi, Bluetooth™, or any other channel that transmits information without a wired connection. For example, a user of the wearable extended reality device may input default settings into the wearable extended reality device by entering information about the default settings through a keyboard wirelessly paired with the wearable extended reality device. In one example, pairing may enable modification of a virtual object using the keyboard. For example, the virtual object may include a representation of text entered using the keyboard.

[0124] Other disclosed embodiments may include a method for controlling a viewpoint in an extended reality environment using a physical touch controller. By way of example, FIG. 9 illustrates a flowchart illustrating an exemplary process 900 for changing a viewpoint of a scene, consistent with some embodiments of the present disclosure. The method includes outputting a first display signal reflecting a first viewpoint of the scene for presentation via the wearable extended reality device, as shown in block 910. The method also includes receiving, via the touch sensor, a first input signal caused by a first multi-finger interaction with the touch sensor, as shown in block 912. The method further includes outputting, in response to the first input signal, a second display signal configured to cause a second viewpoint of the scene to be presented via the wearable extended reality device by changing the first viewpoint of the scene. This step is shown in block 914. The method also includes receiving, via the touch sensor, a second input signal caused by a second multi-finger interaction with the touch sensor, as shown in block 916. Block 918 depicts outputting, in response to the second input signal, a third display signal configured to cause a third perspective of the scene to be presented via the wearable extended reality device by changing the second perspective of the scene for presentation via the wearable extended reality device.

[0125] Some embodiments of the present disclosure may relate to enabling gesture interaction with invisible virtual objects. A virtual object displayed by a wearable extended reality device may be docked to a physical object (e.g., such that the virtual object can have a fixed position relative to the physical object). If the location where the virtual object is docked is outside a portion of the field of view associated with the display system of the wearable extended reality device such that the virtual object is not displayed by the display system, the user may interact with the virtual object through gesture interaction with the docking location. In another example, if the virtual object is docked to a portion of the field of view associated with the display system of the wearable extended reality device but is obscured by another object (physical or virtual) such that the virtual object is not displayed by the display system, the user may interact with the virtual object through gesture interaction with the docking location. The gesture interaction with the docking location may be captured (e.g., by an image sensor) and used to determine the user's interaction with the virtual object. Further details regarding enabling gesture interaction with invisible virtual objects are described below.

[0126] Some disclosed embodiments may relate to enabling gestural interaction with invisible virtual objects, and the embodiments are presented as methods, systems, devices, and non-transitory computer-readable media. When any of the presentation formats is described below, it should be understood that the respective disclosures are equally applicable to the others. For example, one or more processes embodied in a non-transitory computer-readable medium may be performed as a method, system, or device. Some aspects of such processes may be performed electronically over a network, which may be wired, wireless, or both. Other aspects of such processes may be performed using non-electronic means. In the broadest sense, a process is not limited to a particular physical and / or electronic means, but rather may be accomplished using many different means. For example, some disclosed embodiments may include a method, system, or device for enabling gestural interaction with invisible virtual objects. The method may include various processes described herein. The system may include at least one processor configured to perform the various processes described herein. The device may include at least one processor configured to perform various operations as described herein.

[0127] A non-transitory computer-readable medium may include instructions that, when executed by at least one processor, cause the at least one processor to perform various processes as described herein. A non-transitory computer-readable medium may include any type of physical memory in which information or data readable by at least one processor may be stored. A non-transitory computer-readable medium may include, for example, random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drive, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), flash drive, disk, any optical data storage medium, any physical medium with a pattern of holes, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), FLASH-EPROM or any other flash memory, non-volatile random access memory (NVRAM), cache, register, any other memory chip or cartridge, or networked versions thereof. A non-transitory computer-readable medium may refer to multiple structures, such as multiple non-transitory computer-readable media located in local or remote locations. Additionally, one or more non-transitory computer-readable media may be utilized in performing the computer-implemented method. Thus, non-transitory computer-readable media may include tangible items and may exclude carrier waves or transient signals.

[0128] The instructions included in the non-transitory computer-readable medium may include, for example, software instructions, computer programs, computer code, executable instructions, source code, machine instructions, machine language programs, or any other type of instructions for a computing device. The instructions included in the non-transitory computer-readable medium may be based on one or more of various types of desired programming languages ​​and may include (e.g., embody) various processes for enabling gestural interaction with invisible virtual objects as described herein.

[0129] At least one processor may execute instructions contained on a non-transitory computer-readable medium to perform various operations for enabling gestural interaction with invisible virtual objects as described herein. The processor may include, for example, an integrated circuit, microchip, microcontroller, microprocessor, central processing device (CPU), graphics processing device (GPU), digital signal processor (DSP), field programmable gate array (FPGA), or other unit suitable for executing instructions or performing logical operations. The processor may include a single-core or multi-core processor. In some examples, the processor may be a single-core processor configured with virtualization technology. The processor may implement virtualization technology or other functionality to provide, for example, the ability to run, control, execute, manipulate, or store multiple software processes, applications, or programs. In another example, the processor may include a multi-core processor (e.g., dual-core, quad-core, or any desired number of cores) configured to provide parallel processing capabilities to enable a device associated with the processor to run multiple processes simultaneously. Other types of processor configurations may be implemented to provide the functionality described herein.

[0130] Some disclosed embodiments may relate to enabling gestural interaction with invisible virtual objects. Gestures may include any finger or hand movement, such as dragging, pinching, spreading, swiping, tapping, pointing, scrolling, rotating, flicking, touching, zooming in, zooming out, thumbs up, thumbs down, touch and hold, or any other movement of the hand. In some examples, gestures may include movements of the eyes, mouth, face, or other parts of a person's body. A virtual object may refer to a visual representation rendered by a computing device and configured to represent an object. A virtual object may include, for example, an inanimate virtual object, an animate virtual object, virtual furniture, virtual decorative objects, a virtual widget, a virtual screen, or any other type of virtual display. A user may interact with the virtual object via gestures. In some examples, as described in more detail herein, the virtual object need not be displayed to the user for the user to interact with it. For example, if a user docks a volume control virtual object to the corner of a desk, a sensor (e.g., a camera) may detect the touch gesture each time the user touches the corner of the desk, enabling volume adjustment even if the volume control virtual object is not visible at the time the interaction is initiated. Docking a virtual object to a physical object may, for example, cause the virtual object to have a fixed position relative to the physical object. Further details regarding various processes for enabling gesture interactions with invisible virtual objects (e.g., including docking) are described below.

[0131] Some disclosed embodiments may include receiving image data captured by at least one image sensor of a wearable extended reality device (WER-device). The image data may include a representation of multiple physical objects within a field of view associated with the at least one image sensor of the wearable extended reality device. For example, a user may use the wearable extended reality device. The wearable extended reality device may include at least one image sensor. The at least one image sensor may include, or be implemented similarly to, image sensor 372 or image sensor 472. In some examples, the at least one image sensor may be configured to continuously or periodically capture image data of a location where the wearable extended reality device is present (e.g., an image of a physical spatial setting that the at least one image sensor may be facing). Capturing the image data may include receiving, by the image sensor, a light signal emitted from an environment in which the image sensor is present, such as visible light, infrared light, electromagnetic waves, or any other desired radiation. Capturing the image data may include, for example, converting the received light signal into data that can be stored or processed by a computing device. The at least one processor can receive image data captured by the at least one image sensor of the wearable extended reality device. In some examples, the at least one processor may include or be implemented similarly to processing device 360, processing device 460, processing device 560, or any other processor associated with the wearable extended reality device.

[0132] At least one image sensor of a wearable extended reality device can have a field of view. The field of view can refer to the spatial extent that can be observed or detected at any given moment. For example, the field of view of an image sensor can include a solid angle over which the image sensor can be sensitive to radiation (e.g., visible light, infrared light, or other optical signals). In some examples, the field of view can include an angle of view. The field of view can be measured horizontally, vertically, diagonally, or in any other desired manner.

[0133] The image data captured by the at least one image sensor of the wearable extended reality device can include representations of multiple physical objects within a field of view associated with the at least one image sensor of the wearable extended reality device. The multiple physical objects can include a desk, a table, a keyboard, a mouse, a touchpad, a lamp, a cup, a phone, a mobile device, a printer, a screen, a shelf, a machine, a vehicle, a door, a window, a chair, a button, a surface, or any other type of physical item that can be observed by an image sensor. The multiple physical objects can be within a field of view of the at least one image sensor of the wearable extended reality device, and image data including representations of the multiple physical objects can be captured by the at least one image sensor.

[0134] 10 is a flowchart illustrating an example process 1000 for enabling gestural interaction with invisible virtual objects consistent with some embodiments of the present disclosure. Referring to FIG. 10, at step 1010, instructions contained on a non-transitory computer-readable medium, when executed by a processor, can cause the processor to receive image data captured by at least one image sensor of a wearable extended reality device, the image data including a representation of a plurality of physical objects within a field of view associated with the at least one image sensor of the wearable extended reality device.

[0135] FIG. 11 is a schematic diagram illustrating a field of view that is horizontally consistent with some embodiments of the present disclosure. Referring to FIG. 11 , a wearable extended reality device 1110 can include at least one image sensor. The field of view of the at least one image sensor can have a horizontal extent 1112. FIG. 12 is a schematic diagram illustrating a field of view that is vertically consistent with some embodiments of the present disclosure. Referring to FIG. 12 , the field of view of the at least one image sensor of the wearable extended reality device 1110 can have a vertical extent 1210.

[0136] FIG. 13 is a schematic diagram illustrating a user using an exemplary wearable extended reality device consistent with some embodiments of the present disclosure. One or more elements shown in FIG. 13 may be the same as or similar to one or more elements described in connection with FIG. 1. For example, with reference to FIG. 13, a user 100 may be sitting behind a table 102. The user 100 may use a wearable extended reality device (e.g., wearable extended reality device 110). A keyboard 104 and a mouse 106 may be located on the table 102. The wearable extended reality device (e.g., wearable extended reality device 110) may display virtual content, such as a virtual screen 112 and virtual widgets 114C, 114D, and 114E, to the user 100. The wearable extended reality device (e.g., wearable extended reality device 110) may include at least one image sensor. The at least one image sensor may have a field of view 1310, which may be represented by a solid angle from a point on the wearable extended reality device. The at least one image sensor may be configured to capture image data including a representation of a plurality of physical objects within the field of view 1310. The plurality of physical objects may include, for example, a table 102, a keyboard 104, and a mouse 106.

[0137] 14, 15, 16, 17, 18, and 19 illustrate various usage snapshots of an exemplary system for enabling gestural interaction with invisible virtual objects consistent with some embodiments of the present disclosure. FIGS. 14, 15, 16, 17, 18, and 19 may illustrate one or more elements as described in connection with FIG. 13 from another perspective (e.g., the perspective of a user 100, the perspective of a wearable extended reality device for the user 100, or the perspective of at least one image sensor for the wearable extended reality device for the user 100). Referring to FIG. 14, a keyboard 104 and a mouse 106 may be positioned on a table 102. The wearable extended reality device may display virtual content, such as a virtual screen 112 and virtual widgets 114C, 114D, and 114E, to the user. The wearable extended reality device may include at least one image sensor. The at least one image sensor may have a field of view 1310 and may be configured to capture image data including a representation of a plurality of physical objects within the field of view 1310. The plurality of physical objects may include, for example, a table 102, a keyboard 104, and a mouse 106.

[0138] Some disclosed embodiments may include displaying multiple virtual objects in a portion of the field of view, the portion of the field of view being associated with a display system of the wearable extended reality device. As previously described, the virtual objects may include, for example, inanimate virtual objects, animate virtual objects, virtual furniture, virtual decorative objects, virtual widgets, virtual screens, or any other type of virtual display. The wearable extended reality device may display the multiple virtual objects to a user via the display system. The display system of the wearable extended reality device may include, for example, an optical head-mounted display, a monocular head-mounted display, a binocular head-mounted display, a see-through head-mounted display, a helmet-mounted display, or any other type of device configured to display images to a user. In some examples, the display system of the wearable extended reality device may include a display configured to reflect a projected image and allow the user to view through the display. The display may be based on waveguide technology, diffractive optics, holographic optics, polarization optics, reflective optics, or other types of technology for combining an image projected by a computing device with an optical signal emitted from a physical object.

[0139] The display system of the wearable extended reality device may be configured to display virtual content (e.g., one or more virtual objects) to a user in a portion of a field of view of at least one image sensor of the wearable extended reality device. The portion of the field of view may be associated with the display system of the wearable extended reality device. For example, the virtual content output by the display system of the wearable extended reality device may occupy a solid angle that may be within the solid angle representing the field of view of the at least one image sensor of the wearable extended reality device. In some examples, the display system of the wearable extended reality device may have a field of view through which the virtual content can be displayed. The field of view of the display system may correspond to a portion of the field of view of the at least one image sensor. In some examples, the solid angle representing the field of view of the display system may have a vertex (e.g., a point from which an object can be viewed) that is the same as or approximates a vertex of the solid angle representing the field of view of the at least one image sensor.

[0140] Referring to FIG. 10 , in step 1012, instructions contained in a non-transitory computer-readable medium, when executed by a processor, can cause the processor to display a plurality of virtual objects in a portion of a field of view, the portion of the field of view being associated with a display system of a wearable extended reality device.

[0141] 11 , a wearable extended reality device 1110 can include at least one image sensor. The field of view of the at least one image sensor can have a horizontal range 1112. A portion of the field of view of the at least one image sensor can have a horizontal range 1114. The horizontal range 1114 can be within the horizontal range 1112. In some examples, the vertices of the horizontal range 1112 can be the same as or close to the vertices of the horizontal range 1114 (e.g., within less than 1 mm of each other, within less than 1 cm of each other, within less than 10 cm of each other, etc.).

[0142] 12 , the field of view of at least one image sensor of wearable extended reality device 1110 can have vertical range 1210. A portion of the field of view of at least one image sensor can have vertical range 1212. Vertical range 1212 can be within vertical range 1210. In some examples, the vertices of vertical range 1210 can be the same as or close to (e.g., within 1 mm of each other, within 1 cm of each other, within 10 cm of each other, etc.) ...2.

[0143] 13 , a wearable extended reality device (e.g., wearable extended reality device 110) may include at least one image sensor. The at least one image sensor may have a field of view 1310, which may be represented by a solid angle from a point on the wearable extended reality device. A portion of the field of view 1310 may be 1312. The field of view portion 1312 may be represented by a solid angle within the solid angle representing the field of view 1310. The field of view portion 1312 may be associated with a display system of the wearable extended reality device. The wearable extended reality device may display multiple virtual objects (e.g., virtual screen 112 and virtual widgets 114C, 114D, 114E) in the field of view portion 1312. Continuing with the example, and referring to FIG. 14, field of view portion 1312 may be within field of view 1310, and the wearable extended reality device may display multiple virtual objects (e.g., virtual screen 112 and virtual widgets 114C, 114D, 114E) in field of view portion 1312.

[0144] In some examples, the field of view associated with at least one image sensor of the wearable extended reality device may be wider than a portion of the field of view associated with the display system of the wearable extended reality device. In some examples, the horizontal and / or vertical range of the portion of the field of view associated with the display system may be between 45 and 90 degrees. In other implementations, the portion of the field of view associated with the display system may be increased to a human field of view or field of view. For example, the portion of the field of view associated with the display system may be increased to approximately 200 degrees for the horizontal range and approximately 150 degrees for the vertical range. As another example, the field of view associated with the at least one image sensor may be increased accordingly to be wider than the portion of the field of view associated with the display system.

[0145] In some examples, the horizontal range of the field of view associated with at least one image sensor of the wearable extended reality device may exceed 120 degrees, and the horizontal range of a portion of the field of view associated with a display system of the wearable extended reality device may be less than 120 degrees. In some examples, the vertical range of the field of view associated with at least one image sensor of the wearable extended reality device may exceed 120 degrees, and the vertical range of a portion of the field of view associated with a display system of the wearable extended reality device may be less than 120 degrees.

[0146] In some examples, the horizontal range of the field of view associated with at least one image sensor of the wearable extended reality device may exceed 90 degrees, and the horizontal range of a portion of the field of view associated with a display system of the wearable extended reality device may be less than 90 degrees. In some examples, the vertical range of the field of view associated with at least one image sensor of the wearable extended reality device may exceed 90 degrees, and the vertical range of a portion of the field of view associated with a display system of the wearable extended reality device may be less than 90 degrees.

[0147] In some examples, the horizontal range of the field of view associated with at least one image sensor of the wearable extended reality device may exceed 150 degrees, and the horizontal range of a portion of the field of view associated with a display system of the wearable extended reality device may be less than 150 degrees. In some examples, the vertical range of the field of view associated with at least one image sensor of the wearable extended reality device may exceed 130 degrees, and the vertical range of a portion of the field of view associated with a display system of the wearable extended reality device may be less than 130 degrees.

[0148] Additionally or alternatively, the horizontal and / or vertical extent of a portion of the field of view associated with the display system of the wearable extended reality device may be configured to be any desired number of degrees for displaying virtual content to a user. For example, the field of view associated with at least one image sensor of the wearable extended reality device may be configured to be wider (e.g., in terms of horizontal and / or vertical extent) than a portion of the field of view associated with the display system of the wearable extended reality device.

[0149] In some examples, the plurality of virtual objects displayed in the portion of the field of view associated with the display system of the wearable extended reality device may include a first virtual object that may run on a first operating system and a second virtual object that may run on a second operating system. An operating system may include software that controls the operation of a computer and directs the processing of programs. Multiple operating systems may be used to control and / or present the virtual objects. For example, a first operating system may be implemented in association with a keyboard (e.g., keyboard 104), and a second operating system may be implemented on the wearable extended reality device (e.g., wearable extended reality device 110). Additionally or alternatively, an input unit (e.g., input unit 202), a wearable extended reality device (e.g., extended reality unit 204), or a remote processing unit (e.g., remote processing unit 208) may have their own operating systems. In another example, a first operating system can run on a remote cloud and a second operating system can run on a local device (e.g., a wearable extended reality device, a keyboard, a smartphone in proximity to the wearable extended reality device, etc.). A first virtual object can run on one of the operating systems. A second virtual object may run on another one of the operating systems.

[0150] Some disclosed embodiments may include receiving a selection of a specific physical object from a plurality of physical objects. For example, at least one image sensor of the wearable extended reality device may continuously or periodically monitor a scene within a field of view of the at least one image sensor, and at least one processor may detect an indication of a selection of the specific physical object from the plurality of physical objects based on the monitored scene. The indication of the selection of the specific physical object may include, for example, a gesture by a user indicating the selection of the specific physical object (e.g., pointing at the specific physical object, touching and holding on the specific physical object, or tapping on the specific physical object). In some examples, the at least one processor may determine a specific point or region on or near the selected specific physical object based on the gesture by the user. The specific point or region may be used to receive a docking virtual object. In some examples, the selected specific physical object may be a part of a physical object (e.g., a specific point or region on a table). In some other examples, the selection of a particular physical object from a plurality of physical objects may be received from an external device, from a data structure specifying the selection of physical objects, from a memory unit, from a user, from an automated process (e.g., from an object detection algorithm used to analyze image data to detect and thereby select the particular physical object), etc.

[0151] In some examples, the selection of a particular physical object may be from a list or menu including multiple physical objects. For example, based on receiving image data captured by at least one image sensor including a representation of multiple physical objects, the at least one processor can analyze the image data to assign a label to each of the multiple physical objects and generate a list or menu including the labels identifying each physical object. A user of the wearable extended reality device can invoke the list or menu and select a particular physical object from the list or menu. The user can further indicate a particular point or area on or near the selected particular physical object to receive the docking virtual object.

[0152] The particular physical object may include any physical object selected from a plurality of physical objects. In some examples, the particular physical object may include a portion of a touchpad. In some examples, the particular physical object may include a non-control surface. A non-control surface may include a surface that is not configured to provide control signals to a computing device, such as the surface of a table, a surface of clothing, etc. In some examples, the particular physical object may include a keyboard. In some examples, the particular physical object may include a mouse. In some examples, the particular physical object may include a table or a portion thereof. In some examples, the particular physical object may include a sensor.

[0153] 10 , in step 1014, instructions included on a non-transitory computer-readable medium, when executed by a processor, can cause the processor to receive a selection of a specific physical object from a plurality of physical objects. For example, referring to FIG. 15 , hand gesture 1510 can indicate a tap on table 102. The at least one processor can detect the hand gesture 1510, for example, based on image data captured by at least one image sensor having field of view 1310, and can determine an indication of a selection of the specific physical object (e.g., table 102) based on the hand gesture 1510. In some examples, the at least one processor can further determine, based on the hand gesture 1510, a specific point or area on or near the selected specific physical object (e.g., the point or area of ​​table 102 tapped by hand gesture 1510) to receive the docking virtual object. In some examples, the selected specific physical object may be a portion of the physical object (e.g., the specific point or area on table 102).

[0154] Some disclosed embodiments may include receiving a selection of a specific virtual object from a plurality of virtual objects for association with a specific physical object. For example, at least one image sensor of the wearable extended reality device may continuously or periodically monitor a scene within a field of view of the at least one image sensor. The at least one processor may detect an indication of a selection of the specific virtual object from the plurality of virtual objects based on the monitored scene. The indication of the selection of the specific virtual object may include, for example, a gesture by a user indicating selection of the specific virtual object (e.g., pointing at the specific virtual object, touching and holding on the specific virtual object, or tapping on the specific virtual object). In some examples, the selection of the specific virtual object may be from a list or menu including multiple virtual objects. For example, the at least one processor may generate a list or menu including multiple virtual objects. A user of the wearable extended reality device may invoke the list or menu and select a specific virtual object from the list or menu. The specific virtual object may be selected for association with the specific physical object. In some other examples, the selection of a particular virtual object from a plurality of virtual objects to associate with a particular physical object may be received from an external device, from a data structure that associates the desired selection of virtual objects with different physical objects, from a memory unit, from a user, from an automated process (e.g., from an algorithm that ranks the compatibility of virtual objects with physical objects), etc.

[0155] 10 , at step 1016, instructions contained on a non-transitory computer-readable medium, when executed by a processor, can cause the processor to receive a selection of a specific virtual object from a plurality of virtual objects for association with a specific physical object. Referring to FIG. 16 , hand gesture 1610 can indicate a tap on virtual widget 114E. At least one processor can detect hand gesture 1610 based on image data captured by at least one image sensor having field of view 1310, for example, and can determine, based on hand gesture 1610, an indication of a selection of a specific virtual object (e.g., virtual widget 114E) for association with the specific physical object (e.g., table 102).

[0156] In some examples, both the selection of a specific physical object and the selection of a specific virtual object may be determined from an analysis of image data captured by at least one image sensor of the wearable extended reality device. The image data may include a representation of one or more scenes within a field of view of the at least one image sensor. One or more gestures may be indicative of a selection of a specific physical object and / or a selection of a specific virtual object based on the image data. In some examples, both the selection of a specific physical object and the selection of a specific virtual object may be determined from the detection of a single predetermined gesture. The single predetermined gesture may include, for example, any one of a variety of gestures including a tap, a touch and hold, a pointing, or any other desired gesture indicating object selection. In one example, a data structure may associate gestures with the selection of a pair of a physical object and a virtual object, and the data structure may be accessed based on the single predetermined gesture to select the specific physical object and the specific virtual object. In some other examples, image data captured by at least one image sensor of the wearable extended reality device may be analyzed using an object detection algorithm to detect a particular physical object of a particular category (and / or the location of the particular physical object), and the particular physical object may be selected to be the detected particular physical object. Further, a particular virtual object may be selected based on the particular physical object detected in the image data and / or based on characteristics of the particular physical object determined by analyzing the image data. For example, the image data may be analyzed to determine dimensions of the particular physical object, and a virtual object that fits the determined dimensions may be selected as the particular virtual object.

[0157] Some disclosed embodiments may include docking a specific virtual object with a specific physical object. Docking a specific virtual object to a specific physical object may include, for example, placing the specific virtual object in a fixed position relative to the specific physical object. Additionally or alternatively, docking a specific virtual object to a specific physical object may include attaching the specific virtual object to the specific physical object. Attachment may be performed in any manner that establishes an association. For example, depending on the system designer's choice, a physical command (e.g., a hard or soft button press), a voice command, a gesture, a hover, or any other predefined action to trigger docking may be used. The specific virtual object may be attached to a specific point or area on or near the specific physical object. The specific point or area on or near the specific physical object may be specified, for example, by a user of the wearable extended reality device. Docking a specific virtual object to a specific physical object may be based on a selection of a specific physical object and a selection of a specific virtual object. In some examples, the particular physical object selected may be a portion of the physical object (e.g., a particular point or area on a table), and the particular virtual object may be docked to the portion of the physical object.

[0158] In some examples, docking the particular virtual object to the particular physical object may be based on receiving a user input indication to dock the particular virtual object to the particular physical object (e.g., a gesture to drag the particular virtual object to the particular physical object). Docking the particular virtual object to the particular physical object may cause the particular virtual object to remain in a position that may be fixed relative to the particular physical object. Based on the docking, the particular virtual object may be displayed as being in a position that may be fixed relative to the particular physical object.

[0159] In some examples, docking a particular virtual object to a particular physical object may include connecting the position of the particular virtual object to the particular physical object. The docked particular virtual object may move along with the particular physical object to which it is docked. The particular physical object may be associated with a dockable area where the virtual object may be positioned to dock to the particular physical object. A dockable area may include, for example, an area (e.g., a surface) of the particular physical object and / or an area surrounding the particular physical object. For example, the dockable area of ​​a table may include the surface of the table. As another example, the dockable area of ​​a keyboard may include the area surrounding the keyboard (e.g., when the keyboard is placed on a support surface) and / or the surface of the keyboard (e.g., a portion thereof that does not have buttons or keys). In some examples, the particular virtual object may be dragged to a dockable area (e.g., an area of ​​the particular physical object). In some examples, as a particular virtual object is moved, a dockable area is highlighted (e.g., using virtual content displayed by the wearable extended reality device), and a representation of the dockable area is visually displayed to the user, allowing the particular virtual object to be positioned and docked.

[0160] Referring to FIG. 10, in step 1018, instructions contained in a non-transitory computer-readable medium when executed by a processor may cause the processor to dock a particular virtual object with a particular physical object.

[0161] 17 , hand gestures 1710, 1712, and 1714 may indicate the dragging of a virtual widget 114E onto the table 102. At least one processor may detect the hand gestures 1710, 1712, and 1714 based on image data captured by at least one image sensor having a field of view 1310, for example, and may determine a display of docking a particular virtual object (e.g., the virtual widget 114E) with a particular physical object (e.g., the table 102) based on the hand gestures 1710, 1712, and 1714. The particular virtual object may be docked to a particular point or region on the particular physical object or may approximate the particular physical object. The particular point or region may be specified by a user. In some examples, the selected particular physical object may be a portion of the physical object (e.g., a particular point or region on the table 102), and the particular virtual object may be docked to the portion of the physical object. Referring to FIG. 18, a particular virtual object (eg, virtual widget 114E) is docked to a particular physical object (eg, table 102, or a particular point or area on table 102).

[0162] In some examples, the particular physical object may be a keyboard, and at least some of the plurality of virtual objects may be associated with at least one rule that defines a behavior of the virtual object when the wearable extended reality device moves away from the docked keyboard. The virtual object behavior may include, for example, undocking a docked virtual object, deactivating a docked virtual object, hiding a docked virtual object, rendering a virtual object undockable, or other desired action. The at least one rule that defines the virtual object behavior may be configured to cause the at least one processor to execute one or more of the virtual object behaviors on one or more corresponding virtual objects based on the wearable extended reality device moving a threshold distance (e.g., 1 meter, 2 meters, 3 meters, 5 meters, 10 meters, or any other desired length) from the docked keyboard. In some examples, the at least one rule may be based on at least one of a keyboard type, a time of day, or another virtual object docked to the keyboard. The rule may include any instruction that causes an effect based on an occurrence or a condition of occurrence. For example, moving the keyboard relative to the virtual object may cause some change to the virtual object (e.g., the virtual object may move with the keyboard). Alternatively or additionally, the effect may be conditional. For example, at least one rule may be configured to cause the at least one processor to perform a first virtual object action on a virtual object for a keyboard in an office and a second virtual object action on a virtual object for a keyboard in a home. As another example, the at least one rule may be configured to cause the at least one processor to perform a first virtual object behavior on a virtual object during non-business hours and a second virtual object behavior on the virtual object during business hours.As a further example, the at least one rule may be configured to cause the at least one processor to perform a first virtual object behavior of the virtual object when no other virtual objects are docked to the keyboard, and to perform a second virtual object behavior of the virtual object when the other virtual objects are docked to the keyboard.

[0163] In some embodiments, when the particular physical object and the particular virtual object are outside a portion of a field of view associated with a display system of the wearable extended reality device such that the particular virtual object is not visible to a user of the wearable extended reality device, these embodiments may include receiving a gesture input indicating that a hand is interacting with the particular virtual object. After the particular virtual object is docked with the particular physical object, the user may move the wearable extended reality device, which may change the position and / or orientation of the wearable extended reality device. Based on the change in position and / or orientation, the portion of the field of view associated with the display system of the wearable extended reality device may not cover the location where the particular virtual object is docked. Thus, the particular virtual object may not be displayed to the user and may become invisible to the user. When the location where the particular virtual object is docked is outside a portion of a field of view associated with the display system, the user may make a gesture to interact with the location where the particular virtual object is docked.

[0164] At least one image sensor of the wearable extended reality device can capture image data within the field of view of the at least one image sensor and can detect gesture input. The gesture input can include, for example, dragging, pinching, spreading, swiping, tapping, pointing, grabbing, scrolling, rotating, flicking, touching, zooming in, zooming out, thumbs up, thumbs down, touch and hold, or any other movement of one or more fingers or hands. In some examples, the gesture input can be determined from an analysis of image data captured by the at least one image sensor of the wearable extended reality device. Based on the image data (e.g., based on analyzing the image data with a gesture recognition algorithm), the at least one processor can determine that the gesture input can indicate that a hand or a portion thereof is interacting with a particular virtual object. The hand interacting with the particular virtual object can be determined based on a hand interacting with a particular physical object. In some examples, the hand interacting with the particular virtual object can be a hand of a user of the wearable extended reality device. In some examples, the hand interacting with a particular virtual object may be the hand of someone other than the user of the wearable extended reality device.

[0165] In some examples, the gesture input may be determined from an analysis of additional sensor data acquired by an additional sensor associated with an input device connectable to the wearable extended reality device. The additional sensor may be located on, connected to, or associated with any other physical element. For example, the additional sensor may be associated with a keyboard. Thus, even if the user looks away from the keyboard, rendering the keyboard outside the user's field of view, the additional sensor on the keyboard can detect the gesture. The additional sensor is preferably, but not necessarily, an image sensor. Furthermore, a keyboard-related sensor is merely an example. The additional sensor may be associated with any item. For example, the additional sensor may include a sensor on a glove, such as a haptic glove, a wired glove, or a data glove. The additional sensor may include a hand position sensor (e.g., an image sensor or a proximity sensor included in an item). Additionally or alternatively, the additional sensor may include a sensor described herein with respect to a hand position sensor included in a keyboard. The sensor may be included in an input device other than a keyboard. The additional sensor data may include the position and / or posture of a hand or hand portion. In some examples, the particular physical object may include a portion of a touchpad, and the gesture input may be determined based on an analysis of touch input received from the touchpad. In some examples, the particular physical object may include a sensor, and the gesture input may be received from the sensor and / or may be determined based on data received from the sensor.In some examples, when a particular physical object and a particular virtual object are within a portion of a field of view associated with a display system of the wearable extended reality device, but the particular virtual object is not visible to a user of the wearable extended reality device because it is obscured by another object (virtual or physical), some disclosed embodiments may include receiving gesture input indicating that a hand is interacting with the particular virtual object from analysis of further sensor data acquired by a further sensor associated with an input device connectable to the wearable extended reality device, for example, as described above.

[0166] Referring to FIG. 10 , in step 1020, instructions contained on a non-transitory computer-readable medium, when executed by a processor, cause the processor to receive gesture input indicating a hand is interacting with a particular virtual object when the particular physical object and the particular virtual object are outside a portion of a field of view associated with a display system of the wearable extended reality device, such that the particular virtual object is not visible to a user of the wearable extended reality device.

[0167] 19 , the position and / or orientation of the wearable extended reality device may change, and the field of view 1310 and field of view portion 1312 may change their respective coverage areas. A particular point or area where a particular virtual object (e.g., virtual widget 114E) is docked may move outside of the field of view portion 1312, making the virtual widget 114E invisible to the user. A gesture 1910 may indicate an interaction with the point or area where the virtual widget 114E is docked. The gesture 1910 may occur within the field of view 1310 and may be captured by at least one image sensor of the wearable extended reality device. Based on the gesture 1910, the at least one processor may determine that a hand may be interacting with the virtual widget 114E.

[0168] Some disclosed embodiments may include triggering an output associated with a particular virtual object in response to a gesture input. The output associated with the particular virtual object may be based on the gesture input and may be generated as if the gesture input were interacting with the particular virtual object, even if the particular virtual object is not visible to a user of the wearable extended reality device. For example, if the gesture input is a tap, the output associated with the particular virtual object may include activating the particular virtual object. If the gesture input is a scroll up or scroll down, the output associated with the particular virtual object may include, for example, moving a scroll bar up or down. In some examples, triggering the output associated with the particular virtual object may include modifying a previous output associated with the particular virtual object. Thus, a user may be permitted to manipulate a particular virtual object even if the virtual object is outside the user's field of view. Such operations may include, for example, changing a volume (if the virtual object is a volume control), rotating or repositioning another virtual object associated with the particular virtual object (if the particular virtual object is a positioning control), changing a display parameter (if the particular virtual object is a display parameter control), moving a particular virtual object from outside the user's field of view into the user's field of view, and invoking a function or any other action, depending on the system designer's selection.

[0169] 10 , in step 1022, instructions contained in a non-transitory computer-readable medium, when executed by a processor, can cause the processor to provide an output associated with a particular virtual object in response to a gesture input. Referring to FIG. 19 , gesture 1910 may include, for example, scrolling up. In response to gesture 1910, at least one processor can cause an output associated with virtual widget 114E. For example, virtual widget 114E may be a volume bar, and the output associated with virtual widget 114E may increase the volume.

[0170] As alluded to above, in some examples, the particular virtual object may be a presentation control, and the output may include a change in a presentation parameter associated with the presentation control. The presentation control may include, for example, a start button, a next page button, a previous page button, an end button, a pause button, a last seen button, a volume bar, a zoom in button, a zoom out button, a viewpoint selection, a security control, a privacy mode control, or any other function for controlling a presentation. In response to a gesture input indicating an interaction with a presentation control (e.g., which may be invisible to the user), at least one processor may cause a change in a presentation parameter associated with the presentation control (e.g., a change in the page number of the presentation). In some examples, the presentation parameter may include at least one of a volume, a zoom level, a viewpoint, a security, or a privacy mode. For example, the volume of the presentation may be changed. As another example, the zoom level of the presentation may be changed, or the viewpoint of the presentation may be changed. For example, a change in a presentation parameter may change the security level of the presentation to change a viewer's access to the presentation based on security credentials or levels associated with the viewer. As yet another example, a change in a presentation parameter may change the privacy mode of the presentation to change a viewer's access to the presentation based on the viewer's trust level.

[0171] In some examples, the particular virtual object may be an icon of an application, and the output may include virtual content associated with the application within a portion of a field of view associated with a display system of the wearable extended reality device. For example, the particular virtual object may be an icon of an application, such as a weather application, an email application, a web browser application, an instant messaging application, a shopping application, a navigation application, or any other computer program for performing one or more tasks. In response to gesture input indicating an interaction with the application icon (e.g., which may be invisible to the user), the at least one processor may, for example, launch the application and display virtual content of the application within a portion of a field of view associated with a display system of the wearable extended reality device.

[0172] Some disclosed embodiments may include recommending default positions for multiple virtual objects based on associated functionality. The default position may be a specific point or area where the virtual objects may be docked. The at least one processor may determine functionality associated with multiple virtual objects displayed in a portion of a field of view associated with a display system of the wearable extended reality device, for example, by accessing a data structure that associates virtual objects with functionality. Based on the functionality, the at least one processor may determine default positions for the multiple virtual objects. For example, if a virtual object is associated with a function that controls volume, the default position for the virtual object may include, for example, a top area of ​​a keyboard, a corner of a table, or any other desired location for docking the virtual object. If the virtual object is associated with a function of an application (e.g., an email application), the default position for the virtual object may include, for example, a center area of ​​a table or any other desired location for docking the virtual object. In another example, if a particular virtual object function is associated with one or more gestures that benefit from haptic feedback, the default position for the particular virtual object may be on a surface. The at least one processor may provide a default position recommendation to a user, for example, when a virtual object is selected for docking. In some examples, the recommended default position may be displayed to the user as an overlay on top of a physical object in a portion of the field of view associated with the display system of the wearable extended reality device.

[0173] In some examples, a wearable extended reality device may be configured to pair with multiple different keyboards. Some disclosed embodiments may include varying the display of a particular virtual object based on the selected paired keyboard. The different keyboards may be particular physical objects for docking a particular virtual object. For example, a first keyboard may be in a home and associated with a first configuration of a particular virtual object. A second keyboard may be in an office and associated with a second configuration of a particular virtual object. The first configuration may be different from the second configuration. The different keyboards may be paired with a wearable extended reality device (e.g., usable as an input device for the wearable extended reality device). A particular virtual object (e.g., a volume bar) may be docked to a first keyboard and a second keyboard. A particular virtual object may be configured in different configurations and may appear differently when docked with different keyboards. For example, a particular virtual object may be configured in a first setting when docked with a first keyboard at home and in a second setting when docked with a second keyboard at office. As another example, a particular virtual object may appear to have a larger size when docked with a first keyboard at home than when docked with a second keyboard at office.

[0174] Some disclosed embodiments may include virtually displaying a specific virtual object in proximity to the specific physical object upon identifying that the specific physical object enters a portion of a field of view associated with a display system of the wearable extended reality device. The specific physical object (e.g., a specific point or area on a table) may enter a portion of a field of view associated with the display system of the wearable extended reality device. The at least one processor may identify the entry of the specific physical object into the portion of the field of view, for example, based on image data captured by at least one image sensor of the wearable extended reality device. The newly captured image data may be compared to stored image data, and the physical object may be recognized based on the comparison. This may occur even if only a portion of the physical object enters the field of view, as the comparison may be performed on only a portion of the physical object (e.g., the comparison may be performed on less than a complete set of image data of the physical object). Upon recognition of the physical object or a portion thereof, the at least one processor may cause a virtual presentation of the specific virtual object in proximity to the specific physical object (e.g., even if the specific physical object is not completely within the portion of the field of view). In some examples, a particular virtual object may be docked in a position proximate to a particular physical object (e.g., a keyboard). For example, a volume bar may be docked in a position next to the keyboard. The at least one processor may cause a virtual presentation of the particular virtual object in proximity to the particular physical object upon identifying entry of the particular physical object in the portion of the field of view.

[0175] Some disclosed embodiments may include determining whether a hand interacting with a particular virtual object is the hand of a user of a wearable extended reality device. (As used throughout, references to a hand include a complete hand or a portion of a hand.) Hand source recognition may be performed in any one of several ways, or a combination of ways. For example, based on the relative position of the hand, the system may determine that the hand belongs to a user of the wearable extended reality device. This may occur because the wearer's hand may have an orientation in the field of view that is a telltale sign that the hand is the wearer's hand. This may be determined by performing image analysis on an image of the current hand and comparing it to stored images or image data associated with the orientation of the wearer's hand. Similarly, the hands of people other than the wearer may have different orientations, and the system may determine that a detected hand belongs to someone other than the wearer, just as it may determine that the hand belongs to one of the wearers of the extended reality device.

[0176] As another example, hands of different individuals may be different. The system may come to recognize a hand as that of the extended reality device wearer by examining unique characteristics of the skin or structure. As the wearer uses the system, characteristics of the wearer's hand may be stored in a data structure, and image analysis may be performed to confirm that the current hand in an image is the wearer's hand. Similarly, hands of individuals other than the wearer may be recognized, allowing the system to distinguish between multiple individuals based on the characteristics of their hands. This characteristic may enable unique system control. For example, when multiple individuals interact with the same virtual object in the same virtual space, the control and movement of the virtual object may vary based on who is interacting with the virtual object.

[0177] Thus, the at least one processor can determine, for example, based on image data captured by at least one image sensor of the wearable extended reality device, whether a hand interacting with a particular virtual object (e.g., gesture 1910) is a hand of a user of the wearable extended reality device. The at least one processor can analyze the image data for hand identification, for example, by determining hand characteristics based on the image data and comparing the determined hand characteristics to stored characteristics of the hand of the user of the wearable extended reality device. In some examples, the at least one processor can perform hand identification based on a particular object associated with the hand (e.g., a particular ring to identify the user's hand). Some disclosed embodiments may include triggering an output associated with the particular virtual object in response to a hand and gesture input interacting with the particular virtual object being a hand of a user of the wearable extended reality device. For example, if the hand interacting with the particular virtual object (e.g., gesture 1910) is a hand of a user of the wearable extended reality device, the at least one processor can trigger an output associated with the particular virtual object. Further disclosed embodiments may include refraining from triggering an output associated with a particular virtual object in response to a hand interacting with the particular virtual object not being a hand of a user of the wearable extended reality device. For example, if a hand interacting with the particular virtual object (e.g., gesture 1910) is not a hand of a user of the wearable extended reality device, the at least one processor may refrain from triggering an output associated with the particular virtual object.

[0178] When working in an extended reality environment, a user may have difficulty knowing whether the system recognizes the intended object through a kinematic selection (e.g., through a gesture or through gaze) or when the kinematic selection is precise enough to indicate a single object. Therefore, it may be useful to provide the user with intermediate feedback indicating that the kinematic selection is not precise enough to indicate a single object. For example, at the start of a gesture, if the gesture is not specific enough, a group of virtual objects in the direction of the gesture may be highlighted. As the user refines the intended selection, surrounding objects may become less highlighted until only the selected object remains highlighted. This may indicate to the user that the initial gesture is in the correct direction but needs to be further refined to indicate a single virtual object.

[0179] Various embodiments may be described with reference to systems, methods, apparatus, and / or computer-readable media. One disclosure is intended to be a full disclosure. For example, it should be understood that disclosure of one or more processes embodied in a non-transitory computer-readable medium, as described herein, may also constitute a disclosure of a method implemented by the computer-readable medium, as well as a disclosure of a system and / or apparatus for implementing the processes embodied in the non-transitory computer-readable medium, e.g., via at least one processor. Thus, in some embodiments, a non-transitory computer-readable medium may include instructions that, when executed by at least one processor, cause the at least one processor to perform incremental convergence operations in an augmented environment. Some aspects of such processes may be performed electronically over a network, which may be wired, wireless, or both. Other aspects of such processes may be performed using non-electronic means. In the broadest sense, the processes disclosed herein are not limited to particular physical and / or electronic means; rather, they may be accomplished using any number of different means. As previously mentioned, the term non-transitory computer-readable medium should be interpreted expansively to encompass any medium capable of storing data in any memory in a manner that can be read by any computing device having at least one processor to execute operations, methods, or any other instructions stored in the memory.

[0180] Non-transitory computer-readable media can include, for example, random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), flash drive, disk, any optical data storage medium, any physical medium with a pattern of holes, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), FLASH-EPROM or any other flash memory, non-volatile random access memory (NVRAM), cache, registers, any other memory chip or cartridge, or networked versions thereof. Non-transitory computer-readable media can refer to multiple structures, such as multiple non-transitory computer-readable media located in local or remote locations. Furthermore, one or more non-transitory computer-readable media can be utilized in performing a computer-implemented method. Thus, non-transitory computer-readable media can include tangible items and can exclude carrier waves or transient signals.

[0181] The instructions contained in the non-transitory computer-readable medium may include, for example, software instructions, computer programs, computer code, executable instructions, source code, machine instructions, machine language programs, or any other type of instructions for a computing device. The instructions contained in the non-transitory computer-readable medium may be based on one or more various types of desired programming languages ​​and may include (e.g., embody) various processes for detecting, measuring, processing, and / or implementing incremental convergence in the extended reality environments described herein or for providing feedback based thereon.

[0182] At least one processor may be configured to execute instructions contained on a non-transitory computer-readable medium to cause various processes to be performed to implement incremental convergence in the extended reality environment described herein. The processor may include, for example, an integrated circuit, microchip, microcontroller, microprocessor, central processing device (CPU), graphics processing device (GPU), digital signal processor (DSP), field programmable gate array (FPGA), or other unit suitable for executing instructions or performing logical operations. The processor may include a single-core or multi-core processor. In some examples, the processor may be a single-core processor configured with virtualization technology. The processor may implement virtualization technology or other functionality to provide, for example, the ability to run, control, execute, manipulate, or store multiple software processes, applications, or programs. In another example, the processor may include a multi-core processor (e.g., dual-core, quad-core, or any desired number of cores) configured to provide parallel processing capabilities to enable a device associated with the processor to run multiple processes simultaneously. Other types of processor configurations may be implemented to provide the functionality described herein.

[0183] Some disclosed embodiments may relate to performing incremental convergence operations in an extended reality environment (such as a virtual reality environment, an augmented reality environment, or a mixed reality environment). The term incremental convergence may relate to or otherwise represent any increase or decrease, as well as the start and stop of a user's physical interaction with the extended reality environment. The user's physical interaction with the extended reality environment may serve as a motor input that can enable the user to select or otherwise interact with a virtual object within the coordinate system of the extended reality environment. The interaction with the extended reality environment may include hand gestures, body gestures, eye movements, and / or any gesture-based interaction or combination of gesture-based interactions. The incremental convergence operation may relate to a process for honing in on a selection or item. This may occur gradually over time, such that targeting abruptly approaches or approaches a final selection or identified item. The gradual convergence may occur based on feedback that helps direct the selection. For example, feedback may be provided to a user during a kinematic selection process of a virtual object within an extended reality environment when the user approaches or otherwise interacts with at least one of a plurality of virtual objects displayed in a user interface of the extended reality environment.

[0184] Some disclosed embodiments may include displaying multiple distributed virtual objects across multiple virtual regions. A virtual object may relate to virtual content, including any type of data representation, that is visually displayed to a user in an extended reality environment via an extended reality device and with which the user can interact. Virtual objects may be considered distributed if they are not all located in the exact same location. For example, virtual objects may be considered distributed if they appear in different locations on a virtual display or extended reality environment. Objects located adjacent to each other are an example of distributed objects. A virtual display or extended reality environment may have regions that are close to each other or far apart from each other. In either case, the objects may be considered distributed across multiple regions.

[0185] A virtual object may include any visual presentation rendered by a computer or processing device, such as an inanimate or animate virtual object. In some cases, a virtual object may be configured to change over time or in response to a trigger. In some embodiments, a virtual object of a plurality of distributed virtual objects may include at least one virtual widget and / or option in a virtual menu. A virtual object, such as at least one virtual widget and / or an option in a virtual menu, may be used to issue a command, initiate a dialogue sequence, change an interaction mode, and / or give a user quick access to information without having to fully open the associated virtual object. Additionally or alternatively, a virtual object may include at least one icon that can activate a script to trigger an action associated with the particular virtual object associated with the icon and / or may be linked to an associated program or application.

[0186] The plurality of distributed virtual objects may relate to any number of virtual objects that may be visually displayed or presented to a user via a user interface of an extended reality environment. For example, the plurality of distributed virtual objects may be distributed across a virtual computer screen (also referred to herein as a virtual display) that displays virtual objects generated by an operating system. In some embodiments, the plurality of virtual regions are plurality of virtual regions of an extended reality environment, and displaying the plurality of distributed virtual objects may include causing a wearable extended reality device to display the plurality of distributed virtual objects.

[0187] The user interface of the extended reality environment may be configured to display multiple virtual regions. The multiple virtual regions may relate to at least two separate or connected subsets of space within the overall space of the user interface. Each of the multiple virtual regions may be defined by at least one of its boundary, an interior region, an exterior region, or elements (e.g., virtual objects) included in the virtual region. The boundary may or may not be visually represented. For example, in a visual sense, the boundary of a given virtual region may be represented by any type of closed geometric shape that encompasses a subset of the overall space of the user interface and can form an interface between the interior and exterior regions of the virtual region. The interior region of the virtual region may relate to a subset of the space inside the boundary of the virtual region and may include at least one of multiple distributed virtual objects. Thus, a given virtual region may include a single virtual object or multiple virtual objects. The exterior region of the virtual region may relate to a subset of the space outside the boundary of the virtual region and may include a virtual object included inside the virtual region or multiple distributed virtual objects excluding virtual objects. A virtual object located in an exterior region of a virtual region may be located in an interior region of another one of the virtual regions. In other embodiments, color or other visual characteristics may indicate different regions. In other embodiments, regions may be activated together without a perceptible divider.

[0188] The multiple virtual regions may include at least a first virtual region and a second virtual region different from the first virtual region. This term may indicate that the first virtual region includes at least one virtual object of the multiple distributed virtual objects that is not included in the second virtual region, and the second virtual region may include at least one virtual object of the multiple distributed virtual objects that is not included in the first virtual region. Thus, the multiple virtual regions include at least one non-overlapping region with a given virtual object. Additionally or alternatively, not all of the virtual objects in the first virtual region may be included in the second virtual region, and not all of the virtual objects in the second virtual region may be included in the first virtual region. In some embodiments, the first virtual region may be selected based on an initial movement input. As described in more detail below, the initial movement input may include any gesture-based interaction in which a user begins to show interest in or otherwise pay attention to a virtual region, such as the first virtual region, for example, by initially pointing and / or looking at the first virtual region.

[0189] In some embodiments, the boundaries of the first virtual region may be adaptable in real time in response to motion input. For example, the boundaries of the first virtual region may be dynamically adaptable to include or exclude different numbers of the plurality of distributed virtual objects based on the user's motion input within the coordinate system of the extended reality environment. In one example, the boundaries of the first virtual region may encompass a first number of the plurality of virtual objects based on a threshold confidence that the user may intend to make a kinematic selection of one of the first number of the plurality of virtual objects. In one non-limiting example, if at least one processor determines that the user may intend to select at least one of several adjacent virtual objects of the plurality of virtual objects based on the user pointing at the virtual object, the boundaries may be defined to include each of the virtual objects that the user may intend to interact with above a threshold confidence, e.g., above 50% confidence. In another example, the boundaries of the first virtual region may encompass a second number of the plurality of virtual objects that is less than the first number of the plurality of virtual objects based on a higher threshold confidence that the user may intend to make a kinematic selection of one of the second number of the plurality of virtual objects.

[0190] 20 is a flowchart illustrating an example process 2000 for providing feedback based on incremental convergence in an extended reality environment, consistent with certain embodiments of the present disclosure. Referring to step 2010 of FIG. 20 , instructions included on a non-transitory computer-readable medium, when executed by at least one processor, can cause the at least one processor to display multiple distributed virtual objects across multiple virtual regions. By way of example, FIG. 21 illustrates a non-limiting example of an extended reality environment displaying multiple distributed virtual objects across multiple virtual regions, consistent with certain embodiments of the present disclosure.

[0191] 21 , a user interface 2110 is presented within an exemplary extended reality environment from the perspective of a user of the extended reality environment. The user interface 2110 can display virtual content to a user, including a plurality of distributed virtual objects 2112, across the user interface 2110, such that the user can interact with the plurality of distributed virtual objects 2112. The plurality of distributed virtual objects 2112 presented to a user of the extended reality environment can include at least one interactive virtual widget and / or option within a virtual menu. Within a given virtual object of the plurality of distributed virtual objects 2112, there can be additional virtual objects with which the user can interact. In some examples, the user interface 2110 can be associated with a virtual computer screen configured to display a plurality of distributed virtual objects 2112 generated by an operating system to a user, such that the user can interact with the plurality of distributed virtual objects 2112.

[0192] The plurality of distributed virtual objects 2112 may be displayed across multiple virtual regions of the user interface 2110. For example, the user interface 2110 may include a first virtual region 2114 including a first subset of the plurality of distributed virtual objects 2112 and a second virtual region 2116 including a second subset of the plurality of distributed virtual objects 2112. The boundary of the first virtual region 2114 is depicted as a short-dashed line and is shown to encompass the first subset of the plurality of distributed virtual objects 2112, and the boundary of the second virtual region 2116 is depicted as a short-dashed line and is shown to encompass the second subset of the plurality of distributed virtual objects 2112, such that each of the plurality of distributed virtual objects 2112 is included in either the first virtual region 2114 or the second virtual region 2116. It should be understood that the boundaries of the virtual regions may be invisible to the user and are shown herein for illustrative purposes. Additionally, the boundaries of the first virtual area 2114 and the second virtual area 2116 in this figure are depicted for simplicity to include arbitrarily selected first and second subsets of the plurality of distributed virtual objects 2112 and may vary to include different numbers of the plurality of distributed virtual objects 2112 based on several factors, including the user's past and / or current interactions with the extended reality environment.

[0193] Some disclosed embodiments can include receiving an initial motor input trending toward the first virtual region. The term motor input can relate to natural human movements and / or gestures of a user. Such movements can occur in a real coordinate system that can be tracked and / or captured as input data that can be translated into virtual actions within a virtual coordinate system of the extended reality environment. The user's motor input can include hand gestures, body gestures (e.g., movements of the user's head or body), eye movements or other eye movements, and / or any other gesture-based interaction, facial movement, or a combination of gesture-based interactions. In another example, the user's motor input can include movement of a computer cursor in a virtual display and / or the extended reality environment. The user's motor input can be detected and translated into virtual interactions with the extended reality environment, thereby enabling the user to select or otherwise interact with virtual objects within the extended reality environment.

[0194] In some embodiments of the present disclosure, receiving the motor input may involve capturing and / or recognizing at least one motor input by components of the wearable extended reality device, such as a stereoscopic head-mounted display, a head motion tracking sensor (e.g., gyroscope, accelerometer, magnetometer, image sensor, structured light sensor), an eye-tracking sensor, and / or any other component capable of recognizing at least one of a user's various sensory motor inputs and converting the sensed motor input into a readable electrical signal. In some examples, these components may be utilized to detect or measure the velocity and / or acceleration of the user's motor input, and to detect or otherwise measure the time difference between particular motor inputs in order to distinguish a user's intentions based on the user's subtle gesture-based interactions.

[0195] Furthermore, the term initial motor input may refer to any motor input that causes a user to begin to show interest in or otherwise pay attention to a virtual area, such as a first virtual area including at least one virtual object. A user may express interest in or otherwise pay attention to a first virtual area including at least one virtual object by generating a motor input that includes at least one slight movement toward the first virtual area including at least one virtual object. This slight movement toward the first virtual area may allow the user to receive nearly seamless feedback of the sensed motor input within the extended reality environment and intuitively discover the results of the sensed motor input. For example, a user attempting to make a kinematic selection of a virtual object within the extended reality environment through an initial motor input, such as initially pointing and / or looking at a first virtual area including a group of virtual objects, may be provided with initial and / or intermediate feedback based on the initial motor input if the interaction intent is not specific enough to identify an intent to select a particular virtual object of interest.

[0196] In some examples, the first region and / or the second region may be predetermined, for example, based on a system configuration. In some other examples, the first region and / or the second region may be dynamically created, modified, and / or destroyed. In some examples, the first region and / or the second region may be determined based on a plurality of distributed virtual objects. In one example, changes to the plurality of distributed virtual objects (e.g., adding a virtual object, removing a virtual object, changing a virtual object's position, changing a virtual object's state, etc.) can cause changes to the first region and / or the second region. Some non-limiting examples of such states of virtual objects include a virtual object not associated with a notification, a virtual object associated with a notification, a closed virtual object, an open virtual object, an active virtual object, an inactive virtual object, etc.

[0197] In one example, at a given time (e.g., at system startup, upon change, etc.), a clustering algorithm may be used to cluster virtual objects into regions based on, for example, at least one of the virtual object's location, the virtual object's identity, or the virtual object's state. In another example, at a given time (e.g., at system startup, upon change, etc.), a segmentation algorithm may be used to segment at least a portion of the extended reality environment and / or at least a portion of the virtual display into regions.

[0198] In some examples, the first region and / or the second region may be determined based on the dimensions of the virtual display and / or the extended reality environment. In one example, a change in the dimensions of the virtual display and / or the extended reality environment can cause the first region and / or the second region to change. In one example, at a given time (e.g., at system startup, at a change, etc.), the virtual display and / or the extended reality environment may be divided into, for example, a grid of a predetermined number of equally sized (or approximately equally sized) regions, regions of a predetermined fixed size, and / or regions of various sizes.

[0199] In some examples, the first region and / or the second region may be determined based on conditions related to the extended reality device (such as ambient lighting conditions in the extended reality device's environment, movement of the extended reality device, or mode of use of the extended reality device). In one example, a change in conditions related to the extended reality device can cause a change in the first region and / or the second region. In one example, brighter ambient lighting in the extended reality device's environment can trigger use of a larger region. In another example, a moving extended reality device can have a larger region than a non-moving extended reality device.

[0200] In some examples, the first region and / or the second region may be determined based on a motor input, such as based on at least one of a direction, a position, a pace, or an acceleration associated with the motor input. For example, a machine learning model may be trained using training examples to determine regions based on the motor input. One example of such training examples may include a sample motor input and a sample distribution of a sample virtual object, along with a label indicating a desired region corresponding to the sample motor input and the sample distribution. In another example, a motor input associated with a faster movement may cause the determination of a region farther from a position associated with the motor input than a motor input associated with a slower movement.

[0201] 20, instructions included in a non-transitory computer-readable medium, when executed by at least one processor, may cause the at least one processor to receive an initial motion input tending toward a first virtual region. In one example, the initial motion input may be received after displaying multiple distributed virtual objects across the multiple virtual regions according to step 2010. By way of example, FIG. 22 illustrates a non-limiting example of a user's initial motion input tending toward a first virtual region of the extended reality environment of FIG. 21, consistent with some embodiments of the present disclosure.

[0202] 22 , the user interface 2210 of the extended reality environment includes a first subset of the plurality of distributed virtual objects 2212 in a first virtual region 2214 grouped in a different region than a second subset of the plurality of distributed virtual objects 2216 in a second virtual region 2218. The first virtual region 2214 may be dynamically defined to include any number of the plurality of distributed virtual objects, and a user 2220 begins to show interest in or otherwise pay attention to the plurality of distributed virtual objects by generating an initial motor input including at least one gesture toward the virtual objects. For example, the first subset of the plurality of distributed virtual objects 2212 to be included in the first virtual region 2214 may be determined based on the initial motor input from the user 2220. If it is determined that the user 2220 may wish to interact with the first subset of the plurality of distributed virtual objects 2212, the first virtual region 2214 may be defined to include the virtual objects 2212. In response to any given initial motor input, the number of virtual objects 2212 included in the first virtual region 2214 may vary based on the specificity and / or determined confidence level associated with the user's 2220 initial motor input.

[0203] 22 , a user 2220 is shown expressing interest in at least one virtual object of a first subset of the plurality of distributed virtual objects 2212 in a first virtual region 2214 by a hand gesture pointing to the first virtual region 2214 that includes the first subset of the plurality of distributed virtual objects 2212. At the time shown in FIG. 22 , the user 2220's initial motor input, which is intended to make a kinematic selection of at least one of the first subset of the plurality of distributed virtual objects 2212, may not be specific enough to clearly detect a particular one of the virtual objects 2212 with which the user 2220 may wish to interact. However, the at least one processor can determine that the user 2220's initial motor input is not intended to make a kinematic selection of at least one of a second subset of the plurality of distributed virtual objects 2212 based on the user's hand gesture.

[0204] It should be understood that the hand of user 2220 shown in the figure does not necessarily represent a virtual image of the hand as seen by user 2220. Rather, in the illustrated example, the illustrated virtual hand of user 2220 is intended to be an illustration of one type of motor input in a real coordinate system that can be captured and translated into a virtual interaction in a corresponding virtual coordinate system of the extended reality environment, thereby allowing the user to select or otherwise interact with a virtual object in the extended reality environment. However, in some embodiments, a virtual image of user 2220's hand can also be superimposed on the extended reality environment. Furthermore, in some embodiments, the initial motor input of user 2220 may additionally or alternatively include an eye gesture by user 2220.

[0205] In some embodiments, the initial motor input may include a combination of gaze detection and gesture detection. The gaze detection and gesture detection of the initial motor input may relate to capturing and / or recognizing a user's hand and / or body gestures and / or eye movements. In a general sense, gaze detection and gesture detection may be performed in a manner similar to the detection of the user's motor input described above. The gaze detection and gesture detection of the user's gestures and eye movements may be performed simultaneously and both may be used to cause at least one processor to determine the user's intention of the combined initial motor input. In one example, if the detected gaze is toward a first virtual area and the detected gesture is toward a second virtual area, the non-transitory computer-readable medium may cause at least one processor to highlight a group of virtual objects in the first virtual area. In another example, if the detected gaze is toward the first virtual region and the detected gesture is toward an area without virtual objects, the non-transitory computer-readable medium can cause the at least one processor to highlight a group of virtual objects in the first virtual region.

[0206] In some examples, determining a confidence level associated with a motor input, such as an initial motor input, can include determining a confidence level associated with gaze detection and a confidence level associated with gesture detection. Additionally or alternatively, a decision to prioritize a detected gaze over a detected gesture, and vice versa, may be informed or otherwise weighted based on analysis of motor inputs, including past and / or real-time motor inputs, using previously stored user interactions with the extended reality environment, pre-selected and / or analyzed user preferences, artificial intelligence, machine learning, deep learning, and / or neural network processing techniques.

[0207] 22 , a user 2220 may simultaneously indicate a first interest in at least one virtual object of a first subset of the plurality of distributed virtual objects 2212 in the first virtual region 2214 with a first gesture toward the first virtual region 2214, and may indicate a second interest in at least one virtual object of a second subset of the plurality of distributed virtual objects 2216 in the second virtual region 2218 with a second gesture toward the second virtual region 2218. Alternatively, the user's second interest may be directed, with the second gesture, toward a region other than the first virtual region 2214 that does not include any virtual objects.

[0208] For example, the user 2220 may simultaneously indicate one interest in at least one virtual object in one virtual region via a gaze and a second interest in at least another virtual object or a region without virtual objects in a different virtual region via a hand gesture. In some cases, when a gesture toward the first virtual region 2214 is associated with the user's 2220 gaze and a gesture toward the second virtual region 2218 is associated with a hand gesture, the non-transitory computer-readable medium can cause the at least one processor to highlight a group of virtual objects 2212 in the first virtual region 2214 based on the detected gaze and the detected gesture. In other examples, when a gesture toward the first virtual region 2214 is associated with the user's 2220 gaze and a hand gesture tends toward a region without virtual objects, the non-transitory computer-readable medium can cause the at least one processor to highlight a group of virtual objects 2212 in the first virtual region 2214 based on the detected gaze and the detected gesture.

[0209] Some disclosed embodiments may include highlighting a group of virtual objects in a first virtual region based on an initial motor input. The group of virtual objects in the first virtual region that may be highlighted in response to the initial motor input may include any set of virtual objects in the region for which a user may show interest or pay attention. This may occur, for example, through a gesture and / or gaze in the direction of the group. The number of virtual objects included in the group of virtual objects in the first virtual region in response to any given initial motor input may depend on the specificity of the user's motor input and / or a threshold confidence level regarding the intent of the motor input. Furthermore, the first virtual region may be dynamically defined to include a group of adjacent virtual objects for which a user begins to show interest or otherwise pay attention by generating an initial motor input for the group of virtual objects. In some embodiments, the highlighted group of virtual objects in the first virtual region may include at least two virtual objects of a plurality of distributed virtual objects. In one example, the highlighted group of virtual objects in the first virtual region may include all of the virtual objects in the first virtual region. In another example, the group of highlighted virtual objects in the first virtual region may include some but not all of the virtual objects in the first virtual region.

[0210] A group of virtual objects in a first virtual region may be highlighted in response to an initial motion input tending toward the group of virtual objects in the first virtual region. In some embodiments, highlighting the group of virtual objects may involve presenting any user-perceivable visual indicator that may serve to highlight or otherwise distinguish the group of objects in the first virtual region relative to virtual objects outside the first virtual region (e.g., virtual objects in a second region). In some embodiments, highlighting the group of virtual objects in the first virtual region based on the initial motion input may include modifying the appearance of each virtual object in the group of virtual objects in the first virtual region. For example, highlighting may include enlarging at least one virtual object in the group, changing a color of at least one virtual object in the group, e.g., causing or otherwise changing a shadow or glow around a frame of at least one virtual object in the group, and / or flashing at least one virtual object in the group, and / or moving at least one virtual object in the group relative to virtual objects in the second virtual region. Additionally or alternatively, highlighting the group of virtual objects in the first virtual area based on the initial movement input may include providing a visual representation of the group of virtual objects in the first virtual area based on the initial movement input.

[0211] In some embodiments, highlighting may include separately highlighting each of the virtual objects in the group of virtual objects in the first virtual region and / or highlighting any portion of the first virtual region that includes the group of virtual objects. The group of virtual objects may be highlighted equally or may be highlighted to varying degrees depending on the specificity with which a user may express interest in any given one or more virtual objects in the group of virtual objects. In some embodiments, highlighting the group of virtual objects includes causing the wearable extended reality device to highlight the group of virtual objects.

[0212] In other embodiments, highlighting the group of virtual objects in the first virtual area based on the initial motor input may include displaying or adding a control that enables interaction with at least one virtual object in the group of virtual objects in the first virtual area. For example, the displayed control may include a button on a virtual object, such as a virtual widget, that, when selected by a user, may activate a script for or otherwise link to a program or application. In other embodiments, the highlighting may relate to any user-perceivable physical indicator that may help to emphasize or otherwise distinguish the group of objects in the first virtual area, for example, by providing physical feedback (e.g., haptic feedback) to the user in response to the motor input.

[0213] 20, instructions contained in a non-transitory computer-readable medium, when executed by at least one processor, can cause the at least one processor to highlight a group of virtual objects in a first virtual area based on an initial movement input. By way of example, FIG. 22 illustrates one non-limiting example of an instance in which a group of virtual objects in a first virtual area is highlighted based on a user's initial movement input, consistent with some embodiments of the present disclosure.

[0214] 22 , when a user 2220 shows interest in at least one virtual object of a group of virtual objects included in a first subset of the plurality of distributed virtual objects 2212 in a first virtual region 2214, the group of virtual objects 2212 in the direction of an initial motor input may be highlighted in a manner visually perceptible to the user 2220. When it is determined that the user 2220 may potentially desire to kinematically select one of the virtual objects in the group of virtual objects 2212, the group of virtual objects 2212 is highlighted in response to the user 2220's initial motor input tending toward the first virtual region 2214. In the example shown in FIG. 22 , the visual appearance of each virtual object of the group of virtual objects 2212 in the first virtual region 2214 is shown to be emphasized or otherwise distinguished from the second subset of distributed virtual objects 2212 in the second virtual region 2218 due to a shadow added around the frame of the virtual object 2212.

[0215] Additionally or alternatively, highlighting the at least one virtual object of the group of virtual objects 2212 in the first virtual region 2214 may include displaying or adding a control associated with the at least one virtual object of the group of virtual objects 2212 with which the user 2220 can interact. Referring to Figure 23, the user 2310 shows a controller 2318 included in the highlighted virtual object 2312 that enables user interaction with the at least one virtual object 2312 in the group of virtual objects in the first virtual region 2314. The highlighting of the at least one virtual object of the group of virtual objects may help to provide near-seamless feedback to the user within the extended reality environment based on the sensed motor input, allowing the user to intuitively discover the results of the sensed motor input.

[0216] Some disclosed embodiments may include receiving a refined motor input tending toward a particular virtual object from among a group of highlighted virtual objects. In a general sense, receiving the refined motor input may be similar to receiving the motor input described above. The refined motor input may be associated with a more precise gesture, gaze, or other action of the user. Such greater precision may enable corresponding selection refinement. For example, in response to perception of the group of highlighted virtual objects, the user may increase the accuracy or precision of the motor input relative to the initial motor input toward a single virtual object of interest from the group of highlighted virtual objects. The refined motor input may be received, detected, or otherwise recognized such that a confidence associated with the refined motor input may be determined by at least one processor to be greater than a determined confidence associated with the initial motor input.

[0217] In some embodiments, receiving the refined motor input may result in canceling the highlighting of at least some of the virtual objects in the group of virtual objects. For example, a user attempting to perform a kinematic selection of virtual objects in an extended reality environment with an initial motor input may be provided with intermediate feedback, such as a highlighted group of virtual objects, based on the initial motor input if the user's interaction intent is less specific. As the user reaches toward a more specific region of the first virtual area toward a specific virtual object that the user may intend to select, subsequent feedback may be provided to the user by de-highlighting virtual objects surrounding the specific virtual object until only the single virtual object intended to be selected remains highlighted based on the user's refined movement. Alternatively, all virtual objects in the group of virtual objects may cease to be highlighted in response to the refined motor input. Additionally or alternatively, de-highlighting at least some of the virtual objects in the group of virtual objects may include ceasing to provide visual representations of at least some of the virtual objects in the group of virtual objects.

[0218] In some embodiments, the initial motor input may be associated with a first type of motor input, and the refined motor input may be associated with a second type of motor input that is different from the first type of motor input. For example, the first motor input may be gaze, and the second motor input may be a gesture such as pointing. Alternatively, the first motor input may be a head rotation toward a particular area of ​​the display, and the second motor input may be a focused gaze on a particular area or item. In a general sense, the first type of motor input and the second type of motor input may include any combination of gesture-based interactions, such as hand gestures, body gestures (e.g., movements of the user's head or body), and eye movements.

[0219] Referring to step 2016 of FIG. 20 , instructions included on a non-transitory computer-readable medium, when executed by at least one processor, can cause the at least one processor to receive a refined motor input tending toward a particular virtual object from among a group of highlighted virtual objects. In one example, the refined motor input tending toward a particular virtual object from among a group of highlighted virtual objects may be received after highlighting a group of virtual objects in the first virtual area per step 2014 and / or after receiving an initial motor input per step 2012. By way of example, FIG. 23 illustrates a non-limiting example of a user's refined motor input compared to the initial motor input shown in FIG. 22 , where the refined motor input tends toward a particular virtual object from among a group of highlighted virtual objects, consistent with some embodiments of the present disclosure. While both the initial motor input and the refined motor input shown in FIGS. 22 and 23 illustrate a user's hand gestures, it should be understood that each of the initial motor input and the refined motor input can include any combination of gesture-based interactions, including hand gestures and / or eye movements.

[0220] 23 , a user 2310 is shown, via refined motor input, indicating a particular interest in a particular virtual object 2312 from among a group of previously highlighted virtual objects in a first virtual region 2314. At the time shown in FIG. 23 , the user's 2310's refined motor input has a clearer tendency toward the particular virtual object 2312 over other virtual objects 2316 in the first virtual region 2314. Thus, the confidence factor associated with the refined motor input shown in FIG. 23 is higher than the confidence factor associated with the initial motor input shown in FIG. 22 , and the refined motor input indicates that the user 2310, more specifically, may wish to make a kinematic selection of the particular virtual object 2312. In response to the refined motor input, subsequent feedback can be provided to the user 2310 by de-highlighting other virtual objects 2312 in the first virtual region 2314 around the particular virtual object 2312 until only the particular virtual object 2316 determined by the at least one processor to be intended for selection above a predetermined confidence factor remains highlighted based on the user's refined movement.

[0221] Some disclosed embodiments may include triggering a function associated with a particular virtual object based on sophisticated motor input. The term "triggering a function" associated with a particular virtual object may include any occurrence resulting from the identification of the particular virtual object. Such occurrences may include changing the appearance of the virtual object or initiating the operation of code associated with the virtual object. Thus, selection of a virtual object may result in the presentation of a virtual menu, the presentation of additional content, or the triggering of an application. When a function associated with a particular virtual object is triggered, the virtual content and / or functionality of the particular virtual object may change from a disengaged or otherwise highlighted state based on the initial motor input to an engaged state in response to selection or engagement of the virtual object via sophisticated motor input. In some embodiments, the function associated with a particular virtual object includes causing the wearable extended reality device to begin displaying particular virtual content.

[0222] The functionality associated with a particular virtual object may include causing an action associated with the particular virtual object, such as being additionally highlighted. Additionally or alternatively, the functionality associated with a particular virtual object may include magnifying the particular virtual object, revealing additional virtual objects for selection within or otherwise related to the particular virtual object, showing or hiding additional viewing areas containing virtual content associated with the particular virtual object, and / or initiating any virtual content or process functionally associated with the particular virtual object based on sophisticated motor input. In some embodiments, if the particular virtual object is an icon, the triggered functionality may include causing an action associated with the particular virtual object, including activating a script associated with the icon.

[0223] 20, instructions contained on a non-transitory computer-readable medium, when executed by at least one processor, can cause the at least one processor to trigger a function associated with a particular virtual object based on the refined motor input. By way of example, FIG. 23 illustrates a non-limiting example of a refined motor input provided by a user that triggers a resulting function associated with a particular virtual object in an extended reality environment, consistent with some embodiments of the present disclosure.

[0224] 23 , in response to the refined motor input of the user 2310, at least one function associated with the particular virtual object 2312 intended for selection may be triggered by the at least one processor, which causes an action associated with the particular virtual object 2312. For example, one function associated with the particular virtual object 2312 that is triggered in this example is to enlarge the particular virtual object in response to the refined motor input of the user 2310. Another function associated with the particular virtual object 2312 that is triggered in this example is to reveal an additional virtual controller 2318 for selection within the particular virtual object 2312. As shown in FIG. 24 , once an action associated with the particular virtual object 2412 is triggered, such as revealing an additional virtual object 2414 for selection within the particular virtual object 2412, the user 2410 can interact with the additional virtual object 2414.

[0225] Other embodiments may include tracking the amount of time that has elapsed since receiving the initial motion input. Time tracking may be used to determine whether to trigger a function or cancel highlighting based on the amount of time that has elapsed between receiving the initial motion input and receiving the refined motion input. For example, if the refined motion input is received within a predetermined period of time following the initial motion input, a function associated with a particular virtual object may be triggered. If the refined motion input is not received within the predetermined period of time, highlighting of a group of virtual objects in the first virtual area may be canceled.

[0226] In some embodiments, the term "predetermined period following an initial movement input" may relate to a measurement of time after the initial movement input is received. For example, the "predetermined period following an initial movement input" may relate to a period of time starting after the initial movement input is detected or otherwise recognized and / or translated into a virtual interaction with the extended reality environment, highlighting a group of virtual objects in a first virtual region. In some cases, tracking of elapsed time may begin when a user initiates the initial movement input from a rest position or from another preliminary movement input. In other examples, tracking of elapsed time may begin when a user terminates or otherwise decelerates the initial movement input below a threshold level. The time at which receipt of a refined movement input begins may be based on a measurement of the distance of the user's initial movement input from a particular one of the group of virtual objects and / or virtual regions containing said virtual objects, and / or a measurement of, or change in, velocity and / or acceleration associated with the user's initial movement input.

[0227] Other embodiments may include receiving a preliminary motor input trending toward the second virtual region before receiving the initial motor input, and highlighting a group of virtual objects in the second virtual region based on the preliminary motor input. In a general sense, the preliminary motor input may be similar to the motor input described above. The term preliminary motor input may refer to any motor input that may cause a user to begin to show interest in or otherwise pay attention to an area containing at least one virtual object before showing interest in another area containing at least one different virtual object. In some examples, at least one virtual object may be in the group of virtual objects from the second virtual region that is included in the group of virtual objects in the first virtual region upon receiving the initial motor input following the preliminary motor input.

[0228] Upon receiving the initial movement input, highlighting of the group of virtual objects in the second virtual region can be canceled and the group of virtual objects in the first virtual region can be highlighted. In one embodiment, removing the highlighting can involve restoring or changing the group of virtual objects that was highlighted in response to the preparatory movement input to an unhighlighted state in response to the initial movement input. For example, the group of virtual objects in the second virtual region that was highlighted in response to the preparatory movement input can be restored to an unhighlighted state, and the new group of virtual objects in the first virtual region can transition from an unhighlighted state to a highlighted state in response to the initial movement input. In one example, removing the highlighting of the group of virtual objects in the second virtual region can occur when the group of virtual objects in the first virtual region is highlighted. In another example, removing the highlighting of the group of virtual objects in the second virtual region can begin to fade or otherwise transition to an unhighlighted state as the group of virtual objects in the first virtual region is highlighted or transitions to a highlighted state.

[0229] 25 is a flowchart illustrating an example process 2500 for highlighting a group of highlighted virtual objects in response to a preliminary movement input and for canceling the highlighting of the group of virtual objects in response to an initial movement input, consistent with some embodiments of the present disclosure. Referring to FIG. 25 , at step 2510, instructions contained on a non-transitory computer-readable medium, when executed by at least one processor, can cause the at least one processor to receive a preliminary movement input trending toward a second virtual region prior to receiving the initial movement input. At step 2512, the at least one processor can highlight the group of virtual objects in the second virtual region based on the preliminary movement input. At step 2514, upon receiving the initial movement input, the at least one processor can cancel the highlighting of the group of virtual objects in the second virtual region and highlight the group of virtual objects in the first virtual region.

[0230] Other embodiments may include determining a confidence level associated with the initial motion input and selecting a visual characteristic of the highlighting based on the confidence level. The term confidence level may relate to a threshold confidence that some of the plurality of distributed virtual objects may be intended to be selected by a user based on a motion input, such as the initial motion input. If the number of the plurality of distributed virtual objects is determined by the at least one processor to fall within the threshold confidence for the likelihood of being selected based on the initial motion input, the group of virtual objects may be included in the group of virtual objects that is highlighted based on the initial motion input. In one non-limiting example of determining a confidence level associated with the initial motion input, if it is determined that the user is interested in or otherwise attending to a number of virtual objects at or above an assessed confidence level, e.g., a 50% confidence level, the interaction may highlight the group of virtual objects. If it is determined that the user is attending to some virtual objects below an assessed confidence level, e.g., a 50% confidence level, the interaction may not highlight the virtual objects.

[0231] In some embodiments, the non-transitory computer-readable medium may cause at least one processor to analyze past virtual object selections to determine a user profile and, based on the user profile, determine a confidence level associated with a motor input, such as an initial motor input. Analyzing past virtual object selections may include any artificial intelligence, machine learning, deep learning, and / or neural network processing techniques. Such techniques may enable machine learning by absorbing large amounts of unstructured data, such as sensed motor inputs, pre-selection recognition processes, and / or videos, and user preferences analyzed over a period of time. Any suitable computing system or group of computing systems may be used to perform the analysis of past and / or current virtual object selections or any other interactions within an extended reality environment using artificial intelligence. Furthermore, data corresponding to the analyzed past and / or current virtual object selections, the user profile, and the confidence levels associated with the user's past and / or current motor inputs may be stored in and / or accessed from any suitable databases and / or data structures, including one or more memory devices. For example, at least one database may be configured and operable to record and / or reference a user's past interactions with the extended reality environment to identify a confidence level associated with a new motor input.

[0232] Other embodiments may include determining a confidence level associated with a refined motor input and refraining from triggering a function associated with a particular virtual object when the confidence level is below a threshold. In a general sense, determining a confidence level associated with a refined motor input may be performed in a manner similar to determining a confidence level associated with an initial motor input, described above. If a particular virtual object is determined by at least one processor to fall within a threshold confidence level for its likelihood of being triggered based on the refined motor input, triggering of a function associated with the particular virtual object may not be initiated, or may otherwise be prevented from being initiated. It should be understood that determining a confidence level associated with a refined motor input, including gestures and eye movements, may include determining a confidence level associated with gaze detection and a confidence level associated with gesture detection. The decision to prioritize a detected gaze over a detected gesture, or vice versa, for refined motor input may be signaled or otherwise weighted in a manner similar to the initial motor input, described above.

[0233] Ceasing a function associated with a particular virtual object in response to a refined motor input may be performed in a similar manner to canceling the highlighting of at least some of the virtual objects in a group of virtual objects in response to the motor input. Ceasing a function associated with a particular virtual object may involve restoring the particular virtual object to a non-engagement state or returning it to a highlighted state, depending on a low confidence level associated with the refined motor input. In one non-limiting example of determining a confidence level associated with a refined motor input, if a user's expressed interest in a particular virtual object is determined to be below an assessed confidence level, e.g., a 90% confidence level, the interaction is found to include a non-specific intent of selection, and triggering of a function associated with the particular virtual object may not be initiated or may be prevented from being initiated. If a user's expressed interest in a particular virtual object is determined to be above an assessed confidence level, e.g., a 90% confidence level, the interaction is found to include a specific intent of selection, and an intermediate action associated with the particular virtual object may be generated, such as zooming in on the virtual object or presenting additional virtual objects for selection within the particular virtual object prior to selection of the virtual object. If it is determined that the user has shown interest in a particular virtual object within a confidence level of approximately 100%, the interaction may trigger some action associated with the particular virtual object.

[0234] In some embodiments, the non-transitory computer-readable medium may cause at least one processor to access a user action history indicating at least one action performed prior to the initial motor input and to determine a confidence level associated with the refined motor input based on the user action history. In a general sense, the action history may be stored, accessed, and / or analyzed in a manner similar to the confidence levels associated with past virtual object selections, user profiles, and previous motor inputs of a user, as described above. Furthermore, the user behavior history may relate to a behavioral history including data from an ongoing session and / or previous sessions, and may relate to any type of gesture-based interaction. In some examples, the user behavior history may relate to data including past and / or current patterns of interaction with an extended reality environment. The behavioral history relating to the ongoing session and / or previous sessions may be used by the at least one processor to determine a confidence level associated with the newly initiated refined motor input.

[0235] In some embodiments, the threshold confidence level may be selected based on a particular virtual object. In one example, a user's previous selection history, user profile, and / or user behavior history associated with a particular virtual object may cause the threshold confidence level for selection of the virtual object to be greater or lower than the threshold confidence level for selection of another virtual object, such as a less frequently used virtual object. In another example, if selection of a particular virtual object could cause user confusion (e.g., pausing or terminating a stream of data) or data loss for the user (e.g., closing an application or powering off a device), the threshold may be selected to require a higher confidence level before triggering a function associated with the particular virtual object. Additionally or alternatively, the threshold confidence level for a particular virtual object may be increased and / or decreased based on a determined increase and / or decrease in the need for the virtual object based on at least one external factor, such as notifications and / or alerts associated with the virtual object or concurrently running programs attempting to send or receive information from a program associated with the virtual object.

[0236] Some embodiments may include determining a first confidence level associated with the initial motor input, ceasing to highlight the group of virtual objects in the first virtual area if the first confidence level is lower than a first threshold, determining a second confidence level associated with the refined motor input, and ceasing to trigger a function associated with the particular virtual object if the second confidence level is lower than a second threshold. In a general sense, the first confidence level associated with the initial motor input and the second confidence level associated with the refined motor input may be similar to the confidence level associated with the initial motor input and the confidence level associated with the refined motor input, respectively, as described above. Furthermore, in a general sense, ceasing to highlight the group of virtual objects and ceasing to trigger a function may be performed in a manner similar to ceasing to trigger a function described above. In some embodiments, the second threshold for ceasing to trigger a function associated with the particular virtual object may be greater than the first threshold for highlighting the group of virtual objects in the first virtual area.

[0237] 26 is a flowchart illustrating an example process 2600 for determining a confidence level associated with a motor input and not highlighting or aborting the triggering of a function associated with the virtual object if the confidence level is below a threshold level. Referring to FIG. 26 , at step 2610, instructions included on a non-transitory computer-readable medium, when executed by at least one processor, may cause the at least one processor to determine a first confidence level associated with an initial motor input. At step 2612, the at least one processor may cause the at least one processor to abort highlighting a group of virtual objects in a first virtual area if the first confidence level is below a first threshold. At step 2614, the at least one processor may determine a second confidence level associated with the refined motor input. At step 2616, the at least one processor may abort triggering a function associated with a particular virtual object if the second confidence level is below a second threshold.

[0238] A user of a wearable extended reality device may be immersed in the displayed extended reality environment and become unaware of the surrounding physical environment. The physical environment can be considered to include the area in which the user of the wearable extended reality device is located. For example, if the user is located indoors, the physical environment may be the room in which the user is located. As another example, if the user is located outdoors, the physical environment may be a predetermined radius around the user's location. In some embodiments, the predetermined radius may be determined by the range of one or more sensors in communication with the wearable extended reality device. For example, the one or more sensors may include an audio sensor 471, an image sensor 472, a motion sensor 473, an environmental sensor 474, or other sensors 475 as shown in FIG. 4 . In another example, the physical environment may be the area visible to the user if the user were not using the wearable extended reality device. In another example, the physical environment may be the area visible to an image sensor included in the wearable extended reality device. In some embodiments, the wearable extended reality device may be configured to detect changes to the physical environment around the user and automatically adjust one or more display parameters of the extended reality display provided by the wearable extended reality device, thereby enhancing the user's awareness of the environmental change. For example, a change to the physical environment may include another person (i.e., a non-user) moving toward or into the field of view of the wearable extended reality device, and in response to this change, virtual objects may be moved out of the way to allow the user to see the other person. As another example, a change to the physical environment may include a change in ambient light levels near the user, and in response to this change, the brightness of the extended reality display may be changed so that the extended reality display has better visibility given the current ambient light level.

[0239] A person (i.e., a non-user) walking by a user of a wearable extended reality device may be unaware of the user's immersion in the extended reality environment. In some embodiments, an indication may be provided to a non-user that the user of the wearable extended reality device may not be able to see. For example, an indicator light (e.g., one or more light indicators 451 as shown in FIG. 4 ) may be provided on an exterior portion of the wearable extended reality device to alert the non-user that the user may not be able to see. As another example, a virtual object presented by the wearable extended reality device may be moved out of the way to allow the user to view (or get a better view of) one or more non-users.

[0240] Some disclosed embodiments may include methods, systems, and computer-readable media for facilitating environmentally adaptive extended reality displays in a physical environment (three forms of articulation). This disclosure is intended to encompass all three forms of articulation, and it should be understood that any details described, even if described in relation to only one form of articulation, are intended to disclose all three.

[0241] An environmentally adaptive extended reality display is one that changes one or more display parameters (i.e., adapts the extended reality display) in response to changes in the physical environment around a user of the extended reality display. In this context, it should be understood that the term "display" can refer to the display itself as well as software or a controller associated with the display, regardless of whether the software or controller is physically located within the display housing. Changing display parameters can include adjusting how the extended reality display (and / or extended reality environment) is perceived by a user of a wearable extended reality device. Display parameters and methods of adjusting display parameters are described in more detail below.

[0242] Some disclosed embodiments may be implemented via a non-transitory computer-readable medium including instructions for performing the operations of the method. In some embodiments, the method may be implemented on a system including at least one processor configured to perform the operations of the method. In some embodiments, the method may be implemented by one or more processors associated with a wearable extended reality device. For example, a first processor may be disposed within the wear...

Claims

1. 1. A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations for moving virtual content between virtual planes in three-dimensional space, the operations comprising: using a wearable extended reality device to virtually display a plurality of virtual objects on a plurality of virtual planes, the plurality of virtual planes including a first virtual plane and a second virtual plane; outputting a first display signal for presentation via the wearable extended reality device, the first display signal reflecting a virtual object at a first position on the first virtual plane; receiving an in-plane input signal to move the virtual object to a second position on the first virtual plane; causing the wearable extended reality device to virtually display an in-plane movement of the virtual object from the first position to the second position in response to the in-plane input signal; while the virtual object is at the second location, receiving an inter-plane input signal to move the virtual object to a third location on the second virtual plane; causing the wearable extended reality device to virtually display inter-plane movement of the virtual object from the second position to the third position in response to the inter-plane input signal; 1. A non-transitory computer readable medium comprising:

2. 2. The non-transitory computer-readable medium of claim 1, wherein the first virtual plane is associated with a first distance from the wearable extended reality device and the second virtual plane is associated with a second distance from the wearable extended reality device, and the first distance associated with the first virtual plane is greater than the second distance associated with the second virtual plane.

3. 3. The non-transitory computer-readable medium of claim 2, wherein a first curvature of the first virtual plane is substantially identical to a second curvature of the second virtual plane, and the operations further include modifying an extended reality display of the virtual object to reflect a difference between the first distance and the second distance in response to the inter-surface input signal.

4. 3. The non-transitory computer-readable medium of claim 2, wherein a first curvature of the first virtual plane differs from a second curvature of the second virtual plane, and the operations further include modifying, in response to the inter-surface input signal, a display of the virtual object to reflect a difference between the first distance and the second distance and a difference between the first curvature and the second curvature.

5. 2. The non-transitory computer-readable medium of claim 1 , wherein the first virtual plane is associated with a first distance from a physical input device and the second virtual plane is associated with a second distance from the physical input device, and the first distance associated with the first virtual plane is greater than the second distance associated with the second virtual plane.

6. 2. The non-transitory computer-readable medium of claim 1 , wherein the first virtual plane is associated with a first distance from an edge of a physical surface and the second virtual plane is associated with a second distance from the edge of the physical surface, the first distance associated with the first virtual plane being greater than the second distance associated with the second virtual plane.

7. 2. The non-transitory computer-readable medium of claim 1 , wherein the first virtual plane is curved and the in-plane movement of the virtual object from the first location to the second location comprises three-dimensional movement.

8. The non-transitory computer-readable medium of claim 1 , wherein the first imaginary plane and the second imaginary plane are convex surfaces.

9. 2. The non-transitory computer-readable medium of claim 1 , wherein the in-plane movement includes moving the virtual object along two orthogonal axes, movement along a first axis including changing a dimension of the virtual object and movement along a second axis excluding changing a dimension of the virtual object.

10. 10. The non-transitory computer-readable medium of claim 9, wherein the operations further comprise modifying the dimensions of the virtual object based on a curvature of the first virtual plane.

11. 2. The non-transitory computer-readable medium of claim 1, wherein both the in-plane input signal and the inter-surface input signal are received from a touch sensor, and the operations further include identifying a first multi-finger interaction as a command to cause in-plane movement and identifying a second multi-finger interaction different from the first multi-finger interaction as a command to cause inter-surface movement.

12. 10. The non-transitory computer-readable medium of claim 1, wherein the operations further include receiving a preliminary input signal for selecting a desired curvature in the first virtual plane, and wherein the in-plane movement is determined based on the selected curvature of the first virtual plane.

13. 13. The non-transitory computer-readable medium of claim 12, wherein selecting the curvature in the first virtual plane affects a curvature of the second virtual plane.

14. The operation includes: A step of virtually displaying a plurality of virtual objects on a plurality of virtual planes using a wearable extended reality device, the plurality of virtual planes including a first virtual plane and a second virtual plane; displaying, on the wearable extended reality device, in-plane movement of the virtual object and the another virtual object in response to the in-plane input signal; displaying inter-surface movement of the virtual object and the another virtual object on the wearable extended reality device in response to the inter-surface input signal; The non-transitory computer-readable medium of claim 1 , further comprising:

15. The operation includes: receiving an input signal indicating the virtual object is to be docked to a physical object; determining a relative movement between the physical object and the wearable extended reality device; determining whether to move the virtual object to a different virtual plane based on the determined movement; The non-transitory computer-readable medium of claim 1 , further comprising:

16. 16. The non-transitory computer-readable medium of claim 15, wherein the actions further include determining a relative movement caused by a movement of the physical object, and causing the inter-plane movement of the virtual object to the different virtual planes in response to the determined relative movement.

17. 16. The non-transitory computer-readable medium of claim 15, wherein the actions further include determining the relative movement caused by movement of the wearable extended reality device, and preventing the inter-plane movement of the virtual object to the different virtual plane in response to the determined relative movement.

18. The operation includes: receiving an input signal indicating that the first virtual plane and the second virtual plane are to be docked to a physical object; While the virtual object is in the second position and prior to receiving the inter-surface input signal, determining a movement of the physical object to a new position; determining a new position in the first virtual plane and in the second virtual plane based on the new position of the physical object; in response to the identified movement of the physical object, causing the wearable extended reality device to display a movement of the virtual object from the second position on the first virtual plane at the original position of the first virtual plane to the second position on the first virtual plane at the new position of the first virtual plane; Further comprising: the display of the inter-plane movement of the virtual object from the second position to the third position is a display of a movement of the virtual object from the second position on the first virtual plane at the new position on the first virtual plane to the third position on the second virtual plane at the new position on the second virtual plane. The non-transitory computer-readable medium of claim 1 .

19. 1. A system for moving virtual content between planes in three-dimensional space, the system comprising: At least one processor, the processor comprising: using a wearable extended reality device to virtually present a plurality of virtual objects on a plurality of virtual planes, the plurality of virtual planes including a first virtual plane and a second virtual plane; outputting a first display signal for presentation via the wearable extended reality device, the first display signal reflecting a virtual object at a first position on the first virtual plane; receiving an in-plane input signal for moving the virtual object to a second position on the first virtual plane; displaying, on the wearable extended reality device, an in-plane movement of the virtual object from the first position to the second position in response to the in-plane input signal; receiving an inter-plane input signal to move the virtual object to a third position on the second virtual plane while the virtual object is at the second position; displaying, on the wearable extended reality device, an inter-surface movement of the virtual object from the second position to the third position in response to the inter-surface input signal; The system is configured as follows.

20. 1. A method for moving virtual content between planes in three-dimensional space, the method comprising: using a wearable extended reality device to virtually present a plurality of virtual objects on a plurality of virtual planes, the plurality of virtual planes including a first virtual plane and a second virtual plane; outputting a first display signal for presentation via the wearable extended reality device, the first display signal reflecting a virtual object at a first position on the first virtual plane; receiving an in-plane input signal to move the virtual object to a second position on the first virtual plane; causing the wearable extended reality device to display an in-plane movement of the virtual object from the first location to the second location in response to the in-plane input signal; while the virtual object is at the second location, receiving an inter-plane input signal to move the virtual object to a third location on the second virtual plane; causing the wearable extended reality device to display an inter-surface movement of the virtual object from the second position to the third position in response to the inter-surface input signal; The method includes:

Images