System for detecting air and surface gestures usable for artificial real environment using sensors at wrist wearable devices and methods of use thereof

Abstract

An artificial reality system comprising an optional head wearable device and wrist wearable device for displaying an artificial reality environment identifies and detects a shape of a surface, and optionally, identifies and detects a distance of the surface from the wrist wearable device, and performing an operation available to the user including a haptic event by detecting a gesture performed by the user, and optionally, detecting a distance of the wrist wearable device from the surface during the execution of the gesture such that the corresponding operation is performed based on the type of the gesture. For example, the wrist wearable device may be configured to cause a first set of operations corresponding to air gestures to be performed within an air threshold distance of the surface, such that a second set of operations corresponding to near-surface gestures to be performed within a surface threshold distance of the surface. For example, these operations may update a display at an optional head wearable device, control device attributes on other electronic devices, or a response to a message.

Description

Technical Field

[0001] The present disclosure generally relates to wearable devices (e.g., wrist wearable devices and head wearable devices) and methods for detecting different types of gestures using the wearable devices, including but not limited to wearable devices configured to detect gestures performed in an artificial reality environment using various sensing capabilities (e.g., time-of-flight sensors, electromyography (EMG) sensors, inertial measurement unit (IMU) sensors, etc.). Background Art

[0002] Artificial reality (e.g., augmented reality (AR), virtual reality (VR), etc.) environments can provide users with an immersive experience, allowing users to interact with a user interface using a head-wearable device that displays the artificial reality environment.

[0003] In some examples of user interaction with an artificial reality environment, a handheld device (e.g., a game controller) can be used to detect the user's movement, including gestures performed by the user. In some examples, the detection of the user's movement causes an adjustment to the display of the artificial reality environment. However, such artificial reality environments, handheld devices, and wearable devices may cause many inconvenient, awkward, and socially unacceptable interactions because they require the user's full attention, large gestures that require a large amount of available space (at least an arm's length away), and require the user to expend considerable energy.

[0004] Further, such systems may require the user to carry and manipulate a handheld device (e.g., a controller) that facilitates such a system to detect the user's gestures, and / or require the user to wear multiple electronic devices at each potential contact point on the user's body (e.g., a fingertip), which is often cumbersome or inconvenient and fails to take advantage of wearable devices that the user wears on a daily basis.

[0005] Finally, certain VR devices and methods associated therewith may isolate a user from physical surfaces with which they could potentially come into physical contact (e.g., by creating a fixed guardian boundary around the user, and / or by providing an indication to the user that they are about to come into physical contact with a physical object), rather than allowing the user to interact with such physical surfaces and / or other features and objects within the user's actual physical environment. Summary of the invention

[0006] The features and advantages described in the specification are not necessarily all inclusive, and in particular, some additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, description and claims provided in this disclosure. In addition, it should be noted that the terms used in this specification are primarily selected for readability and instructional purposes and are not necessarily selected to describe or define the subject matter described herein.

[0007] According to a first aspect of the present disclosure, a method is provided, the method comprising: when a head wearable device is presenting an artificial reality environment including a user interface object, wherein the user interface object responds to a first group of operations corresponding to corresponding air gestures and a second group of operations corresponding to corresponding surface gestures: based on detecting the execution of an air gesture corresponding to an operation in the first group of operations using a first group of sensors of a wrist wearable device, the head wearable device: (i) executes the operation in the first group of operations, and (ii) updates the user interface object in combination with the execution of the operation in the first group of operations, wherein the air gesture is performed outside a surface threshold distance of a physical surface, and the physical surface is at least partially located in the artificial reality environment; and based on detecting the execution of a surface gesture corresponding to an operation in the second group of operations using a second group of sensors of the wrist wearable device, the head wearable device: (i) executes the operation in the second group of operations, and (ii) updates the user interface object in combination with the execution of the operation in the second group of operations, at least one sensor of the second group of sensors is not in the first group of sensors, and the surface gesture is performed within a surface threshold distance of the physical surface.

[0008] In some embodiments, the method further includes causing a haptic event at the head wearable device, the wrist wearable device, or another connected device in response to detecting that the wrist wearable device is within a surface edge threshold distance of a surface edge of the physical surface.

[0009] In some embodiments, the second group of sensors includes: a first time-of-flight sensor arranged and configured to measure a first angle relative to the wrist wearable device; and a second time-of-flight sensor arranged and configured to measure a second angle relative to the wrist wearable device.

[0010] In some embodiments, the method further comprises: based on detecting that the wrist wearable device is farther from the physical surface than a threshold distance from the surface: causing a reduction in power of at least one sensor from the second set of sensors.

[0011] In some embodiments, performance of the surface gesture is detected using a second set of sensors of the wrist wearable device after determining that the surface gesture is performed within a surface threshold distance of the physical surface.

[0012] In some embodiments, performance of the surface gesture is detected using a second set of sensors of the wrist wearable device after determining that at least a portion of the surface gesture contacts the physical surface.

[0013] In some embodiments, the second group of sensors includes a biopotential signal sensor; and a determination that two or more of the multiple contacts are performing a pinch gesture is based on a change in a pinch threshold distance between two or more of the multiple contacts of the surface gesture, which is detected by the biopotential signal sensor.

[0014] In some embodiments, detecting the execution of a surface gesture also includes: when a user interface object is in focus based on one of an air gesture or a surface gesture: detecting a first wrist angle of the wrist wearable device using a second group of sensors; and detecting a second wrist angle of the wrist wearable device using a second group of sensors; wherein identifying the above operations in the second group of operations includes: determining a rotation angle between the first wrist angle and the second wrist angle.

[0015] In some embodiments, the second group of operations includes an enable indication activation operation, wherein execution of the surface gesture includes: a first gesture portion corresponding to selecting a user interface object in the artificial reality environment; and a second gesture portion corresponding to activating the user interface object, wherein the second gesture portion includes a downward movement of the user's hand.

[0016] In some embodiments, performance of an air gesture is detected while the user interface object remains in focus, and is also based in part on: using a first set of sensors to determine that the air gesture is performed within an air threshold distance relative to a physical surface, the air threshold distance being greater than a surface threshold distance.

[0017] In some embodiments, the maximum value of the threshold distance in air is 50 centimeters.

[0018] In some embodiments, the method further includes: based on determining that another air gesture is detected after the air gesture without being performed within a threshold air distance: causing the head wearable device to perform an operation in a third set of operations corresponding to the position-independent gesture.

[0019] In some embodiments, the user interface object includes a virtual object located within a threshold air distance of the physical surface and beyond the threshold air distance of the surface; and the air gesture includes virtual contact with the virtual object.

[0020] In some embodiments, the physical surface comprises a portion of a body part of the user.

[0021] In some embodiments, the portion of the user's body part is selected from the group consisting of: (i) a palm portion; or (ii) a leg portion.

[0022] In some embodiments, the method further includes: in response to detecting a position-independent gesture at any location within a surface threshold distance of the physical surface, causing the head wearable device to: (i) perform an operation from a set of position-independent operations, and (ii) update the artificial reality environment, wherein the position-independent gesture is selected from the group consisting of: (i) a thumbs up gesture; and (ii) a thumbs down gesture; and in response to detecting the position-independent gesture, causing a response to be applied to a received message corresponding to a message object displayed within the artificial reality environment.

[0023] In some embodiments, the surface gesture is detected using a surface electromyography (sEMG) sensor from the second set of sensors to determine whether a user of the wrist wearable device has transmitted an electrical signal to the physical surface.

[0024] In some embodiments, at least one sensor of the first set of sensors and the second set of sensors includes a shared electrode, wherein the shared electrode is configured to perform more than one sensing capability.

[0025] According to a second aspect of the present disclosure, a wrist wearable device is provided, the wrist wearable device comprising: one or more processors; and a memory, the memory comprising a plurality of instructions, which, when executed by the one or more processors, enable the wrist wearable device to perform the following operations: when a head wearable device is presenting an artificial reality environment including a user interface object, wherein the user interface object responds to a first group of operations corresponding to corresponding air gestures and a second group of operations corresponding to corresponding surface gestures: based on detecting, using a first group of sensors of the wrist wearable device, that an air gesture corresponding to an operation in the first group of operations is performed, the head wearable device: (i) performs The invention relates to a method for performing an in-air gesture comprising: performing an operation in a first group of operations, and (ii) updating a user interface object in conjunction with the performance of the operation in the first group of operations, wherein the in-air gesture is performed outside a surface threshold distance of a physical surface that is at least partially located within the artificial reality environment; and based on detecting the performance of a surface gesture corresponding to an operation in the second group of operations using a second group of sensors of the wrist wearable device, causing the head wearable device to: (i) perform an operation in the second group of operations, and (ii) update the user interface object in conjunction with the performance of the operation in the second group of operations, wherein at least one sensor in the second group of sensors is not in the first group of sensors, and the surface gesture is performed within a surface threshold distance of the physical surface.

[0026] According to a third aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, which includes a plurality of instructions. When the plurality of instructions are executed by one or more processors of an electronic device, the following operations are performed: when a head wearable device is presenting an artificial reality environment including a user interface object, wherein the user interface object responds to a first group of operations corresponding to corresponding air gestures and a second group of operations corresponding to corresponding surface gestures: based on detecting an air gesture corresponding to an operation in the first group of operations using a first group of sensors of a wrist wearable device, the head wearable device: (i) performs an operation in the first group of operations; , and (ii) updating the user interface object in conjunction with the execution of the operation in the first group of operations, wherein the air gesture is performed outside a surface threshold distance of a physical surface that is at least partially located within the artificial reality environment; and based on detecting the execution of a surface gesture corresponding to an operation in the second group of operations using a second group of sensors of the wrist wearable device, causing the head wearable device to: (i) execute the operation in the second group of operations, and (ii) update the user interface object in conjunction with the execution of the operation in the second group of operations, at least one sensor in the second group of sensors is not in the first group of sensors, and the surface gesture is performed within a surface threshold distance of the physical surface.

[0027] It will be appreciated that any feature described herein as being suitable for incorporation into one or more aspects or embodiments of the present disclosure is intended to be universal in any and all aspects and embodiments of the present disclosure. Other aspects of the present disclosure may be understood by those skilled in the art from the specification, claims and drawings of the present disclosure. The foregoing general description and the following detailed description are exemplary and explanatory, rather than limiting, the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to understand the present disclosure in more detail, a more specific description may be made by reference to the features of various embodiments, some of which are shown in the accompanying drawings. However, the accompanying drawings only show the relevant features of the present disclosure, and therefore these drawings are not necessarily considered to be limiting, as those skilled in the art will recognize when reading the present disclosure that the present specification may admit other effective features.

[0029] Figures 1A to 1J An example user scenario with an artificial reality system (e.g., including at least augmented reality glasses and a wrist wearable device) in accordance with some embodiments is shown.

[0030] FIG. 2A to FIG. 2D Another example user scenario with an artificial reality system (e.g., including at least AR glasses and a wrist wearable device) in accordance with some embodiments is shown.

[0031] FIG. 3A to FIG. 3I Another example user scenario with an artificial reality system (e.g., including at least AR glasses and a wrist wearable device) in accordance with some embodiments is shown.

[0032] FIG. 4A to FIG. 4D Another example user scenario with an artificial reality system (e.g., including at least AR glasses and a wrist wearable device) in accordance with some embodiments is shown.

[0033] FIG. 5A to FIG. 5H Another example user scenario with an artificial reality system (e.g., including at least a wrist wearable device) in accordance with some embodiments is shown.

[0034] FIG. 6A to FIG. 6C Another example user scenario with an artificial reality system (e.g., including at least a wrist wearable device) in accordance with some embodiments is shown.

[0035] 7A to 7F Another example user scenario with an artificial reality system (e.g., including at least a wrist wearable device) in accordance with some embodiments is shown.

[0036] FIG. 8A to FIG. 8E Another example user scenario with an artificial reality system (e.g., including at least AR glasses and a wrist wearable device) in accordance with some embodiments is shown.

[0037] 9A to 9C is a flow chart illustrating an example method of making mid-air gestures and surface gestures available to a user in accordance with some embodiments.

[0038] FIG. 10A to FIG. 10C is a flow chart illustrating an example method for gesture detection and content rendering determination using a time-of-flight sensor in an artificial reality environment in accordance with some embodiments.

[0039] Fig.11A and Fig. 11B is a flow chart illustrating an example method of controlling at least one physical electronic device using interaction with an artificial reality environment in accordance with some embodiments.

[0040] Fig. 12A and Fig. 12B is a block diagram illustrating an example artificial reality system in accordance with some embodiments.

[0041] Fig.13A Example AR glasses (which may be used with some embodiments of artificial reality systems) are shown in accordance with some embodiments.

[0042] Fig. 13B Example VR glasses according to some embodiments are shown (which can be used with some embodiments of artificial reality systems).

[0043] Fig.14A and Fig. 14B An example wrist wearable device is shown in accordance with some embodiments.

[0044] Figure 15A to Figure 15V An example user scenario with an artificial reality system (e.g., including at least augmented reality glasses and a wrist wearable device) in accordance with some embodiments is shown.

[0045] Fig.16A and Fig. 16B is a flow chart illustrating a method for controlling user interface objects using subtle hand gestures in an artificial reality environment.

[0046] According to common practice, the various features shown in the drawings are not necessarily drawn to scale, and like reference numerals may be used to denote like features throughout the specification and drawings. DETAILED DESCRIPTION

[0047] Therefore, exploration around different types of gestures for interacting with artificial reality environments and testing around a properly defined gesture space can provide technical improvements, especially when wrist wearable devices are used to control various aspects of artificial reality environments.

[0048] Therefore, it would be beneficial to address one or more of the above-mentioned problems, shortcomings, or areas for further exploration.

[0049] The systems (e.g., wearable devices) and methods described herein can address at least one of the above-described disadvantages by allowing a user to interact with an artificial reality environment using one or more wearable devices (e.g., wrist wearable devices) that include sensors for detecting gestures performed by a user of the one or more wearable devices. The sensors at the wearable devices can include time-of-flight sensors (e.g., for detecting spatial distances) and EMG sensors (e.g., for detecting muscle responses).

[0050] As described herein, user gestures can correspond to operations for adjusting the display of an artificial reality environment. For example, a tap gesture at a virtual button displayed as appearing at a physical surface can cause an update to a user interface displayed in the air (e.g., "in-air") in front of the user. In other words, a gesture performed by a user can cause an operation to be performed directly (e.g., at a virtual object to which the gesture is directed) or indirectly (e.g., at a different virtual object or a different physical electronic device) to update the visual aspects of the artificial reality environment. Other user gestures can correspond to operations that cause non-visual updates either in the artificial reality environment or at another electronic device. For example, a "thumbs-up" gesture in front of a virtual screen element displayed in front of the user can cause an operation to be performed to save the current state of a virtual object in an artificial reality environment, which can be performed without any visual updates to the artificial reality environment.

[0051] As described herein, the artificial reality system can be configured to detect "surface" gestures that occur at or near a physical surface, as well as "air gestures" that occur farther away from the surface but still within a threshold distance (e.g., an air threshold distance) detectable by a sensor of the artificial reality system (e.g., a time-of-flight sensor). As described herein, the artificial reality system can also be configured to detect "position-independent gestures" that do not depend on a spatial relationship to a physical surface (e.g., using a sensor of a wearable device or another connected device). In addition, certain aspects of the physical environment, when present, can cause a head wearable device (also referred to as a head-mounted wearable device, a head-mounted display device, or simply a head-mounted device or a head wearable device, and the head-mounted device is also a wearable device because it is worn on the user's head) to update the display of a virtual object that is intended to be displayed within a certain proximity of a physical surface, the physical surface including a physical surface having a curved portion (e.g., an irregular surface shape).

[0052] After receiving or detecting a gesture from a user, the wearable device described herein may provide the following data to a computing device: the data causes the computing device to perform an operation of updating the presentation of a user interface (e.g., a user interface presented by a head wearable device). The computing device may be another wearable device or an intermediate device (e.g., a smartphone). In some instances, the wearable device (or intermediate device) is configured to cause an operation to be performed on another electronic device (e.g., an audio speaker, a household appliance (e.g., a light bulb), and / or a smartphone).

[0053] As an illustrative example, suppose a person (Lynn) wants to browse various websites on the Internet. Conventionally, Lynn needs to walk to her personal computer or use a mobile electronic device such as a smartphone. Both options have disadvantages. Using a mobile device, the small screen size (i.e., limited "real estate") can make it difficult to view the content, especially when the requested user interface elements for performing the desired operation occupy a significant portion of the screen. Using a personal computer, Lynn needs to walk to the computer (e.g., in her office) and stay there while she browses.

[0054] The artificial reality system described herein allows Lynn to browse anywhere because the wearable device (e.g., artificial reality glasses and wristband) is also a mobile device and provides a larger (e.g., virtual) and / or faster and efficient display for interacting with content (which is independent of the larger size of any physical display component). In this example, Lynn can view a website on a user interface presented via her AR glasses and can navigate with gestures detected by sensors in the AR glasses, wristband, or both without having to use another electronic device (e.g., a smartphone and / or laptop). In addition, navigation controls and other user interface elements can be displayed at a physical surface (e.g., to simulate a keyboard or touch screen experience), while content can be displayed in the air. In this way, user interface elements do not obstruct the content that Lynn wants to view, thereby providing an improved human-computer interface based at least on the above-mentioned efficiency improvements. Additionally, in this example, Lynn can manipulate content and user interfaces using a combination of air gestures (e.g., a pinch gesture to adjust magnification of content), surface gestures (e.g., a tap gesture on a virtual button at or near a physical surface or a swipe gesture on a virtual slider), and position-independent gestures, even though these gestures are performed at a variety of different locations relative to the user. Those skilled in the art will recognize from the embodiments described herein that a surface gesture can be a gesture that involves actual contact between a part of the user's body (e.g., a fingertip) and a physical surface with which they are interacting (e.g., a surface contact gesture), or a surface gesture can be a gesture that is within a threshold distance of a physical surface (e.g., a surface threshold distance) but does not make physical contact with the physical surface (e.g., a near-surface gesture). In some embodiments, a surface gesture that is a surface contact gesture can cause a different operation to be performed than a corresponding near-surface gesture. In embodiments where the surface gesture includes a gesture within a surface threshold distance of a physical surface, an air gesture can be a gesture performed outside the surface threshold distance of the physical surface but within an air threshold distance of the physical surface.

[0055] Continuing with the example, suppose that after browsing, Lynn wants to watch local sporting events on TV. Conventionally, Lynn would sit on her couch and watch sporting events on TV with a remote control. Using the artificial reality system described herein, Lynn can watch sporting events on TV (which can be displayed on a physical TV screen or in her AR glasses) without having to find her remote control and provide manual user input to the physical buttons of her remote control. The artificial reality system described herein can present virtual affordance representations (e.g., two-dimensional or three-dimensional virtual objects that can be likened to buttons and sliders on a two-dimensional computer screen) for controlling the TV (which can be displayed by Lynn's AR glasses), and the system can forward commands to the TV (e.g., via Bluetooth or wireless fidelity (Wi-Fi) protocols). In this example, when Lynn sits on her couch, there is no plane (e.g., a table or desk) near her to project the virtual affordance representation on it. Therefore, the virtual affordance representation is presented on a curved surface (e.g., the armrest of the sofa or Lynn's leg or palm). When the virtual affordance is presented on a curved surface, the system can adjust the display properties of the virtual affordance (e.g., to have a shape complementary to the curvature of the surface) so that Lynn can use surface gestures similar to those used to manipulate a conventional remote control.

[0056] Continuing with the example further, suppose Lynn wants to turn off the lights in her apartment while watching a sporting event. Conventionally, Lynn needs to stand up and walk to the light switch to turn off the lights. Using the artificial reality system described herein, Lynn can manipulate the lights without having to go to the physical light switch. For example, Lynn can turn to the light switch and perform a gesture corresponding to turning off the switch (e.g., a tap gesture on her palm). The artificial reality system can detect Lynn's orientation change from the TV to the light switch and replace (or supplement) the virtual affordance representation of the TV with a virtual affordance representation of the light switch. The system can forward commands to the light switch (e.g., via Bluetooth or Zigbee protocol). In this example, the virtual affordance representation of the light switch does not need to be visually displayed to Lynn, for example, by a head wearable device, but can correspond to a specific gesture (tapping someone's palm) when facing the light switch. For example, she can adjust the light switch after removing her AR glasses while watching a physical TV.

[0057] In this way, the systems and methods described herein can provide a more efficient human-computer interface because they allow users to interact with artificial reality systems without being significantly disturbed by, for example, a computer screen or other components of the associated electronic device. For example, a user can receive a message from another user of a different electronic device at a wrist wearable device, and apply a response to the message by performing, for example, a position-independent "thumbs-up" gesture (e.g., a user performs a position-independent "thumbs-up" gesture so that a "thumbs-up" response is applied to the message in the message thread user interface and is received by the other user). These improvements allow wearable devices to be designed so that they are comfortable, multifunctional, practical and socially accepted for daily use. In addition, these improvements allow users to interact with a computer (e.g., a physical monitor or keyboard) and / or a user interface without requiring a fixed position or orientation for interaction. The user interface can be moved according to the user's position and orientation. In addition, the user does not necessarily need to interact directly with the electronic device (e.g., a speaker or light switch) to interact with it, and can also modify the operations performed at the corresponding electronic device by performing gestures at different relative positions and using different types of user movements to access different operations of the corresponding electronic device. Additionally, users can use the same mid-air gesture to modify which electronic device they are interacting with. All of this furthers the goal of enabling more users to adopt emerging technologies in the artificial reality (AR) and AR spaces (e.g., the metaverse) in more use cases, especially not just for gaming in large open spaces.

[0058] In addition, the systems and methods described herein can allow for a more efficient and simplified human-machine interface, because they can provide users with more options for interacting with electronic devices and digital media without cluttering the user interface of the electronic device with a graphical representation of each available operation that the electronic device can perform. Therefore, these improvements simplify the user interface by providing fewer visual elements and simplify the user input for interacting with such an interface. For example, as described herein, it is possible to interact with a single virtual object by at least a surface gesture, an air gesture, or a position-independent gesture, and each type of gesture can have its own gesture space, thereby further defining the potential interaction mode available to the user. Therefore, the user can perform more operations without having to deal with the interference of having more user interface controls, clicking on countless option windows, etc. Some of the multiple gestures and operations described herein can be performed without displaying any user interface, which allows users to interact with digital technology more seamlessly when they perform their daily tasks in the physical world.

[0059] In addition, the systems and methods described herein can achieve more efficient and ergonomic human-computer interfaces because they do not require the user to provide user input at physical objects located at different locations within their reachable range of motion and / or do not require the user to provide physical force in a less ergonomic environment, which may lead to, for example, carpal tunnel syndrome. In addition, it does not require the user to use additional output devices that are different from the above-mentioned input devices. For example, physical contact with a physical surface can be detected without any device or sensor at the contact point (e.g., a user's fingertip or an electronic device (e.g., a touch-sensitive display) that includes the physical surface being interacted with.

[0060] According to some embodiments, a method for making air gestures or surface gestures available to a user is provided. The method includes: presenting an artificial reality environment including a user interface via a head wearable device. The user interface responds to a first set of operations corresponding to the corresponding air gestures and a second set of operations corresponding to the corresponding surface gestures. The method also includes: when presenting the user interface, in response to detecting the execution of an air gesture corresponding to an operation in the first set of operations using a first set of sensors of a wrist wearable device, causing the head wearable device to execute an operation in the first set of operations to update the presentation of the user interface. The method also includes: when presenting the user interface, in response to detecting the execution of a surface gesture corresponding to an operation in the second set of operations using a second set of sensors of a wrist wearable device (at least one sensor of the second set of sensors is not in the first set of sensors) performed at a physical surface, causing the head wearable device to execute an operation in the second set of operations to update the presentation of the user interface.

[0061] According to some embodiments, a method for gesture detection and content rendering determination using a time-of-flight sensor in an artificial reality environment is provided. The method includes: receiving data about a physical surface from one or more time-of-flight sensors communicatively coupled to a wrist wearable device, wherein the wrist wearable device is communicatively coupled to a head wearable device, and the head wearable device is configured to display a virtual object in an artificial reality environment presented by the head wearable device. The method also includes: determining that the physical surface has a curved portion based on the data, so that at least a portion of the virtual object is displayed at the curved portion, including updating the display of the virtual object according to the curved portion.

[0062] According to some embodiments, a method for controlling at least one other electronic device using interactions in an artificial reality environment is provided. The method includes: detecting that a user of the wrist wearable device is within an activation threshold distance of the electronic device that responds to a user gesture based on data from a first set of sensors of the wrist wearable device. The method also includes: when the user is within the activation threshold distance of the electronic device, using a second set of sensors of the wrist wearable device to detect a user gesture corresponding to an operation at the electronic device. The method also includes: in response to detecting the user gesture, causing the electronic device to perform the operation. In some embodiments, the artificial reality environment is completely non-visual, meaning that it does not contain visual user interface objects (e.g., virtual objects). In some embodiments, the user does not need to wear any head wearable device to interact with the artificial reality environment.

[0063] The systems (e.g., wearable devices) and methods described herein address at least one of the above-described disadvantages by allowing a user to interact with an artificial reality environment using one or more wearable devices (e.g., wrist wearable devices) that include sensors that use an imaging sensor of at least one wearable device to detect gestures performed by a user of the one or more wearable devices. Other sensors at the wearable device may include a time-of-flight sensor (e.g., for detecting spatial distance) and an EMG sensor (e.g., for detecting muscle response). As described herein, an air gesture may correspond to an operation that causes a focus selector to change position within a user interface of an artificial reality environment.

[0064] According to some embodiments, a method for controlling a user interface object in an artificial reality environment based on subtle hand gestures of a user is provided. The method receives an indication from a wrist wearable device including an imaging sensor to perform an air gesture when the user interface object is in focus at a user interface presented to the user via a head wearable device, the air gesture including movement of the user's hand in a direction relative to a starting position of the user's hand. When the user wears the wrist wearable device, the imaging sensor is oriented toward the user's hand, and the movement is detected by the imaging sensor included on the wrist wearable device. The method also includes: in response to the indication, controlling the user interface object presented to the user via the head wearable device based on determining the movement of the user's hand, wherein controlling the user interface object is based on the direction of the movement relative to the starting position of the user's hand.

[0065] According to some embodiments, a method for making air gestures and surface gestures available to a user is provided. The method includes: presenting an artificial reality environment including a user interface via a head wearable device. The user interface responds to a first set of operations corresponding to the corresponding air gestures and a second set of operations corresponding to the corresponding surface gestures. The method also includes: when presenting the user interface, in response to detecting the execution of an air gesture corresponding to an operation in the first set of operations using a first set of sensors of a wrist wearable device, causing the head wearable device to execute an operation in the first set of operations to update the presentation of the user interface. The method also includes: when presenting the user interface, in response to detecting the execution of a surface gesture corresponding to an operation in the second set of operations using a second set of sensors of a wrist wearable device (at least one sensor of the second set of sensors is not in the first set of sensors) performed at a physical surface, causing the head wearable device to execute an operation in the second set of operations to update the presentation of the user interface.

[0066] In some embodiments, an artificial reality system (e.g., a wrist wearable device or a head wearable device) includes one or more processors, a memory, a display, and one or more programs stored in the memory. The one or more programs are configured to be executed by the one or more processors. The one or more programs include instructions for executing (or causing execution of) any of the various methods described herein (e.g., including methods 900, 1000, and 1100 described in detail below).

[0067] In some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by a computing device (e.g., a wrist wearable device or a head wearable device, or another connected device (e.g., a smartphone, or a desktop computer, or a laptop computer)) having one or more processors, a memory, and a display (in some embodiments, the display may be optional, for example, for an example connected device that can coordinate operations to be performed at a wrist wearable device and / or a head wearable device and thus has processing resources and power resources). The one or more programs include instructions for performing (or causing the performance of) any of the methods described herein (e.g., including methods 900, 1000, and 1100 described in detail below).

[0068] Thus, methods, systems, and computer-readable storage media for detecting mid-air and surface gestures in an artificial reality environment are disclosed. These methods can supplement or replace conventional methods of interacting with an artificial reality environment.

[0069] Numerous details are described herein to provide a thorough understanding of the example embodiments shown in the accompanying drawings. However, some embodiments may be practiced without many of these specific details, and the scope of the claims is limited only by those features and aspects specifically recited in the claims. In addition, it is not necessary to provide an exhaustive description of well-known processes, components, and materials in order to avoid obscuring the relevant aspects of the embodiments described herein.

[0070] Embodiments of the present disclosure may include various types of artificial reality systems or embodiments of artificial reality systems, or may be implemented in combination with various types of artificial reality systems or embodiments of artificial reality systems. As described herein, artificial reality is any superimposed functional and / or sensory detectable presentation provided by an artificial reality system in a user's physical environment. Such artificial reality may include and / or represent virtual reality (VR), augmented reality (AR), mixed artificial reality (MAR), or some combination of these realities and / or a variation of one of these realities. For example, a user may perform a sliding air gesture to cause a song to be skipped by a music providing application programming interface provided for playback at, for example, a home speaker. In some embodiments of an AR system, ambient light may pass through a display element of a corresponding head wearable device presenting various aspects of the AR system. In some embodiments, ambient light may pass through various aspects of the AR system. For example, a visual user interface element (e.g., a notification user interface element) may be presented at a head wearable device, and a certain amount of ambient light (e.g., 15% to 50% of ambient light) may pass through the user interface element so that the user can distinguish at least a portion of the physical environment on which the user interface element is being displayed.

[0071] Artificial reality content may include fully generated content or generated content combined with captured (e.g., real-world) content. Artificial reality content may include video, audio, tactile events, or some combination thereof, any of which may be presented in a single channel or in multiple channels (e.g., stereoscopic video that produces a three-dimensional effect to the viewer). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in the artificial reality and / or are otherwise used in the artificial reality (e.g., to perform actions in the artificial reality).

[0072] Artificial reality systems can be implemented in a variety of different form factors and configurations. Some artificial reality systems include near-eye displays (NEDs) that provide visibility into the real world (e.g., Fig.13A AR system 1320 in an artificial reality (e.g., Fig. 13B While some artificial reality devices are stand-alone systems, other artificial reality devices communicate and / or coordinate with external devices to provide an artificial reality experience to a user. Examples of such external devices include handheld controllers, mobile devices, desktop computers, devices worn by the user (e.g., Figures 1A to 1J ), a wrist-wearable device 102 in the user's wrist wearable device, one or more other user-worn devices, and / or any other suitable external system.

[0073] Figures 1A to 1J An example user scenario with an artificial reality system (e.g., including at least augmented reality glasses and a wrist wearable device) in accordance with some embodiments is shown. Figure 1A A user 101 and an artificial reality system 100 are shown, which includes a wrist wearable device 102 and a head wearable device 104 (e.g., AR glasses). Figures 1A to 1J The user 101 in is performing a gesture corresponding to an operation to be performed by the artificial reality system 100. Figures 1A to 1J In the specific example shown, user 101 is interacting with a virtual object (e.g., Figures 1A to 1E as well as Fig. 1I and Figure 1J The virtual object 118 and Figure 1F to Figure 1H In some embodiments, the virtual objects 118 and 126 (e.g., virtual buttons) displayed by the head wearable device 104 are similar to buttons of a physical electronic device such as a computer keyboard. Other example virtual objects include any of the following examples and derivatives, including combinations of the following:

[0074] A button, including a button having: a three-dimensional depth; and a plurality of visual display configurations associated with a plurality of states of the button when the button is pressed and / or otherwise interacted with. In some embodiments, the head wearable device may present a plurality of buttons, and some of the plurality of buttons may be three-dimensional, while other buttons of the plurality of buttons are displayed as two-dimensional buttons without three-dimensional depth.

[0075] · A slider, including a slider having a vertical portion corresponding to the adjustment range of the corresponding slider, the slider having an optional "handle" enable representation, which "handle" enable representation can optionally have three-dimensional depth and can be used to set control options in an artificial reality environment or at another electronic device.

[0076] ·Switch, including a switch or a switch on another electronic device having a toggle indication that moves between two or more different positions corresponding to two or more discrete states (e.g., "on" or "off") of an aspect of an artificial reality environment.

[0077] ·Menus, including menus having selectable affordances for adjusting options and performing other operations related to configuration of the head wearable device and / or the wrist wearable device and / or one or more additional electronic devices.

[0078] A rod, including a rod that appears to have three-dimensional depth when presented in an artificial reality environment presented by a head wearable device.

[0079] A dial, including a dial that appears to have three-dimensional depth when displayed by the head wearable device such that a user can perform a gesture at a simulated position of the dial to cause the dial to rotate about an axis of the dial at the simulated position where the dial is displayed.

[0080] a virtual keyboard, including a virtual keyboard that appears to have three-dimensional depth when displayed by the head wearable device,

[0081] And may include virtual buttons representing individual buttons of a physical keyboard.

[0082] • A virtual screen element, including a virtual screen element configurable to display at least two-dimensional content within the virtual screen element.

[0083] The above list of potential virtual objects that can be displayed by the head wearable device 104 is only some examples of virtual objects that can be visually displayed. In some embodiments, the user can create additional virtual objects by performing the methods and operations described herein.

[0084] In addition, although Figures 1A to 1J Three-dimensional virtual objects are shown, but in some embodiments, the user interface includes one or more two-dimensional virtual objects displayed at a physical surface (or in mid-air).

[0085] Figure 1AAlso shown is a visual aid in the form of dashed lines that represent a surface threshold distance 106 (e.g., 1 inch, 3 inches, or 12 inches from the surface) and an air threshold distance 108 (e.g., 6 inches, 12 inches, or 18 inches from the surface). As discussed above in the Summary of the Invention, those skilled in the art will recognize that a surface gesture can be a gesture that involves actual physical contact between a part of a user's body and a physical surface with which they are interacting (e.g., a surface contact gesture), or a surface gesture can be a gesture that is within a surface threshold distance of a physical surface (e.g., surface threshold distance 106) but not in physical contact with the physical surface (e.g., a near-surface gesture). In some embodiments, a surface gesture that is a surface contact gesture can cause a different operation to be performed than a corresponding near-surface gesture. In embodiments where the surface gesture includes a gesture within a surface threshold distance of a physical surface, an air gesture can be a gesture that is performed outside a surface threshold distance of a physical surface (e.g., surface threshold distance 106) but within an air threshold distance of the physical surface (e.g., air threshold distance 108).

[0086] As will be discussed below, a particular set of operations corresponds to gestures performed by user 101 within a surface threshold distance 106 of physical surface 110. For example, the set of operations includes Figure 1B Surface gestures in 109 and Figure 1C The operation of responding to the surface gesture 122 in the embodiment of the present invention is performed using data from a set of sensors 156 to detect Figures 1A to 1J In the example of surface gestures in the air, the set of sensors 156 includes one or more time-of-flight sensors 158. A gesture performed outside the air threshold distance 108 may be classified by the user 101 as an air gesture (e.g., Figure 1D The air gestures 123 and Figure 1E Position-independent gestures 125 in FIG. 1 ). Using data from a set of sensors 152 to detect Figures 1A to 1J In the air gesture, the set of sensors 152 includes one or more EMG sensors 154.

[0087] Figure 1A The enlarged view 10 in FIG. 1 shows a physical surface 110 near a wrist wearable device 102 and shows that the wrist wearable device 102 has sensors 150, which include a set of sensors 152 and a set of sensors 156. Figures 1A to 1FIn the example of , the set of sensors 152 is used to detect air gestures and the set of sensors 156 is used to detect surface gestures, as will be explained in detail below. In some embodiments, one or more sensors in the sensors 150 belong to both the set of sensors 152 and the set of sensors 156. For example, the set of sensors 152 and the set of sensors 156 do not need to be mutually exclusive. In addition, in some embodiments, the set of sensors 152 can be used to detect or characterize some surface gestures, and the set of sensors 156 can be used to detect or characterize some air gestures. In some embodiments, the power of one or more sensors in the set of sensors 152 is reduced when the user 101 is performing a surface gesture (e.g., when the wrist wearable device 102 is within a surface threshold distance), and the surface gesture may include a surface contact gesture and a near-surface gesture. In some embodiments, when the wrist wearable device 102 remains at the surface threshold distance 106 for a predefined duration (e.g., a "stay" duration), a determination is made to reduce the power of one or more sensors in the set of sensors 152. In some embodiments, while the user 101 is performing an air gesture (or while the wrist wearable device 102 is outside the surface threshold distance 106 (e.g., and has been outside the surface threshold distance for a predetermined amount of time)), power to one or more sensors of the set of sensors 156 is reduced.

[0088] exist Figure 1A In the example shown, the set of sensors 156 includes one or more time-of-flight sensors that can emit radiation at different angles. Figure 1A As shown, each of the one or more time-of-flight sensors can emit rays 113, 115, and 117 at different incident angles relative to the wrist wearable device 102, and these rays 113, 115, and 117 are used to detect at different angles relative to the wrist wearable device 102. Figure 1AAs shown, ray 113 points at an angle vertically downward from the wrist wearable device 102, ray 115 points at an angle offset in a forward direction relative to the wrist wearable device 102, and ray 117 points at an angle offset in a backward direction relative to the wrist wearable device. In some embodiments, multiple time-of-flight sensors are arranged to point at angles offset from each other by a predefined offset angle (e.g., 15 degrees, 20 degrees, or 30 degrees). In some embodiments, one or more time-of-flight sensors at the wrist wearable device 102 are configured and / or arranged to be pivotable, slidable, and / or otherwise adjustable to point at various incident angles relative to the wrist wearable device. In some embodiments, the one or more time-of-flight sensors are formed as an array of multiple individual time-of-flight sensors (e.g., an array having four rows and four columns of sensors), and the array of time-of-flight sensors can be used to generate a two-dimensional depth profile of a coverage area (e.g., as explained in more detail below, in one example, the coverage area can include a physical surface (e.g., a sofa or a portion of a user's body), and the two-dimensional depth profile can be used to detect contoured surfaces and edges of the surface). In some embodiments, data collected by sensors of the wrist wearable device 102 can be transmitted to an intermediate computing device (e.g., one or more computing devices in the plurality of computing devices 1224 in FIG. 12 , which may be, for example, a smartphone, a portable computing unit with a processor but without a display or other peripherals, etc.). In some embodiments, the intermediate computing device can then provide instructions to the head wearable device 104 to cause operations to be performed to update the AR or VR user interface at the head wearable device 104.

[0089] Figure 1B The user 101 is shown performing a surface gesture 109 at a virtual object 118 (e.g., a virtual touch gesture in which the user's index finger performs a gesture at a location corresponding to a simulated location of the virtual object 118), which is detected by a set of sensors 156 of the wrist wearable device 102 (e.g., detected by a time-of-flight sensor 158). Performing the surface gesture 109 at the virtual object 118 causes an operation to be performed at the head wearable device 104 (e.g., to update the display of the virtual object 118). In some embodiments, the user 101 receives a tactile event 151 (represented by a pair of reference numerals 151a and 151b) at the wrist wearable device 102 based on making virtual contact with the virtual object 118. Figure 1B In the example of , detection of the surface gesture 109 causes an operation to be performed that changes the appearance of the virtual object 118 (e.g., to indicate that the user 101 has activated a selectable functionality of the virtual object 118).

[0090] Figure 1CThe user 101 is shown performing a surface gesture 122 at a virtual object 118, which is detected by a set of sensors 156 (e.g., a time-of-flight sensor 158) of the wrist wearable device 102. Specifically, Figure 1C A virtual push gesture is illustrated in which a user's index finger moves from the top of a virtual object toward the bottom of the virtual object, close to a physical surface. Surface gesture 122 is performed within a surface threshold distance 106 of physical surface 110. Execution of surface gesture 122 directed toward virtual object 118 causes an operation to be performed at head wearable device 104 (e.g., to update a display of a user interface of artificial reality system 100 (e.g., a display of virtual object 118). In some embodiments, user 101 receives a tactile event 153 (represented by a pair of reference numerals 153a and 153b) at wrist wearable device 102 based on making virtual contact with virtual object 118, and receives a second (e.g., different) tactile event at wrist wearable device 102 when user 101 makes physical contact with physical surface 110 during execution of surface gesture 122. In some embodiments, tactile event 153 is configured to simulate the sensation of touching a physical button (e.g., tactile event 153 causes wrist wearable device 102 to apply a force in a direction opposite to the movement of surface gesture 122). Figure 1C In the example of , detection of the surface gesture 122 causes an operation to be performed that changes the appearance of the virtual object 118 to indicate that the user 101 has activated a selectable function of the virtual object 118. Figure 1C , the display of virtual object 118 is updated to indicate that virtual object 118 is in a pressed state from surface gesture 122 (as shown by virtual object animation 124, which makes the virtual object appear to be "pressed" toward the surface).

[0091] Figure 1D User 101 is shown performing an air gesture 123. Specifically, Figure 1DA hold gesture is shown in which the user's hand remains in one position (e.g., a fixed wrist position) for more than a predefined hold time period. For example, the predefined hold time period may be 1 second to 1.5 seconds. In some embodiments, the predefined hold time period may be different for different corresponding gestures (e.g., a pointing gesture may have a hold time period of 1.5 seconds, while a pinch gesture may have a hold time period of 0.75 seconds). The hold gesture may also be referred to as a stateful gesture above a virtual object 118 that is beyond the surface threshold distance 106 but within the air threshold distance 108. The air gesture 123 is detected by a set of sensors 152 (e.g., EMG sensors 154) of the wrist wearable device 102. Performing the air gesture 123 above the virtual object 118 causes an operation to be performed at the head wearable device 104 to update the display of the user interface of the artificial reality system 100, including the display of the virtual object 118. In some embodiments, user 101 receives a haptic event 155 (indicated by a pair of reference numerals 155a and 155b) at wrist wearable device 102 in accordance with causing an operation corresponding to virtual object 118 to be performed in accordance with air gesture 123 (e.g., wrist wearable device provides a soft, continuous vibration indicating that user 101 has caused an additional operation to be accessible at virtual object 118). Figure 1D In the example of FIG. 1 , performing an air gesture 123 causes an operation to be performed that changes the appearance of the virtual object 118 to indicate that the user 101 has activated a selectable function of the virtual object 118 (e.g., Figure 1B Specifically, in Figure 1D In the embodiment of the present invention, as shown by the corresponding virtual object 121, performing an air gesture 123 causes a save operation to be performed, which causes a file being displayed on another virtual object in the artificial reality system 100 to be saved for storage in the wrist wearable device 102, the head wearable device 104, one or more other electronic devices and / or a remote server. In response to the save operation, the user 101 receives a different Figure 1C A tactile event 155 of the tactile event 153 (indicated by a pair of reference numerals 153a and 153b) in FIG.

[0092] Figure 1E A user 101 is shown performing a position-independent gesture 125. Specifically, Figure 1EA vertical swipe gesture is illustrated in which a user moves their hand in a sweeping motion, upward beyond a surface threshold distance 106 and an air threshold distance 108. The position-independent gesture 125 is detected by a set of sensors 152 (e.g., EMG sensors 154) of the wrist wearable device 102. Performing the position-independent gesture 125 over the virtual object 118 causes an operation in a third set of operations corresponding to the position-independent gesture to be performed at the head wearable device 104 to update the display of the user interface of the artificial reality system 100, including the display of the virtual object 118. In some embodiments, the user 101 receives a tactile event at the wrist wearable device 102 in response to activating the virtual object 118 by performing the position-independent gesture 125 (e.g., the wrist wearable provides a soft, continuous vibration indicating that the user 101 has reference-locked the virtual object 118). Figure 1E In the example of FIG. 1 , performing the position-independent gesture 125 causes an operation to be performed that changes the appearance of the virtual object 118 to indicate that the user 101 has activated a selectable function of the virtual object 118 (e.g., FIG. 3A to FIG. 3I As shown, an operation is performed at a virtual screen element of a second virtual object).

[0093] and Figures 1B to 1E The symbolic view of the sensor 150 shown side by side includes Figures 1B to 1E100 . As shown in the symbolic view of sensor 150 , more than one time-of-flight sensor 158 (e.g., s1, s2, s3) may be used to detect the execution of various gestures. In some embodiments, additional sensors may be used to detect various aspects of the actions (including gestures) performed by user 101, as well as other aspects of the user's physical environment. For example, IMU sensor 160 may be used to detect specific forces, angular rates, and / or orientations of body parts of user 101 and / or their surroundings at any one of wrist wearable device 102, head wearable device 104, or another electronic device. In some embodiments, impedance sensor 162 provides a signal to the user's body that may be used to detect the resistance of the user's body to determine whether user 101 has contacted a physical surface (e.g., physical surface 110) or another physical object. In some embodiments, one or more camera sensors 164 are used (e.g., in addition to or in place of the above sensors) to detect user motion and gestures. An example of an impedance sensor is a surface electromyography (sEMG) sensor. In some embodiments, the impedance sensor 162 includes at least one sEMG sensor. It will be appreciated by those skilled in the art that, although some embodiments discuss the use of EMG sensors when the operations described herein are performed, other types of biopotential signal sensors, including other neuromuscular signal sensors, can be used to perform similar operations. As described herein, a biopotential signal sensor is a sensor that measures electrical signals generated by an organism (e.g., a human or an animal). As described herein, a neuromuscular signal sensor is a biopotential signal sensor that specifically measures electrical signals generated by a neuromuscular system. Examples of biopotential signal sensors include electroencephalography (EEG) sensors that measure the electrical activity of the brain, electrocardiography (ECG) sensors that measure the electrical activity of the heart, and electromyography (EMG) sensors that measure the electrical activity of muscles. An EMG sensor is an example of a neuromuscular signal sensor.

[0094] Figure 1F and the following Figure 1G to Figure 1J User 101 is shown performing a series of interactions, including performing user gestures that cause various operations at artificial reality system 100. Figure 1G to Figure 1J An example that can also be shown is that user 101 performs a series of continuous gestures (e.g., performed within a predefined threshold time or with a time less than the threshold time between each individual sub-gesture of the mixed gesture). In an example where user 101 performs a mixed gesture including a series of individual gestures (e.g., sub-gestures), Figure 1G to Figure 1JThe user 101 is shown performing a surface gesture 128 (e.g., a vertical slide gesture) over a virtual object 126 (which includes a set of virtual buttons) and then performing additional gestures (e.g., position-independent gesture 130—a “reverse pinch” gesture and position-independent gesture 132—a pinch gesture) outside of the surface edge 112. Figures 1A to 1E Any of the multiple example processes and devices described may be used with Figure 1F to Figure 1J In other words, the mixed gesture includes two or more sub-gestures, which include multiple gestures with one gesture type (e.g., a position-independent gesture of a thumbs-up and a position-independent gesture of pointing to a point) and multiple gestures with multiple gesture types (e.g., a surface gesture and an air gesture). In some embodiments, the mixed gesture causes an operation corresponding to each individual sub-gesture (e.g., including any of the individual surface gestures, air gestures, and position-independent gestures described herein) to be performed. In some embodiments, the mixed operation may be different from the operation corresponding to one or more of the multiple individual sub-gestures. In some embodiments, in addition to performing an operation corresponding to the mixed gesture (e.g., a mixed gesture including surface gesture 128 and position-independent gesture 130) from a set of mixed operations, one or more operations of the multiple operations corresponding to the multiple individual gestures may also be performed. As shown in the enlarged view of surface gesture 128, when the wrist wearable device 102 crosses the surface edge 112 during the execution of surface gesture 128, a tactile event 157 (represented by a pair of reference numerals 157a and 157b) may be provided. In some embodiments, a sub-gesture of a mixed gesture may also be a mixed gesture.

[0095] Figure 1G The user 101 is shown performing a surface gesture 127 on a virtual object 126, which is recognized as a surface gesture by a set of sensors 156 (including but not limited to a time-of-flight sensor 158) of the wrist wearable device 102 by detecting that the surface gesture 127 is performed within a surface threshold distance 106 of the physical surface 110. Performing the surface gesture 128 at the virtual object 126 causes an operation in the second set of operations to be performed at the head wearable device 104 to update the display of the user interface of the artificial reality system 100, including the display of the virtual object 126. In some embodiments, the user 101 receives a tactile event at the wrist wearable device 102 as follows: activating an operation at the virtual object 126 according to the surface gesture 127. Figure 1G In the example of FIG. 1 , performing a surface gesture 127 causes an operation to be performed that changes the appearance of the virtual object 126 to indicate that the user 101 has activated a selectable function of the virtual object 126 (e.g., FIG. 3A to FIG. 3I), performing operations at a virtual screen element of a second virtual object as shown in ).

[0096] Figure 1H A user 101 is shown performing a surface gesture 128 at a virtual object 126, an initial portion of the surface gesture 128 being recognized by a set of sensors 156 (including, but not limited to, a time-of-flight sensor 158) of the wrist wearable device 102 as a surface gesture performed within a surface threshold distance 106, and a second portion corresponding to when the gesture is performed outside of a surface edge 112. The surface gesture 128 may also be an example of a sub-gesture of a hybrid gesture that includes the surface gesture 128 and, for example, the position-independent gesture 130. A hybrid gesture that includes two or more sub-gestures (e.g., a surface gesture 128 and a position-independent gesture 130) may cause different operations to be performed, and / or cause one or more operations to be performed that are different from those that would be performed by performing the operations separately. In some embodiments, the user 101 receives a tactile event 157 (represented by a pair of reference numerals 157a and 157b) at the wrist wearable device 102 in response to making virtual contact with the virtual object 126. In Figure 1H In the example of FIG. 1 , performing a surface gesture 128 causes an operation to be performed that changes the appearance of the virtual object 126 to indicate that the user 101 has activated a selectable function of the virtual object 126 (e.g., FIG. 3A to FIG. 3I ), performing operations at a virtual screen element of a second virtual object as shown in ).

[0097] Fig. 1I A user 101 is shown performing a position-independent gesture 130 (e.g., a “reverse pinch gesture”) directed at a virtual object 118 while the user's wrist wearable device 102 is outside of a surface edge 112. The position-independent gesture 130 is recognized as a position-independent gesture by sensors 150 of the wrist wearable device 102 (including, but not limited to, an EMG sensor 154 of a set of sensors 152, and an IMU sensor 160). Performing the position-independent gesture 130 at the virtual object 118 causes an operation to be performed at the head wearable device 104 to update the display of the user interface of the artificial reality system 100, including the display of the virtual object 126. Fig. 1I In the example of FIG. 1 , performing the position-independent gesture 130 causes the following operation to be performed: the operation causes the head wearable device 104 to adjust the visual appearance of the virtual object 118 to indicate that the user 101 has activated a selectable function of the virtual object 118 (e.g., performing an operation at a virtual screen element of a second virtual object, such as a FIG. 3A to FIG. 3I ), such as the virtual object 320 shown in FIG.

[0098] Figure 1JUser 101 is shown performing a position-independent gesture 132 (e.g., a “pinch” gesture) at a virtual object 118 while the user's wrist wearable device 102 is outside of a surface edge 112. The position-independent gesture 132 is recognized as a position-independent gesture by sensors 150 of the wrist wearable device 102 (including, but not limited to, a set of sensors 152 of EMG sensors 154, and IMU sensors 160). Performing the position-independent gesture 132 at the virtual object 118 causes an operation in a third set of operations to be performed at the head wearable device 104 to update the display of the user interface of the artificial reality system 100, including the display of the virtual object 118. Figure 1J In the example of FIG. 1 , performing the position-independent gesture 132 causes an operation to be performed that changes the appearance of the virtual object 118 to indicate that the user 101 has activated an available operation of the virtual object 118 (e.g., FIG. 3A to FIG. 3I ), performing operations at a virtual screen element of a second virtual object as shown in ).

[0099] and Figure 1G to Figure 1J The symbolic view of the sensor 150 shown side by side includes the Figure 1G to Figure 1J A chart of predicted data corresponding to data collected by EMG sensors and time-of-flight sensors during the execution of various gestures shown. As shown in the symbolic view of sensor 150, more than one time-of-flight sensor 158 (e.g., s1, s2, s3) can be used to detect the execution of various gestures. In some embodiments, additional sensors can be used to detect various aspects of the actions (including gestures) performed by user 101, as well as other aspects of the user's physical environment. For example, IMU sensor 160 can be used to detect specific forces, angular rates and / or orientations of body parts of user 101 and / or the user's surroundings at any device in wrist wearable device 102, head wearable device 104, or another electronic device. In some embodiments, impedance sensor 162 includes one or more sEMG sensors. In some embodiments, (e.g., in addition to the above sensors, or in place of the above sensors), one or more other sensors including camera sensor 164 are used to detect user movements and gestures.

[0100] FIG. 2A to FIG. 2D An example of a user scenario including a virtual object and a user gesture above a physical surface 210 (e.g., an air gesture) occurring within an artificial reality system 200 according to some embodiments is shown. The artificial reality system 200 includes a head wearable device 204 (e.g., Fig.13A ) and the wrist wearable device 202.

[0101] FIG. 2A to FIG. 2D A virtual object 220 is also shown. FIG. 2A to FIG. 2DIn the example shown, the aerial virtual object is a virtual screen (also sometimes referred to as a virtual display or virtual user interface). Those skilled in the art will recognize that the virtual object 220 can be a virtual object having various simulated sizes and simulated positions relative to the user's field of view. It will also be recognized that the spatial relationship between the virtual object 220 and the user 201 is simulated by the head wearable device 204. In some embodiments, the time-of-flight sensor 258 is used to determine the distance between the head wearable device 204 and the wrist wearable device 202 to simulate the spatial relationship between the virtual object 220 and the user 201. In some embodiments, gaze tracking and data from an IMU sensor (e.g., IMU sensor 260) can be used in addition to or as an alternative to other sensors to allow direct interaction with (and / or manipulation of) virtual objects (e.g., interaction with virtual objects by directly touching them, for example). FIG. 2B to FIG. 2D ). Also refer to FIG. 3A to FIG. 3I An example of indirect manipulation is shown, where a virtual object can be manipulated without directly touching it.

[0102] exist FIG. 2A to FIG. 2D , user 201 is standing near a table with a physical surface 210. The user is wearing a wrist wearable device 202 and a head wearable device 204. A virtual object 220 is displayed above the physical surface 210, and there is a reference lock enable representation 222 displayed in a dragging manner below the virtual object 220. The reference lock enable representation 222 can be used to control the position of the virtual object 220 by allowing the user 201 to "lock" a first "reference" position of the wrist wearable device 202 relative to the reference lock enable representation 222. In some embodiments, the user 201 is able to make a relative movement from a first reference position that causes relative movement through the first virtual object 220, even though the wrist wearable device 202 is not at the same position as the reference lock enable representation 222 (e.g., the user 101 is not required to simulate contact with the reference lock enable representation 222). The virtual object 220 also has an interactive display portion 226, in which content (e.g., a website) can be displayed for the user 201 to interact with. The virtual object 220 also has a display control portion 224. In some embodiments, the virtual object 220 is displayed without the display control portion 224. In some embodiments, the display control portion 224 is displayed at a physical surface (e.g., the physical surface 210). In some embodiments, one or more display controls are present in addition to or in lieu of the controls displayed in the display control portion 224.

[0103] Figure 2AAlso shown are visual aids representing surface threshold distance 206 and air threshold distance 208. As will be discussed below, air threshold distance 208 is related to an air gesture performed by user 201 (e.g., Figure 2B The air gestures in 242 and Figure 2C The air gesture 244 in FIG. Figure 2A Also shown is a visual aid representing surface edge 212 of physical surface 210. In some embodiments, for example interactions in artificial reality system 200, surface 210 has more than one relevant surface edge. In some embodiments, only position-independent gestures can be detected when wrist wearable device 202 is outside of one or more corresponding surface edges of physical surface 210. In some embodiments, surface gestures and / or air gestures can be detected when the wrist wearable device is within a threshold distance outside of one of the plurality of surface edges (e.g., surface edge 212).

[0104] like Figure 2A As shown in FIG. 2E , the sensors 250 of the wrist wearable device 202 include a set of sensors 252, which include an EMG sensor 254. In some embodiments, EMG data from the EMG sensor 254 is used to determine whether the user has performed a position-independent gesture, such as the position-independent gesture 240. The sensors 250 of the wrist wearable device 202 also include a set of sensors 256, which include a time-of-flight sensor 258. In some embodiments, data from the time-of-flight sensor 258 is used to determine whether the user 201 has performed a position-independent gesture. In some embodiments, data from the time-of-flight sensor 258 is not used to determine whether the user 201 has performed a position-independent gesture. For example, when the wrist wearable device 202 exceeds the surface edge 212, the time-of-flight sensor 258 is disabled (in a low-power state). In some embodiments, additional sensors can be used to detect various aspects of the actions (including gestures) performed by the user 201, as well as other aspects of the user's physical environment. For example, the IMU sensor 260 can be used to detect a specific force, angular rate, and / or orientation of a body part of the user 201 and / or the user's surrounding environment at any one of the wrist wearable device 102, the head wearable device 104, or another electronic device. In some embodiments, the impedance sensor 262 includes one or more sEMG sensors. In some embodiments, one or more other sensors 264 can be used to detect air gestures, and the one or more other sensors 264 include one or more imaging sensors.

[0105] exist Figure 2A In FIG. 2 , user 201 is performing a position-independent gesture 240 (a “thumbs-up” gesture) for initiating display of a virtual object 220 within artificial reality system 200. FIG. 4A to FIG. 4D As shown, position-independent gesture 240 and other position-independent gestures (eg Figure 4A In some embodiments, a position-independent gesture 440) can be performed at different locations relative to the physical surface 210 so that the same operation relative to the artificial reality system 200 is performed. In some embodiments, if the position-independent gesture is performed within a surface threshold distance and / or an air threshold distance, the position-independent gesture is not recognized. In some embodiments, a position-independent gesture is a gesture performed outside the edge of the surface and outside the air threshold distance. In some embodiments, the position-independent gesture is not recognized when the wearable device is in a particular state (e.g., a dormant state or other non-gesture state). In some embodiments, the virtual object 220 corresponds to content that is currently active and / or displayed at the wrist wearable device 202 or another electronic device.

[0106] exist Figure 2B , user 201 is performing air gesture 242 at affordance 228-3 within virtual object 220. Figure 2B In the example provided, the user receives a haptic event 221 at the wrist wearable device 202 in response to an air gesture 242 making virtual contact with the affordance 228-3.

[0107] like Figure 2B As shown, the sensors 250 of the wrist wearable device 202 include a set of sensors 252, which includes an EMG sensor 254. EMG data from the EMG sensor 254 (as shown by the predicted data of the EMG sensor 254 in the symbolic view of the sensor 250) is used to determine whether the user 201 has performed the air gesture 242. The sensors 250 of the wrist wearable device 202 include a set of sensors 256, which includes a time-of-flight sensor 258. In some embodiments, data from the time-of-flight sensor 258 is not used to determine whether the user 201 has performed the air gesture 242. For example, the EMG sensor 254 and / or the IMU sensor 260 can be used to determine whether the user 201 has performed the air gesture 242.

[0108] exist Figure 2C In FIG. 2 , the user is performing an air gesture 244 pointing to the display control portion 224 of the virtual object 220. Figure 2C In the example provided, user 201 receives a haptic event from wrist wearable device 202 in response to air gesture 244 of making virtual contact with affordance 228-3. In some embodiments, user 201 receives an intermediate haptic event as part of the haptic event when performing discrete increments of a display scrolling operation in response to air gesture 244.

[0109] like Figure 2CAs shown, the sensors 250 of the wrist wearable device 202 include a set of sensors 252, which include an EMG sensor 254. EMG data from the EMG sensor 254 is used to determine whether the user 201 has performed the air gesture 244. The sensors 250 of the wrist wearable device 202 include a set of sensors 256, which include a time-of-flight sensor 258. In some embodiments, the time-of-flight data from the time-of-flight sensor 258 is not used to determine whether the user 201 has performed the air gesture 244. The prophetic time-of-flight and EMG sensor data are shown in the symbolic view of the sensor 250.

[0110] exist Figure 2D , user 201 is performing an air gesture 246 directed toward reference lock affordance 222. Figure 2D In the example provided, user 201 receives a haptic event at wrist wearable device 202 based on performing an air gesture 246 to reference lock with reference lock affordance 222 .

[0111] like Figure 2D As shown, the sensors 250 of the wrist wearable device 202 include a set of sensors 252, which includes an EMG sensor 254. EMG data from the EMG sensor 254 is used to determine whether the user 201 has performed the air gesture 246. The sensors 250 of the wrist wearable device 202 include a set of sensors 256, which includes a time-of-flight sensor 258. In some embodiments, the time-of-flight data from the time-of-flight sensor 258 is not used to determine whether the user 201 has performed the air gesture 246. The predicted time-of-flight and EMG sensor data are shown in the symbolic view of the sensor 250.

[0112] FIG. 3A to FIG. 3I An example of a user scene including virtual objects and user gestures with an artificial reality system 300 according to some embodiments is shown, which can be displayed at, above, or otherwise relative to a physical surface 310. The artificial reality system 300 includes a wrist wearable device 302 and a head wearable device 304 (e.g., Fig.13A 1320 in the AR system). FIG. 3A to FIG. 3I User 301 in FIG. 3 is standing near a table having a physical surface 310 .

[0113] FIG. 3A to FIG. 3I An example of a virtual object 320 is shown. FIG. 3A to FIG. 3IIn the example shown, the virtual object 320 is a virtual screen element. The spatial relationship between the virtual object 320 and the user 201 is simulated by the display of the head wearable device 304. The time-of-flight sensor can be used to determine the distance between the head wearable device 204 and the wrist wearable device 202 to simulate the spatial relationship between the virtual object 320 and the user 301.

[0114] exist FIG. 3A to FIG. 3I , user 301 is standing near a table with a physical surface 310. User 301 is wearing a wrist wearable device 302 and a head wearable device 304. A virtual object 320 is displayed above the physical surface 310. And there is a reference lock enable representation 322 displayed in a dragging manner below the virtual object 320. Virtual object 320 also has an interactive display portion 326, where content (e.g., a browser, a PDF viewing application, an image editing application, etc.) can be displayed for user 301 to interact with. And virtual object 320 has a display control portion 324, and user 301 can use display control portion 324 to indirectly manipulate interactive display portion 326. Display control portion 324 does not need to be displayed as part of virtual object 320, but can also be displayed at physical surface 310. In addition to display control portion 324, there can also be one or more display control portions.

[0115] like FIG. 3A to FIG. 3IAs shown, the sensor 350 of the wrist wearable device 302 includes a group of sensors 352, which includes an EMG sensor 354. In some embodiments, EMG data from the EMG sensor 354 is used to determine whether the user has performed a position-independent gesture. The sensor 350 of the wrist wearable device 302 also includes a group of sensors 356, which includes a time-of-flight sensor 358. In some embodiments, data from the time-of-flight sensor 358 is used to determine whether the user 301 has performed a position-independent gesture. In some embodiments, data from the time-of-flight sensor 358 is not used to determine whether the user 301 has performed a position-independent gesture. For example, when the wrist wearable device 302 exceeds the surface edge 312, the time-of-flight sensor 358 is disabled (in a low-power state). In some embodiments, additional sensors can be used to detect various aspects of the actions (including gestures) performed by the user 301, as well as other aspects of the user's physical environment. For example, IMU sensor 360 can be used to detect a specific force, angular rate, and / or orientation of a body part of user 301 and / or the user's surrounding environment at any one of wrist wearable device 302, head wearable device 304, or another electronic device. In some embodiments, impedance sensor 362 includes one or more sEMG sensors. In some embodiments, one or more other sensors 364 can be used to detect air gestures, and the one or more other sensors include one or more imaging sensors.

[0116] exist Figure 3A In the embodiment, user 301 is performing a surface gesture 340 (e.g., a “surface tap” gesture in which the user's finger makes physical contact with physical surface 310) for initiating display of virtual object 320, which surface gesture 340 is recognized as a surface gesture by a set of sensors 356 (including but not limited to time-of-flight sensor 358) of wrist wearable device 302. Performance of surface gesture 340 at physical surface 310 causes performance of an operation in a second set of operations at head wearable device 304 to update the display of a user interface of artificial reality system 300, including updating the display of virtual object 320 by head wearable device 304. In some embodiments, user 301 receives a tactile event at wrist wearable device 302 based on contacting physical surface 310. In Figure 3A In the example of , performing the surface tap gesture 340 causes an operation to be performed that changes the appearance of the virtual object 320 to indicate that the user 301 has activated a selectable function of the virtual object 320 .

[0117] exist Figure 3B, user 301 performs a surface gesture 342 at an affordance representation of a second virtual object 328, and surface gesture 342 is recognized as a surface gesture by a set of sensors 356 (including but not limited to a time-of-flight sensor 358) of wrist wearable device 302. Performing surface gesture 342 at physical surface 310 causes an operation from a set of operations corresponding to the air gesture to be performed at head wearable device 304 to update the display of a user interface of artificial reality system 300, including the display of virtual object 320. In some embodiments, user 301 receives a tactile event 311 (represented by a pair of reference numerals 311a and 311b) at wrist wearable device 302 based on making virtual contact with second virtual object 328. In Figure 3B In the example of , performing the surface gesture 342 causes an operation to be performed that changes the appearance of the virtual object 320 to indicate that the user 301 has activated a selectable function of the virtual object 320 .

[0118] exist Figure 3C , user 301 is performing a surface gesture 344 (e.g., a horizontal swipe gesture at or near physical surface 310), which is recognized as a surface gesture by a second set of sensors 356 (including but not limited to time-of-flight sensors 358) of wrist wearable device 302. Execution of surface gesture 344 causes an operation in a set of air operations to be performed at head wearable device 304 to update the display of a user interface of artificial reality system 300, including display of virtual object 320 and affordance 330 (which is visually depicted as a smiling clown on interactive display portion 326 of affordance 330). In some embodiments, user 301 receives a tactile event at wrist wearable device 302 in accordance with execution of surface gesture 344. Figure 3C In the example of , performing surface gesture 344 causes an operation to be performed that changes the appearance of virtual object 320 (including the position of affordance 330).

[0119] exist Figure 3D, user 301 is performing a position-independent gesture 346 (a pinch gesture within a surface threshold distance of the physical surface 210), and although the wrist wearable device 302 recognizes that the position-independent gesture 346 is performed within the surface threshold distance 306 of the physical surface 310, the position-independent gesture 346 is recognized as a position-independent gesture by a set of sensors 356 (including but not limited to a time-of-flight sensor 358) of the wrist wearable device 302. Performing the position-independent gesture 346 above the physical surface 310 causes a set of position-independent gesture operations to be performed at the head wearable device 304 to update the display of the user interface of the artificial reality system 300, including the display of the virtual object 320 and the enable representation 330 (the enable representation 330 is visually reduced in response to the position-independent gesture 346). In some embodiments, the user 301 receives a tactile event 313 (represented by a pair of reference numerals 313a and 313b) at the wrist wearable device 302 in accordance with performing the position-independent gesture 346. In Figure 3D In the example of , user 301 performs position-independent gesture 346 to perform an operation from a set of position-independent gestures that changes the appearance of virtual object 320 (including the position of affordance 330 at virtual object 320).

[0120] exist Figure 3E , user 301 is performing a surface gesture 348 (a double tap gesture performed within a surface threshold distance 306 of a physical surface 310), which surface gesture 348 is recognized as a surface gesture by a set of sensors 356 (including but not limited to a time-of-flight sensor 358) of a wrist wearable device 302. Performing surface gesture 348 at the physical surface 310 causes an operation from a set of operations corresponding to the surface gesture to be performed at the head wearable device 304 to update the display of the user interface of the artificial reality system 300, including the display of virtual objects 320 and affordances 330 (displaying additional options for adjusting the appearance of affordances 330). User 301 receives a tactile event at wrist wearable device 302 in accordance with performing surface gesture 348. Specifically, user 301 receives a separate tactile sub-event each time they make physical contact with the physical surface 310. In . Figure 3E In the example of , performing surface gesture 348 causes an operation to be performed that changes the appearance of virtual object 320 (including the appearance of affordance 330).

[0121] exist Figure 3F, user 301 performs position-independent gesture 341 above physical surface 310, and position-independent gesture 341 is recognized as a position-independent gesture by sensor 350 of wrist wearable device 302. Performing position-independent gesture 341 at physical surface 310 causes an operation in a third group of operations corresponding to the position-independent gesture to be performed at head wearable device 304 to update the display of the user interface of artificial reality system 300, including the display of virtual object 320 and enable representation 330 (saving the current state of enable representation 330). User 301 receives a tactile event at wrist wearable device 302 in accordance with performing position-independent gesture 341. In Figure 3F In the example of , performing position-independent gesture 341 results in performing an operation that changes the appearance of virtual object 320 (including the appearance of affordance 330).

[0122] exist Figure 3G , user 301 is performing an air gesture 343 (a virtual tap gesture at a simulated location of a virtual object 320 above a physical surface 310), which is recognized as an air gesture by a set of sensors 352 including an EMG sensor 354. Performing the air gesture 343 at the physical surface 310 causes an operation from a set of operations corresponding to the air gesture to be performed at the head wearable device 304 to update the display of the user interface of the artificial reality system 300, including the display of the virtual object 320 and the enable representation 330 (refer to locking the enable representation 330 in the virtual object 320). User 301 receives a tactile event 317 at the wrist wearable device 302 in response to performing the position-independent gesture 341. Figure 3F In the example of , performing position-independent gesture 341 results in performing an operation that changes the appearance of virtual object 320 (including the appearance of affordance 330).

[0123] exist Figure 3H , user 301 is performing position-independent gesture 345 (a virtual pull gesture directed toward an affordance 330 displayed at a virtual object 320), which is recognized as an air gesture by a set of sensors 352 including an EMG sensor 354. Performing position-independent gesture 345 at physical surface 310 causes an operation from a set of operations corresponding to the air gesture to be performed at head wearable device 304 to update the display of the user interface of artificial reality system 300, including the display of virtual object 320 and affordance 330 (refer to locking affordance 330 in virtual object 320). Figure 3H In the example in , performing position-independent gesture 345 results in performing an operation that changes the appearance of virtual object 320, including the appearance of affordance 330, which visually pulls affordance 330 away from virtual object 320.

[0124] exist Fig. 3I , user 301 is performing position-independent gesture 347 (a continuous virtual pull gesture directed toward affordance 330 displayed at virtual object 320), which is recognized as a position-independent gesture by a set of sensors 352 including EMG sensor 354. Performing position-independent gesture 347 at physical surface 310 causes an operation in a third set of operations to be performed at head wearable device 304 to update the display of a user interface of artificial reality system 300, including the display of virtual object 320 and affordance 330 (reference affordance 330 locked in virtual object 320). Fig. 3I In the example of FIG. 1 , performing position-independent gesture 345 causes an operation for changing the appearance of virtual object 320 from a set of operations corresponding to the position-independent gesture to be performed, where changing the appearance of virtual object 320 includes changing the appearance of affordance 330 (e.g., Fig. 3I As shown, the affordance 330 is visually pulled away from the virtual object 320 and becomes a three-dimensional virtual object 332).

[0125] FIG. 4A to FIG. 4D An example of a user scene including virtual objects and user gestures that can be displayed at, above, or otherwise relative to a physical surface with an artificial reality system 400 according to some embodiments is shown. The artificial reality system 400 includes a wrist wearable device 402 and a head wearable device 404 (e.g., Fig.13A 1320 in the AR system). FIG. 4A to FIG. 4D User 401 in FIG. 4 is standing near a table having a physical surface 410 .

[0126] FIG. 4A to FIG. 4D An example of a virtual object 420 is shown. FIG. 4A to FIG. 4D In the example shown, virtual object 420 is a virtual screen element. The spatial relationship between virtual object 420 and user 401 is simulated by head wearable device 404. In other words, virtual object 420 is provided to user 401 to appear at a simulated three-dimensional position in the physical environment of user 401. A time-of-flight sensor can be used to determine the distance between head wearable device 404 and wrist wearable device 402 to simulate the spatial relationship between virtual object 420 and user 401.

[0127] exist FIG. 4A to FIG. 4DIn FIG. 4 , user 401 stands near a table having a physical surface 410. User 401 wears a wrist wearable device 402 and a head wearable device 404. A virtual object 420 is displayed above the physical surface 410. And there is a reference lock enable indication 422 displayed in a dragging manner below the virtual object 420. The virtual object 420 also has an interactive display portion 426, in which content (e.g., a browser) can be displayed for user 401 to interact with. FIG. 4A to FIG. 4D A key aspect of the examples shown in is that certain position-independent gestures cause the same operation to be performed in the artificial reality system 400 regardless of where the position-independent gestures are performed relative to the physical surface 410. However, those skilled in the art will recognize that in some cases, due to criteria including, for example, the position of the wrist wearable device and / or the state of the sensor 450, the user 401 performing the position-independent gesture will not be recognized as such.

[0128] FIG. 4A to FIG. 4D The sensor at the wrist wearable device 402 in the embodiment may include any sensor in a plurality of sensors 450, the plurality of sensors 450 including a group of sensors 452 and a group of sensors 456, the group of sensors 452 including an EMG sensor 454, the group of sensors 456 including a time-of-flight sensor 458. In some embodiments, data from the time-of-flight sensor 458 is not used to determine whether the user 401 has performed a position-independent gesture. For example, when the wrist wearable device 402 exceeds the surface edge 412, the time-of-flight sensor 458 is disabled (in a low-power state). In some embodiments, additional sensors may be used to detect various aspects of the actions (including gestures) performed by the user 401 and other aspects of the user's physical environment. For example, an IMU sensor 460 may be used to detect a specific force, angular rate, and / or orientation of a body part of the user 401, and / or the user's surrounding environment at any one of the wrist wearable device 402, the head wearable device 404, or another electronic device. In some embodiments, the impedance sensor 462 includes one or more sEMG sensors. In some embodiments, other sensors 464 may be used, including, for example, imaging sensors, audio sensors, etc.

[0129] exist Figure 4A , user 401 performs position-independent gesture 440 (a "thumbs-up" gesture performed while the user's hand is above physical surface 410 and within surface threshold distance 406 of physical surface 410). Figure 4A, position-independent gesture 440 is performed within surface threshold distance 406 of physical surface 410. User 401 performs position-independent gesture 440 above physical surface 410 so that head wearable device 404 performs an operation of a set of position-independent operations for saving the current state of two-dimensional affordance representation 430 of virtual object 420.

[0130] exist Figure 4B 4, user 401 performs position-independent gesture 442 (a "thumbs-up" gesture performed while the user's hand is in contact with physical surface 410). User 401's performance of thumbs-up gesture 442 within the surface threshold distance causes the head wearable device to perform an operation in a set of position-independent operations for saving the current state of two-dimensional affordance 430 in interactive display portion 426 of virtual object 420. It should be noted that even if thumbs-up gesture 442 is performed within surface threshold distance 406, it does not cause an operation in the second set of operations to be performed, even though the second set of operations is typically associated with surface gestures.

[0131] exist Figure 4C 4, user 401 performs position-independent gesture 444 (a "thumbs-up" gesture performed while the user's hand is within air-threshold distance 406 of physical surface 410). User 401's performance of position-independent gesture 444 within air-threshold distance 408 causes head wearable device 404 to perform an operation in a set of position-independent operations for saving the current state of affordance 430 of virtual object 420. Those skilled in the art will recognize that even if position-independent gesture 444 is performed within air-threshold distance 408, it does not cause an operation in the second set of operations to be performed, even though the second set of operations is typically associated with surface gestures.

[0132] exist Figure 4D 4, user 401 performs position-independent gesture 446 (a "thumbs-up" gesture performed when the user's hand is outside of surface edge 412 of the physical surface and is neither within surface threshold distance 406 nor air threshold distance 408). User 401's performance of position-independent gesture 446 within air threshold distance 408 causes head wearable device 404 to perform an operation in a set of position-independent operations for saving a current state of two-dimensional affordance 430 of virtual object 420. It should be noted that even if position-independent gesture 446 is performed within air threshold distance 408, it does not cause an operation in the first set of operations to be performed, even though the first set of operations is typically associated with surface gestures.

[0133] FIG. 5A to FIG. 5HAn example of a user scenario with an artificial reality system including virtual objects and user gestures that cause operations to be performed to adjust control options of a physical object (e.g., an electronic device) that can be provided as a representation of the physical object (e.g., with augmented visual display features overlaid on the electronic device). The artificial reality system 500 includes a wrist wearable device 502 and a head wearable device 504 (e.g., Fig.13A 1320 in the AR system). FIG. 5A to FIG. 5H The user 501 in FIG. 5 is standing near a table with a physical surface 510. On the physical surface 510 there is a physical object, an electronic device 570.

[0134] exist Figure 5A In FIG. 5 , user 501 wears wrist wearable device 502 and head wearable device 504. User 501 faces gaze direction 566 pointing to electronic device 572. FIG. 5A to FIG. 5H In the example shown, electronic device 570 responds to the user's gesture only when user 501 has a gaze direction 566 that is within a predefined gaze range of electronic device 570. Figure 5A , there is also an electronic device 570 capable of interacting with the user 501. The device is located at the physical surface 510 of the coffee table and has a visual aid to indicate that there is a surface threshold distance 506 and an air threshold distance 508 associated with the table. There is also a surface edge 512 associated with the coffee table. In some embodiments, the user 501 can perform surface gestures within the surface threshold distance 506 of the physical surface 510 and perform air gestures within the air threshold distance 508. However, FIG. 5A to FIG. 5H An embodiment is shown in which the user 501 performs a gesture outside of each respective area (eg, away from the surface).

[0135] exist Figure 5B , user 501 has a gaze direction 568 within a predefined gaze range directed toward electronic device 570. In some embodiments, when user 501 has a gaze direction (e.g., gaze direction 568) within a predefined gaze range of electronic device 570, user 501 may perform a gesture directed toward electronic device 570 that causes an operation related to a function of electronic device 570 to be performed. In the example shown, electronic device 570 is a smart home speaker, and user 501 is able to perform a gesture that causes electronic device 570 to play music. In some embodiments, a gesture may cause an operation to be performed at one or more additional devices that have additional control options that may be different from the control options of electronic device 570.

[0136] exist Figure 5B, user 501 is performing a virtual pull gesture 542 (e.g., a waving gesture) directed toward electronic device 570. In response to detecting virtual pull gesture 542 based on data from a set of sensors of wrist wearable device 502, head wearable device 504 is caused to display virtual object 520. Virtual object 520 is displayed at a simulated location near electronic device 570. In some embodiments, virtual pull gesture 542 must satisfy one or more additional gesture criteria in order for an operation to display virtual object 520 to be performed. For example, certain levels of EMG detection may be considered when determining whether to cause an operation to be performed at head wearable device 504 for displaying virtual object 520. Figure 5B As shown, the virtual object 520 can be displayed at a primary display position 560 relative to the electronic device 570.

[0137] exist Figure 5C In FIG. 5 , user 501 is performing an air gesture 544 (e.g., a “virtual pull” gesture) directed toward virtual object 520. Air gesture 544 is similar to Figure 5B However, since the air gesture 544 is directed to the virtual object 520 instead of the electronic device 570, the execution of the virtual pull gesture 542 is performed. Figure 5B The operation corresponding to the displayed virtual object 520 is different from the operation performed in the embodiment. Specifically, in response to detecting based on data from a set of sensors of the wrist wearable device 502 that Figure 5C An air gesture 544 pointing toward the virtual object 520 causes an operation to be performed at the head wearable device 504 for updating the display of the virtual object 520 within the artificial reality system 500.

[0138] exist Figure 5D In FIG. 5 , user 501 is performing a virtual sliding gesture 546 at interactive display portion 526 of virtual object 520, which virtual sliding gesture 546 can be detected as either an air gesture or a position-independent gesture. Figure 5D In the example shown, user 501 is performing a virtual slide gesture 546 at a two-dimensional affordance 530 within an interactive display portion 526 of a virtual object 520 . Figure 5D504. The virtual sliding gesture 546 in the air causes several operations to be performed at the head wearable device and / or electronic device 570. As one of the several operations, an operation is performed at the head wearable device 504 that causes the display of the virtual object 520 to be updated in the artificial reality system 500. The update to the display changes the content displayed in the interactive display portion 526 of the virtual object 520. In addition, the air gesture 544 causes the following operations to be performed at the wrist wearable device 502: when the user 501 makes a virtual touch contact with the simulated location of the virtual object 520, the operation causes a haptic event at the wrist wearable device 502. In addition, when the user's virtual touch contact with the simulated location of the virtual object 520 continues the air gesture 544, a second haptic event is provided at the wrist wearable device. In some embodiments, if the virtual object 520 is a physical object with material properties, the haptic event provided as part of the operations performed at the wrist wearable device 502 corresponds to an actual force that will be applied to the user 501.

[0139] exist Figure 5E , user 501 is performing a virtual tap gesture 548 at a two-dimensional affordance 530 within an interactive display portion 526 of a virtual object 520. Figure 5E In the example shown, user 501 is selecting a song from a playlist to be played at electronic device 570. When user 501 makes simulated contact with two-dimensional affordance 530, a tactile event 590 is provided at wrist wearable device. Data from the first set of sensors is used to detect a virtual tap gesture 548 at a simulated location of two-dimensional affordance 530. For example, an EMG sensor and / or an optical imaging sensor may detect the position and / or movement of a user's hand. Virtual tap gesture 548 causes an operation of the first set of operations for updating the display of virtual object 520 in artificial reality system 500 to be performed at head wearable device 504. Although a variety of specific updates are possible, in some embodiments, the present invention may be used in conjunction with the user's hand to perform a virtual tap gesture 548. Figure 5E In the example of , the user receives a visual indication in the form of user interface element 528 that a song has been selected, displayed separately from virtual object 520. In some embodiments, user 501 can perform a pinch gesture to hide user interface element 528.

[0140] exist Fig. 5F5 , a user is performing a virtual push gesture 541 directed toward a virtual object 520, which causes an operation to be performed at the head wearable device 504 for updating the display of the virtual object 520 so that the display performs an animated motion toward the electronic device 570. As the virtual object 520 is animated along its path toward the electronic device 570, the electronic device 570 performs an operation to cause an auditory confirmation to be emitted from the electronic device 570, indicating to the user that the song selection has been confirmed at the electronic device 570. The animated motion of the virtual object 520 continues until the virtual object 520 reaches a primary display position 560 relative to the electronic device 570.

[0141] exist Figure 5G In FIG. 5 , when the song selected by user 501 starts playing at electronic device 570, virtual object 520 is animated to a hidden state. The animation of virtual object 520 is consistent with the virtual object reaching the primary display position 560 relative to electronic device 570. Once the virtual object reaches the primary display position 560 relative to electronic device 570, an operation is performed at head wearable device 504 to update the display of virtual object 520. Figure 5G As shown, the update to the display of virtual object 520 is an animation sequence that hides virtual object 520. According to virtual object 520 being hidden and not displayed in artificial reality system 500, user 501 also receives haptic events at wrist wearable device 502.

[0142] exist Figure 5H , a user performs a surface gesture at his palm. Specifically, user 501 is performing surface gesture 543 (e.g., a virtual sliding gesture in a particular direction along the palm of user 501). The surface gesture is detected using data from a second set of sensors of the wrist wearable device, causing an operation in the second set of operations to update the display of virtual object 520, and specifically to change the value of two-dimensional volume control indication 532, to be performed at head wearable device 504. Surface gesture 543 also causes an operation to be performed at electronic device 570 to reduce the volume of music played at electronic device 570.

[0143] FIG. 6A to FIG. 6C An example of a user scenario with an artificial reality system including virtual objects and user gestures that cause operations to be performed to adjust control options of a responsive electronic device is shown. The artificial reality system 600 includes a wrist wearable device 602, but notably, does not include a head wearable device. FIG. 6A to FIG. 6C The user 601 in FIG. 1 is standing near a table having a physical surface 610. On the physical surface 610 is an electronic device 670. FIG. 6A to FIG. 6C601 has a visual aid that indicates that there is a surface threshold distance 606 and an air threshold distance 608 associated with the physical surface 610 of the table. There is also a surface edge 612 associated with the coffee table. In some embodiments, the user 601 can perform a surface gesture within the surface threshold distance 606 of the physical surface 610 and perform an air gesture within the air threshold distance 608. However, FIG. 6A to FIG. 6C An embodiment is shown in which user 601 is performing a gesture outside of each respective region (eg, away from physical surface 610 ).

[0144] exist Fig. 6A In FIG. 6 , user 601 wears wrist wearable device 602. User 601 faces a gaze direction 666 pointing away from electronic device 670. However, regardless of the user's gaze direction 666, in FIG. 6A to FIG. 6C In the example shown, the electronic device 670 still responds to the gesture performed by the user 601. This is called an "eyes away" response to the user's gesture, which will be discussed further below. Fig. 6A In the example, user 601 specifically performs gesture 640 (a “pointing gesture” pointing to electronic device 670 ) at electronic device 670 . According to user 601 performing 640 , electronic device 670 starts playing music through a speaker of electronic device 670 .

[0145] exist Figure 6B In FIG. 6 , user 601 is performing a surface gesture, specifically a surface gesture 642 in a first direction at the palm of the user. The surface gesture 642 causes a volume control operation to be performed at electronic device 670. Figure 6B In the example shown, the volume control operation corresponding to the user's virtual sliding gesture is a volume reduction operation. Figure 5H As shown, in head wearable devices (e.g., Figure 5H The virtual object is displayed at the head wearable device 504).

[0146] exist Figure 6C In FIG. 6 , user 601 is performing a surface gesture, specifically a surface gesture 644 at the user's palm. Surface gesture 644 is in Figure 6B In accordance with user 601 performing surface gesture 644 in the second direction, performing the same gesture at the responding device as at the responding device Figure 6B Specifically, in Figure 6C , when the user 601 performs a surface gesture 644 at the user's palm in the second direction, the surface gesture 644 causes a song to be changed at the electronic device 670.

[0147] 7A to 7FAn example of a user scenario with an artificial reality system including a virtual object and user gestures that cause operations to be performed for adjusting control options of an electronic device is shown. The artificial reality system 700 includes a wrist wearable device 702 and optionally includes a head wearable device (e.g., Fig.13A 1320 in the AR system). 7A to 7F A user 701 in FIG. 7 is standing near a table with a physical surface 710. On the physical surface 710 are a physical object and an electronic device 770. In some embodiments, the head wearable device displays a representation of the physical object that is different from the actual appearance of the physical object (e.g., enhanced).

[0148] exist Fig. 7A , user 701 is performing a position-independent gesture 740 directed toward electronic device 770. Wrist wearable device 702 detects position-independent gesture 740 directed toward electronic device 770 using data from the first set of sensors. Based on wrist wearable device 702 detecting position-independent gesture 740 directed toward electronic device 770, the wrist wearable device electronically communicates with electronic device 770 to cause electronic device 770 to begin playing music that user 701 had previously played from wrist wearable device 702. Electronic device 770 begins playing the song when the song stops playing on wrist wearable device 702.

[0149] exist Figure 7B In FIG. 7 , user 701 is performing a pointing gesture 742 at electronic device 772. Since the user's pointing gesture 742 is not directed to electronic device 770, electronic device 770 performs a pointing gesture 742 at electronic device 772. Fig. 7A The pointing gesture 740 performed in the figure starts playing the song. Based on the pointing gesture 742 directed to the electronic device 772 detected by the wrist wearable device 702, the wrist wearable device communicates electronically with the electronic device 772 so that the electronic device 772 activates the light source (e.g., a light bulb) of the electronic device 772.

[0150] exist Figure 7C , user 701 is performing an air gesture 744 that is not directed toward electronic device 770 or electronic device 772. However, because user 701 most recently interacted with electronic device 772, the user's air gesture 744 is reference-locked to electronic device 772. Based on wrist wearable device 702 detecting air gesture 744, wrist wearable device 702 electronically communicates with electronic device 772 to cause electronic device 772 to reduce a brightness level of a light source of electronic device 772.

[0151] exist Fig.7D, user 701 is performing position-independent gesture 746 (e.g., a master control gesture that is not directed toward electronic device 770 or electronic device 772). However, because position-independent gesture 746 is a gesture in a set of master command gestures enabled by user 701 for execution in conjunction with wrist wearable device 702, position-independent gesture 746 causes an operation to be performed at all electronic devices in room 790. Based on wrist wearable device 702 detecting position-independent gesture 746, wrist wearable device 702 electronically communicates with electronic device 770 and electronic device 772 to (a) further reduce the brightness level of the light source of electronic device 772, and (b) reduce the volume level of music played from electronic device 770. Fig.7D In the example of , the location-independent gesture 746 does not cause an operation to be performed at the third electronic device 774 or the fourth electronic device 776 because those devices are not in the same room as the user (eg, they are both in room 792).

[0152] exist Fig. 7E In , the user puts on a head wearable device and can see... Fig. 7E In the embodiment, virtual object 720 and virtual object 722 are displayed in artificial reality system 700. Virtual object 720 displays the control settings of electronic device 770. And virtual object 722 displays the control settings of electronic device 772. Fig. 7E In the example shown, based on the user performing a position-independent gesture 746 (eg Fig.7D In some embodiments, the virtual objects are continuously displayed in conjunction with the user's interaction with the electronic device. In some embodiments, the virtual objects are not displayed (e.g., as shown in the above reference 6A to 7D described).

[0153] exist Figure 7F In FIG. 7 , user 701 is performing a position-independent gesture 748 directed toward fourth electronic device 776. Based on wrist wearable device 702 detecting position-independent gesture 748 directed toward fourth electronic device 776, wrist wearable device 702 electronically communicates with fourth electronic device 776 so that fourth electronic device 776 starts playing music from a speaker of fourth electronic device 776. Figure 7F In the example shown, the music is played from the moment the music is stopped from playing from the speaker of the electronic device 770. In this example, the music is stopped from playing from the electronic device 770 when the user 701 enters the room 792. In this example, the music is stopped from playing from the electronic device 770 in response to the position-independent gesture 748.

[0154] FIG. 8A to FIG. 8EAnother example user scenario with an artificial reality system (e.g., including at least AR glasses and a wrist wearable device) according to some embodiments is shown. Artificial reality system 800 includes wrist wearable device 802 and head wearable device 804.

[0155] exist Fig. 8A In FIG, user 801 is sitting on sofa 824 and watching TV on TV set 822. Fig. 8A 804, the head wearable device 804 does not display user interface elements, and the user is simply interacting with their physical environment. In some embodiments, the head wearable device provides user interface elements that represent content displayed on the television 822, so the user 801 can enjoy the same content without being in close proximity to the television 822. The sofa 824 has two armrests that include curved portions 806 of the physical surface 808. Although in FIG. 8A to FIG. 8E The curved portion in the example shown has a single, smooth field of curvature around the handrail of the surface, but in some embodiments, the curved portion includes more complex geometries and / or shapes. Fig. 8A As shown in the enlarged view in FIG, the sensor 850 includes multiple groups of sensors for detecting gestures of the user 801 and proximity to various physical objects and / or physical electronic devices. As shown in the enlarged view, the sensor 850 includes one or more of the following: a group of sensors 852 (including one or more EMG sensors 854), a group of sensors 856 (including one or more time-of-flight sensors 858), one or more IMU sensors 860, one or more sensors 862, and one or more other sensors 864. Fig. 8A Also shown is a remote control device 820 configured to communicate with a television set 822 .

[0156] exist Figure 8B , user 801 turns to a gaze direction 830 that causes them to have an angle within a threshold gaze angle range, causing the head wearable device 804 to display a virtual object 810 that includes a plurality of virtual buttons (e.g., virtual button 816-1) displayed along an armrest curvature 814 of a curved portion 806. The displayed portion of the virtual object 812 causes the display at the head wearable device 804 to be updated so that the portion of the virtual object 812 appears to be displayed on the physical surface of the armrest of the sofa 824. According to some embodiments, the virtual object 812 corresponds to the remote control device 820 (e.g., replicates the functionality of the remote control device 820). The head wearable device 804 displays the virtual object 812 based on communication with the television 822 (e.g., the virtual object 812 includes controls corresponding to the operation of the television 822).

[0157] exist Figure 8C, a user 801 activates a control of a television by performing a surface gesture 842 at a virtual button 816-1 of a virtual object 812 displayed at a curved portion 806. A surface threshold distance for the surface gesture at the curved portion 806 changes based on the shape of the curved portion 806. The wrist wearable device 802 or the head wearable device 804 communicates with the television 822 in response to activation of the virtual button 816-1 to adjust the operation of the television 822.

[0158] Now go to Fig.8D In some embodiments, the virtual button 816 - 1 and other enabling representations of the virtual object 810 are configured and arranged to be displayed at a horizontal plane 851 above the curved portion 806 to provide a more consistent interaction experience to the user 801 at the curved portion 806 .

[0159] Fig. 8E Another way in which the head wearable device 804 can display a virtual object 812 on a curved portion is shown. Fig. 8E , the physical surface is a portion (e.g., a leg portion) of the leg 865 of the user 801. FIG. 8A to FIG. 8D , even when the physical surface is part of the user's body (e.g., the leg 865 of the user 801), a portion of the virtual object 812 can be displayed at the curved surface portion 806. In some embodiments, other parts of the user's body (e.g., the palm of the user) can be used to perform the same operation as the reference. FIG. 8A to FIG. 8D The operations described are similar to the operations described above.

[0160] In addition, although the user scenarios described in the series with reference to FIG. 1 to FIG. 8 are described as separate sequences, in some embodiments, the user scenarios are combined with each other. FIG. 8A to FIG. 8E The method described in the sequence for detecting curved parts of a physical surface can be used with reference Figures 1A to 4D The presented sequence is used in combination with the method described. In other words, the method for detecting gestures performed at various distances from a physical surface can be adapted to take into account curved surface parts, e.g. FIG. 8A to FIG. 8E Similarly, for execution and reference FIG. 5A to FIG. 7F The method of the described electronic device corresponding to the operation of the gesture is optionally based on the following Figures 1A to 4D The gestures may be performed at various distances from the physical surface shown, and such a system may also take into account curved portions of such physical surfaces.

[0161] 9A to 9C9 is a flow chart illustrating a method 900 of making air gestures and surface gestures available to a user in accordance with some embodiments. The method 900 is performed at an artificial reality system (e.g., artificial reality system 100, 200, 300, 400, 500, 600, 700, or 800) having one or more processors and memory. In some embodiments, the artificial reality system is Fig. 12A In some embodiments, the artificial reality system is Fig. 12B In some embodiments, the artificial reality system includes one or more wearable devices, such as a wrist wearable device 102 or a head wearable device 104. In some embodiments, the artificial reality system includes one or more handheld devices, such as a controller or a smart phone. In some embodiments, the memory stores one or more programs, which are configured to be executed by the one or more processors. 9A to 9C At least some of the operations shown in the figure correspond to operations stored in a computer memory or a computer-readable storage medium (e.g., Fig. 12B In some embodiments, the artificial reality system is a wearable device, such as a computer system 1272 or a memory 1278 of the computer system 1272 or a memory 1256 of the auxiliary device 1252. Figures 1A to 1J A wrist wearable device 102 or a head wearable device 104 in the embodiment.

[0162] The artificial reality system is provided via a head wearable device (e.g. Fig.13A The AR system 1320 in the embodiment presents (902) a user interface (e.g., Figure 1A The artificial reality environment of the virtual object 118 in the specification and corresponding Figures 1A to 8E Various additional examples of artificial reality environments are described and shown in . In some embodiments, presenting the artificial reality environment includes providing audio feedback and / or tactile feedback.

[0163] Artificial reality systems respond to mid-air gestures (e.g. Figure 2B For example, the system can detect an air gesture performed by a user that involves the user's hand making virtual contact with an air virtual object and sliding over the virtual object in a horizontal direction (e.g., Figure 2C 244) and respond to the air gesture.

[0164] The air gesture corresponds to (906) causing the computer system to perform a first set of operations (e.g., updating the Figure 2BIn some embodiments, the first set of operations corresponds to operations performed to update the display of the aerial virtual object (e.g., Figure 3G In some embodiments, the air gesture can also cause an operation to update the display of the virtual object at the physical surface. In some embodiments, the air gesture causes a non-visual update to the artificial reality environment.

[0165] Artificial reality systems respond to surface gestures (e.g. Figure 1B For example, the artificial reality system can detect that a user performs a surface gesture (e.g., Figure 1C 122) and respond to the surface gesture.

[0166] These surface gestures correspond (910) to a second set of operations (e.g., operations corresponding to the surface gestures) that are caused to be performed by the artificial reality system. As an example, Figure 1C Changing the size of the virtual object 118 in response to the user performing a surface gesture 122 is shown.

[0167] In some embodiments, the artificial reality system causes a user interface presented by the head wearable device to respond to a set of position-independent operations corresponding to the position-independent gesture (912). For example, FIG. 4A to FIG. 4D The user 401 is shown performing position-independent gestures at different locations relative to the physical surface, and these gestures cause the same operation (which is saving the current state of the two-dimensional enabling representation 430 at the virtual object 420) to be performed in the artificial reality environment.

[0168] In some embodiments, the user interface presented by the head wearable device by the artificial reality system includes (914) two or more virtual objects (e.g., Figure 3B In some embodiments, one of the virtual objects is presented at a location corresponding to the location of the wrist wearable device (e.g., directly above a display of the wrist wearable device).

[0169] In some embodiments, at least a portion of the virtual object is displayed (916) at a location on the physical surface (e.g., Figures 1A to 1F In some embodiments, the virtual object is displayed on a curved portion of the physical surface (e.g., by taking into account the handrail curvature 814, Figure 8B The virtual button 816-1 in is shown as appearing on the curved surface portion 806 of the sofa).

[0170] In some embodiments, at least a portion of the second virtual object is displayed (918) in the air (e.g., FIG. 2A to FIG. 2D For example, Figure 3A An example is shown with a virtual object 320 displayed in midair above a physical surface 310 and a second virtual object 328 displayed at the physical surface 310. Figure 5E In some embodiments, multiple different virtual objects (e.g., interactive display portion 526 and user interface element 528) are displayed in midair. In some embodiments, multiple different virtual objects are displayed on a physical surface. In some embodiments, a virtual object is displayed at a first position on a first surface, and a second virtual object is displayed at a second position on a second surface. In some embodiments, FIG. 3A to FIG. 3I As shown, the user 301 can perform an indirect operation at the second virtual object 328 to cause another virtual object (eg, Figure 3A An operation is performed at a virtual object 320 in the example.

[0171] In some embodiments, the artificial reality system detects (920) an air gesture using a first set of sensors. For example, Figure 2B The air gesture 242 in the wrist wearable device can be detected by a set of sensors 252, which can include an EMG sensor 254. In some embodiments, an IMU sensor (e.g., IMU sensor 260) is used to detect the air gesture instead of or in addition to the EMG sensor 254, for example, based on the current operating mode of the wrist wearable device. The operating mode can be determined by the current power level at the wrist wearable device 202 and / or the power saving option configured by the user 201.

[0172] In some embodiments, detecting an air gesture includes detecting (922) an air gesture performed within a threshold air distance from the physical surface. Figures 1A to 8E The various figures in FIG. 1 show the distance to the air threshold (e.g., Figure 2B A visual aid corresponding to the aerial threshold distance 208).

[0173] In some embodiments, the artificial reality system identifies (924) a first operation corresponding to an in-air gesture. In some embodiments, the in-air gesture includes (926) a virtual contact with an in-air virtual object. For example, in-air gesture 242 makes a virtual contact with a simulated location of affordance 228-3 displayed at virtual object 220.

[0174] In some embodiments, the artificial reality system causes the head wearable device or another electronic device to update (928) the presentation of the user interface based on performing the first operation. In some embodiments, performing the first operation causes a non-visual update to the user interface. For example, Fig. 6A , gesture 640 causes an operation (e.g., a first operation) to be performed to adjust the audio volume of a speaker at electronic device 670. In some embodiments, the performance of the first operation is detected by another electronic device, which causes the volume of a speaker at a head wearable device or another wearable device worn by the user to be updated.

[0175] In some embodiments, the artificial reality system detects (930) a surface gesture using a second set of sensors. Figure 1B and Figure 1C The gestures performed in (eg, surface gesture 109 and surface gesture 122 ) are performed within surface threshold distance 106 of physical surface 110 and are detected by sensor 150 .

[0176] In some embodiments, the second set of sensors includes (932) a first time-of-flight sensor for measuring a first angle relative to the wrist wearable device. For example, one or more rays (e.g., ray 113) of rays 113, 115, or 117 can be emitted by a first time-of-flight sensor in the plurality of time-of-flight sensors 158.

[0177] In some embodiments, the second set of sensors includes (934) a second time-of-flight sensor for measuring a second angle relative to the wrist wearable device. For example, Figure 1A The ray 115 shown in FIG. 1 may be emitted by a second time-of-flight sensor in plurality of time-of-flight sensors 158 .

[0178] In some embodiments, the second set of sensors includes (936) a third time-of-flight sensor (e.g., Figure 1A Ray 117 shown in FIG. 1 may be emitted from a third time-of-flight sensor in plurality of time-of-flight sensors 158 ).

[0179] In some embodiments, data from three time-of-flight sensors is used (938) to detect (identify or recognize) a surface plane. For example, rays 113, 115, and 117 can be triangulated to infer the surface plane created by multiple incident locations on the surface to which the rays are directed. In some embodiments, determining the surface plane allows the sensor to be calibrated based on the relative position of the sensor at the wrist wearable device. Determining such a surface plane can allow the wrist wearable device 102 to more accurately determine the relative position of a gesture performed in the artificial reality system 100.

[0180] In some embodiments, the second set of sensors includes (940) a time-of-flight sensor that is not included in the first set of sensors. For example, set of sensors 152 may include one or more time-of-flight sensors that are not included in set of sensors 156.

[0181] In some embodiments, the surface gesture contacts (942) the user's body (or clothing worn by the user). For example, Fig. 8E The user's legs in the example may be surfaces for detecting surface gestures. In some embodiments, the user's hands, arms, shoulders, or torso are considered surfaces for surface gestures (e.g., Figure 6B The user's palm is considered the surface for surface gesture 642).

[0182] In some embodiments, a surface gesture is performed (944) at the palm of the user. For example, Figure 5H The surface gesture 543 in FIG. 5 is performed at the palm of the user, and the surface gesture 543 is determined as a surface gesture by a set of sensors at the wrist wearable device 502 .

[0183] Now go to Fig. 9C According to some embodiments, the system identifies (946) a second operation corresponding to the surface gesture. Figure 6B and Figure 6C As shown, a user can swipe in a first direction to change the volume of an electronic device, and can swipe in a second direction different from the first direction to skip songs being played by the same electronic device.

[0184] In some embodiments, the surface gesture includes (948) a change in wrist angle, and the second operation is identified based on the change in wrist angle. For example, a user may select a virtual screen element (e.g., Figure 2A The user may then perform a virtual touch gesture at a simulated location of a virtual object 220 in FIG. 1 , and then the user may rotate their wrist relative to the first position to a second orientation such that there is an angle difference between the user's first wrist angle and the user's second wrist angle. As another example, Figure 3H The position-independent gesture 345 (e.g., a virtual pull gesture) shown in FIG. 4 involves a rotation of the user's wrist toward the user's body. In some embodiments, a change in wrist angle is detected while the user interface object targeted by the surface gesture is in focus (e.g., locked to a reference). For example, a user can perform a first gesture that selects a user interface object as a target for an additional gesture, and then perform a subsequent gesture while the user interface object is in focus. For example, in FIG. Figure 3G and Figure 3H, the user performs a first gesture that causes selection of the virtual object 332, and performs a position-independent gesture 345 for interacting with the virtual object 332 while the virtual object 332 is in focus.

[0185] In some embodiments, the surface gesture includes (950) multiple physical contacts, and the second operation is identified based on identifying two or more of the multiple physical contacts. For example, Figure 3E Surface gesture 348 in 340 includes two contacts (e.g., two taps) with physical surface 310. As another example, a surface pinch gesture involves two moving contacts on the surface (e.g., Figure 3D 346 , a position-independent gesture performed at the physical surface 310 .

[0186] In some embodiments, the two or more physical contacts include a pinch gesture, and the performance of the pinch gesture is determined (952) based on data from the EMG sensor. Figure 3D Predicted EMG sensor data for EMG sensor 354 during position-independent gesture 346 is shown in accordance with some embodiments.

[0187] In some embodiments, the surface gesture has an intensity (954), and the second operation is identified based on the intensity satisfying one or more criteria. For example, Figure 3E Surface gesture 348 in can require that each individual tap made by user 301 on physical surface 310 satisfies a tap intensity threshold.

[0188] In some embodiments, a second operation is identified (956) based on detecting a surface gesture at a location corresponding to a slider affordance in the user interface.

[0189] In some embodiments, a second operation is also identified (958) based on the motion having a speed range within a predefined controlled slide threshold. For example, a gesture may involve a transition from a first position within a surface threshold distance to a second position within an airborne virtual threshold distance of the physical surface, and the speed and acceleration at which the user moves from the first position to the second position may indicate that the particular gesture is a controlled slide gesture or a swipe gesture, which may each correspond to a separate respective operation in a set of operations.

[0190] In some embodiments, the head wearable device updates (960) the user interface according to the execution of the second operation. Figure 1B When a surface gesture 109 is performed, the display of the virtual object 118 is visually updated to indicate the execution of the surface gesture 109.

[0191] In some embodiments, the surface gesture includes (962) a downward movement of the user's hand to an affordance displayed at the physical surface, and updating the user interface includes activating the affordance. Figure 1C , user 101 performs surface gesture 122 downward toward an affordance (e.g., virtual object 118), and performing surface gesture 122 activates the affordance (e.g., to perform an operation corresponding to pressing virtual object 118).

[0192] FIG. 10A to FIG. 10C 1 is a flow chart illustrating an example method 1000 for gesture detection and content rendering determination using a time-of-flight sensor in an artificial reality environment. The method 1000 is performed at an artificial reality system (e.g., artificial reality system 100, 200, 300, 400, 500, 600, 700, or 800) having one or more processors and memory. In some embodiments, the artificial reality system is Fig. 12A In some embodiments, the artificial reality system is Fig. 12B In some embodiments, the artificial reality system includes one or more wearable devices, such as Figures 1A to 1J In some embodiments, the artificial reality system includes one or more handheld devices, such as a controller or a smart phone. In some embodiments, the memory stores one or more programs, which are configured to be executed by the one or more processors. FIG. 10A to FIG. 10C At least some of the operations illustrated in the above may correspond to instructions stored in a computer memory or computer-readable storage medium (e.g., the memory 1278 of the computer system 1272 or the memory 1256 of the auxiliary device 1252). In some embodiments, the artificial reality system is a wearable device, such as Figures 1A to 1J A wrist wearable device 102 or a head wearable device 104 in the embodiment.

[0193] The artificial reality system receives (1002) data about a physical surface from one or more time-of-flight sensors coupled to a wrist wearable device. For example, a set of sensors 856 at the wrist wearable device 802 can be used to detect that the surface 806 has a curved portion. Figure 8B , the wrist wearable device 802 can detect the armrest curvature 814.

[0194] In some embodiments, the artificial reality system includes (1004) a wrist wearable device (e.g., wrist wearable device 802) communicatively coupled to a head wearable device (e.g., head wearable device 804) that displays a virtual object in an artificial reality environment presented by the head wearable device. For example, the head wearable device 804 displays a virtual object that includes Figure 8B A virtual object of the virtual button 816-1 in FIG.

[0195] In some embodiments, the artificial reality system determines (1006) that the physical surface has a curved portion. For example, Figure 8B The display of the virtual object may be based on determining that the curved portion 806 has the armrest curvature 814 .

[0196] In some embodiments, the artificial reality system determines ( 1008 ) that the physical surface has a curved portion based on the shape data obtained by the wrist wearable device.

[0197] In some embodiments, the curved surface portion has (1010) a first angle that is substantially vertical. For example, a user may perform a gesture that causes the wrist wearable device to detect a vertical wall as a relevant physical surface. In such embodiments, the head wearable device may display a virtual object including a set of control options that is different from the virtual object that would be displayed if the relevant physical surface had an angle below a threshold display option angle.

[0198] In some embodiments, the artificial reality system prompts (1012) the user to adjust the position and / or angle of the wrist wearable device to collect data from one or more time-of-flight sensors about the curved surface portion in order to display a portion of the virtual object at the curved surface portion. For example, if the sensor at the wrist wearable device does not recognize a complete portion of the physical surface with respect to which the user is attempting to perform a gesture, the head wearable device can display a prompt virtual object that indicates to the user that they need to adjust the position of the wrist wearable device in order to perform a gesture (e.g., a surface gesture or an air gesture) with respect to the physical surface.

[0199] In some embodiments, the head wearable device displays (1014) at least a portion of the virtual object on the curved surface portion, including adjusting display properties of the virtual object according to the curved surface portion. FIG. 8B to FIG. 8E Various embodiments are shown for displaying a portion of a virtual object having a virtual button enable representation (e.g., 816-1) along a curved portion 806 based on an armrest curvature 814 of the curved portion 806.

[0200] In some embodiments, the head wearable device displays (1016) information about one or more real-world properties of the surface. For example, a user can perform a touch contact at a physical surface that causes the head wearable device to display information about a material property of the physical surface, such as a current temperature or conductivity of the physical surface.

[0201] In some embodiments, at least one of the plurality of real-world properties of the surface is (1018) based on data from one or more additional sensors at the wrist wearable device that are different from the one or more time-of-flight sensors. For example, in some embodiments, a thermal sensor at the wrist wearable device may be used to detect the temperature of the surface. In some embodiments, capacitive sensors and / or other conductivity detection sensors may be used to detect one or more electrical properties of the surface.

[0202] In some embodiments, the real-world properties of the surface include (1020) a plurality of visual aspects of the surface. For example, these visual aspects include color, texture, and the like.

[0203] Now go to Fig. 10B According to some embodiments, the virtual object corresponds (1022) to one of a plurality of potentially displayed virtual objects, for example, a plurality of virtual objects corresponding to the same function but having different visual appearances.

[0204] In some embodiments, the determination (1024) of which of the multiple potentially displayed virtual objects the virtual object corresponds to is dependent on at least one surface criterion of the curved surface portion. Continuing with the previous example, a virtual object of the multiple virtual objects is selected based on a property of the surface (e.g., angle, size, and orientation). In some embodiments, the surface criterion includes (1026) a surface area of ​​the curved surface portion. In some embodiments, the surface criterion includes (1028) a surface angle of the curved surface portion.

[0205] In some embodiments, the artificial reality system determines (1030) whether the wrist position is within a threshold distance of the surface of the curved portion. Figure 1B and Figure 1C , the system determines that surface gestures 109 and 122 are performed within a surface threshold distance 106 .

[0206] In some embodiments, the head wearable device hides (1034) display of at least a portion of the virtual object at the curved surface portion in response to determining (1032) that the wrist position of the wrist wearable device is farther than a visual display threshold from the surface contact area of ​​the curved surface portion. In some embodiments, the air threshold distance is a visual display threshold distance. In some embodiments, the surface threshold distance is a visual display threshold distance. In some embodiments, the visual display threshold distance corresponds to a reachable distance of the user.

[0207] In some embodiments, the artificial reality system reduces (1036) power to a sensor of the wrist wearable device in response to determining (1032) that the wrist position of the wrist wearable device is further than a visual display threshold distance from the surface contact area of ​​the curved portion. In some embodiments, the system reduces power to (e.g., disables) one or more projectors and circuits based on the determination (1032).

[0208] In some embodiments, the sensor that is subject to power reduction is (1038) one of: an EMG sensor; a time-of-flight sensor; and / or an IMU sensor. For example, when user 201 executes Figure 2B In some embodiments, when the wrist wearable device 202 performs an air gesture 242 in the air, the wrist wearable device 202 can determine that the user 201 has performed one or more gestures outside the surface threshold distance 206, or the wrist wearable device 202 has been outside the surface threshold distance 206 for a predefined duration threshold, so the artificial reality system 200 reduces the power of one or more sensors in a set of sensors 256, the set of sensors 256 including one or more time-of-flight sensors in the plurality of time-of-flight sensors 258. In some embodiments, the artificial reality system 200 is configured to reduce the power of one or more IMU sensors 260 based on determining that the device has a threshold power level using the EMG sensor 254.

[0209] Now go to Fig. 10C In some embodiments, the head wearable device displays (1040) a second virtual object separated from the curved surface portion, the second virtual object comprising a two-dimensional virtual screen element. For example, Figure 3A The virtual object 320 in may optionally be associated with FIG. 8B to FIG. 8E The virtual objects displayed at the curved surface portion 806 in the figure are displayed in combination.

[0210] In some embodiments, the head wearable device displays (1044) a visual indicator of the gesture location of the first gesture on a two-dimensional virtual screen element of a second virtual object in response to detecting (1042) the first gesture at the virtual object at the curved surface portion. For example, when user 301 performs surface gesture 344 along physical surface 310, affordance 330 on interactive display portion 326 of virtual object 320 may be displayed based on the second virtual object 320. Figure 3C The position of the wrist wearable device 302 changes as the position changes.

[0211] In some embodiments, the artificial reality system causes the head wearable device to update (1048) a visual indicator of the location of the gesture on the two-dimensional virtual screen element in response to (1046) detecting the second gesture at the virtual object at the curved surface portion. In some embodiments, the first gesture and the second gesture are located at different locations of the virtual object at the curved surface portion (1050).

[0212] In some embodiments, the combination of the first gesture and the second gesture causes the artificial reality system to initiate switching between virtual applications at the two-dimensional virtual screen element 1052. For example, the combination of gestures may be a first position-independent pinch gesture for displaying a three-dimensional rotating view of open virtual applications in the artificial reality system, and the second gesture may be a virtual swipe gesture for scrolling through the open virtual applications.

[0213] Fig.11A and Fig. 11B 1 is a flow chart illustrating an example method for controlling at least one physical electronic device using interaction with an artificial reality environment. The method 1100 is performed at an artificial reality system (e.g., artificial reality systems 100, 200, 300, 400, 500, 600, 700, 800) having one or more processors and memory. In some embodiments, the artificial reality system is Fig. 12A In some embodiments, the artificial reality system is Fig. 12B In some embodiments, the artificial reality system includes one or more wearable devices, such as, Figures 1A to 1J In some embodiments, the artificial reality system includes one or more handheld devices, such as a controller or a smart phone. In some embodiments, the memory stores one or more programs, which are configured to be executed by the one or more processors. In some embodiments, the memory stores one or more programs, which are configured to be executed by the one or more processors. Fig.11A and Fig. 11BAt least some of the operations shown in the figure correspond to instructions stored in a computer memory or computer-readable storage medium (e.g., the memory 1278 of the computer system 1272 or the memory 1256 of the auxiliary device 1252). In some embodiments, the artificial reality system is a wearable device, such as Figures 1A to 1J The wrist wearable device 102 or the head wearable device 104 in the embodiment of the present invention can be used to perform the operation of the method 1100 when the head wearable device presents an artificial reality environment including a virtual object and a representation of at least one physical object to the user.

[0214] Go to Fig.11A The artificial reality system detects (1102) based on data from a first set of sensors of the wrist wearable device that a user of the wrist wearable device is near an electronic device (e.g., a physical electronic device) that is responsive to one or more user gestures. In some embodiments, the user is determined to be near the device based on determining that the user is within a first activation threshold distance of the electronic device. In some embodiments, the user is further determined to be near a virtual object (e.g., Figure 5B 520) to determine that the user is near the electronic device. Figure 5A and Figure 5B As shown, detecting that the user 501 is near the electronic device is also based on whether the user's gaze direction is within a predefined gaze threshold range.

[0215] In some embodiments, the artificial reality system provides (1104) to the user an indication that one or more user gestures are available at the electronic device (e.g., Figure 5B In some embodiments, the indication is received on the head wearable device or another connected device (e.g., Figures 1A to 1J In some embodiments, the indication is provided at a display of the wrist wearable device 102 in the physical electronic device. In some embodiments, the indication is provided based on determining that the user is within a first activation threshold distance of the physical electronic device. In some embodiments, the indication indicates that one or more gestures detectable by a second set of sensors of the wrist wearable device can be used to cause an operation to be performed at the physical electronic device. In some embodiments, the indication is provided at a virtual object.

[0216] In some embodiments, the indication of the user input to the user is (1106) a haptic event at one or more of a wrist wearable device, a head wearable device, or another connected electronic device. Figure 5A In the embodiment of the present invention, when the user 501 changes their orientation so that the gaze direction 566 is directed toward the electronic device 572, the head wearable device 504 provides a tactile event (e.g., a slight vibration of a lens or other component of the head wearable device 504).

[0217] In some embodiments, when the user is within the activation threshold distance of the electronic device, the wrist wearable device detects (1108) a user gesture corresponding to an operation at the electronic device (e.g., user 501 is in the activation threshold distance of the electronic device) using a second set of sensors of the wrist wearable device. Figure 5B The gesture 542 performed in is detected by sensors of the wrist wearable device 502, which may include a time-of-flight sensor and / or an EMG sensor).

[0218] In some embodiments, the artificial reality system responds to (1110) detecting a user gesture so that an operation is performed at the electronic device. For example, when the user 501 performs Figure 5B When performing gesture 542 in , virtual object 520 can be displayed at a primary display position 560 relative to electronic device 570. In some embodiments, the primary display position 560 has a fixed position and / or orientation relative to electronic device 570.

[0219] In some embodiments, the user gesture includes (1112) a movement from a first position to a second position. For example, gesture 542 can have a first position associated with the beginning of gesture 542 and a second position associated with the end of gesture 542.

[0220] In some embodiments, between the first position and the second position of the user gesture, a tactile event is generated (1114) at the wrist wearable device indicating a change to a controlled option at the electronic device. For example, when the user performs a surface gesture 543 for adjusting the volume of the electronic device 570, the adjustment may involve more than one discrete volume level. When the user performs a first portion of the surface gesture 543 so that the volume is adjusted to a first volume level, a tactile event (e.g., volume) indicating to the user that the controlled option (e.g., volume) at the electronic device 570 has been adjusted to the first volume level may be generated at the wrist wearable device and / or the head wearable device. Figure 5A tactile events in ).

[0221] In some embodiments, the electronic device is (1116) a speaker, and the operation performed at the speaker adjusts the volume level of the speaker. For example, 7A to 7F Each of the electronic devices 770 and 776 includes a speaker, and various gestures performed by the user 701 can enable operations to adjust the volume level of one or both of the corresponding speakers of the electronic devices 770 and 776.

[0222] Now go to Fig. 11B In some embodiments, the electronic device is (1118) an adjustable light source, and the user gesture causes the adjustment of the adjustable light source (e.g., Figure 7FFor example, when the user performs Fig.7D After the position-independent gesture 746 in , the brightness of the electronic device 772 is adjusted, and Fig. 7E The adjusted brightness level is displayed at the virtual object 722 displayed by the head wearable device 704.

[0223] In some embodiments, the artificial reality system causes the head wearable device to display (1120) a virtual object near the adjustable light source, and adjusts the brightness of the adjustable light source in response to a user gesture to update the display of the virtual object near the adjustable light source. For example, Fig. 7E The virtual object 722 in the head wearable device 704 includes the current levels of brightness, color, and saturation, and the display of the virtual object 722 can be adjusted at the head wearable device 704 based on the user performing one or more gestures pointing at the electronic device 772).

[0224] In some embodiments, the artificial reality system causes the head wearable device to display (1122) a virtual object including a user interface element for controlling the electronic device. For example, Figure 5D The virtual object 520 in the example includes an interactive display portion 526 for interacting with the virtual object 520 to select a song to be played at the electronic device 570.

[0225] In some embodiments, the artificial reality system adjusts (1124) the presentation of the display of the virtual object in response to detecting a user gesture at the virtual object. Figure 5C As shown, virtual object 520 appears to move so as to appear closer to user 501 and thereby be accessible to user 501 for viewing and / or virtual touch interaction.

[0226] In some embodiments, adjusting the presentation of the display includes presenting (1130) an animation of the virtual object moving toward the electronic device and then ceasing to display the virtual object. Fig. 5F and Figure 5G In 5G, the virtual object 520 first moves closer to the electronic device 570, and then in 5G, the virtual object 520 is hidden from the view at the head wearable device 504.

[0227] In some embodiments, the artificial reality system detects (1132) based on data from the head wearable device that the user is moving toward a first gaze direction relative to the electronic device (e.g., Figure 5A 568 relative to the electronic device 570).

[0228] In some embodiments, the artificial reality system determines (1134) whether the first gaze direction is within a predetermined threshold angle relative to the electronic device in response to detecting the user gesture. Figure 5B , the head wearable device 504 determines that the gaze direction 568 of the user 501 is within a predetermined threshold range of the electronic device 570, so the artificial reality system causes the head wearable device 504 to display the virtual object 520. In other words, the data received from the head wearable device can be used to determine the user's orientation so that the user has a first gaze direction, where the first gaze direction is within a threshold angle range of a first direction between the user's position and the position of the physical electronic device. In addition, the artificial reality system can detect a user gesture when the user's first gaze direction is within a threshold angle range of the first direction, so that an operation is performed at the physical electronic device.

[0229] In some embodiments, the artificial reality system detects (1136) based on data from the head wearable device that the user's orientation has changed such that they now have a second gaze direction relative to the electronic device. Figure 5B , gaze direction 568 is a second gaze direction relative to gaze direction 566.

[0230] In some embodiments, the artificial reality system then refrains (1138) from indicating to the user that the user gesture is available at the second electronic device (e.g., forgoes providing the indication). Figure 5A and Figure 5B , user 501 changes their gaze direction from gaze direction 566 to gaze direction 568. When the user faces gaze direction 566, head wearable device 504 determines that gaze direction 566 is not within a predetermined threshold angle of electronic device 570, and therefore does not display virtual object 520. However, the same sequence can be performed in reverse as recorded in operation 1138.

[0231] In some embodiments, method 1100 includes: based on data from a second set of sensors of the wrist wearable device, detecting a second user gesture different from the above user gesture, the second user gesture corresponding to a second operation on the physical electronic device different from the above operation. And the method includes: in response to detecting the second user gesture, causing the second operation to be performed at the physical electronic device.

[0232] In some embodiments, the operations of methods 900 and / or 1000 and / or aspects of these operations are used in combination with the operations of method 1100. For example, in some embodiments, method 1100 includes: detecting, based on data from a second set of sensors of the wrist wearable device, a third user gesture corresponding to a third operation at the physical electronic device that is different from the above operation, the third user gesture having the same hand motion as the above user gesture, wherein (i) the above user gesture is a surface gesture within a surface threshold distance of the physical surface, and (ii) the third user gesture is an air gesture outside the surface threshold distance of the physical surface.

[0233] Having thus described example sequences and methods of operation employing the same, attention is now turned to a system-level description of the hardware and software upon which (and with which) these methods may be implemented.

[0234] Example System-Level Block Diagram

[0235] Fig. 12A and Fig. 12B is a block diagram illustrating example components for an artificial reality system in accordance with some embodiments. Fig. 12A 1 is a block diagram illustrating an artificial reality system 1200 according to some embodiments. Although some example features are shown, various other features are not shown for the sake of simplicity and in order not to obscure the relevant aspects of the example embodiments disclosed herein. To this end, as a non-limiting example, the system 1200 includes one or more wearable devices 1202, which can be used in conjunction with one or more computing devices 1224. In some embodiments, the system 1200 provides the functionality of a virtual reality device, an AR device, a mixed-reality device, a hybrid-reality device, or a combination thereof. In some embodiments, the system 1200 provides the functionality of a user interface and / or one or more user applications (e.g., games, word processors, messaging applications, calendars, clocks, etc.). In some embodiments, the system 1200 is an instance of any one of the artificial reality systems 100, 200, 300, 400, 500, 600, 700, or 800 described above.

[0236] In some embodiments, the system 1200 provides the functionality of controlling one or more computing devices 1224 or providing commands to the one or more computing devices 1224 based on the wearable device 1202 determining the user's motion or expected motion. The relative positions of these motions or expected motions (e.g., gestures) can be detected using a time-of-flight sensor 1218 and / or an EMG sensor 1219 in addition to other sensors at the wearable device 1202. When a neuromuscular signal detected through a neuromuscular pathway before the user performs a motion or before the user completes a motion can be determined as a motion, the motion is an expected motion. Motions can be detected based on detected neuromuscular signals, but motions can also be detected (using a fusion of various sensor inputs), or alternatively, other types of sensors (e.g., a camera focused on observing hand motions and / or using data from an inertial measurement unit that can detect characteristic vibration sequences or other data types corresponding to air gestures) are used to detect motions. The one or more computing devices 1224 include one or more wearable devices (e.g., front-facing displays), smartphones, tablets, smart watches, laptops, computer systems, augmented reality systems, robots, vehicles, avatars, user interfaces, wearable devices 1202, and / or other electronic devices and / or control interfaces.

[0237] Wearable device 1202 includes a wearable structure worn by a user. In some embodiments, wearable device 1202 is an instance of any wrist wearable device among the plurality of wrist wearable devices described previously (e.g., wrist wearable device 102) or any head wearable device among the plurality of head wearable devices described previously (e.g., head wearable device 104). In some embodiments, wearable device 1202 collects information about a portion of a user's body (e.g., one or more positions and one or more orientations of a hand or two hands and one or more fingers of the user relative to a physical surface such as physical surface 110) that can be used as input to execute one or more commands of computing device 1224. In some embodiments, the collected information about a portion of a user's body (e.g., one or more positions and one or more orientations of a hand or two hands and one or more fingers of the user relative to a physical surface such as physical surface 110) can be used as input to execute one or more commands at wearable device 1224 (e.g., for selecting content to be presented on an electronic display 1226 of wearable device 1202 or controlling one or more applications 1214 stored locally on wearable device 1202). The information collected about the portion of the user's body may include neuromuscular signals that the one or more processors 1204 of the wearable device 1202 may use to determine the motor action that the user intends to perform with their hands and / or fingers. The information collected about the user's body may also include information about surface features of one or more portions of the user's body. For example, this information may be used to determine whether a portion of the user's body has a curved portion (e.g., Fig. 8E The user's legs 862).

[0238] exist Fig. 12A In the example shown, wearable device 1202 includes one or more processors 1204, memory 1206, sensors 1216 (including time-of-flight sensor 1218), electronic display 1220, and communication interface 1222. In some embodiments, memory 1206 includes one or more of interface operations 1212 (e.g., first set of air gesture operations 1208, second set of surface gesture operations 1210) and applications 1214. Wearable device 1202 may include Fig. 12A Additional components not shown in the figure, such as a power source (e.g., an integrated battery, a connection to an external power source), a tactile feedback generator, etc. In some embodiments, Fig. 12A One or more of the multiple components shown in are housed within the conductive hemispherical shape of the wearable device.

[0239] In some embodiments, sensor 1216 includes one or more hardware devices that contact the user's skin (e.g., the wrist of user 101). In some embodiments, sensor 1216 (which may be coupled additionally or alternatively to the head wearable device discussed herein) detects neuromuscular signals from neuromuscular pathways in and on the user's wrist, and sensor 1216 may also detect proximity to the physical surface of wearable device 1202 based on time-of-flight sensing data. In some embodiments, sensor 1216 is configured to detect different finger movements, wrist movements, arm movements, index finger movements, middle finger movements, ring finger movements, pinky finger movements, thumb movements, hand thumbs, etc. based on different neuromuscular signals detected from the user's skin. In some embodiments, sensor 1216 is used in pairs and / or other combinations to form corresponding channels and / or planes for detecting neuromuscular signals and / or gestures relative to a physical surface, where each channel is a pair of sensors. In some embodiments, wearable device 1202 includes at least three time-of-flight sensors 1218, which can be used in combination to detect a "surface plane" on a physical surface. In some embodiments, the sensors 1216 of the wearable device 1202 include a time-of-flight sensor 1218 and / or an EMG sensor 1219 .

[0240] In some embodiments, the one or more processors 1204 are configured to receive data detected by the sensor 1216 and determine whether a user gesture performed by a user corresponds to an air gesture or a surface gesture (e.g., FIG. 1A to FIG. 1I The input command, when provided to the computing device 1224, causes the computing device to perform an action (e.g., as described in detail above, various air gestures can cause various operations to be performed to update the display of a user interface at the head wearable device). Alternatively, in some embodiments, one or more input commands are used to cause the wearable device 1202 to locally perform one or more actions (e.g., present a display on the electronic display 1220, operate one or more applications 1214, cause an operation to be performed at another electronic device, etc.). For example, the wearable device 1202 can be a wrist wearable device (e.g., a smart watch), and the one or more input commands can be used to cause the smart watch to perform one or more actions.

[0241] The one or more applications 1214 stored in the memory 1206 may be productivity-based applications (e.g., calendars, organizers, word processors), social applications (e.g., social platforms), games, etc. In some embodiments, the one or more applications 1214 are presented to the user via the electronic display 1220. In some embodiments, the one or more applications 1214 are used to facilitate the transfer of information (e.g., to another application running on the computing device 1224). In some embodiments, the user can provide one or more input commands to the application 1214 operating on the wearable device 1202 based on the determined motion action, so that the application 1214 executes the one or more input commands.

[0242] In addition, different groups of gestures (including multiple groups of air gesture operations 1208 and multiple groups of surface gesture operations 1210) can be stored in memory 1206. This allows the wearable device 1202 to provide user-specific execution. More specifically, the wearable device 1202 can be customized to perform as efficiently as possible for each user.

[0243] The communication interface 1222 implements input and output to the computing device 1224. In some embodiments, the communication interface 1222 is a single communication channel, such as a USB. In other embodiments, the communication interface 1222 includes several different communication channels that operate together or independently. For example, the communication interface 1222 may include a separate communication channel for sending input commands to the computing device 1224 so that the computing device 1224 performs one or more actions. In some embodiments, data from the sensor 1216 is sent to the computing device 1224, and the computing device 1224 then interprets the appropriate input response based on the received data. One or more communication channels of the communication interface 1222 can be implemented as a wired connection or a wireless connection. In some embodiments, the communication interface 1222 includes hardware capable of communicating data using any of a variety of protocols: custom or standard wireless protocols (e.g., Institute of Electrical and Electronics Engineers 802.15.4 standard (IEEE 802.15.4), Wi-Fi, ZigBee, low-power wireless personal area network based on IPv6 (6LoWPAN), Thread, Z-Wave, Bluetooth Smart, Industrial Automation Association 100.11a standard (ISA100.11a), Wireless Addressable Remote Sensor Highway (WirelessHART), or MiWi), custom or standard wired protocols (e.g., Ethernet or Power Line Communications Alliance (HomePlug)), and / or any other suitable communication protocol, including communication protocols that have not been developed as of the filing date of this document.

[0244] In some embodiments, computing device 1224 presents media to the user. Examples of media presented by computing device 1224 include images, videos, audio, or some combination thereof. Additional examples of media include virtual reality applications and / or AR applications that are executed to process input data from sensor 1216 on wearable device 1202. In some embodiments, media content is based on information received from one or more applications 1232 (e.g., productivity applications, social applications, games, etc.). Computing device 1224 includes an electronic display 1226 for presenting media content to the user. In various embodiments, electronic display 1226 includes a single electronic display or multiple electronic displays (e.g., one display for each eye of the user). Computing device 1224 includes a communication interface 1228 that can input and output other devices in system 1200. Communication interface 1228 is similar to communication interface 1222 described above.

[0245] In some embodiments, the computing device 1224 receives instructions (or commands) from the wearable device 1202 (e.g., a request to process sensor data associated with the performance of surface gestures, air gestures, and / or position-independent gestures). In response to receiving the instructions, the computing device 1224 performs one or more actions associated with the instructions (e.g., executing one or more input commands in an AR or VR environment). Alternatively, in some embodiments, the computing device 1224 receives instructions from an external device communicatively coupled to the wearable device 1202, and performs one or more actions associated with the instructions in response to receiving the instructions. In some embodiments, the computing device 1224 receives instructions from the wearable device 1202, and in response to receiving the instructions, provides instructions to an external device communicatively coupled to the computing device 1224, which performs one or more actions according to the instructions. Although not shown, in embodiments including different external devices, the external device can be connected to the wearable device 1202 and / or the computing device 1224 via a wired connection or a wireless connection. The external device may be a remote gaming console, an additional display, an additional head mounted display, and / or may be any other additional electronic device capable of being coupled to the wrist wearable device 1202 and / or the computing device 1224 .

[0246] In some embodiments, computing device 1224 provides information to wearable device 1202, which in turn causes the wearable device to present information to the user. The information provided by computing device 1224 to wearable device 1202 may include media content (which may be displayed on electronic display 1220 of wearable device 1202), organizational data (e.g., calendars, phone numbers, invitations, instructions), files (e.g., word processing files, spreadsheets, or other files that can be processed locally from wearable device 1202).

[0247] In some embodiments, the computing device 1224 is implemented as an integrated system on chip, a microcontroller, a desktop or laptop computer, a server computer, a tablet computer, or a smart phone or other mobile device. Therefore, the computing device 1224 includes components common to computing devices, such as one or more processors 1234, random access memory, storage devices, network interfaces, and input and output interfaces (I / O interfaces), etc. The processor may be or include one or more microprocessors or application specific integrated circuits (application specific integrated circuits, ASICs). The memory 1230 may be or include random access memory (RAM), read-only memory (ROM), dynamic random access memory (DRAM), static random access memory (SRAM), and MRAM (magnetic random access memory), and the memory 1230 may include firmware, such as static data or fixed instructions, basic input and output systems (BIOS), system functions, configuration data, and other routines used during the operation of the computing device and the processor. The memory 1230 also provides a storage area for data and instructions associated with the data and applications processed by the processor.

[0248] Storage devices provide non-volatile, large-capacity or long-term storage of data or instructions in computing devices. Storage devices can take the form of magnetic disks or solid-state disks, tapes, compact disks (CDs), digital video disks (DVDs) or other relatively high-capacity addressable or serial storage media. Multiple storage devices can be provided to computing devices or multiple storage devices can be made available to computing devices. Some of these storage devices (e.g., network storage devices or cloud-based storage devices) can be outside computing devices. Network interfaces include interfaces to networks, and the network interfaces can be implemented as wired interfaces or wireless interfaces. I / O interfaces connect processors to peripheral devices (not shown), such as and depending on computing devices, sensors, displays, cameras, color sensors, microphones, keyboards, and USB devices.

[0249] exist Fig. 12A In the example shown, computing device 1224 also includes application 1232. In some embodiments, application 1232 is implemented as a software module stored on a storage device and executed by processor 1234. Some embodiments of computing device 1224 include additional components or are combined with Fig. 12A Similarly, the various functions described further below may be distributed among the various components of the computing device 1224 in a manner different from that described here.

[0250] Each application 1232 is a set of instructions that, when executed by the processor, generates specific content for presentation to the user. For example, application 1232 may include a virtual reality application that generates virtual reality content (e.g., a virtual reality environment) and further generates virtual reality content in response to receiving input from wearable device 1202 (based on the determined user motion action). Examples of virtual reality applications include gaming applications, conference applications, and video playback applications. Additional examples of application 1232 may include productivity-based applications (e.g., calendars, organizers, word processors, etc.), social-based applications (e.g., social media platforms, dating platforms, etc.), entertainment (e.g., performances, games, movies, etc.), and travel (e.g., ride-sharing applications, hotel applications, airline applications, etc.). In some embodiments, application 1232 includes React.js, React VR, and / or React Native functions associated with performing operations to update display properties at electronic display 1226.

[0251] In some embodiments, computing device 1224 allows application 1232 to operate in conjunction with wearable device 1202. In some embodiments, computing device 1224 receives information from sensor 1216 of wearable device 1202 and provides the information to application 1232. Application 1232 determines, based on the received information, media content and / or a type of haptic event (e.g., a touch event) to be provided to computing device 1224 (or wearable device 1202) for presentation to a user via electronic display 1226. Figure 1B For example, if computing device 1224 receives information from sensor 1216 on wearable device 1202 indicating that a user has performed an action (e.g., a virtual touch gesture, a surface touch gesture, a “thumbs-up” gesture, etc.), application 1232 generates content for presentation by computing device 1224 (or wearable device 1202) that reflects the user's instruction based on the motion action determined by wearable device 1202. Similarly, in some embodiments, application 1232 (e.g., an application saved locally to wearable device 1202) directly receives information from sensor 1216 on wearable device 1202 and, based on the information (e.g., the motion action determined by wearable device 1202), provides media content to computing device 1224 for presentation to the user.

[0252] Fig. 12B12 is a block diagram illustrating a system 1250 according to some embodiments. Although some example features are shown, various other features are not shown for the sake of simplicity and so as not to obscure relevant aspects of the example embodiments disclosed herein. To this end, as a non-limiting example, system 1250 includes auxiliary devices 1252-1 and 1252-2 used in conjunction with computer system 1272 (e.g., computing device 1224). In some embodiments, system 1250 is an instance of any of artificial reality systems 100, 200, 300, 400, 500, 600, 700, 800, or 1200.

[0253] Example accessory device 1252 (two example embodiments of which are represented by Fig. 12B 1252-1 and 1252-2 in the example) include, for example, one or more processors / cores 1254 (hereinafter referred to as "processors"), memory 1256, one or more actuators 1260, one or more communication components 1264, and / or one or more sensors 1258. In some embodiments, these components are interconnected via a communication bus 1266. Reference to these components of the accessory device 1252 encompasses embodiments in which one or more of these components (and combinations thereof) are included. In some embodiments, one or more sensors 1258 and one or more transducers 1262 are the same component. In some embodiments, the example accessory device 1252 includes one or more cameras 1270. In some embodiments (not shown), the accessory device 1252 includes a wearable structure. In some embodiments, the accessory device and the wearable structure are formed integrally. In some embodiments, the accessory device and the wearable structure are different structures, but are part of the system 1250. In some embodiments, one or more of the multiple accessory devices 1252 are Figures 1A to 1J Any wrist wearable device in the plurality of wrist wearable devices previously described in (e.g., wrist wearable device 102) or any head wearable device in the head wearable devices previously described in (e.g., head wearable device 104). In some embodiments, there is an additional accessory device, such as a handheld accessory device, that allows for better detection of user gestures and / or physical surfaces.

[0254] For example, accessory device 1252-1 can be a ring used in conjunction with a wearable structure to adjust the fit of the wearable structure using data measurements obtained by sensor 1258-1. In another example, accessory device 1252-1 and accessory device 1252-2 are different wristbands to be worn on each wrist of the user.

[0255] In some embodiments, a single processor 1254 (e.g., processor 1254-1 of accessory device 1252-1) executes software modules for controlling multiple accessory devices 1252 (e.g., accessory devices 1252-1...1252-n). In some embodiments, a single accessory device 1252 (e.g., accessory device 1252-2) includes multiple processors 1254 (e.g., processor 1254-2), such as one or more actuator processors, one or more communication component processors, one or more sensor processors, and / or one or more transducer processors. In some embodiments, the one or more actuator processors are configured to adjust the fit of the wearable structure. In some embodiments, the one or more communication processors are configured to control communications sent by the communication component 1264 and / or receive communications through the communication component 1264. In some embodiments, the one or more sensor processors are configured to control the operation of the sensor 1258 and / or receive output from the sensor 1258. In some embodiments, the one or more transducer processors are configured to control the operation of the transducer 1262.

[0256] In some embodiments, communication component 1264 of accessory device 1252 includes a communication component antenna for communicating with computer system 1272. In some embodiments, communication component 1274 includes a complementary communication component antenna for communicating with communication component 1264. In some embodiments, data included in the communication signal prompts computer system 1272 that accessory device 1252 is ready for use. In some embodiments, computer system 1272 sends instructions to accessory device 1252, and accessory device 1252, in response to receiving the instructions, instructs the transmit and receive electrodes to provide coupling information between the receive electrodes and the user.

[0257] In some embodiments, one or more actuators 1260 are used to adjust the fit of the wearable structure on the user's limb. In some embodiments, one or more actuators 1260 are also used to provide tactile feedback to the user. For example, each actuator 1260 can apply vibration stimulation, pressure stimulation, shear stimulation, or some combination thereof to the user. In some embodiments, one or more actuators 1260 are hydraulic actuators, pneumatic actuators, electric actuators, and / or mechanical actuators.

[0258] In some embodiments, one or more transducers 1262 are used to send and receive one or more signals 1268. In some embodiments, one or more sensors 1258 are used to send and receive one or more signals 1268. In some embodiments, one or more sensors 1258 and one or more transducers 1262 are part of the same component used to send and receive one or more signals 1268. Signal 1268 can be an electromagnetic wave, a mechanical wave, an electrical signal, or any wave / signal capable of being transmitted through a medium. As used herein, a medium includes the wearer's skin, flesh, bone, blood vessels, or some combination thereof.

[0259] In addition to sending signals (e.g., electrical signals), the accessory device 1252 is also configured to receive (e.g., detect, sense) signals sent by itself or by another accessory device 1252. For illustration, the accessory device 1252-1 can send multiple signals through a medium such as a user's limbs, and the second accessory device 1252-2 can receive the multiple signals sent by the accessory device 1252-1 through the medium. In addition, the accessory device 1252 that receives the sent signals can use the received signals to determine whether the accessory device is in contact with the user.

[0260] In some embodiments, the one or more transducers 1262 of the accessory device 1252-1 include one or more transducers configured to generate and / or receive signals. In some embodiments, an integrated circuit (not shown) such as a controller circuit and / or a signal generator of the accessory device 1252-1 controls the behavior of the transducers 1262. In some embodiments, the transmit electrode and / or the receive electrode are part of the one or more transducers 1262 of the accessory device 1252-1. Alternatively, the transmit electrode and / or the receive electrode can be part of the one or more sensors 1258-1 of the accessory device 1252-1, or the transmit electrode can be part of the transducer 1262 and the receive electrode can be part of the sensor 1258-1 (or vice versa).

[0261] In some embodiments, sensor 1258 includes one or more time-of-flight sensors and one or more EMG sensors. In some embodiments, sensor 1258 includes one or more of a transmitting electrode and a receiving electrode for obtaining coupling information. In some embodiments, sensor 1258 includes one or more neuromuscular sensors. In some embodiments, the neuromuscular sensor includes one or more surface electromyography (sEMG) sensors, one or more mechanomyography sensors, and / or one or more sonomyography sensors. Additional non-limiting examples of sensor 1258 (and sensor 1290) include, for example, infrared sensors, pyroelectric sensors, ultrasonic sensors, microphone sensors, laser sensors, optical sensors, Doppler sensors, gyroscope sensors, accelerometer sensors, resonant liquid crystal (LC) sensors, capacitive sensors, acoustic sensors, and / or inductive sensors. In some embodiments, sensor 1258 (and sensor 1290) is configured to collect additional data about the user (e.g., the impedance of the user's body). Examples of sensor data output by these sensors may include body temperature data, infrared rangefinder data, motion data, activity recognition data, contour detection and recognition data, gesture data, heart rate data, and other wearable device data (e.g., biometric readings and outputs, accelerometer data).

[0262] The computer system 1272 is a computing device for executing an artificial reality application (e.g., a virtual reality application, an AR application, etc.) to process input data from sensors 1290 on the head wearable device 1282 and sensors 1258 on the accessory device 1252. The computer system 1272 provides output data to at least: (i) a display 1284 on the head wearable device 1282; and (ii) one or more accessory devices 1252. In some embodiments, the head wearable device 1282 is an example of the head wearable device 104 or the AR system 1320.

[0263] Computer system 1272 includes one or more processors / cores 1276, memory 1278, one or more communication components 1274, and / or one or more cameras 1280. In some embodiments, these components are interconnected via a communication bus 1294. Reference to these components of computer system 1272 encompasses embodiments in which one or more of these components (and combinations thereof) are included.

[0264] In some embodiments, the computer system 1272 is a separate device coupled to a head-mounted display (e.g., a display of a head-wearable device) 1282. For example, the computer system 1272 has one or more processors / one or more cores 1276 for controlling one or more functions of the computer system 1272, and the head-mounted display 1282 has one or more processors / one or more cores 1286 for controlling one or more functions of the head-mounted display 1282. Alternatively, in some embodiments, the head-mounted display 1282 is a component of the computer system 1272. For example, the one or more processors 1276 control the functions of the computer system 1272 and the head-mounted display 1282. Furthermore, in some embodiments, the head-mounted display 1282 includes one or more processors 1286 that communicate with the one or more processors 1276 of the computer system 1272. In some embodiments, communication between the computer system 1272 and the head-mounted display 1282 occurs via a wired (or wireless) connection between the communication bus 1294 and the communication bus 1292. In some embodiments, the computer system 1272 and the head mounted display 1282 share a single communication bus. In some embodiments, the head mounted display 1282 is separate from the computer system 1272.

[0265] Computer system 1272 can be any suitable computer device, such as a laptop computer, a tablet device, a netbook, a personal digital assistant, a mobile phone, a smart phone, an artificial reality console or device (e.g., a virtual reality device or an AR device, etc.), a gaming device, a computer server, or any other computing device. Computer system 1272 is sometimes referred to as a host or host system. In some embodiments, computer system 1272 includes other user interface components for adding functionality to computer system 1272, such as a keyboard, a touch screen display, a mouse, a trackpad, and / or any number of supplemental I / O devices.

[0266] In some embodiments, one or more cameras 1280 of the computer system 1272 are used to facilitate the artificial reality experience. In some embodiments, the computer system 1272 provides images captured by the one or more cameras 1280 to a display 1284 of the head-mounted display 1282, and the display 1284 in turn displays the provided images. In some embodiments, a processor 1286 of the head-mounted display 1282 processes the provided images. Note that in some embodiments, one or more of the plurality of cameras 1280 are part of the head-mounted display 1282. In some embodiments, data from the camera 1280 of the head-mounted display can be combined with data from a time-of-flight sensor at the wrist wearable device 102 for more accurate gesture detection.

[0267] The head-mounted display 1282 presents media to the user. Examples of media presented by the head-mounted display 1282 include images, video, audio, or some combination thereof. In some embodiments, the audio is presented via an external device (e.g., a speaker and / or headphones) that receives audio information from the head-mounted display 1282, the computer system 1272, or both, and presents audio data based on the audio information. The displayed image may be virtual reality, augmented reality, or mixed reality. The display 1284 displays an image to the user based on the data received from the computer system 1272. In various embodiments, the display 1284 includes a single electronic display or multiple electronic displays (e.g., one display for each eye of the user).

[0268] Sensors 1290 include one or more hardware devices that detect spatial and motion information about the head-mounted display 1282. In some embodiments, sensors 1290 include one or more neuromuscular sensors. In some embodiments, neuromuscular sensors include one or more surface electromyography (sEMG) sensors, one or more myograph sensors, and / or one or more sonomyogram sensors. Spatial and motion information may include information about the position, orientation, speed, rotation, and acceleration of the head-mounted display 1282. For example, sensors 1290 may include one or more inertial measurement units that detect rotation of a user's head while the user is wearing the head-mounted display 1282. In some embodiments, sensors 1290 include one or more cameras positioned on the head-mounted display 1282. In some embodiments, the head-mounted display 1282 includes one or more sensors 1290. In some embodiments, one or more of the plurality of sensors 1290 are part of the computer system 1272.

[0269] Having thus described an example block diagram, attention is now directed to examples of wearable devices that may be used in conjunction with the techniques described herein. Example Wearable Devices (e.g., wrist wearable devices such as smart watches, AR glasses, VR goggles / head-mounted viewers, each of which may form various artificial reality systems for use with the techniques described herein)

[0270] Fig.13A An example AR system 1320 is shown in accordance with some embodiments. Fig.13AIn the AR system 1320, an eyeglass device having a frame 1324 is configured to hold a left display device 1328-1 and a right display device 1328-2 in front of the user's eyes. The display devices 1328-1 and 1328-2 can work together or independently to present an image or a series of images to the user. Although the AR system 1320 includes two displays, the various embodiments of the present disclosure can be implemented in an AR system with a single near-eye display (NED) or more than two NEDs. In some embodiments, the AR system 1320 is an instance of any head wearable device (e.g., head wearable device 104) among the multiple head wearable devices described above.

[0271] In some embodiments, AR system 1320 includes one or more sensors, such as sensors 1330 and 1332 (e.g., examples of sensor 503, Figure 5A ). For example, sensors 1330 and 1332 can generate measurement signals in response to the movement of AR system 1320, and sensors 1330 and 1332 can be located on substantially any portion of frame 1310. Each sensor can be a positioning sensor, an inertial measurement unit (IMU), a depth camera assembly, or any combination thereof. In some embodiments, AR system 1320 includes a plurality of sensors. Fig.13A In the embodiment where the sensor includes an IMU, the IMU can generate calibration data based on the measurement signal from the sensor. Examples of sensors include, but are not limited to, accelerometers, gyroscopes, magnetometers, other suitable types of sensors that detect motion, sensors for error correction of the IMU, or some combination thereof.

[0272] In some embodiments, the AR system 1320 includes a microphone array having a plurality of acoustic sensors 1326-1 to 1326-8, which are collectively referred to as acoustic sensors 1326. Acoustic sensors 1326 can be transducers that detect changes in air pressure caused by sound waves. In some embodiments, each acoustic sensor 1326 is configured to detect sound and convert the detected sound into an electronic format (e.g., an analog format or a digital format). In some embodiments, the microphone array includes ten acoustic sensors: 1326-1 and 1326-2 designed to be placed in the user's respective ears, acoustic sensors 1326-3, 1326-4, 1326-5, 1326-6, 1326-7, and 1326-8 positioned at different locations on the frame 1324, and acoustic sensors positioned on corresponding neckbands, wherein the neckband is an optional component of the system and is not present in certain embodiments of the artificial reality system discussed herein.

[0273] The configuration of the acoustic sensor 1326 of the microphone array can vary. Fig.13A 1324, but the number of acoustic sensors 1326 can be more than ten or less than ten. In some cases, using more acoustic sensors 1326 can increase the amount of audio information collected and / or the sensitivity and accuracy of the audio information. In contrast, in some cases, using a smaller number of acoustic sensors 1326 can reduce the computing power required for the controller 1336 to process the collected audio information. In addition, the position of each acoustic sensor 1326 of the microphone array can vary. For example, the position of the acoustic sensor 1326 can include a defined coordinate of a defined position on the user on the frame 1324, an orientation associated with each acoustic sensor, or some combination thereof.

[0274] Acoustic sensors 1326-1 and 1326-2 can be positioned on different parts of the user's ear, such as behind the pinna, or inside the auricle or ear cavity. In some embodiments, in addition to the acoustic sensor 1326 inside the ear canal, additional acoustic sensors are also provided on or around the ear. In some cases, positioning the acoustic sensor next to the user's ear canal enables the microphone array to collect information about how the sound reaches the ear canal. By positioning at least two of the multiple acoustic sensors 1326 on both sides of the user's head (e.g., as binaural microphones), the AR device 1320 can simulate binaural hearing and collect a 3D stereo sound field around the user's head. In some embodiments, the acoustic sensors 1326-1 and 1326-2 are connected to the AR system 1320 via a wired connection 1230, while in other embodiments, the acoustic sensors 1326-1 and 1326-2 are connected to the AR system 1320 via a wireless connection (e.g., a Bluetooth connection). In some embodiments, the AR system 1320 does not include acoustic sensors 1326 - 1 and 1326 - 2 .

[0275] The acoustic sensors 1326 on the frame 1324 may be positioned along the length of the temples, across the nose bridge, above or below the display device 1328, or some combination thereof. The acoustic sensors 1326 may be oriented so that the microphone array can detect sounds in a wide range of directions around the user wearing the AR system 1320. In some embodiments, a calibration process may be performed during manufacture of the AR system 1320 to determine the relative positioning of each acoustic sensor 1326 in the microphone array.

[0276] In some embodiments, the eyeglass device also includes an external device (e.g., a pairing device), or is communicatively coupled to an external device (e.g., a pairing device), such as the optional neckband described above. In some embodiments, the optional neckband is coupled to the eyeglass device via one or more connectors. These connectors can be wired or wireless connectors, and these connectors can include electronic and / or non-electronic (e.g., structural) components. In some embodiments, the eyeglass device and the neckband operate independently without any wired or wireless connection between them. In some embodiments, the components of the eyeglass device and the components of the neckband are located on one or more additional peripheral devices that are paired with the eyeglass device, the neckband, or some combination thereof. In addition, the neckband is intended to represent any suitable type or form of paired device. Therefore, the following discussion of the neckband can also be applied to various other paired devices, such as smart watches, smart phones, wristbands, other wearable devices, handheld controllers, tablet computers, or laptop computers.

[0277] In some cases, pairing an external device (e.g., an optional neckband) with the AR eyewear device enables the AR eyewear device to achieve the form factor of a pair of glasses while still providing sufficient battery power and computing power for the expanded capabilities. Some or all of the battery power, computing resources, and / or additional features of the AR system 1320 can be provided by a paired device, or shared between a paired device and the eyewear device, thereby reducing the weight, heat distribution, and form factor of the eyewear device overall while still retaining the desired functionality. For example, a neckband can allow components that would otherwise be included on the eyewear device to be included in the neckband, thereby shifting the weight load from the user's head to the user's shoulders. In some embodiments, the neckband has a larger surface area through which heat is diffused and dissipated to the surrounding environment. Therefore, the neckband can allow for greater battery power and computing power than would otherwise be possible on a standalone eyewear device. Because the weight carried in a neckband can be less intrusive to the user than the weight carried in an eyeglass device, the user can tolerate wearing a lighter eyeglass device and carrying or wearing a paired device for longer periods of time than the user could tolerate wearing a heavy, stand-alone eyeglass device, thereby enabling the user to more fully integrate the artificial reality environment into their daily activities.

[0278] In some embodiments, the optional neckband is communicatively coupled to the eyewear device and / or other devices (e.g., wrist wearable device 102). These other devices can provide certain functions (e.g., tracking, positioning, depth map construction, processing, storage, etc.) to the AR system 1320. In some embodiments, the neckband includes a controller and a power source. In some embodiments, the acoustic sensor of the neckband is configured to detect sound and convert the detected sound into an electronic format (analog or digital).

[0279] The controller of the neckband processes information generated by sensors on the neckband and / or the AR system 1320. For example, the controller can process information from the acoustic sensor 1326. For each detected sound, the controller 1325 can perform a direction of arrival (DOA) estimate to estimate the direction in which the detected sound arrives at the microphone array. When the microphone array detects a sound, the controller can fill the audio data set with this information. In an embodiment where the AR system 1320 includes an IMU, the controller 1336 can calculate all inertial calculations and spatial calculations from the IMU located on the eyeglass device. The connector can transmit information between the eyeglass device and the neckband and between the eyeglass device and the controller. The information can be in the form of optical data, electronic data, wireless data, or any other transmittable data form. Moving the processing of information generated by the eyeglass device to the neckband can reduce the weight and heat of the eyeglass device, making the eyeglass device more comfortable and safe for the user.

[0280] In some embodiments, the power supply in the neckband supplies power to the eyeglass device and the neckband. The power supply may include, but is not limited to, a lithium-ion battery, a lithium-polymer battery, a disposable lithium battery, an alkaline battery, or any other form of power storage device. In some embodiments, the power supply is a wired power supply.

[0281] As described above, some artificial reality systems may substantially replace one or more of a user's multiple sensory perceptions of the real world with a virtual experience, rather than blending the artificial reality with the real reality. An example of this type of system is a head-mounted display system that largely or completely covers the user's field of view (e.g., Fig. 13B VR system 1350 in.

[0282] Fig. 13B 1 shows a VR system 1350 (e.g., also referred to herein as VR goggles or a VR head-mounted viewer) according to some embodiments. VR system 1350 includes a head-mounted display 1302. The head-mounted display 1302 includes a front body 1352 and a frame 1354 (e.g., a strip or band) shaped to fit the user's head. In some embodiments, as Figure 8BAs shown, the head mounted display 1302 includes output audio transducers 1356-1 and 1356-2. In some embodiments, the front body 1352 and / or the frame 1354 include one or more electronic components, which include one or more electronic displays, one or more IMUs, one or more tracking transmitters or detectors, and / or any other suitable devices or sensors for creating an artificial reality experience. In some embodiments, the VR system 1350 is an instance of any head wearable device (e.g., head wearable device 104) in the plurality of head wearable devices described above.

[0283] The artificial reality system may include various types of visual feedback mechanisms. For example, the display device in the AR system 1320 and / or the display device in the VR system 1350 may include one or more liquid crystal displays (LCD), one or more light emitting diode (LED) displays, one or more organic LED (OLED) displays, and / or any other suitable type of display screen. The artificial reality system may include a single display screen for both eyes, or a display screen may be provided for each eye, which may provide additional flexibility for zoom adjustment or for correcting the user's refractive error. Some artificial reality systems also include an optical subsystem having one or more lenses (e.g., a conventional concave or convex lens, a Fresnel lens, or an adjustable liquid lens), through which the user can view the display screen.

[0284] In addition to or instead of using a display screen, some artificial reality systems include one or more projection systems. For example, a display device in the AR system 1320 and / or VR system 1350 may include a micro-LED projector that projects light into the display device (e.g., using a waveguide), such as a clear combined lens that allows ambient light to pass through. The display device can refract the projected light into the user's pupil, and can enable the user to view both the artificial reality content and the real world at the same time. The artificial reality system can also be configured to have any other suitable type or form of image projection system.

[0285] The artificial reality system may also include various types of computer vision components and subsystems. For example, artificial reality systems 1320 and 1350 may include one or more optical sensors, such as two-dimensional (2D) or three-dimensional (3D) cameras, time-of-flight depth sensors, single-beam or scanning laser rangefinders, three-dimensional laser radar (3D LiDAR) sensors, and / or any other suitable type or form of optical sensor. The artificial reality system may process data from one or more of these sensors to identify the user's location, map the real world, provide the user with context about the real-world surroundings, and / or to perform various other functions.

[0286] In some embodiments, artificial reality systems 1320 and 1350 include a haptic (tactile) feedback system that can be incorporated into headwear, gloves, clothing, handheld controllers, environmental devices (e.g., chairs or floor mats), and / or any other type of device or system (e.g., wearable devices discussed herein). The haptic feedback system can provide various types of skin feedback, including vibration, thrust, pull, shear, texture, and / or temperature. The haptic feedback system can also provide various types of kinesthetic feedback, such as motion and compliance. Tactile feedback can be implemented using motors, piezoelectric actuators, fluid systems, and / or various other types of feedback mechanisms. The haptic feedback system can be implemented independently of other artificial reality devices, within other artificial reality devices, and / or in combination with other artificial reality devices.

[0287] Fig.14A and Fig. 14B An example wrist wearable device 1400 according to some embodiments is shown. In some embodiments, the wrist wearable device 1400 is any of the plurality of wrist wearable devices described above (e.g., the wrist wearable device 1400 described above). Figures 1A to 1J An example of a wrist wearable device 102 is described. Fig.14AA perspective view of a wrist wearable device 1400 is shown, which includes a watch body 1402 separated from a watch band 1404. The watch body 1402 and the watch band 1404 can have a generally rectangular or circular shape and can be configured to allow a user to wear the wrist wearable device 1400 on a body part (e.g., a wrist). In some embodiments, the wrist wearable device 1400 includes a retaining mechanism 1406 (e.g., a buckle, a hook and loop fastener, etc.) for securing the watch band 1404 to the user's wrist. In some embodiments, the wrist wearable device 1400 includes a coupling mechanism 1406 (e.g., a cradle) for removably coupling the pod or watch body 1402 (via the watch band coupling mechanism 1406 of the watch body 1402) to the watch band 1404.

[0288] The wrist wearable device 1400 is configured to perform the same Figures 1A to 11B The various functions associated with the selection of the contact representation are described below. Fig. 12A and Fig. 12B Described in more detail, the functions performed by the wrist wearable device 1400 may include, but are not limited to: displaying visual content to the user (e.g., visual content displayed on the display 1408), sensing user input (e.g., sensing a touch on the button 1410, sensing biometric data on the sensor 1412, sensing neuromuscular signals on the neuromuscular sensor 1414, etc.); sending and receiving messages (e.g., text, voice, video, etc.); image acquisition; wireless communication (e.g., cellular network, near field network, Wi-Fi network, personal area network, etc.); location determination; financial transactions; providing tactile feedback; prompts; notifications; biometric authentication; health monitoring; sleep monitoring, etc. These functions may be performed independently in the watch body 1402, independently in the watch band 1404, and / or performed in communication between the watch body 1402 and the watch band 1404. In some embodiments, functions may be performed on the wrist wearable device 1400 in conjunction with an artificial reality environment, including but not limited to a virtual reality (VR) environment (including a non-immersive VR environment, a semi-immersive VR environment, and a fully immersive VR environment), an AR environment (including a marker-based AR environment, a markerless AR environment, a location-based AR environment, and a projection-based AR environment), a hybrid reality, and other types of mixed-reality environments. As will be understood by those skilled in the art upon reading the description provided herein, the novel wearable device described herein may be used with any of these types of artificial reality environments.

[0289] The strap 1404 can be configured to be worn by a user so that the inner surface of the strap 1404 is in contact with the user's skin. When worn by the user, the sensor 1412 is in contact with the user's skin. In some embodiments, the sensor 1412 is a biosensor that senses the user's heart rate, saturated oxygen level, body temperature, sweat level, muscle intention, or a combination thereof. The strap 1404 may include a plurality of sensors 1412, which may be distributed on the inner surface and / or outer surface of the strap 1404. Additionally or alternatively, the watch body 1402 may include sensors that are the same as or different from the strap 1404 (or in some embodiments, the strap 1404 may not include any sensors at all). For example, a plurality of sensors may be distributed on the inner surface and / or outer surface of the watch body 1402. As described below with reference to Fig.13A and Fig. 13B As described, the watch body 1402 may include, but is not limited to, a front image sensor 1418A and / or a rear image sensor 1418B, a biometric sensor, an IMU, a heart rate sensor, an oxygen saturation sensor, one or more neuromuscular sensors (e.g., Fig. 12A The sensor 1412 may include an EMG sensor 1219 in the watch body 1402, an altimeter sensor, a temperature sensor, a bioimpedance sensor, a pedometer sensor, an optical sensor, a touch sensor, a sweat sensor, and the like. The sensor 1412 may also include a sensor that provides data about the user's environment, including the user's movement (e.g., IMU), height, position, orientation, gait, or a combination thereof. The sensor 1412 may also include a light sensor (e.g., an infrared light sensor, a visible light sensor) configured to track the position and / or movement of the watch body 1402 and / or the strap 1404. The strap 1404 may use a wired communication method (e.g., a universal asynchronous receiver-transmitter (UART), a USB transceiver, etc.) and / or a wireless communication method (e.g., near field communication, Bluetooth, etc.) to send the data obtained by the sensor 1412 to the watch body 1402. The strap 1404 may be configured to operate (e.g., to collect data using the sensor 1412) independently of whether the watch body 1402 is coupled to the strap 1404 or is separated from the strap 1404.

[0290] The strap 1404 and / or the watch body 1402 may include a haptic device 1416 (e.g., a vibrotactile actuator) configured to provide tactile feedback (e.g., skin sensation and / or kinesthetic sensation, etc.) to the user's skin. The sensor 1412 and / or the haptic device 1416 may be configured to operate in conjunction with a number of applications, including but not limited to health monitoring, social media, playing games, and artificial reality (e.g., applications associated with artificial reality).

[0291] In some examples, strap 1404 may include neuromuscular sensors 1414 (e.g., electromyography (EMG) sensors, mechanomyogram (MMG) sensors, sonomyography (SMG) sensors, etc.). Neuromuscular sensors 1414 may sense the user's intention to perform certain motor actions. The sensed muscle intention may be used to control certain user interfaces displayed on display 1408 of wrist wearable device 1400, and / or may be sent to a device responsible for rendering an artificial reality environment (e.g., a head-mounted display) to perform actions in an associated artificial reality environment, such as to control the movement of a virtual device displayed to the user.

[0292] Signals from the neuromuscular sensors 1414 may be used to provide the user with enhanced interaction with physical objects and / or virtual objects in an artificial reality application generated by the artificial reality system (e.g., user interface objects presented on the display 1408 or another computing device (e.g., a smartphone)). Signals from the neuromuscular sensors 1414 may be obtained (e.g., sensed and recorded) by one or more neuromuscular sensors 1414 of the strap 1404. Although Fig.14A An example with one neuromuscular sensor 1414 is shown, but the strap 1404 may include multiple neuromuscular sensors 1414 arranged circumferentially on the inner surface of the strap 1404 so that the multiple neuromuscular sensors 1414 contact the user's skin. The strap 1404 may include multiple neuromuscular sensors 1414 arranged circumferentially on the inner surface of the strap 1404. The neuromuscular sensors 1414 may sense and record neuromuscular signals from the user when the user performs muscle activation (e.g., movement, gesture, etc.). The muscle activation performed by the user may include: static gestures (e.g., placing the user's palm down on a table); dynamic gestures (e.g., grabbing a physical or virtual object); and covert gestures that are not noticeable to others (e.g., slightly tensing a joint by co-contracting opposing muscles or using sub-muscle activation). The muscle activation performed by the user may include symbolic gestures (e.g., gestures mapped to other gestures, interactions, or commands, for example, based on a gesture vocabulary that specifies the mapping of gestures to commands).

[0293] In some embodiments, the wrist wearable device 1400 includes a coupling mechanism (also referred to as a cradle) for removably coupling the watch body 1402 to the strap 1404. The user can remove the watch body 1402 from the strap 1404 to reduce the burden of the wrist wearable device 1400 on the user. In some embodiments, the wrist wearable device 1400 includes a coupling surface 1420 on the watch body 1402, and / or a coupling mechanism 1406 (e.g., a cradle, a tracker band, a support base, a buckle). The user can perform any type of action for coupling the watch body 1402 to the strap 1404 and separating the watch body 1402 from the strap 1404. For example, the user can perform the following on the watch body 1402 relative to the strap 1404 to attach the watch body 1402 to the strap 1404 and detach the watch body 1402 from the strap 1404: twist; slide; turn; push; pull; turn; or a combination thereof.

[0294] like Fig.14A As shown in the example of , the coupling mechanism 1406 of the watch band may include a frame or housing that allows the coupling surface 1420 of the watch body 1402 to be retained within the coupling mechanism 1406 of the watch band. The watch body 1402 can be removably coupled to the watch band 1404 by: friction fit; magnetic coupling; rotation-based connector; shear pin coupler; retaining spring; one or more magnets; clamp; pin; hook and loop fastener; or a combination thereof. In some examples, the watch body 1402 can be separated from the watch band 1404 by actuation of the release mechanism 1440. The release mechanism 1440 can include, but is not limited to, a button, a knob, a plunger, a handle, a lever, a fastener, a buckle, a dial, a latch, or a combination thereof.

[0295] The wrist wearable device 1400 can include a single release mechanism 1440 or multiple release mechanisms 1440 (e.g., two release mechanisms 1440, such as spring-loaded buttons, positioned on opposite sides of the wrist wearable device 1400). Fig.14A As shown, the release mechanism 1440 can be positioned on the watch body 1402 and / or the coupling mechanism 1406 of the watch band. Fig.14AThe release mechanism 1440 is shown positioned at a corner of the watch body 1402 and a corner of the strap coupling mechanism 1406, but the release mechanism 1440 can be positioned at any location on the watch body 1402 and / or the strap coupling mechanism 1406 that is convenient for actuation by a user of the wrist wearable device 1400. The user of the wrist wearable device 1400 can actuate the release mechanism 1440 by pushing, turning, lifting, pressing, moving, or performing other actions on the release mechanism 1440. Actuation of the release mechanism 1440 can release (e.g., separate) the watch body 1402 from the strap coupling mechanism 1406 and the strap 1404, thereby allowing the user to use the watch body 1402 independently of the strap 1404. For example, separating the watch body 1402 from the strap 1404 can allow the user to use the rear image sensor 1418B to capture an image.

[0296] Fig. 14B 1400 , showing sensors along the wrist wearable device 1400 . The sensors may include various sensors along the inner surface of the strap 1404 and / or on the inner surface of the watch body 1402 . Different sensors may be disposed at different locations along the strap 1404 so that the sensors may detect signals at various locations on the user's wrist. In some embodiments, there are sensors along the front edge 1405 of the wrist wearable device 1400 that are used to detect, for example, the proximity of the wrist wearable device 1400 to a physical surface (e.g., Figures 1A to 1J In some embodiments, multiple types of sensors are provided along the front edge 1405 of the wrist wearable device 1400. For example, the sensors (e.g., the time-of-flight sensors 1464 and 1474) located on the first portion of the front edge 1405 of the wrist wearable device 1400 can be sensors configured to emit light (e.g., Figure 1AThe sensors on the second portion of the wrist wearable device 1400 (e.g., sensors 1466, 1468, 1470, and 1472) may be proximity sensors of a different type than the time-of-flight sensors 1464 and 1474. In some embodiments, the time-of-flight sensors on the first portion (e.g., time-of-flight sensors 1464 and 1474) are configured for slower but more intensive sampling than the proximity sensors on the second portion (e.g., sensors 1466, 1468, 1470, and 1472). In some embodiments, the time-of-flight sensors 1464 and 1474 on the first portion of the wrist wearable device 1400 are configured to primarily detect the distance of the wrist wearable device 1400 to the physical surface. In some embodiments, the proximity sensors 1466, 1468, 1470, and 1472 are configured to primarily detect hand motions and / or gestures performed by a user of the wrist wearable device 1400.

[0297] In some embodiments, the watch body 1402 of the wrist wearable device 1400 includes components of an EMG sensor (e.g., EMG ground ("GND") 1478, EMG reference ("REF") 1480, and EMG sensors 1428 and 1432). In some embodiments, the watch body of the wrist wearable device also includes sensors 1412 and 1430, which are configured to transmit electrical signals to the body of the user of the wrist wearable device 1400 and use the electrical signals to detect when a part of the user's body contacts a physical surface. In some embodiments, the watch body 1402 of the wrist wearable device 1400 also includes a haptic controller 1490 for providing haptic events to the user of the wrist wearable device 1400. In some embodiments, the haptic controller 1490 on the body 1402 of the wrist wearable device 1400 is configured to receive signals from one or more other electronic devices other than the wrist wearable device 1400 (e.g., via a transducer 1262 configured to transmit one or more signals 1268 between, for example, the accessory device 1252 and the wrist wearable device 1400).

[0298] In some embodiments, the strap 1404 of the wrist wearable device 1400 includes an inner surface 1407, which is configured to contact the user when the user wears the wrist wearable device according to some embodiments. In some embodiments, the inner surface 1407 of the wrist wearable device 1400 includes one or more sensors arranged at different, separate locations (e.g., sensor location 1436, sensor and / or sensor location 1454) along the inner surface 1407. In some embodiments, each sensor location in the plurality of sensor locations includes an EMG sensor arranged at the corresponding sensor location. In some embodiments, at least one sensor location in the plurality of sensor locations may include two or more sensors, the two or more sensors including two different types of EMG sensors, or an EMG sensor and an impedance sensor (e.g., an EMG sensor 1437 and an impedance sensor 1438 located at the sensor location 1436 of the wrist wearable device 1400). In some embodiments, a single sensor disposed at a corresponding sensor location may have more than one sensing capability (eg, a single sensor disposed at sensor location 1450 may have the sensing capabilities of both EMG sensor 1437 and impedance sensor 1438).

[0299] Figure 15A to Figure 15V An example user scenario with an artificial reality system (e.g., including at least augmented reality glasses and a wrist wearable device) in accordance with some embodiments is shown.

[0300] Fig.15A A user 1501 and a computing system 1500 are shown, which includes a wrist wearable device 1502 and a head wearable device 1504 (e.g., such as Fig.13A The AR system 1320 shown, etc., is an AR glasses). Figure 15A to Figure 15V The user 1501 in FIG. 1 is performing a gesture corresponding to causing an operation to be performed within the computing system 1500 . Figure 15A to Figure 15V In the specific example shown, user 1501 has a user interface object (e.g., FIG. 15C to FIG. 15J ) as shown in the circular user interface object 1510 .

[0301] In some embodiments, the user interface object presented in the field of view 1505 of the user 1501 may be a three-dimensional virtual object. In some embodiments, the virtual object is part of a virtual user interface displayed at a physical surface (e.g., the physical surface 1508 of the table in front of the user 1501).

[0302] Now go to Fig.15A, user 1501 is sitting at a desk with physical surface 1508. In some embodiments, physical surface 1508 can be used in conjunction with any of the above-described plurality of mid-air gestures and / or surface gestures. In some embodiments, such mid-air gestures and / or surface gestures can be used in conjunction with Figure 15A to Figure 15V The gestures described in 1501 are performed in combination with or as an alternative to the gestures described in 1502. User 1501 wears a wrist wearable device 1502 and a head wearable device 1504. The head wearable device 1504 can present user interface objects within the field of view 1505 of the user 1501. Fig.15A The head wearable device 104 is shown displaying a notification user interface element 1511. The notification user interface element is displaying text information to the user 1501: "New user interface (UI) feature available: flip your wrist to open the application menu". In some embodiments, the notification user interface element 1511 may include a selectable user interface element that the user 1501 can interact with, and the selectable user interface element is configured to allow the user to adjust settings related to the new user interface feature. The hand 1503 of the user 1501 on the same arm as the wrist wearable device 1502 is located on the side of the user 1501.

[0303] In some embodiments, there are acceptable starting positions for the operations described herein to be performed. In some embodiments, an acceptable starting position includes the user 1501's arm extended downward at the side of the user 1501, such as Fig.15A As shown. In some embodiments, the wrist wearable device 1502 and / or the head wearable device 1504 can provide an indication to the user 1501 that the hand 1503 is in an acceptable starting position. In some embodiments, the indication can be a non-visual indication (e.g., an auditory indication, a tactile event, etc.). In some embodiments, after the hand 1503 is in an acceptable starting position, an indication can be provided to the user 1501 that the hand 1503 is or will not be in an acceptable starting position. In some embodiments, the acceptable starting position of the hand 1503 is dynamically determined based on sensor data from an imaging sensor of the wrist wearable device 1502. For example, if the hand 1503 of the user 1501 is in a position such that the imaging sensor 1506 of the wrist wearable device 1502 cannot detect the movement of the hand 1503 of the user 1501, the user 1501 can be provided with a notification that the air gesture will not be detected until the user adjusts the position of the hand 1503 so that the hand 1503 is in an acceptable starting position.

[0304] Fig. 15B A bottom view of a hand 1503 of a user 1501 is shown on the right (e.g., from the perspective of the top fingertips facing directly toward the user), and a field of view 1505 of the user 1501 is shown on the left. Fig. 15B , no user interface elements are displayed in the field of view 1505 of the user 1501. In some embodiments, a user interface may be displayed that indicates that the hand 1503 of the user 1501 is in an acceptable starting position for performing a gesture that causes an operation to be performed at the user interface.

[0305] Now go to Fig. 15C , the bottom view of hand 1503 shows that hand 1503 has performed hand rotation 1570, where hand rotation 1570 corresponds to a rotation angle 1550 around an axis of the user's arm direction as seen from the bottom view of hand 1503. Based on user 1501 having performed hand rotation 1570, circular user interface object 1510 is presented by head wearable device 1504 in field of view 1505 of user 1501. In some embodiments, the speed at which circular user interface object 1510 animates into field of view 1505 (e.g., animation transition length) is based on the rotation speed at which user 1501 performed hand rotation 1570. In some embodiments, hand rotation 1570 must have a minimum speed (e.g., two meters per second) to meet gesture criteria. In some embodiments, one or more IMU sensors located at wrist wearable device 1502 can be used to detect an air gesture including hand rotation 170 and / or the speed of hand rotation 1570.

[0306] Now go to Fig.15D , the bottom view of hand 1503 shows hand 1503 performing hand rotation 1572, where hand rotation 1572 corresponds to rotation angle 1550, but in the opposite direction of hand rotation 1570 (e.g., in some embodiments, hand rotation 1570 is clockwise and hand rotation 1572 is counterclockwise). Circular user interface object 1510 remains presented by head wearable device 1504 in field of view 1505 of user 1501. In some embodiments, the circular user interface object will not be displayed in the field of view 1505 of user 1501. Fig. 15B Rather than being displayed at the point in the sequence shown, it will be presented only when the user's 1501 hand 1503 reaches the initial rotation of the hand (e.g., after a flip gesture and another flip gesture in the opposite direction).

[0307] Fig.15E A front view of the hand 1503 of the user 1501 is shown (e.g., from a perspective directly facing the top of the hand 1503 of the user 1501). Fig.15EAs shown, circular user interface object 1510 includes selectable options user interface elements 1514a-1514e, which user 1501 can interact with by performing air gestures. Focus selector 1512 is displayed within circular user interface object 1510, and focus indicator 1516 is displayed on selectable options user interface element 1514-a, which corresponds to focus selector 1512 located within selectable options user interface element 1514a.

[0308] Now go to Fig.15F , the front view of the user 1501's hand 1503 shows that the user 1501 has performed an air gesture including a hand rotation 1572, which is in the air gesture Fig.15F , there is a rotation angle 1552 in the "z" direction of the Cartesian plane shown in . In some embodiments, imaging sensor 1506 detects that hand 1503 has performed an air gesture by determining that the hand has rotated rotation angle 1552. In some embodiments, this determination is made by tracking the movement of a specific location on hand 1503 (e.g., a point on a finger, a knuckle, a ring on a user's finger, or another wearable electronic device).

[0309] Figure 15G A side view of a hand 1503 of a user 1501 is shown (e.g., from a perspective directly facing the thumb and index finger side of the hand 1503). A circular user interface object 1510 presented within a field of view 1505 of the user 1501 shows a focus selector 1512 located within a selectable options user interface element 1514-b (e.g., a focus indicator 1516 is displayed at the location of the selectable options user interface element 1514-b).

[0310] Now go to Fig.15H , the side view of the hand 1503 of the user 1501 shows that the user 1501 has performed an air gesture that includes a hand rotation 1576 in the "x" direction with a rotation angle 1554. In some embodiments, the imaging sensor 1506 detects that the hand 1503 has rotated the rotation angle 1554 by determining how much of the hand 1503 is in the field of view of the imaging sensor 1506. For example, a larger amount of the hand in the field of view of the imaging sensor 1506 may mean that the user's hand is rotating in the positive "x" direction of the Cartesian plane, such as Fig.15H shown.

[0311] Fig.15IA side view of a hand 1503 of a user 1501 is shown. A circular user interface object 1510 presented within a field of view 1505 of the user 1501 shows a focus selector 1512 located within a selectable options user interface element 1514-a (e.g., a focus indicator 1516 is displayed at the location of the selectable options user interface element 1514-a).

[0312] Now go to Fig.15J , a side view of hand 1503 of user 1501 shows that user 1501 is performing another air gesture (e.g., a pinch gesture) that includes contact 1578 between thumb 1507 of hand 1503 and index finger 1509 of hand 1503. As will be discussed in more detail below, user 1501 can perform air gestures corresponding to different operations performed by computing system 1500 by performing pinch gestures using different combinations of the user's fingers. Fig.15J , when the focus selector 1512 is within the selectable options user interface element 1514-a, a selection indicator is displayed within the selectable options user interface element 1514-a, indicating that the user 1501 has selected the selectable options user interface element 1514-1 (e.g., to open the corresponding camera application) at the user interface presented in the field of view 1505 of the user 1501. In some embodiments, similar to Fig.15J Different aspects of a pinch gesture performed in the user interface may cause different operations to be performed within the user interface. For example, a discontinuous pinch gesture may cause an operation to open a camera application, while a held pinch gesture may cause a different operation (e.g., causing additional options related to the camera application to be presented at the user interface).

[0313] Figure 15K A side view of a hand 1503 of user 1501 is shown. Circular user interface object 1510 is no longer presented within field of view 1505 of user 1501, and instead, a camera view indicator 1520 is presented within field of view 1505 of user 1501, which view indicator 1520 may indicate that a portion of the field of view will be captured as an image if user 1501 performs a gesture that causes an image to be captured by the head wearable device. A notification user interface element is also presented within field of view 1505 of user 1501, the notification user interface element stating: "Pinch to capture", which statement is intended to instruct user 1501 how to cause an image to be captured.

[0314] Now go to Figure 15L , a side view of the hand 1503 of the user 1501 shows that the user 1501 is performing another mid-air gesture (e.g., a pinch gesture) including contact 1578 between the thumb 1507 and the index finger 1509. Figure 15LIn some embodiments, the user can perform a pinch gesture corresponding to Figure 15L 1504. The position of the camera view indicator can be adjusted by a pinch gesture of contacts 1578 in the middle of the screen. For example, the user can perform another mid-air gesture that can cause the focus selector 1512 to move to a different position, thereby changing the position of the camera view indicator 1520 centered on the focus selector 1512. In some embodiments, the head wearable device can detect the gaze direction of the user 1501 based on, for example, an eye tracking module located in the head wearable device 1504. In some embodiments, the gaze direction can be determined by detecting the angle of the user's head (e.g., the camera view indicator can be at a predetermined position of the field of view 1505 of the user 1501, thereby causing the field of view 1505 to change so that the position of the camera view indicator 1520 is adjusted accordingly.

[0315] Figure 15M A front view of the hand 1503 of the user 1501 is shown (e.g., from a perspective directly facing the top of the hand 1503 of the user 1501). Figure 15M As shown, the gallery user interface element 1526 is presented within the field of view 1505 of the user 1501. The gallery user interface element 1526 includes gallery thumbnail user interface elements 1528-a to 1528-e, wherein one or more gallery thumbnail user interface elements may include images that have been captured or saved by the user 1501 (e.g., images that have been captured by the head wearable device 1504, such as Figure 15L A focus indicator 1529 is displayed on the gallery thumbnail user interface element 1528-e, indicating that the gallery thumbnail user interface element 1528-e is currently in focus within the gallery user interface element 1526. Figure 15M As shown, in order for the user 1501 to perform an air gesture corresponding to a navigation operation at the user interface presented by the head wearable device 1504, there is no need for an explicit focus selector object (e.g., FIG. 15E to FIG. 15L ).

[0316] Now go to Fig.15N , the front view of hand 1503 shows that user 1501 is performing another air gesture including hand rotation 1582 with rotation angle 1554. Focus indicator 1529 is repositioned (e.g., moved) from gallery thumbnail user interface element 1528-e to gallery thumbnail user interface element 1528-b based on the air gesture, indicating, for example, that if user 1501 performs a gesture that causes a selection operation, gallery thumbnail user interface element 1528-b will be selected for display in, for example, an image editing application.

[0317] Fig.15O A side view of a hand 1503 of a user 1501 is shown. Fig.15O , user 1501 is performing a pinch gesture including contact 1584 between thumb 1507 and index finger 1509. Image editing user interface element 1530 is displayed in field of view 1505, and focus selector 1512 is presented within image editing user interface element 1530. Based on user 1501 performing a pinch gesture including contact 1584, selection indicator 1536 is presented on selectable editing option user interface element 1534-b (e.g., a drawing tool). In some embodiments, once user 1501 activates the drawing tool, computing system 1500 can enable drawing based on tracking the position of the pupil of user 1501 by the eye tracking module.

[0318] Figure 15P A side view of a hand 1503 of a user 1501 is shown. Figure 15P , user 1501 keeps Fig.15O 1501 while performing a hybrid gesture including hand motion 1586. As described herein, the term "hybrid gesture" refers to a gesture that includes aspects corresponding to other discrete gestures (e.g., a pinch gesture combined with hand motion, a double pinch gesture, etc.). Focus selector 1512 is represented by pencil icon 1512-b, indicating that selectable edit option user interface element 1534-b is selected. Based on user 1501 holding down from Fig.15O When the contact 1584 of the pinch gesture performs hand movement 1586, the pencil icon 1512-b moves within the image 1537 displayed within the image editing user interface element 1530. Based on the movement of the pencil icon 1512-b, the image 1537 is edited by the pencil icon 1512-b.

[0319] Figure 15Q A front view of a hand 1503 of a user 1501 is shown. A scrollable user interface element 1538 is displayed within the field of view 1505 of the user 1501 (e.g., a scrollable user interface element of a social media application). The scrollable user interface element 1538 includes a displayed posting user interface element 1542-1 that includes an image and text content (e.g., similar to a post displayed on an Instagram feed). The scrollable user interface element 1538 also includes a scroll position indicator user interface element 1540 that indicates the position of the scrollable user interface element 1538 currently displayed in the field of view 1505.

[0320] Now go to Figure 15R, user 1501 is performing an air gesture including hand rotation 1588 with rotation angle 1556. Based on user 1501 performing an air gesture including hand rotation 1588, different positions of the scrollable user interface element are presented within scrollable user interface element 1538. Figure 15R As shown, the repositioning of the displayed portion of scrollable user interface element 1538 causes scrolling movement 1544 of scroll position indicator user interface element 1540 to indicate that a different position of scrollable user interface element 1538 is currently displayed at scrollable user interface element 1538 .

[0321] 15S shows a side view of the hand 1503 of the user 1501. Figure 15S , user 1501 is performing a double pinch gesture including two discontinuous contacts 1590 between thumb 1507 and index finger 1509 of user 1501. Performing the double pinch gesture causes an operation to be performed at displayed posting user interface element 1542-2 displayed within scrollable user interface element 1538. Specifically, performing the double tap gesture causes a response (e.g., like) to be applied to displayed posting user interface element 1542-2. In some embodiments, the performance of the operation caused by the performance of the double pinch gesture depends on whether user 1501 performs the gesture within a threshold distance of a physical surface near the user (e.g., within a surface threshold distance, within an air threshold distance, etc.).

[0322] Figure 15T A side view of a hand 1503 of a user 1501 is shown. Figure 15T In FIG. 1 , user 1501 is making a fist with thumb 1507 resting on index finger 1509. Scrollable user interface element 1538 is displayed in field of view 1505. Figure 15S 1542-2 is displayed, and a focus indicator 1548 is displayed at the location of the first response in response selector user interface element 1546, indicating that a particular response (the "clapping" emoji response) is in focus at response selector user interface element 1546.

[0323] Now go to Figure 15U, the side view illustrates user 1501 performing an air gesture that includes a thumb motion 1592 along index finger 1509 of user 1501 away from the palm of user 1501. Based on thumb motion 1592, focus indicator 1548 may be repositioned to be located at a different response within response selector user interface element 1546. In some embodiments, the distance of repositioning of focus indicator 1548 and / or the speed of the animation of displaying the repositioning of focus indicator 1548 may be based on a quantity associated with thumb motion 1592. For example, if thumb motion 1592 meets a speed criterion, focus indicator 1548 may be repositioned to a response that is subsequent to one or more responses between the positions of focus indicator 1548 before the air gesture was performed. In some embodiments, focus indicator 1548 may be repositioned during performance of the air gesture so that user 1501 receives intermediate feedback that the gesture is causing an operation to be performed. For example, after user 1501 performs a first portion of thumb motion 1592, the focus indicator may be repositioned by responding to one response within selector user interface element 1546. Then, after user 1501 performs a second portion of thumb motion 1592, the focus indicator may be repositioned by responding to another response within selector user interface element 1546.

[0324] exist Figure 15V , the side view shows user 1501 performing an air gesture including thumb motion 1594 (a downward pressing motion toward index finger 1509) within computing system 1500. Based on user 1501 performing thumb motion 1594, response 1549 (e.g., a fire emoji) is applied to displayed posting user interface element 1542-2. In some embodiments, thumb motion 1594 must satisfy an intensity threshold in order to cause the intended operation to be performed at the user interface presented in field of view 1505 presented by head wearable device 1504.

[0325] Having thus described a system block diagram and then an example wearable device, attention is now turned to certain example embodiments.

[0326] Fig.16A and Fig. 16B 1 is a flow chart illustrating a method 1600 of making air gestures and surface gestures available to a user according to some embodiments. The method 1600 is performed in a system having one or more processors and memory (e.g., Fig. 12A In some embodiments, the artificial reality system includes one or more wearable devices, such as a wrist wearable device 1502 and a head wearable device 1504 (which may include Fig. 12AIn some embodiments, the computing system includes one or more handheld devices, such as a controller or a smartphone. In some embodiments, the memory stores one or more programs configured to be executed by the one or more processors. 9A to 9C At least some of the operations shown in the figure correspond to operations stored in a computer memory or a computer-readable storage medium (e.g., Figure 2B In some embodiments, the computing system is a wearable device, such as a memory 1278 of the computing system 1272 and / or a memory 1256 of the accessory device 1252. Figure 1A To the wrist wearable device 102 or the head wearable device 1504 in Figure 1V.

[0327] To perform the operations of method 1600 at a computing system, a user (e.g., Figure 15A to Figure 15V The user 101 in FIG. 1 is wearing ( 1602 ) a device including an imaging sensor (e.g., FIG. 15B to FIG. 15D When the user wears the wrist wearable device, the imaging sensor faces (1604) the user's hand. In some embodiments, there can be more than one imaging sensor facing the user's hand. For example, FIG. 15B to FIG. 15D The imaging sensor 1506 shown in FIG. 15 may include a photodiode array including, for example, eight individual imaging sensors located at different and separate locations on the wrist wearable device 1502 .

[0328] In some embodiments, there is more than one imaging sensor at the wrist wearable device. In some embodiments, the user wears a head wearable device (e.g., Figure 15A to Figure 15VIn some embodiments, a head wearable device 1504 is provided, and the head wearable device includes at least one imaging sensor. In some embodiments, a combination of sensors (including imaging sensors) having optimal parameters, such as accuracy and power usage, for detecting gestures is determined using sensor fusion techniques. For example, if an imaging sensor at a wrist wearable device is receiving a relatively strong signal of a hand movement corresponding to an air gesture, the computing system 1500 can determine that the power of the imaging sensor of the head wearable device can be reduced based on data from the imaging sensor of the wrist wearable device. In some embodiments, the imaging sensor at the head wearable device can be used exclusively, and the power at the imaging sensor of the wrist wearable device can be reduced while the imaging sensor of the head wearable device is able to accurately detect the air gesture performed by the user. In some embodiments, determining which sensors to reduce power is based on a set of gestures currently available to the user. For example, although there is no operation corresponding to a pinch gesture at the computing system 1500, the power of the vibration sensor at the wrist wearable device can be reduced.

[0329] In some embodiments, before presenting a user interface object to the user at the head wearable device (1606), a first indication is received from the wrist wearable device to perform a first air gesture, the first air gesture comprising a rotation of the user's wrist having at least a threshold angular range (e.g., a flip angle), wherein the rotation is detected by an imaging sensor facing the user's hand. In some embodiments, the first air gesture comprises a first flip motion in a first direction (e.g., a clockwise direction) having a predefined flip angle. In some embodiments, the first air gesture comprises a second flip motion in a second direction (e.g., a counterclockwise direction) having a predefined flip angle. In some embodiments, the first air gesture comprises both the first flip motion and the second flip motion, each having a predefined flip angle. In other words, FIG. 15B to FIG. 15D Treated as a continuous sequence, it can be based on user execution Fig. 15C 1570 to display the circular user interface object 1510, and / or may be based on the user performing Fig.15D 1572 to display the circular user interface object 1510. In some embodiments, a full wrist flip outward and inward is required to display the user interface, while in some embodiments, only an outward flip is required. In either case, a haptic event can be received at one of the wrist wearable device or the head wearable device worn by the user based on the user performing a gesture to display the circular user interface object 1510.

[0330] In some embodiments, the computing system causes the user interface object to be displayed within a user interface presented at the head wearable device in response to (1608) the first indication. For example, FIG. 15B to FIG. 15D It is shown that the hand 1503 of the user 1501 is rotated through an angular range, and as a result of the gesture involving the rotation of the hand 1503, a circular user interface object 1510 is presented to the user 1501 at the head wearable device 1504.

[0331] The computing system presents (1610) a user interface to the user via the head wearable device. For example, the circular user interface object 1510 is Figure 15A to Figure 15V The sequence shown is a portion of an artificial reality user interface displayed to the user 1501. In some embodiments, the user interface presented to the user includes one or more virtual objects as follows: The one or more virtual objects are configured to appear in the user's field of view (e.g., Figure 15A to Figure 15V Appears as a three-dimensional object within the field of view 1505).

[0332] In some embodiments, prior to receiving a second indication to perform the air gesture (1612), detecting using an imaging sensor that the user's hand is in a first position corresponding to an acceptable starting position. For example, an acceptable starting position may be achieved when the user's 1501 hand 1503 is extended downwardly at a side of the user 1501. In some embodiments, there may be multiple acceptable starting positions, and the user 1501 may be prompted with the position of the hand 1503 for each of the multiple acceptable starting positions.

[0333] In some embodiments, based on detecting (1614) that the user's hand is in an acceptable starting position, a visual notification (or, e.g., audio, tactile, etc.) is provided to the user that the user's hand is in a first position corresponding to the acceptable starting position. In some embodiments, additional user input and / or conditions are detected before providing the user with an indication that the air gesture can be used to control the user interface object. For example, according to some embodiments, the user can activate the air gesture space by performing a pinch gesture at the starting position.

[0334] The computing system receives (1616) an instruction from the wrist wearable device to perform an air gesture, the air gesture comprising movement of the user's hand in a direction relative to a starting position of the user's hand. In some embodiments, the starting position of the user's 1501 hand may be determined by Fig.15E 15. Representation in which the hand 1503 of the user 1501 is stretched straight down. Fig.15F An example of the movement of the user's 1501 hand 1503 relative to the starting position of the user's 1501 hand 1503 is shown.

[0335] The motion is detected by an imaging sensor (e.g., a camera, a charge-coupled device (CCD), an active pixel sensor (complementary metal oxide semiconductor (CMOS)), etc.) included on the wrist wearable device (1618). In some embodiments, different imaging sensors may be used in various situations, or a combination of two or more imaging sensors may be used. In some embodiments, sensor fusion is used to determine the best combination of imaging sensors to use in a given situation. For example, Figure 15A to Figure 15V The wrist wearable device 1502 depicted in FIG. 1 can include more than one imaging sensor (eg, imaging sensor 1506 ) such that a determination of what mid-air gesture the user 1501 is performing is based on data from the more than one imaging sensor.

[0336] In some embodiments, the directions correspond to different angular orientations of the user's hand relative to an initial wrist angle of the user's wrist. For example, a first wrist orientation may correspond to the user's hand being extended generally straight from the user's wrist, while a second wrist orientation may correspond to the user's hand being tilted forward from the user's wrist (e.g., the user's hand being tilted forward away from the user's body, such as Fig.15F In some embodiments, the user's relative wrist orientation is relative to the current angle of the user's arm. For example, when the user's arms are stretched out in front of them, the relative wrist orientation of their wrists is different than when the user's arms are stretched straight down (e.g., at the user's side).

[0337] In some embodiments, EMG sensors are not used (1620) to detect air gestures. In some embodiments, only one or more imaging sensors are used to detect the performance of at least some of the plurality of air gestures. In some embodiments, based on a set of available gestures and / or available operations currently available to the user at the wrist wearable device and / or the head wearable device, power of one or more sensors in a first set of sensors, which may include EMG sensors, is reduced. For example, Figure 15A to Figure 15V Any of the multiple air gestures shown may be detected without an EMG sensor. In some embodiments, an air gesture including a pinch gesture having contact between two fingers of the user (e.g., Fig.15O In some embodiments, one or more IMU sensors may be used to detect mid-air gestures that include movement of a user's hand.

[0338] In response to the indication, the computing system controls (1622) a user interface object on a display of the head wearable device based on the determined movement of the user's hand. Fig.15F , focus selector 1512 is shown as moving from selectable option user interface element 1514-a to selectable menu option user interface element 1514-d based on the movement of hand 1503 of user 1501. In some embodiments, the determination of the movement can be performed by a head wearable device (e.g., Figure 15A to Figure 15V In some embodiments, the determination may be performed by a wrist wearable device or another intermediate device. In some embodiments, the determination of movement may be performed by a combination of electronic devices.

[0339] Control of the user interface object is based (1624) on the direction of the motion relative to the starting position of the user's hand. Fig.15F and Fig.15H , the focus selector 1512 moves in different directions based on the user 1501 moving their hand 1503 in different directions relative to the starting position of the hand 1503.

[0340] In some embodiments, control of the user interface object is based (1626) on a quantity associated with the motion, where the quantity associated with the motion is one of: a distance of the motion; a speed of the motion; or a time period over which the motion occurs. In some embodiments, the control can be based on more than one quantity. For example, the control can be based on a distance that the user's hand moves during a first motion and a time that the user holds a second position after the first motion before the user's hand returns to the starting position. For example, Figure 15R The distance of the rolling motion 1544 caused by the mid-air gesture including the hand rotation 1588 can be based on the angle 1556 and the amount of time the user keeps the hand 1503 in the rotated position after the hand rotation 1588.

[0341] In some embodiments, data from one or more sensors included on the wrist wearable device is calibrated such that corresponding weights are applied to corresponding sensors based on user criteria. For example, one user may have a larger motion profile associated with a particular gesture than another user, such that different weights should be applied to data from an imaging sensor at the wrist wearable device for each user (e.g., a lower weight is applied to sensor data that detects motion from a user with a larger motion profile). In some embodiments, the corresponding weights applied to the corresponding sensors are configurable by the user (e.g., if the user wants interactions to correspond to more subtle gestures, the user can increase the associated weight applied to the imaging sensor data).

[0342] In some embodiments, controlling the user interface object includes (1628) moving between a plurality of available options for the user interface object. In some embodiments, controlling the user interface object causes a focus selector to change position (e.g., visually move, animate) from a first selected aspect of the user interface to a second selected aspect of the user interface, such as between a plurality of displayed menus. For example, FIG. 15E to FIG. 15H The focus selector in moves in response to an air gesture performed by user 1501. Another example is Figure 15M and Fig.15N The focus indicator 1529 shown in FIG. 1 is based on the user performing the following steps: Fig.15N 1528 - 1529 , and an air gesture of hand rotation 1582 is performed to move from gallery thumbnail user interface element 1528 - e to gallery thumbnail user interface element 1528 - b .

[0343] In some embodiments, the air gesture is (1630) a waving gesture that causes the focus selector to move circumferentially around the circular menu user interface object (e.g., in a clockwise or counterclockwise direction). In other words, the movement of the user interface object does not always have a one-to-one correspondence with the direction of movement associated with the air gesture. For example, if the first direction is right, the indicator may move in a clockwise direction, and if the first direction is left and / or forward relative to the initial wrist angle, the indicator may move in a clockwise direction, and if the first direction is right and / or backward relative to the initial wrist angle, the indicator may move in a counterclockwise direction. In some embodiments, Fig.15F The wrist gesture shown in corresponds to a clockwise movement of user interface objects (e.g., from selecting a camera application menu option to selecting an editor menu option located clockwise from the camera application menu option).

[0344] In some embodiments, the indication is a first indication and the air gesture is a first air gesture, and method 1600 further includes: receiving a second indication from the wrist wearable device to perform a second air gesture, the second air gesture being different from the first air gesture. In some embodiments, the computing system performs an operation different from the control of the user interface object caused by the first indication based on the computing system receiving the second indication.

[0345] In some embodiments, the motion corresponding to the first air gesture is in a first direction relative to the user (e.g., in a direction extending forward and backward from the user), and the control of the user interface object in response to the first indication includes moving a position of a selector within the user interface object (e.g., a focus selector, a scrollable sidebar, etc.). In some embodiments, another motion corresponding to the second air gesture is in a second direction relative to the user, where the second direction is substantially orthogonal to the first direction (e.g., left and right), and the operation performed in response to the second indication causes display of another user interface object that is different from the user interface object controlled by the first air gesture.

[0346] In some embodiments, the first air gesture causes the user interface object to move in a direction corresponding to the direction of motion of the user's hand during the first air gesture. In some embodiments, the second air gesture is a pinch gesture, wherein the pinch gesture includes contact between a thumb of the user and another finger of the user, and the operation performed in response to the second indication causes an image to be captured by a camera at the wrist wearable device or at the head wearable device. In some embodiments, the direction the camera faces when capturing the image is based on the position of the focus selector.

[0347] In some embodiments, while (1632) a first option in the plurality of available options is in focus at the user interface, one or more sensors (e.g., a vibration sensor, a camera sensor, etc.) of the wrist wearable device that are not EMG sensors are used to detect a pinch gesture including contact between a thumb of the user and an index finger of the user, the contact having at least a predefined pinch strength (e.g., Figure 15L Contact 1578 between the thumb 1507 and index finger 1509 of the user 101).

[0348] In some embodiments, based on detecting (1634) the pinch gesture, an action associated with the user interface object is performed. Fig.15J As shown, when the user 1501 performs a pinch gesture including contact 1578 between the thumb 1507 and the index finger 1509, it causes a selectable option user interface element 1514-a (e.g., a selectable option corresponding to a camera application) to be presented in the field of view 1505 of the head wearable device 1504.

[0349] In some embodiments, in response to (1636) the second indication, an image is captured by a camera at a wrist wearable device, a head wearable device, or another electronic device. Figure 15L, the user 1501 performs a pinch gesture including contact 1578 between the user's thumb 1507 and index finger 1509, and based on the execution of the pinch gesture, an image is captured by the camera of the head wearable device 1504.

[0350] Having thus described example sequences and methods of operating a user using these example sequences, attention now turns to a system-level description of the hardware and software upon which (or with which) these methods may be implemented.

[0351] Example Embodiments

[0352] Turning now to some example embodiments of the previously described methods, devices, systems, and computer-readable storage media. In short, the following description first discusses paragraphs beginning with symbol A, which relate to aspects of the system responding to air gestures and surface gestures from a user, then discusses paragraphs beginning with symbol B, which relate to aspects of a sensor at a wrist wearable device detecting that a physical surface has a curved portion (e.g., detecting curvature, texture, and / or other topographic features); then discusses paragraphs beginning with symbol C, which relate to detected gestures performed at a wrist wearable device causing an operation to be performed at an electronic device that is not a wrist wearable device.

[0353] (A1) In one aspect, some embodiments include a method (e.g., method 900) of making air gestures and surface gestures available to a user. The method is performed at a wearable device (e.g., wearable device 1202) having a memory (e.g., memory 1206) and one or more processors (e.g., one or more processors 1204). The method includes: (i) presenting an artificial reality environment including a user interface (e.g., application 1232) via a head wearable device, wherein the user interface responds to a first set of operations corresponding to corresponding air gestures and a second set of operations corresponding to corresponding surface gestures; and, when presenting the user interface: (ii) in response to detecting, using a first set of sensors (e.g., a set of sensors 252) of a wrist wearable device, that an air gesture (e.g., air gesture 242) corresponding to an operation in the first set of operations is performed, causing the head wearable device to perform an operation in the first set of operations to update the presentation of the user interface (e.g., update the virtual object 220), and (iii) in response to detecting, using a second set of sensors (e.g., surface gesture 342 at the physical surface 310) of the wrist wearable device, that a surface gesture (e.g., surface gesture 342 at the physical surface 310) corresponding to an operation in the second set of operations is performed at a physical surface, causing the head wearable device to perform the operation in the second set of operations to update the presentation of the user interface (e.g., Figure 3B), at least one sensor of the second group of sensors is not in the first group of sensors (eg, time-of-flight sensor 258).

[0354] (A2) In some embodiments of A1, the method further includes: using a second set of sensors to detect whether the wrist wearable device is at a surface edge of the physical surface (e.g., FIG. 1A to FIG. 1I The surface edge 112 of the physical surface 110 in the image is within a surface edge threshold distance.

[0355] (A3) In some embodiments of A2, the method further includes: in response to detecting that the wrist wearable device is crossing a surface edge threshold distance of the surface edge, causing a haptic event at one of the head wearable device, the wrist wearable device, or another connected device (e.g., via haptic controller 1490). For example, a haptic event can be provided when wrist wearable device 102 crosses surface edge 112 of physical surface 110 during surface gesture 128.

[0356] (A4) In some embodiments of any one of A1 to A3, detecting the surface gesture further includes: using the second set of sensors to determine that the surface gesture is performed within a surface threshold distance of the physical surface. For example, the time of flight sensor 158 determines that the surface gesture 109 is performed within the surface threshold distance 106 of the physical surface 110.

[0357] (A5) In some embodiments of any one of A1 to A4, detecting performance of the surface gesture further includes: using a second set of sensors (e.g., using a time-of-flight sensor 158) to determine that at least a portion of the surface gesture contacts the physical surface.

[0358] (A6) In some embodiments of any of A1 to A5, detecting the performance of a surface gesture further comprises: determining, using a second set of sensors, that the surface gesture includes multiple contacts. For example, the time-of-flight sensor 358 may determine Figure 3E Surface gesture 348 in 340 includes two different physical contacts with physical surface 310. In some embodiments, the multiple contacts (e.g., a double tap gesture) are made in sequence, while in other embodiments, the multiple contacts (e.g., a surface pinch gesture) are made simultaneously.

[0359] (A7) In some embodiments of A6, detecting the execution of a surface gesture further includes: using a second set of sensors to determine whether two or more of the multiple contacts are performing a pinch gesture (for example, the position-independent gesture 346 in 3C can be detected based on detecting using an EMG sensor 354 that two or more of the multiple contacts are performing a pinch gesture).

[0360] (A8) In some embodiments of A7, the second set of sensors includes EMG sensors, and determining whether two or more contacts are performing a pinch gesture further includes: detecting a change in a pinch threshold distance between two or more contacts in the plurality of contacts of the surface gesture. For example, Figure 3D The position-independent gesture 346 in the position-independent gesture 346 may be determined as a surface gesture, and may be determined as a pinch gesture based on determining that multiple fingers of the user 301 have performed a pinch threshold distance change during the execution of the position-independent gesture 346. In some embodiments, the second group of sensors includes an IMU sensor (e.g., IMU sensor 160), and the IMU sensor detects a pinch threshold distance change between two or more contacts in the multiple contacts of the surface gesture based on, for example, a vibration signal detected via the IMU sensor 160 when the pinch gesture is performed. In some embodiments, the second group of sensors includes a camera sensor (e.g., camera sensor 164), and the camera sensor (e.g., an infrared camera positioned on a wrist wearable device so that its field of view is toward the top of the user's hand) detects a pinch threshold distance change between two or more contacts in the multiple contacts of the surface gesture. For such a gesture involving detecting detailed gesture features (e.g., whether two or more contacts have performed a pinch threshold distance change) as described in A8, an EMG sensor such as EMG sensor 154 may be more sensitive to such detailed gesture features than, for example, IMU sensor 160. However, the EMG sensor may consume power at a higher rate than the IMU sensor and / or camera sensor 164. In some embodiments, the wrist wearable device may activate the EMG sensor 154 when a threshold amount of power remains in the wrist wearable device. In some embodiments, the user can configure when the wrist wearable device uses the EMG sensor and when to use the IMU sensor and / or camera sensor instead. In some embodiments, the wrist wearable device does not include an EMG sensor, and the wrist wearable device uses an IMU sensor, a camera (imaging) sensor, or other sensors for (air and surface) gesture detection.

[0361] In some embodiments, the wrist wearable device discussed herein may include a limited set of sensors that do not include any EMG sensors (or other neuromuscular signal sensors), the limited set of sensors instead only including one or more IMU sensors and one or more camera sensors (e.g., an infrared sensor with a field of view toward the top of the user's hand (which may be an infrared proximity sensor)). In such embodiments, the wrist wearable device may make available to the user a set of gestures appropriately selected based on the capabilities of the limited set of sensors. For example, a wrist wearable device consistent with these embodiments may have the following gestures available: (i) a clenched fist wrist rotation gesture (e.g., the user's wrist moves left and right in rapid succession while the user's hand is clenched into a fist), which can be interpreted as causing (e.g., via a wearable head device) a menu to be displayed within an artificial reality environment; (ii) an open hand horizontal motion gesture, in which the user's wrist moves so that the user's hand moves left or right while the user's hand is not clenched into a fist, which gesture can cause left navigation motion and right navigation motion within a menu (or within another user interface object that is currently in focus within the artificial reality environment); (iii) an open hand horizontal motion gesture, in which the user's wrist moves so that the user's hand moves left or right while the user's hand is not clenched into a fist, which gesture can cause left navigation motion and right navigation motion within a menu (or within another user interface object that is currently in focus within the artificial reality environment); (iv) an air finger tap gesture, in which one of the user's multiple fingers moves to contact the user's thumb a certain number of times, where the certain number of times triggers different types of actions (e.g., a single tap can cause an image of the artificial reality environment to be captured, a double tap can cause the artificial reality environment to allow drawing on the captured image, and a triple tap can cause a video of the artificial reality environment to be captured). In some embodiments, weighting factors for data from one or more IMU sensors and one or more camera sensors can also be selected by the user to allow the user to customize which data should be weighted more heavily when making gesture determinations. (e.g., for wrist wearable devices that only include a limited set of sensors) These additional features can be used as alternatives, or these additional features can also be used to supplement the gesture space available in combination with other wrist wearable devices discussed herein.

[0362] (A9) In some embodiments of any of A1 to A8, detecting the performance of a surface gesture further comprises: detecting a sliding gesture along the physical surface using a second set of sensors. For example, the time-of-flight sensor 358 may determine Figure 3C Surface gesture 344 in FIG. 3 is a sliding gesture along physical surface 310 .

[0363] (A10) In some embodiments of A9, at least a portion of the sliding gesture includes one or more contacts with a physical surface. Figure 3C , EMG sensor 354 may determine that user 301 made physical contact with physical surface 310 during performance of surface gesture 344. As an alternative example, data from IMU sensor 160 or camera sensor 164 is used to determine that the user made physical contact with the surface.

[0364] (A11) In some embodiments of any one of A1 to A10, detecting an air gesture or a surface gesture includes: (e.g., via sensor 1216) detecting that the corresponding gesture is maintained in a position for at least a threshold time (e.g., at least 1 second, 3 seconds, 5 seconds, or 7 seconds).

[0365] (A12) In some embodiments of any one of A1 to A11, detecting the execution of the surface gesture further includes: using a second set of sensors to detect a touch intensity corresponding to the surface gesture, and identifying the above operation from the second set of operations based on determining whether the touch intensity is greater than a press threshold intensity (e.g., 1.66 Pascals). For example, Figure 3E Surface gestures 348 in can determine whether a tap intensity threshold is met for each individual tap made by user 301 on physical surface 310. In some embodiments, touch intensity is detected by an EMG sensor, an impedance sensor, or an IMU sensor.

[0366] (A13) In some embodiments of any one of A1 to A12, detecting the execution of a surface gesture further includes: using a second set of sensors to detect a first wrist angle of the wrist wearable device, using a second set of sensors to detect a second wrist angle of the wrist wearable device, and identifying the above operation from the second set of operations includes: determining a rotation angle between the first wrist angle and the second wrist angle. In some embodiments, the same sequence of operations can be detected as an air gesture within a threshold distance of the corresponding surface. For example, Figure 3H A position-independent gesture 345 (eg, a virtual pull gesture) that includes a change in wrist angle is shown. In some embodiments, the user's wrist angle is determined based on data from an EMG sensor, an IMU sensor, or a camera sensor.

[0367] (A14) In some embodiments of any of A1 to A13, detecting performance of a surface gesture further includes: using a second set of sensors (e.g., using EMG sensors 354) to detect a first contact at a first location within a threshold distance of the surface, using a second set of sensors to detect a second contact at a second location within a threshold distance of the surface, and determining, based on corresponding gesture criteria, that the user performed a steady motion gesture between the first location and the second location. In some embodiments, a steady motion gesture is distinguishable from a swipe gesture because a swipe gesture includes a "flick" motion with a relatively high acceleration, whereas a steady motion gesture is a controlled motion with a relatively steady speed. For example, the two-dimensional volume control affordance 532 can be adjusted by performing a steady motion gesture.

[0368] (A15) In some embodiments of any of A1 to A14, the second set of operations includes affordance activation operations, wherein detection of performance of the surface gesture includes detecting, by the second set of sensors, a first gesture portion corresponding to selecting an affordance of the user interface, and detecting a second gesture portion corresponding to activation of the selected affordance, wherein the second gesture includes a downward movement of the user's hand. For example, Figure 1B A surface gesture 109 is shown selecting a virtual object 118 (e.g., changing its appearance to indicate that it is being selected), and Figure 1C A surface gesture 122 is shown activating a virtual object 118 .

[0369] (A16) In some embodiments of any one of A1 to A15, detecting the performance of the air gesture further includes: using the first set of sensors to determine that the air gesture is performed within the air threshold distance. For example, using data from the EMG sensor 354 or a time-of-flight sensor in the set of sensors 252 to determine that the air gesture 242 is performed between the surface threshold distance 206 and the air threshold distance 208.

[0370] (A17) In some embodiments of A16, the maximum value of the air threshold distance is 50 centimeters. For example, FIG. 2A to FIG. 2D The air threshold distance in can be configured to be in the range of 6 cm to 50 cm.

[0371] (A18) In some embodiments of any one of A16 to A17, the method further includes: based on determining that the air gesture is not performed within the air threshold distance, causing the electronic device to perform an operation in a third group of operations corresponding to the position-independent gesture (e.g., Figure 1JIn some embodiments, the operation is performed in response to determining that the air gesture is not performed within the air threshold distance. In some embodiments, detecting the gesture within the air threshold distance further includes: a gesture detected outside the surface threshold distance. In some embodiments, other criteria are considered. For example, in some embodiments, the EMG sensor indicates that the user is performing a position-independent gesture. The electronic device performs an operation in a third group of operations corresponding to the position-independent gesture based on the indication from the EMG sensor.

[0372] (A19) In some embodiments of any of A1 to A18, the user interface includes a virtual object (e.g., virtual object 220) within an air threshold distance of the physical surface, and the air gesture includes simulated contact with a simulated location of the virtual object (e.g., as Figure 2B In some embodiments, a haptic event is provided based on the simulated contact. For example, in some embodiments, a haptic event is provided at a wrist wearable device in response to determining that the user's hand is located at the same simulated location as the virtual surface of the virtual object.

[0373] (A20) In some embodiments of any one of A1 to A19, the method further includes: in response to detecting the execution of one or more of the air gesture and the surface gesture, causing a tactile event (e.g., such as a touch event) at one of the head wearable device and the wrist wearable device (e.g., via the tactile controller 1490). Figure 2D shown).

[0374] (A21) In some embodiments of A20, the haptic event corresponds to an air gesture that includes simulated contact with a virtual object at a simulated location within an air threshold distance of a physical surface (e.g., Figure 2B Tactile events in 221).

[0375] (A22) In some embodiments of any one of A1 to A21, the physical surface corresponds to a part of the user's body. For example, Fig. 8E 865 of the user in the figure. Examples of parts of the user's body that can be used as physical surfaces include palms, hands, arms, shoulders, legs, chest. In some embodiments, the physical surface corresponds to a piece of clothing of the user.

[0376] (A23) In some embodiments of A22, the portion of the user's body is the user's palm (e.g., Figure 6C palm of the user).

[0377] (A24) In some embodiments of any of A22 to A23, the physical surface does not include a physical surface of a portion of the user's body. In some embodiments, the method detects whether there is clothing or other material on the portion of the user's body and can still respond to gestures on the physical surface or other material (e.g., clothing).

[0378] (A25) In some embodiments of any one of A1 to A24, presenting the user interface further includes: displaying a message object corresponding to the received message at any one of the wrist wearable device, the head wearable device, or another connected device. For example, an instant message or short message service (SMS) is sent to the user and received by the wearable device (e.g., wrist wearable device 102). The received instant message or SMS is displayed to the user (e.g., together with an option for responding).

[0379] (A26) In some embodiments of A25, the message object is displayed above the wrist wearable device. For example, the message object is anchored to the wrist wearable device, similar to Figure 2A How the virtual object 220 is displayed on top of the physical surface 210.

[0380] (A27) In some embodiments of any one of A25 to A26, the message object is a virtual three-dimensional object. In some embodiments, the virtual three-dimensional object is a three-dimensional emoticon virtual object based on the content of the message received from a different user of a different electronic device. In some embodiments, the received message is from another user of a different device. In some embodiments, the other user sends the received message from a phone (different from a wrist wearable device).

[0381] (A28) In some embodiments of any one of A25 to A27, the method further includes: based on receiving the received message at any one of the wrist wearable device, the head wearable device, or another connected device, updating the presentation of the user interface to display one or more selectable enable representations, wherein each of the one or more selectable enable representations, when selected, performs a corresponding operation from a set of message response operations associated with the received message. For example, the message object is displayed with the selectable enable representation, similar to how the display virtual object 220 is displayed with the display control portion 224.

[0382] (A29) In some embodiments of any one of A1 to A28, the method further includes: in response to detecting a position-independent gesture anywhere, performing an operation from a set of position-independent operations for updating the presentation of the user interface, the position-independent gesture being included within a surface threshold distance of the physical surface and an air threshold distance of the physical surface. In some embodiments, in order to be recognized, the position-independent gesture must be near the physical surface (e.g., within 2 feet, within 6 feet, or within 10 feet). In some embodiments, in order to be recognized, the position-independent gesture must be just above the physical surface. In some embodiments, one or more position-independent gestures are not detected when performed at a distance from the wrist wearable device relative to the corresponding physical surface. For example, in some embodiments, the artificial reality system is configured to prevent detection of a position-independent gesture when the wrist wearable device is within one or more of a surface threshold distance, an air threshold distance, and / or a surface edge threshold distance.

[0383] (A30) In some embodiments of A29, the position-independent gesture is one of a thumbs-up gesture and a thumbs-down gesture, and in response to detecting the position-independent gesture, a corresponding response is applied to the received message corresponding to the message object displayed above the wrist wearable device. Examples of other position-independent gestures include a waving gesture, a fist gesture, a nod gesture, a shrug gesture, a clapping gesture, and a shaking head gesture.

[0384] (A31) In some embodiments of any one of A1 to A30, the second set of sensors includes a first time-of-flight sensor and a second time-of-flight sensor, the first time-of-flight sensor being arranged and configured to measure a first angle relative to the wrist wearable device (e.g., corresponding to a first angle Figure 1A ray 105 in the wrist wearable device), the second time-of-flight sensor is arranged and configured to measure a second angle relative to the wrist wearable device (e.g., corresponding to Figure 1A In some embodiments, one or more of the plurality of time-of-flight sensors are time-of-flight sensors that emit laser light. In some embodiments, one or more of the plurality of time-of-flight sensors emit structured light. In some embodiments, one or more of the plurality of time-of-flight sensors are multi-mode sensors that can emit either laser light or structured light.

[0385] (A32) In some embodiments of A31, the second set of sensors further includes a third time-of-flight sensor arranged and configured to measure a third angle relative to the wrist wearable device. For example, Figure 1A Ray 107 in may correspond to a third time-of-flight sensor in time-of-flight sensors 158 .

[0386] (A33) In some embodiments of any one of A1 to A32, each of the plurality of time-of-flight sensors in the second set of sensors has a different sensor location at the wrist wearable device (e.g., Fig. 14B The time-of-flight sensors 1466 , 1468 , and 1470 are disposed at different locations on the watch body 1402 .

[0387] (A34) In some embodiments of any one of A1 to A33, the second group of sensors includes one or more time-of-flight sensors, and at least one of the one or more time-of-flight sensors is an optoelectronic CMOS device.

[0388] (A35) In some embodiments of A34, the optoelectronic CMOS device includes a sensor array. For example, the time-of-flight sensor 158 may include a CMOS sensor array configured to emit Figure 1A Two or more rays among rays 103, 105 and 107 in.

[0389] (A36) In some embodiments of any one of A1 to A35, the method further includes: causing a power reduction of at least one sensor from the second set of sensors based on detecting that the wrist wearable device is farther from the physical surface than a surface threshold distance. In some embodiments, the power reduction includes limiting or disabling power to at least one sensor. In some embodiments, the power reduction also includes reducing power to one or more processing or control circuits of one or more sensors. In some cases, reducing power in this manner increases battery life of the wrist wearable device.

[0390] (A37) In some embodiments of A36, at least one sensor whose received power is reduced is a time-of-flight sensor.

[0391] (A38) In some embodiments of any one of A1 to A37, at least one sensor not in the first group of sensors is one of: a time-of-flight sensor; an EMG sensor; a camera (imaging) sensor; and a sEMG sensor.

[0392] (A39) In some embodiments of any one of A1 to A38, at least one sensor from one of the first set of sensors and the second set of sensors includes a shared electrode, wherein the shared electrode includes more than one sensing capability. For example, Fig. 14B The sensor locations 1436 in may include a set of sensors 1437 and 1438, which may include two or more sensing capabilities.

[0393] (A40) In some embodiments of any one of A1 to A39, the method further includes: in response to detecting the execution of a mixed gesture corresponding to an operation in a set of mixed operations using one or more of the first group of sensors and the second group of sensors of the wrist wearable device, causing the head wearable device to execute the operation in the set of mixed operations to update the presentation of the user interface, wherein the set of mixed operations corresponds to a mixed gesture including two or more of: a surface gesture portion; an air gesture portion; and a position-independent gesture portion. For example, the surface gesture portion of the mixed gesture is detected by an EMG sensor, and the air gesture portion of the mixed gesture is detected by a time-of-flight sensor. For example, any combination of the individual gestures described herein can be used as a mixed gesture (e.g., Figure 1G Surface gestures 128 and Fig. 1I The gesture is performed as part of the execution of the position-independent gesture 130).

[0394] (A41) In some embodiments of A40, the compound gesture further comprises a physical contact gesture portion, wherein the physical contact gesture portion comprises at least one physical surface contact with a physical surface. For example, the compound gesture comprises a surface tap gesture portion followed by a pinch and flick gesture portion, and the compound gesture corresponds to an operation of deleting a user interface element (at the tap location).

[0395] (A42) In some embodiments of any one of A1 to A41, detecting a surface gesture further includes: using a sEMG sensor from the second group of sensors (e.g., an sEMG sensor in sensor 1258) to detect whether a user of the wrist wearable device has transmitted an electrical signal to the physical surface.

[0396] (B1) In another aspect, some embodiments include a method (e.g., method 1000) for gesture detection and content rendering determination using a time-of-flight sensor in an artificial reality environment. The method is performed at a wearable device (e.g., wrist wearable device 1202) having a memory (e.g., memory 1206) and one or more processors (e.g., one or more processors 1204). The method includes: (i) receiving data about a physical surface from one or more time-of-flight sensors (e.g., time-of-flight sensor 1218), the one or more time-of-flight sensors being communicatively coupled to a wrist wearable device (e.g., wrist wearable device 1400), wherein the wrist wearable device is communicatively coupled to a head wearable device (e.g., AR system 1320), the head wearable device (e.g., AR system 1320) being configured to display a virtual object (e.g., virtual object 810) within an artificial reality environment presented by the head wearable device, and (ii) determining based on the data that the physical surface has a curved surface portion (e.g., curved surface portion 806): (iii) causing at least a portion of the virtual object to be displayed at the curved surface portion, including updating the display of the virtual object based on the curved surface portion (e.g., as Figure 8B shown).

[0397] In some embodiments, data from one or more time-of-flight sensors indicates the distance of the wrist wearable device from the surface. In some embodiments, data from one or more time-of-flight sensors indicates whether the surface is associated with an animate object or an inanimate object. In some embodiments, data from one or more time-of-flight sensors indicates whether the surface meets the requirements (e.g., certain minimum dimensions) for displaying virtual objects. In some embodiments, the surface is a user's leg, palm, etc., or an inanimate object. In some embodiments, an inanimate object can be generally planar so that its surface has no curved portions. In some embodiments, the surface can have different edges or other significant surface topologies. In some embodiments, the surface is dynamic, meaning that the surface is moving or otherwise changing shape.

[0398] In some embodiments, at least one of the one or more time-of-flight sensors is pivotally coupled to the wrist wearable device. In some embodiments, at least one of the one or more time-of-flight sensors is located at an electronic device different from the wrist wearable device (e.g., head wearable device 204 in Figure 2). In some embodiments, data about the distance of the wrist wearable device from the surface is based on data from more than one time-of-flight sensor. In some embodiments, the wrist wearable device includes more than one time-of-flight sensor, and the more than one time-of-flight sensor has different angles projected from the wrist wearable device. In some embodiments, the data from one or more time-of-flight sensors comes from at least one time-of-flight sensor at a different electronic device (e.g., a head wearable device or a smartphone). In some embodiments, a semiconductor-based time-of-flight sensor is used.

[0399] In some embodiments, considering the curved surface portion includes determining the surface topology (e.g., Figure 1A In some embodiments, the time-of-flight sensor sweeps over a predefined range of angles to determine the time-of-flight at various locations along the identified surface. In some embodiments, only portions of the surface that can be accurately detected by the time-of-flight sensor are used for display of the virtual object. In some embodiments, the accuracy threshold of the time-of-flight sensor depends on a position corresponding to a relative position of the display of the virtual object.

[0400] (B2) In some embodiments of B1, the virtual object corresponds to one of a plurality of potentially displayed virtual objects, and determining which of the plurality of potentially displayed virtual objects to display depends on one or more surface criteria of the curved portion. For example, a plurality of virtual objects correspond to the same function but have different visual appearances. A virtual object of a plurality of virtual objects may be selected based on properties of the surface (e.g., angle, size, and orientation). In some embodiments, the surface properties are determined based on data from one or more sensors of the wearable device (e.g., an EMG sensor, a sEMG sensor, an imaging sensor, and a time-of-flight sensor).

[0401] (B3) In some embodiments of B2, at least one of the one or more surface criteria is a surface angle of a curved portion. For example, a virtual object is selected based on an assessment of relative visibility (compared to other potentially displayed virtual objects) given surface properties. In some embodiments, the surface angle is determined based on data from one or more time-of-flight sensors.

[0402] (B4) In some embodiments of B3, based on determining that the surface angle is greater than a keyboard display threshold angle (e.g., an angle greater than 30 degrees, 50 degrees, or 70 degrees), a virtual object including device controls (including at least one slider element (e.g., a two-dimensional volume control enable representation 532)) is displayed, and based on determining that the surface angle is less than a keyboard display threshold angle, a virtual object including at least one virtual keyboard element is displayed.

[0403] (B5) In some embodiments of any one of B2 to B4, at least one of the one or more surface criteria is the surface area of ​​the curved portion. For example, when it is determined that the surface area of ​​the curved portion is smaller, a virtual object with a relatively smaller size (compared to other potential virtual objects) is selected.

[0404] (B6) In some embodiments of any one of B1 to B5, the method further includes: prompting a user of the wrist wearable device to adjust at least one of a wrist position or a wrist angle of the wrist wearable device to collect data about the curved surface portion from one or more time-of-flight sensors so as to display a portion of the virtual object at the curved surface portion. For example, in response to a user's reque...

Claims

1. A method comprising: While the head wearable device is presenting an artificial reality environment including a user interface object, wherein the user interface object responds to a first set of operations corresponding to a corresponding air gesture and a second set of operations corresponding to a corresponding surface gesture: Based on detecting, using a first set of sensors of a wrist wearable device, an in-air gesture corresponding to an operation in the first set of operations, causing the head wearable device to: (i) perform an operation in the first set of operations; and (ii) update the user interface object in conjunction with the performance of the operation in the first set of operations, wherein the in-air gesture is performed beyond a surface threshold distance of a physical surface that is at least partially located within the artificial reality environment; and Based on detecting a surface gesture corresponding to an operation in the second group of operations using a second group of sensors of the wrist wearable device, the head wearable device: (i) performs the operation in the second group of operations; and (ii) updates the user interface object in conjunction with the execution of the operation in the second group of operations, at least one sensor of the second group of sensors is not in the first group of sensors, and the surface gesture is performed within the surface threshold distance of the physical surface.

2. The method according to claim 1, further comprising: In response to detecting that the wrist wearable device is within a surface edge threshold distance of a surface edge of the physical surface, a haptic event is caused at the head wearable device, the wrist wearable device, or another connected device.

3. The method according to claim 1 or 2, wherein: The second set of sensors includes: a first time-of-flight sensor arranged and configured to measure a first angle relative to the wrist wearable device; and A second time-of-flight sensor is arranged and configured to measure a second angle relative to the wrist wearable device.

4. The method according to any preceding claim, further comprising: In response to detecting that the wrist wearable device is farther from the physical surface than the surface threshold distance: A power reduction is caused to at least one sensor in the second group of sensors.

5. A method according to any preceding claim, wherein: The performance of the surface gesture is detected using the second set of sensors of the wrist wearable device after determining that the surface gesture is performed within a surface threshold distance of the physical surface.

6. The method according to claim 1, wherein: The performance of the surface gesture is detected using the second set of sensors of the wrist wearable device after determining that at least a portion of the surface gesture contacts the physical surface; and preferably, wherein: The second group of sensors includes biopotential signal sensors; and A determination that two or more of the multiple contacts are performing a pinch gesture is based on a pinch threshold distance change between two or more of the multiple contacts of the surface gesture, the pinch threshold distance change being detected by the biopotential sensor.

7. A method according to any preceding claim, wherein: Detecting the performance of the surface gesture further includes: While the user interface object is in focus based on one of the mid-air gesture or the surface gesture: detecting a first wrist angle of the wrist wearable device using the second set of sensors; and The second group of sensors is used to detect a second wrist angle of the wrist wearable device; wherein, Identifying the operation in the second set of operations includes determining a rotation angle between the first wrist angle and the second wrist angle.

8. A method according to any preceding claim, wherein: The second set of operations includes affordances activating operations, wherein the performing of the surface gesture includes: a first gesture portion corresponding to selecting a user interface object in the artificial reality environment; and A second gesture portion, the second gesture portion corresponding to activating the user interface object, wherein the second gesture portion includes a downward movement of a user's hand.

9. A method according to any preceding claim, wherein: The performance of the mid-air gesture is detected while the user interface object remains in focus and is further based in part on: using the first set of sensors to determine that the mid-air gesture is performed within a mid-air threshold distance relative to the physical surface, the mid-air threshold distance being greater than the surface threshold distance; and preferably, wherein the mid-air threshold distance has a maximum value of 50 centimeters; and / or preferably, wherein: Based on determining that another air gesture is detected after the air gesture without being performed within the air threshold distance: The head wearable device is enabled to perform operations in a third group of operations corresponding to the position-independent gesture.

10. A method according to any preceding claim, wherein: The user interface objects include virtual objects that are within an air threshold distance of the physical surface and beyond a surface threshold distance; and The mid-air gesture includes virtual contact with the virtual object.

11. A method according to any preceding claim, wherein: The physical surface comprises a portion of a body part of the user; and preferably, wherein the portion of the body part of the user is selected from the group consisting of: (i) a palm portion; or (ii) a leg portion.

12. The method according to any preceding claim, further comprising: In response to detecting a position-independent gesture at any location within the surface threshold distance of the physical surface, causing the head wearable device to: (i) perform an operation from a set of position-independent operations; and (ii) update the artificial reality environment, wherein the position-independent gesture is selected from the group consisting of: (i) a thumbs-up gesture; and (ii) a thumbs-down gesture; and In response to detecting the position-independent gesture, a response is applied to a received message, the received message corresponding to a message object displayed within the artificial reality environment.

13. A method according to any preceding claim, wherein: The surface gesture is detected by using a surface electromyography (sEMG) sensor in the second group of sensors to determine whether the user of the wrist wearable device has transmitted an electrical signal to the physical surface; and / or preferably, at least one sensor in the first group of sensors and the second group of sensors includes a shared electrode, wherein the shared electrode is configured to perform more than one sensing capability.

14. A wrist wearable device comprising: one or more processors; as well as A memory comprising instructions that, when executed by the one or more processors, cause the wrist wearable device to perform the following operations: When the head wearable device is presenting an artificial reality environment including a user interface object, wherein the user interface object responds to a first set of operations corresponding to a corresponding air gesture and a second set of operations corresponding to a corresponding surface gesture: Based on detecting, using a first set of sensors of a wrist wearable device, an in-air gesture corresponding to an operation in the first set of operations, causing the head wearable device to: (i) perform an operation in the first set of operations; and (ii) update the user interface object in conjunction with the performance of the operation in the first set of operations, wherein the in-air gesture is performed beyond a surface threshold distance of a physical surface that is at least partially located within the artificial reality environment; and Based on detecting a surface gesture corresponding to an operation in the second group of operations using a second group of sensors of the wrist wearable device, the head wearable device: (i) performs the operation in the second group of operations; and (ii) updates the user interface object in conjunction with the execution of the operation in the second group of operations, at least one sensor of the second group of sensors is not in the first group of sensors, and the surface gesture is performed within the surface threshold distance of the physical surface.

15. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors of an electronic device, cause the following operations to be performed: When the head wearable device is presenting an artificial reality environment including a user interface object, The user interface object responds to a first set of operations corresponding to the corresponding air gestures and a second set of operations corresponding to the corresponding surface gestures: Based on detecting, using a first set of sensors of the wrist wearable device, that an air gesture corresponding to an operation in the first set of operations is performed, causing the head wearable device to: (i) perform an operation in the first set of operations; and (ii) updating the user interface object in conjunction with performance of the operation in the first set of operations, wherein the mid-air gesture is performed beyond a surface threshold distance of a physical surface that is at least partially within the artificial reality environment; Based on detecting a surface gesture corresponding to an operation in the second group of operations using a second group of sensors of the wrist wearable device, the head wearable device: (i) performs the operation in the second group of operations; and (ii) updates the user interface object in conjunction with the execution of the operation in the second group of operations, at least one sensor of the second group of sensors is not in the first group of sensors, and the surface gesture is performed within the surface threshold distance of the physical surface.

Images