Activation forces detected via neuromuscular signal sensors of wearable devices and systems and methods of use thereof

Abstract

Various embodiments described herein include methods and systems for providing a force dimension to an interface element. In one aspect, a method includes: (i) communicatively coupling a wearable device with an activatable device, the wearable device including an electromyography (EMG) sensor, the activatable device including a mechanical user interface element; (ii) acquiring data from the EMG sensor according to activation of the user interface element, the data corresponding to the activation; (iii) determining, based on data from the EMG sensor, whether the activation comprises an activation force that meets a predefined criterion; (iv) in accordance with a determination that the activation force satisfies a predefined criterion, causing a first function corresponding to the user interface element to be performed; and (v) in accordance with a determination that the activation force satisfies a predefined criterion, causing a second function corresponding to the user interface element to be performed.

Description

Technical Field

[0001] The present disclosure generally relates to devices and methods for detecting activation forces corresponding to user postures and hardware activation. Background Art

[0002] Many devices with mechanical switches (e.g., light switches) and buttons do not have the ability to determine the force associated with activation. This limits the functionality of such devices. However, in terms of cost, power consumption, and device size, adding force sensing capabilities to these devices may be inefficient. Summary of the Invention

[0003] The present disclosure describes a system for adding force sensing information to the activation of mechanical devices (e.g., buttons and switches) and other devices that do not have force sensing capabilities. Example systems include a mechanical switch (e.g., a wall-mounted light switch) and a smartwatch (which may more commonly be referred to as a wrist-wearable device). The disclosed system obtains force information from a wearable device (e.g., a wrist-wearable device) and combines the force information with activation information from a mechanical device. For example, the wearable device may include one or more neuromuscular signal sensors (e.g., electromyography (EMG) sensors). In this way, the functionality of mechanical devices and other non-force sensing devices can be extended and improved (e.g., without the need to add force sensing components to each mechanical device).

[0004] According to some embodiments, a method for providing a force dimension to an interface element is provided. The method includes: (i) communicatively coupling a wearable device including one or more EMG sensors to an activatable device including a mechanical user interface element; (ii) obtaining, in response to activation of the mechanical user interface element, data from the one or more EMG sensors corresponding to the activation; (iii) determining, based on the data from the one or more EMG sensors, whether the activation of the mechanical user interface element includes an activation force that meets one or more predefined criteria; (iv) causing a first function corresponding to the mechanical user interface element to be performed in response to determining that the activation of the mechanical user interface element includes the activation force that meets the one or more predefined criteria; and (v) causing a second function corresponding to the mechanical user interface element to be performed in response to determining that the activation of the mechanical user interface element does not include the activation force that meets the one or more predefined criteria, where the second function is different from the first function.

[0005] In some embodiments, the mechanical user interface element is a switch, a joystick, or a button.

[0006] In some embodiments, the activatable device does not include a display.

[0007] In some embodiments: the mechanical user interface element is a first mechanical user interface element; the activatable device includes a second mechanical user interface element; and the second mechanical user interface element is associated with one or more predefined force criteria that are different from the one or more predefined criteria associated with the first mechanical user interface element.

[0008] In some embodiments, the method further includes: receiving, at the wearable device, a request for force data from the activatable device, wherein the wearable device is communicatively coupled to the activatable device in response to the request.

[0009] In some embodiments, the method further includes: receiving, at the activatable device, a broadcast communication from the wearable device that identifies the force capabilities of the wearable device, wherein the wearable device is communicatively coupled to the activatable device in response to the broadcast communication.

[0010] In some embodiments, the wearable device includes a wrist-worn device.

[0011] In some embodiments, the method further includes: providing a notification to a user of the wearable device, the notification including information about being communicatively coupled to the activatable device and / or information about providing data from the one or more neuromuscular signal sensors.

[0012] In some embodiments, the method further includes: providing information about a first function and a second function to the user before detecting an activation of the mechanical user interface element.

[0013] In some embodiments, the information about the first function and the second function is displayed to the user via a head-worn device.

[0014] In some embodiments, the wearable device is communicatively coupled to the activatable device based on the wearable device being in the vicinity of the activatable device.

[0015] In some embodiments, the method further includes: communicatively coupling the wearable device to a second activatable device, the second activatable device including a second mechanical user interface element; obtaining, based on activation of the second mechanical user interface element, additional data from the one or more neuromuscular signal sensors, the additional data corresponding to the activation of the second mechanical user interface element; determining, based on the additional data from the one or more neuromuscular signal sensors, whether the activation of the second mechanical user interface element includes a corresponding force satisfying one or more predefined criteria; causing, based on determining that the activation of the second mechanical user interface element includes the corresponding force satisfying the one or more predefined criteria, a third function corresponding to the second mechanical user interface element to be performed; and causing, based on determining that the activation of the second mechanical user interface element does not include the corresponding force satisfying the one or more predefined criteria, a fourth function corresponding to the second mechanical user interface element to be performed.

[0016] According to another aspect of the present disclosure, a method of providing a force dimension to an interface element is provided, the method including: communicatively coupling a wearable device to an activatable device, the wearable device including one or more neuromuscular signal sensors, the activatable device including a physical user interface element but not including a display; obtaining, based on activation of the physical user interface element, data from the one or more neuromuscular signal sensors, the data corresponding to the activation; determining, based on the data from the one or more neuromuscular signal sensors, whether the activation of the physical user interface element includes a force satisfying one or more predefined criteria; causing, based on determining that the activation of the physical user interface element includes the force satisfying the one or more predefined criteria, a first function corresponding to the physical user interface element to be performed; and causing, based on determining that the activation of the physical user interface element does not include the force satisfying the one or more predefined criteria, a second function corresponding to the physical user interface element to be performed, wherein the second function is different from the first function.

[0017] According to another aspect of the present disclosure, there is provided a wearable device, the wearable device comprising: one or more neuromuscular signal sensors; a control circuit; a memory; and one or more sets of instructions stored in the memory and configured to be executed by the control circuit, the one or more sets of instructions including instructions for: communicatively coupling with an activatable device, the activatable device including a mechanical user interface element; obtaining, based on activation of the mechanical user interface element, data from the one or more neuromuscular signal sensors, the data corresponding to the activation; determining, based on the data from the one or more neuromuscular signal sensors, whether the activation of the mechanical user interface element includes an activation force that meets one or more predefined criteria; causing, based on determining that the activation of the mechanical user interface element includes the activation force that meets the one or more predefined criteria, a first function corresponding to the mechanical user interface element to be executed; and causing, based on determining that the activation of the mechanical user interface element does not include the activation force that meets the one or more predefined criteria, a second function corresponding to the mechanical user interface element to be executed, wherein the second function is different from the first function.

[0018] In some embodiments, the mechanical user interface element is a switch, a joystick, or a button.

[0019] In some embodiments, the activatable device does not include a display.

[0020] In some embodiments: the mechanical user interface element is a first mechanical user interface element; the activatable device includes a second mechanical user interface element; and the second mechanical user interface element is associated with one or more predefined force criteria that are different from the one or more predefined criteria associated with the first mechanical user interface element.

[0021] In some embodiments, the one or more sets of instructions further include instructions for receiving a request for force data from the activatable device, wherein the wearable device is communicatively coupled to the activatable device in response to the request.

[0022] In some embodiments, the one or more sets of instructions further include instructions for providing a notification to a user of the wearable device, the notification including information about communicatively coupling with the activatable device and / or information about providing data from the one or more neuromuscular signal sensors.

[0023] In some embodiments, the wearable device includes a wrist wearable device.

[0024] In some embodiments, a computing device (e.g., a wrist-wearable device or a head-mounted device, or an intermediate device such as a smartphone, a desktop computer, or a laptop computer) includes one or more processors, a memory, a display (in some embodiments, for certain example intermediate devices below, the display may be optional: the intermediate device may coordinate operations at the wrist-wearable device and the head-mounted device and thus has sufficient processing resources and power resources without the need to have its own display), and one or more programs stored in the memory. These programs are configured to be executed by the one or more processors. The one or more programs include instructions for performing (or causing to be performed) any of the methods described herein.

[0025] In some embodiments, a non-transitory computer-readable storage medium stores one or more programs, which are configured to be executed by a computing device having one or more processors and a memory (e.g., a wrist-wearable device or a head-mounted device, or an intermediate device such as a smartphone, a desktop computer, or a laptop computer, which may be configured to coordinate operations at the wrist-wearable device and the head-mounted device). The one or more programs include instructions for performing (or causing to be performed) any of the methods described herein.

[0026] Accordingly, methods, systems, and computer-readable storage media for providing a force dimension to an interface element (e.g., a button, a switch, and / or a joystick) are disclosed. Such methods and systems may supplement or replace conventional methods for providing a force dimension.

[0027] It will be recognized that any feature described herein as being suitable for incorporation into one or more aspects or embodiments of the present disclosure is intended to be generalizable to any and all aspects and embodiments of the present disclosure. Those skilled in the art can understand other aspects of the present disclosure based on the specification, claims, and drawings of the present disclosure. The foregoing general description and the following detailed description are merely exemplary and explanatory and are not limiting of the claims. The features and advantages described in the specification are not necessarily all inclusive, and in particular, given the drawings, specification, and claims provided in the present disclosure, some additional features and advantages will be apparent to those of ordinary skill in the art. In addition, it should be noted that the terminology used in this specification is mainly selected for readability and guidance purposes and is not necessarily selected to depict or limit the subject matter described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to understand the present disclosure in more detail, a more specific description may be obtained by reference to the features of various embodiments, some of which are shown in the accompanying drawings. However, the accompanying drawings only illustrate the relevant features of the present disclosure and are not necessarily considered to be limiting, as the specification may admit other effective features as will be understood by those skilled in the art upon reading the present disclosure.

[0029] Figures 1A to 1G An example user scenario for interacting with a mechanical switch in accordance with some embodiments is shown.

[0030] Figures 2A to 2D Example user scenarios for interacting with an electronic device in accordance with some embodiments are shown.

[0031] Figures 3A to 3D An example user scenario for interacting with a controller according to some embodiments is shown.

[0032] Figure 4 is a flow chart illustrating an example method for providing a force dimension to an interface element according to some embodiments.

[0033] Figure 5A , Figure 5B , Figure 5C-1 , Figure 5C-2 , Figure 5D-1 and Figure 5D-2 An example artificial reality system is shown in accordance with some embodiments.

[0034] Figure 6A and Figure 6B An example wrist wearable device is shown in accordance with some embodiments.

[0035] Figure 7A , Figure 7B-1 , Figure 7B-2 and Figure 7C An example head wearable device is shown in accordance with some embodiments.

[0036] Figure 8A and Figure 8B An example handheld intermediate processing device is shown in accordance with some embodiments.

[0037] Figures 9A to 9C An example textile-based smart garment is shown in accordance with some embodiments.

[0038] According to common practice, the various features shown in the drawings are not necessarily drawn to scale, and like reference numerals may be used throughout the specification and drawings to denote like features. DETAILED DESCRIPTION

[0039] As an example, a user may want to use a mechanical light switch to dim their lights. According to the embodiments described herein, the user has a wrist-wearable device (e.g., the wrist-wearable device 600 referenced in Figure 6A and Figure 6B ), which includes one or more neuromuscular signal sensors capable of detecting the force exerted by the user's hand. In this example, when the user actuates the light switch, the wrist-wearable device detects the actuation force and provides force information to the light switch (or to a separate control device). In this way, the brightness level of the light can be adjusted based on the force information.

[0040] As another example, a user may want to play a video game that responds to force-based button activation. The user in this example has a conventional controller that includes multiple buttons but does not include force sensing. According to the embodiments described herein, the user has a wrist-wearable device that includes one or more neuromuscular signal sensors capable of detecting the force exerted by the user's hand. In this example, when the user presses a button on the controller, for each button press, the wrist-wearable device detects the force applied to the button and provides force information to the video game system.

[0041] Now turning to the drawings. Figures 1A to 1G An example user scenario of interacting with a mechanical switch according to some embodiments is shown. As described below, in Figures 1A to 1G 's example, a light press on the light switch 106 turns on one light, while a deep (forceful) press on the light switch 106 turns on multiple lights.

[0042] Figure 1A A user 102 wearing a wrist-wearable device 104 is shown, which may include some or all of the components of the wrist-wearable device 600. Figure 1A The user 102 in

[0043] Figure 1B stands near the light switch 106, which is communicatively coupled to the lights 108 in the room (e.g., the table lamps 108-1 and 108-3 and the ceiling lamp 108-2). The light switch 106 in this example does not include a force (intensity) sensing component. For example, the light switch 106 can be a mechanical switch. Figure 1B 's example, the user presses the light switch 106 using their index finger (pointer finger). Figure 1BAlso shown are the respective lights 108 in the off state. In some embodiments, the wrist-wearable device 104 includes one or more neuromuscular signal sensors. The one or more neuromuscular signal sensors are adapted to detect neuromuscular signals transmitted through the neuromuscular pathway. The neuromuscular signals have a relative amplitude corresponding to the amount of force applied (e.g., how much force is applied when pressing the light switch 106). In some embodiments, the light switch 106 determines and stores the activation time of each switch press. In some embodiments, the wrist-wearable device 104 determines and stores the timing information of the neuromuscular signal detection. In some embodiments, the system (e.g., the wrist-wearable device 104, the light switch 106, and / or another device) uses the timing information from the light switch 106 and the wrist-wearable device 104 to associate the switch press with the neuromuscular signal.

[0044] Figure 1C Shows Figure 1B A close-up view of the user 102 in pressing the light switch 106 (as indicated by the press 109). Figure 1C Also shown is the wrist-wearable device 104 communicating with the light switch 106, as shown by the line 111. In some embodiments, the wrist-wearable device 104 and the light switch 106 do not communicate directly (e.g., each is communicatively coupled to an intermediate device (e.g., a handheld intermediate processing device 800)). In some embodiments, the communication is initiated by the light switch 106 (e.g., the light switch 106 broadcasts a request for force information and the wrist-wearable device 104 responds). In some embodiments, the communication is initiated by the wrist-wearable device 104 (e.g., the wrist-wearable device 104 announces its force sensing capabilities and the light switch 106 responds). In some embodiments, the communication between the wrist-wearable device 104 and the light switch 106 determines that the wrist-wearable device 104 will provide force information for interacting with the light switch 106. In some embodiments, the communication is a handshake (e.g., establishing a Bluetooth and / or Wi-Fi connection) between the wrist-wearable device 104 and the light switch 106.

[0045] Figure 1CAlso shown is a graph 113 showing the state of the lamp switch 106, and a graph 115 showing the voltage signal from the neuromuscular signal sensor of the wrist-wearable device 104. Graph 113 shows that the lamp switch 106 is activated at time t0. For example, the signal 114 transitions from the '0' state to the '1' state according to the press 109. Graph 115 shows that the neuromuscular signal sensor of the wrist-wearable device 104 detects a voltage signal that has an increase corresponding to the activation force of the press 109 at t0. In some embodiments, the neuromuscular signal sensor is configured to monitor the muscle group corresponding to the user's index finger. For example, the neuromuscular signal sensor is arranged on the wrist-wearable device 104 to contact the corresponding part of the user's wrist corresponding to this muscle group. As Figure 1C shown, the press 109 is a light press because the voltage signal 116 does not meet or exceed the threshold f0.

[0046] Figure 1D Shows the lamp 108-1 that is turned on in response to Figure 1B and Figure 1C the press 109 shown in. In some embodiments, a light press on the lamp switch 106 causes the closest lamp to switch on / off. In some embodiments, one of the lamps 108 is designated as the main lamp, and the state of the main lamp is switched in response to a light press on the lamp switch 106. In some embodiments, the user can specify which one or more lamps are activated (or otherwise controlled) in response to a light press on the lamp switch 106.

[0047] Figure 1E Shows the user 102 pressing the lamp switch 106 with the arm wearing the wrist-wearable device 104. Figure 1E Also shown is that each lamp 108 is in the off state. Figure 1F Shows Figure 1E a close-up view of the user 102 in pressing the lamp switch 106 (as indicated by the press 119). Figure 1F Also shown is that the wrist-wearable device 104 communicates with the lamp switch 106, as shown by the line 111. In some embodiments, this communication is initiated based on the user 102 being near the lamp switch 106 (e.g., within the direct communication range of the lamp switch 106 and the wrist-wearable device 104).

[0048] Figure 1FAlso shown is a graph 121 showing the state of the lamp switch 106, and a graph 123 showing the voltage signal from the neuromuscular signal sensor of the wrist-wearable device 104. Graph 121 shows that the lamp switch 106 is activated at time t1. For example, the signal 122 transitions from the '0' state to the '1' state according to the press 119. Graph 123 shows that the neuromuscular signal sensor of the wrist-wearable device 104 detects a voltage signal 124 that has an increase corresponding to the activation force of the press 119 at t1. As Figure 1F shown, the press 119 is a deep press (sometimes also referred to as a forceful press) because the voltage signal 124 exceeds a threshold f0 during the press 119 (e.g., at time t1).

[0049] Figure 1G Shows the respective lamps 108 that are turned on in response to Figure 1E and Figure 1F the press 119 shown in. In some embodiments, a deep press on the lamp switch 106 causes all the lamps coupled to the lamp switch 106 (e.g., all the lamps in a room) to switch on / off. In some embodiments, a predetermined subset of the respective lamps 108 is activated in response to a deep press on the lamp switch 106. In some embodiments, the user is able to specify which one or more lamps are activated (or otherwise controlled) in response to a deep press on the lamp switch 106.

[0050] Figures 2A to 2D Illustrates an example user scenario of interacting with an electronic device according to some embodiments. Figure 2A Shows a user 102 wearing a wrist-wearable device 104 and a head-wearable device 204 (e.g., augmented reality glasses or a virtual reality headset, e.g., the artificial reality (AR) device 700 and the virtual reality (VR) device 710 described with reference to Figures 7A to 7C and standing near the electronic device 202. In some embodiments, the electronic device 202 does not include a display.

[0051] Figure 2B Shows Figure 2A a close-up view of the electronic device 202 shown in. Figure 2BAlso shown is the communication, represented by line 203, between the wrist-worn device 104 and the electronic device 202. In some embodiments, the communication is initiated by the electronic device 202 (e.g., the electronic device 202 broadcasts a request for force information and the wrist-worn device 104 responds). In some embodiments, the communication is initiated by the wrist-worn device 104 (e.g., the wrist-worn device 104 announces its force sensing capabilities and the electronic device 202 responds). In some embodiments, the communication between the wrist-worn device 104 and the electronic device 202 determines that the wrist-worn device 104 will provide force information for interaction with the electronic device 202. In some embodiments, the communication is a handshake (e.g., establishing a Bluetooth and / or Wi-Fi connection) between the wrist-worn device 104 and the electronic device 202. In some embodiments, the head-worn device 204 (and / or an intermediate device) communicates with the electronic device 202. For example, activation data from the electronic device is transmitted to the head-worn device 204 and then combined with force data from the wrist-worn device 104 to determine the amount of activation force at the electronic device.

[0052] Figure 2C Shown is the user interface 207 of the electronic device 202 presented to the user 102 via the head-worn device 204. In some embodiments, a similar user interface (e.g., in addition to or in place of the user interface 207) is presented via a display component of the electronic device 202. In some embodiments, a similar user interface (e.g., in addition to or in place of the user interface 207) is presented via a display component of the wrist-worn device 104. The user interface 207 includes interaction options 206. In some embodiments, the interaction options are based on sensors available to the system (e.g., based on whether a force sensor is available for use in determining a force component of an input at the electronic device 202). The interaction options 206 include option 206-1 and option 206-2. The option 206-1 is for resuming the playback of a playlist in response to a light press activation of the button 205, and the option 206-2 is for activating a digital assistant function in response to a deep press activation of the button 205.

[0053] Figure 2D Shown is the user interface 207 of the electronic device 202 presented to the user 102 via the head-worn device 204. In Figure 2DIn the example, user 102 does not have a device for measuring the input force (e.g., does not wear the wrist wearable device 104). Due to the lack of the force sensor, the user interface 207 includes interaction options 209 corresponding to the activation of the button 205. Due to the absence of the force sensing component, the interaction option 209 does not have a force component (e.g., does not correspond to a light press or a deep press). In some embodiments, force-based interaction options are presented in the user interface 207 (e.g., the force-based interaction options are grayed out or otherwise indicated as unavailable). In some embodiments, the electronic device 202 and / or the head wearable device 204 notify the user 102 of the force-based interaction (e.g., so that the user can decide to wear the wrist wearable device 104 or another force sensing device).

[0054] Figures 3A to 3D FIG. shows an example user scenario of interacting with a controller according to some embodiments. Figure 3A It shows user 102 wearing the wrist wearable device 104 and holding the controller 302. Figure 3A In, user 102 is sitting and playing a video game, where the video game scene 306 is presented on a display device 304 (e.g., a TV or a monitor). In Figures 3A to 3C the example, the video game responds to force-based input, but the controller 302 does not include a force sensor for determining the activation force of one or more button presses.

[0055] Figure 3B It shows Figure 3A a close-up view of the controller 302 in. Figure 3BAlso shown is the communication between the wrist-wearable device 104 and the controller 302, represented by line 303. In some embodiments, the controller 302 initiates the communication (e.g., the controller 302 broadcasts a request for force information and the wrist-wearable device 104 responds). In some embodiments, the communication is initiated by the wrist-wearable device 104 (e.g., the wrist-wearable device 104 announces its force-sensing capabilities and the controller 302 responds). In some embodiments, the communication between the wrist-wearable device 104 and the controller 302 determines that the wrist-wearable device 104 will provide force information to interact with the controller 302. In some embodiments, the communication is a handshake between the wrist-wearable device 104 and the controller 302 (e.g., establishing a Bluetooth and / or Wi-Fi connection). In some embodiments, the video game system (and / or intermediate device) communicates with the controller 302 and the wrist-wearable device 104. For example, button activation data from the controller is transmitted to the video game console and then combined with force data from the wrist-wearable device 104 received by the console to determine the amount of force for button presses and other controller interactions. In some embodiments, one or more communication channels are established between the wrist-wearable device 104 and the controller 302 and / or the video game console or video game system according to the power-on process.

[0056] Figure 3C Shown is a scene 307 including interaction options 308 presented to the user 102. The scene 307 is presented via the display device 304. In some embodiments, a similar user interface (e.g., in addition to or instead of the scene 307) is presented via a head-wearable device. In some embodiments, the interaction options are based on the sensors available to the system (e.g., based on whether a force sensor is available for use in determining the force component of an input at the controller 302). The interaction options 308 include option 308-1, which is for making the player jump according to the amount of force applied to the first button (labeled 'A') of the controller 302. The interaction options 308 also include option 308-2, which is for slowing down the pace of the game (e.g., controlling the speed at which time elapses in the video game) according to the amount of force applied to the second button (labeled 'R') of the controller 302.

[0057] Figure 3D Shown is a scene 307 of a video game displayed to the user 102 via the display device 304. In Figure 3DIn the example, user 102 does not have a device for measuring the input force (e.g., does not wear the wrist-wearable device 104). Based on the lack of the force sensor, scenario 307 includes interaction option 312 and notification 314. Based on the absence of the force-sensing component, interaction options 312-1 and 312-2 do not have a force component (e.g., do not have different functions for different input forces). Notification 314 notifies user 102 that using a force-sensing device (e.g., the wrist-wearable device 104 or the force-sensing controller) will allow the user to access / perform additional interactions in the game (e.g., as Figure 3C shown).

[0058] Additionally, although the user scenarios described with reference to a series of FIGS. 1 to 3 are described as separate sequences, in some embodiments, the user scenarios are combined with each other. For example, the sequence described with reference to Figures 1A to 1F can occur before (or after) the sequences described with reference to Figures 2A to 2D and Figures 3A to 3D (e.g., all three sequences can occur when user 102 interacts with the devices in his home).

[0059] The user scenarios described with reference to a series of FIGS. 1 to 3 relate to specific user interfaces and applications, such as the user interfaces in Figure 2C and Figure 2D and the video game scenarios in Figure 3C and Figure 3D . However, these sequences, gestures, actions, and operations can be used in combination with other types of user interfaces, menus, and applications, such as web browsing, note-taking, social media, word processing, data entry, and programming, etc.

[0060] Figure 4 FIG. is a flowchart showing a method 400 for providing a force dimension to an interface element according to some embodiments. Method 400 is executed at a computing system (e.g., a wearable device or an intermediate device) having one or more processors and a memory. In some embodiments, the memory stores one or more programs, and the one or more programs are configured to be executed by the one or more processors. Figure 4 At least some of the multiple operations shown in correspond to instructions stored in a computer memory or a computer-readable storage medium (e.g., the memory 650 of the computer system 630). In some embodiments, the computing system is a wearable device such as the wrist-wearable device 104. In some embodiments, the computing system is or includes an intermediate device such as a smartphone, a personal computer, or a video game console.

[0061] The system communicatively couples (402) a wearable device (e.g., wrist wearable device 104) to an activatable device (e.g., electronic device 202), the wearable device including one or more neuromuscular signal sensors (e.g., one or more electromyography sensors), and the activatable device including a mechanical user interface element (e.g., button 205). In some embodiments, the wearable device is a wrist wearable device (e.g., wrist wearable device 600). In some embodiments, the mechanical user interface element is a switch (e.g., light switch 106), a joystick, or a button (e.g., button 205).

[0062] In some embodiments, the activatable device does not include a display. For example, Figures 2A to 2D electronic device 202 in does not include a display. In some embodiments, the activatable device includes one or more indicator lights but does not include a screen.

[0063] In some embodiments, (i) the mechanical user interface element is a first mechanical user interface element; (ii) the activatable device includes a second mechanical user interface element; and (iii) the second mechanical user interface element is associated with one or more predefined force criteria that are different from the one or more predefined criteria associated with the first mechanical user interface element. For example, a first button on controller 302 may have a different force threshold than a second button on controller 302. In some embodiments, the force criteria for a particular user interface element are based on the type of user interface element (e.g., button, joystick, or switch).

[0064] In some embodiments, the wearable device is communicatively coupled to the activatable device based on the wearable device being in the vicinity of the activatable device. For example, the wearable device is coupled to the activatable device via a local area network (LAN) or a direct wireless connection.

[0065] The system obtains (404) data (e.g., data regarding voltage signal 124) from the one or more neuromuscular signal sensors in response to activation of the mechanical user interface element (e.g., in response to Figure 1F light switch 106 in transitioning to the '1' state), the data corresponding to the activation.

[0066] In some embodiments, the system provides information about a first function and a second function to the user (e.g., user interface 207 with interaction options 206) prior to detecting activation of the mechanical user interface element. In some embodiments, the information about the first function and the second function is displayed to the user via a head wearable device (e.g., head wearable device 204).

[0067] Based on data from the one or more neuromuscular signal sensors, the system determines (406) whether the activation of the mechanical user interface element includes an activation force that meets one or more predefined criteria. For example, the system determines whether the activation is a light press or a deep press based on whether the associated activation force exceeds a preset threshold.

[0068] Based on determining that the activation of the mechanical user interface element includes the activation force that meets one or more predefined criteria, the system causes (408) a first function corresponding to the mechanical user interface element to be executed. For example, as Figures 1B to 1D shown, the first function can be to turn on the nearest light, or as Figure 2C shown, the first function can be to resume a playlist.

[0069] Based on determining that the activation of the mechanical user interface element does not include the activation force that meets one or more predefined criteria, the system causes (410) a second function corresponding to the mechanical user interface element to be executed, where the second function is different from the first function. For example, as Figures 1E to 1G shown, the second function can be to turn on a group of lights, or as Figure 2C shown, the second function can be to activate a digital assistant function.

[0070] In some embodiments, the system receives a request for force data from an activatable device at the wearable device, where the wearable device is communicatively coupled to the activatable device in response to the request. In some embodiments, the system receives a broadcast communication from the wearable device at the activatable device; the broadcast communication identifies the force capabilities of the wearable device, where in response to the broadcast communication, the wearable device is communicatively coupled to the activatable device.

[0071] In some embodiments, the system provides a notification to the user of the wearable device; the notification includes information about being communicatively coupled to the activatable device and / or information about providing data from the one or more neuromuscular signal sensors. For example, Figure 3D shows scenario 307 including notification 314.

[0072] In some embodiments, the system (i) communicatively couples the wearable device to a second activatable device (e.g., controller 302) that includes a second mechanical user interface element (e.g., a controller button); (ii) obtains additional data from the one or more neuromuscular signal sensors corresponding to an activation of the second mechanical user interface element; (iii) determines, based on the additional data from the one or more neuromuscular signal sensors, whether the activation of the second mechanical user interface element includes a corresponding activation force that meets one or more predefined criteria; (iv) causes a first function corresponding to the second mechanical user interface element to be performed based on determining that the activation of the second mechanical user interface element includes the corresponding activation force that meets the one or more predefined criteria; and (v) causes a second function corresponding to the second mechanical user interface element to be performed based on determining that the activation of the second mechanical user interface element does not include the corresponding activation force that meets the one or more predefined criteria.

[0073] Having described an example sequence and a method of operation using the example sequence in this manner, attention is now turned to a system-level description of the hardware and software on which (using which) the method may be implemented.

[0074] The above-described devices, which include a system, a wrist wearable device, a head-mounted viewer device, and textile-based smart apparel, are described in more detail below. The specific operations described above may occur as a result of specific hardware, which is described in more detail below. The devices described below are not restrictive, and features may be removed from these devices or additional features may be added to these devices. Different devices may include one or more similar hardware components. For the sake of brevity, similar devices and components are described below. Any differences between the devices and components are described in their respective sections below.

[0075] As described herein, a processor (e.g., a central processing unit (CPU) or a microcontroller unit (MCU)) is an electronic component responsible for executing instructions and controlling the operation of an electronic device (e.g., a wrist wearable device 600, a head wearable device, a handheld intermediate processing device (HIPD) 800, a textile-based smart garment 900, or other computer system). There are various types of processors, which can be used interchangeably or are specifically required by the various embodiments described herein. For example, the processor can be: (i) a general-purpose processor, which is designed to execute a wide range of tasks, such as running software applications, managing an operating system, and performing arithmetic and logical operations; (ii) a microcontroller, which is designed for specific tasks, such as controlling an electronic device, sensors, and motors; (iii) a graphics processing unit (GPU), which is designed to accelerate the creation and rendering of images, videos, and animations (e.g., virtual reality animations, such as 3D modeling); (iv) a field-programmable gate array (FPGA), which can be programmed and reconfigured after manufacturing and / or customized to perform specific tasks, such as signal processing, encryption, and machine learning; (v) a digital signal processor (DSP), which is designed to perform mathematical operations on signals (e.g., audio, video, and radio waves). Those skilled in the art will understand that one or more processors of one or more electronic devices can be used in the various embodiments described herein.

[0076] As described herein, a controller is an electronic component that manages and coordinates the operation of other components within an electronic device (e.g., controls inputs, processes data, and / or generates outputs). Examples of controllers can include: (i) a microcontroller, which includes a small, low-power controller commonly used in embedded systems and Internet of Things (IoT) devices; (ii) a programmable logic controller (PLC), which can be configured for use in industrial automation systems to control and monitor manufacturing processes; (iii) a system-on-a-chip (SoC) controller, which integrates multiple components such as a processor, memory, input / output (I / O) interfaces, and other peripherals into a single chip; and / or a DSP. As described herein, a graphics module is a component or software module designed to process graphics operations and / or graphics processes, and the graphics module can include a hardware module and / or a software module.

[0077] As described herein, a memory refers to an electronic component in a computer or electronic device that stores data and instructions for access and operation by a processor. The devices described herein may include volatile memory and non-volatile memory. Examples of memory may include: (i) random access memory (RAM) configured to temporarily store data and instructions, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state storage devices; (ii) read-only memory (ROM) configured to permanently store data and instructions (e.g., one or more portions in system firmware and / or a boot loader); (iii) flash memory, disk storage devices, optical disk storage devices, other non-volatile solid-state storage devices (e.g., universal serial bus (USB) drives, memory cards, and / or solid-state drives (SSD)) that may be configured to store data in an electronic device; and (iv) cache memory configured to temporarily store frequently accessed data and instructions. As described herein, memory may include structured data (e.g., Structured Query Language (SQL) databases, MongoDB databases, GraphQL data, or JSON data). Other examples of memory may include: (i) profile data that includes user account data, user settings, and / or other user data stored by a user; (ii) sensor data detected and / or otherwise obtained by one or more sensors; (iii) media content data that includes stored image data, audio data, and documents, etc.; (iv) application data that may include data collected and / or otherwise obtained and stored during the use of an application; and / or any other type of data described herein.

[0078] As described herein, the power system of an electronic device is configured to convert input power into a form that can be used to operate the device. The power system can include various components, including: (i) a power source, which can be an alternating current (AC) adapter power source or a direct current (DC) adapter power source; (ii) a charger input, which can be configured to use a wired connection and / or a wireless connection (the charger input can be part of a peripheral interface, such as a USB, micro-USB interface, near-field magnetic coupling, magnetic induction and magnetic resonance charging, and / or radio frequency (RF) charging); (iii) a power management integrated circuit, which is configured to distribute power to the various components of the device and ensure that the device operates within safe limits (e.g., regulate voltage, control current, and / or manage heat dissipation); and / or (iv) a battery, which is configured to store power to provide available power to the components of one or more electronic devices.

[0079] As described herein, a peripheral interface is an electronic component (e.g., an electronic component of an electronic device) that allows an electronic device to communicate with other devices or peripheral devices and can provide a means for inputting and outputting data and signals. Examples of peripheral interfaces can include: (i) a USB interface and / or a micro-USB interface, which are configured to connect a device to an electronic device; (ii) a Bluetooth interface, which is configured to allow multiple devices to communicate with each other, including Bluetooth low energy (BLE); (iii) a near-field communication (NFC) interface, which is configured as a short-range wireless interface for operations such as access control; (iv) a POGO pin, which can be a small, spring-loaded pin configured to provide a charging interface; (v) a wireless charging interface; (vi) a global-position system (GPS) interface; (vii) a Wi-Fi interface, which is used to provide a connection between the device and a wireless network; and (viii) a sensor interface.

[0080] As described herein, a sensor is an electronic component (e.g., an electronic component in an electronic device (such as a wearable device) and / or an electronic component that otherwise communicates electronically with an electronic device) configured to detect physical and environmental changes and generate an electrical signal. Examples of sensors can include: (i) an imaging sensor for collecting imaging data (e.g., including one or more cameras disposed on a corresponding electronic device); (ii) a bioelectrical signal sensor; (iii) an inertial measurement unit (e.g., an IMU) for detecting, e.g., angular velocity, force, magnetic field, and / or acceleration changes; (iv) a heart rate sensor for measuring a user's heart rate; (v) a blood oxygen saturation (SpO2) sensor for measuring a user's blood oxygen saturation and / or other biometric data; (vi) a capacitive sensor for detecting potential changes at a location on a user's body (e.g., a sensor-skin interface) and / or the proximity of other devices or objects; and (vii) a light sensor (e.g., a time-of-flight sensor, an infrared light sensor, or a visible light sensor), and / or a sensor for sensing data from a user or the user's environment. As described herein, a bioelectrical signal sensing component is a device (e.g., a bioelectrical signal sensor) for measuring electrical activity within the body. Some types of bioelectrical signal sensors include: (i) an electroencephalography (EEG) sensor configured to measure electrical activity in the brain to diagnose neurological disorders; (ii) an electrocardiography (ECG or EKG) sensor configured to measure electrical activity of the heart to diagnose heart problems; (iii) an electromyography (EMG) sensor configured to measure electrical activity of muscles and diagnose neuromuscular disorders; (iv) an electrooculography (EOG) sensor configured to measure electrical activity of eye muscles to detect eye movements and diagnose eye disorders.

[0081] As described herein, an application (e.g., software) stored in the memory of an electronic device includes instructions stored in the memory. Examples of such applications include: (i) games; (ii) word processors; (iii) messaging applications; (iv) media streaming applications; (v) financial applications; (vi) calendars; (vii) clocks; (viii) web browsers; (ix) social media applications; (x) camera applications; (xi) web-based applications; (xii) health applications; (xiii) artificial-reality (AR) applications, and / or any other application that can be stored in the memory. The application can operate in conjunction with data and / or one or more components of one or more devices that are device- or communication-coupled to perform one or more operations and / or functions.

[0082] As described herein, the communication interface module can include hardware and / or software capable of data communication using any of the following various protocols: custom wireless protocols or standard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.11a, WirelessHART, or MiWi); custom wired protocols or standard wired protocols (e.g., Ethernet or HomePlug); and / or any other suitable communication protocol (including communication protocols not yet developed as of the filing date of this document). The communication interface is an agency that enables different systems or devices to exchange information and data with each other, and it includes hardware, software, or a combination of both. For example, the communication interface can refer to a physical connector and / or port on a device that enables communication with other devices (e.g., USB, Ethernet, High-Definition Multimedia Interface (HDMI), or Bluetooth). In some embodiments, the communication interface can be a software layer that enables different software programs to communicate with each other (e.g., an application programming interface (API) and protocols such as Hypertext Transfer Protocol (HTTP) and Transmission Control Protocol / Internet Protocol (TCP / IP)).

[0083] As described herein, the graphics module is a component or software module designed to process graphics operations and / or graphical processes, and the graphics module can include a hardware module and / or a software module.

[0084] As described herein, a non-transitory computer-readable storage medium is a physical device or storage medium that can be used to store electronic data in a non-transitory form (e.g., such that the data is stored permanently until the data is intentionally deleted or modified).

[0085] Example AR system

[0086] Figure 5A , Figure 5B , Figure 5C-1 , Figure 5C-2 , Figure 5D-1 and Figure 5D-2 An example AR system is shown in accordance with some embodiments. Figure 5A A first AR system 500a and a first example user interaction using a wrist wearable device 600 , a head wearable device (eg, AR device 700 ), and / or a handheld intermediary processing device (HIPD) 800 are shown. Figure 5B A second AR system 500 b is shown, along with a second example user interaction using the wrist wearable device 600 , the AR device 700 , and / or the HIPD 800 . Figure 5C-1 and Figure 5C-2 A third AR system 500 c and a third example user interaction using a wrist wearable device 600 , a head wearable device (eg, a virtual-reality (VR) device 710 ), and / or a HIPD 800 are shown. Figure 5D-1 and Figure 5D-2 A fourth AR system 500d and a fourth example user interaction are shown, which uses a wrist wearable device 600, a VR device 710, and / or a textile-based smart garment 900 (e.g., a wearable glove, a haptic glove). As will be understood by those skilled in the art upon reading the description provided herein, the above example AR system (described in detail below) can perform the above reference Figures 1A to 3D Various functions and / or operations are described.

[0087] The following references Figure 6A and Figure 6B Describes the wrist wearable device 600 and its components; the following reference Figure 7A FIG. 7D describes the head wearable device and its components; and the following reference Figure 8A and Figure 8B The HIPD 800 and its components are described below. Figures 9A to 9CDescribes a textile-based smart garment 900 and one or more of its components. The wrist-wearable device 600, the head-wearable device, and / or the HIPD 800 may be communicatively coupled via a network 525 (e.g., cellular, near-field, Wi-Fi, personal area network, or wireless local area network (LAN)). Additionally, the wrist-wearable device 600, the head-wearable device, and / or the HIPD 800 may also be communicatively coupled via the network 525 (e.g., cellular, near-field, Wi-Fi, personal area network, or wireless LAN) to one or more servers 530, a computer 540 (e.g., a laptop computer or a computer), a mobile device 550 (e.g., a smartphone or a tablet), and / or other electronic devices. Similarly, when in use, the textile-based smart garment 900 may also be communicatively coupled via the network 525 to the wrist-wearable device 600, the head-wearable device, the HIPD 800, one or more servers 530, the computer 540, the mobile device 550, and / or other electronic devices.

[0088] Go to Figure 5A , shows a user 502 wearing the wrist-wearable device 600 and the AR device 700, and the user places the HIPD 800 on their table. The wrist-wearable device 600, the AR device 700, and the HIPD 800 facilitate the user's interaction with the AR environment. Specifically, as shown in the first AR system 500a, the wrist-wearable device 600, the AR device 700, and / or the HIPD 800 cause one or more avatars 504, digital representations of contacts 506, and virtual objects 508 to be presented. As discussed below, the user 502 may interact with one or more avatars 504, digital representations of contacts 506, and virtual objects 508 via the wrist-wearable device 600, the AR device 700, and / or the HIPD 800.

[0089] The user 502 may use any one of the wrist-wearable device 600, the AR device 700, and / or the HIPD 800 to provide user input. For example, the user 502 may perform one or more gestures to provide user input, and the one or more gestures are detected by the wrist-wearable device 600 (e.g., using one or more EMG sensors and / or IMUs described with reference to Figure 6A and Figure 6B and / or by the AR device 700 (e.g., using one or more described with reference to Figure 7ADetection by one or more image sensors or cameras (described with reference to FIGS. 7A and 7B). Alternatively or additionally, user 502 may provide user input via one or more touch surfaces of wrist wearable device 600, AR device 700, and / or HIPD 800, and / or voice commands captured by microphones of wrist wearable device 600, AR device 700, and / or HIPD 800. In some embodiments, wrist wearable device 600, AR device 700, and / or HIPD 800 include a digital assistant for assisting the user in providing user input (e.g., completing a series of operations, recommending different operations or commands, providing reminders or confirming commands). In some embodiments, user 502 may provide user input via one or more facial gestures and / or facial expressions. For example, the cameras of wrist wearable device 600, AR device 700, and / or HIPD 800 may track the eyes of user 502 for navigating the user interface.

[0090] Wrist wearable device 600, AR device 700, and / or HIPD 800 may operate alone or in combination to allow user 502 to interact with the AR environment. In some embodiments, HIPD 800 is configured to operate as a central hub or control center for wrist wearable device 600, AR device 700, and / or another communicatively coupled device. For example, user 502 may provide input for interacting with the AR environment at any one of wrist wearable device 600, AR device 700, and / or HIPD 800, and HIPD 800 may identify one or more back-end tasks and front-end tasks to effectuate the requested interaction and distribute instructions for performing the one or more back-end tasks and front-end tasks at wrist wearable device 600, AR device 700, and / or HIPD 800. In some embodiments, back-end tasks are background processing tasks not perceptible to the user (e.g., rendering content, decompressing or compressing), while front-end tasks are user-perceptible user-facing tasks (e.g., presenting information to the user or providing feedback to the user). As described with reference to Figure 8A and Figure 8B HIPD 800 may perform back-end tasks and provide operation data corresponding to the performed back-end tasks to wrist wearable device 600 and / or AR device 700 such that wrist wearable device 600 and / or AR device 700 may perform front-end tasks. In this way, HIPD 800 (which has more computing resources and greater thermal headroom than wrist wearable device 600 and / or AR device 700) performs computationally intensive tasks and reduces the computer resource utilization and / or power usage of wrist wearable device 600 and / or AR device 700.

[0091] In the example shown by the first AR system 500a, the HIPD 800 identifies one or more backend tasks and frontend tasks associated with a user request to initiate an AR video call with one or more other users (represented by the avatar 504 and the digital representation 506 of the contact), and distributes instructions to cause the execution of the one or more backend tasks and frontend tasks. Specifically, the HIPD 800 performs backend tasks for processing and / or rendering image data (and other data) associated with the AR video call, and provides the AR device 700 with operation data associated with the performed backend tasks, such that the AR device 700 performs frontend tasks for presenting the AR video call (e.g., presenting the avatar 504 and the digital representation 506 of the contact).

[0092] In some embodiments, the HIPD 800 may operate as a focus or anchor for information presentation. This allows the user 502 to know where the information is presented. For example, as shown by the first AR system 500a, the avatar 504 and the digital representation 506 of the contact are presented above the HIPD 800. Specifically, the HIPD 800 and the AR device 700 operate in combination to determine the location for presenting the avatar 504 and the digital representation 506 of the contact. In some embodiments, information may be presented within a predetermined distance from the HIPD 800 (e.g., within five meters). For example, as shown by the first AR system 500a, the virtual object 508 is presented on a table at a certain distance from the HIPD 800. Similar to the above example, the HIPD 800 and the AR device 700 may operate in combination to determine the location for presenting the virtual object 508. Alternatively, in some embodiments, the presentation of information is not restricted by the HIPD 800. More specifically, the avatar 504, the digital representation 506 of the contact, and the virtual object 508 do not have to be presented within a predetermined distance from the HIPD 800.

[0093] Coordinate user inputs provided at the wrist-worn device 600, the AR device 700, and / or the HIPD 800, such that the user can use any device to initiate, continue, and / or complete an operation. For example, the user 502 may provide a user input to the AR device 700 to cause the AR device 700 to present the virtual object 508, and while the AR device 700 is presenting the virtual object 508, the user 502 may provide one or more gestures via the wrist-worn device 600 to interact with and / or manipulate the virtual object 508.

[0094] Figure 5BShows a user 502 wearing a wrist-wearable device 600 and an AR device 700, and holding a HIPD 800. In the second AR system 500b, the wrist-wearable device 600, the AR device 700, and / or the HIPD 800 are used to receive one or more messages and / or provide one or more messages to the contacts of the user 502. Specifically, the wrist-wearable device 600, the AR device 700, and / or the HIPD 800 detect and coordinate one or more user inputs to initiate a messaging application and prepare a response to the received messages via the messaging application.

[0095] In some embodiments, the user 502 initiates an application on the wrist-wearable device 600, the AR device 700, and / or the HIPD 800 via a user input, which causes the application to be launched on at least one device. For example, in the second AR system 500b, the user 502 performs a gesture associated with a command for launching a messaging application (represented by the messaging user interface 512); the wrist-wearable device 600 detects the gesture and, based on determining that the user 502 is wearing the AR device 700, causes the AR device 700 to present the messaging user interface 512 of the messaging application. The AR device 700 can present the messaging user interface 512 to the user 502 via its display (e.g., as shown in the field of view 510 of the user 502). In some embodiments, the application is launched and can run on the device that detected the user input for launching the application (e.g., the wrist-wearable device 600, the AR device 700, and / or the HIPD 800), and this device provides operation data to another device to cause the presentation of the messaging application. For example, the wrist-wearable device 600 can detect a user input for launching the messaging application, launch and run the messaging application, and provide operation data to the AR device 700 and / or the HIPD 800 to cause the presentation of the messaging application. Alternatively, the application can be launched and run on a device different from the device that detected the user input. For example, the wrist-wearable device 600 can detect a gesture associated with launching the messaging application and can cause the HIPD 800 to run the messaging application and coordinate the presentation of the messaging application.

[0096] In addition, user 502 may provide user input provided at wrist-worn device 600, AR device 700, and / or HIPD 800 to continue and / or complete an operation initiated at another device. For example, after starting a messaging application via wrist-worn device 600 and while AR device 700 presents a messaging user interface 512, user 502 may provide input at HIPD 800 to prepare a response (e.g., as indicated by a swipe gesture performed on HIPD 800). A gesture performed by user 502 on HIPD 800 may be provided and / or displayed on another device. For example, a swipe gesture performed by user 502 on HIPD 800 is displayed on a virtual keyboard of the messaging user interface 512 displayed by AR device 700.

[0097] In some embodiments, wrist-worn device 600, AR device 700, HIPD 800, and / or other communicatively coupled devices may present one or more notifications to user 502. The notification may be an indication of a new message, incoming call, application update, status update, etc. User 502 may select the notification via wrist-worn device 600, AR device 700, or HIPD 800 and cause an application or operation associated with the notification to be presented on at least one device. For example, user 502 may receive a notification of a received message at wrist-worn device 600, AR device 700, HIPD 800, and / or other communicatively coupled devices, provide user input at wrist-worn device 600, AR device 700, and / or HIPD 800 to view the notification, and the device that detects the user input may cause the application associated with the notification to be launched and / or the application associated with the notification to be presented at wrist-worn device 600, AR device 700, and / or HIPD 800.

[0098] Although the above examples describe coordinated input for interacting with a messaging application, those skilled in the art will understand upon reading this description that user input may be coordinated to interact with any number of applications, including but not limited to gaming applications, social media applications, camera applications, web-based applications, and financial applications, etc. For example, AR device 700 may present gaming application data to user 502, and HIPD 800 may use a controller to provide input to the game. Similarly, user 502 may use wrist-worn device 600 to start the camera of AR device 700, and the user may use wrist-worn device 600, AR device 700, and / or HIPD 800 to manipulate image capture (e.g., zoom in or out, or apply filters) and capture image data.

[0099] Moving on to Figure 5C-1 and Figure 5C-2, shows a user 502 wearing a wrist-worn device 600 and a VR device 710, and holding a HIPD 800. In the third AR system 500c, the wrist-worn device 600, the VR device 710, and / or the HIPD 800 are used to interact within an AR environment (such as a VR game or other AR application). Although the VR device 710 presents a representation of a VR game to the user 502 (e.g., the first AR game environment 520), the wrist-worn device 600, the VR device 710, and / or the HIPD 800 detect and coordinate one or more user inputs to allow the user 502 to interact with the VR game.

[0100] In some embodiments, the user 502 may provide user inputs that cause actions in the corresponding AR environment via the wrist-worn device 600, the VR device 710, and / or the HIPD 800. For example, the user 502 in the third AR system 500c (as Figure 5C-1 shown) raises the HIPD 800 in preparation for a swing in the first AR game environment 520. In response to the user 502 raising the HIPD 800, the VR device 710 causes the user's AR representation 522 to perform a similar action (e.g., raise a virtual object, such as a virtual sword 524). In some embodiments, each device uses corresponding sensor data and / or image data to detect user inputs and provide an accurate representation of the movement of the user 502. For example, the image sensor 854 of the HIPD 800 (e.g., the simultaneous localization and mapping (SLAM) camera or other camera discussed below in Figure 8A and Figure 8B ) can be used to detect the position of the 800 relative to the body of the user 502 such that the virtual object can be appropriately positioned within the first AR game environment 520; the sensor data from the wrist-worn device 600 can be used to detect the speed at which the user 502 raises the HIPD 800 such that the user's AR representation 522 and the virtual sword 524 are synchronized with the movement of the user 502; and the image sensor 726 of the VR device 710 ( Figures 7A to 7C ) can be used to represent the body of the user 502, the boundary conditions, or real-world objects within the first AR game environment 520.

[0101] In Figure 5C-2In this case, user 502 performs a downward swing while holding the HIPD 800. The wrist-wearable device 600, the VR device 710, and / or the HIPD 800 detect the downward swing of user 502 and perform corresponding actions in the first AR game environment 520. In some embodiments, the data collected by each device is used to enhance the user's experience within the AR environment. For example, the sensor data of the wrist-wearable device 600 can be used to determine the speed and / or force of the downward swing, and the image sensors of the HIPD 800 and / or the VR device 710 can be used to determine the position of the swing and how the swing should be represented in the first AR game environment 520, which in turn can be used as an input to the AR environment (e.g., a game mechanism that can classify the user's input (e.g., the user performs a tap, a heavy strike, a critical strike, a glancing strike, a miss) using the detected speed, force, position, and / or aspects of the actions of user 502 or can calculate an output (e.g., amount of damage)).

[0102] Although the wrist-wearable device 600, the VR device 710, and / or the HIPD 800 are described as detecting user input, in some embodiments, the user input is detected at a single device (where the single device is responsible for distributing signals to other devices for performing the user input). For example, the HIPD 800 can run an application for generating the first AR game environment 520, provide corresponding data for presenting the first AR game environment 520 to the VR device 710, and detect the movement of 502 (while holding the HIPD 800) such that corresponding actions are performed within the first AR game environment 520. Additionally or alternatively, in some embodiments, the operation data (e.g., sensor data, image data, application data, device data, and / or other data) of one or more devices is provided to a single device (e.g., the HIPD 800) to process the operation data and cause the corresponding device to perform actions associated with the processed operation data.

[0103] In Figure 5D-1 and Figure 5D-2 this case, user 502 wearing the wrist-wearable device 600, the VR device 710, and the textile-based smart garment 900 is shown. In the fourth AR system 500d, the wrist-wearable device 600, the VR device 710, and / or the textile-based smart garment 900 are used in an AR environment (e.g., as referenced above Figures 5A to 5C-2 and Figures 1A to 2DInteract within any of the described AR systems). Although the VR device 710 presents a representation of a VR game (e.g., the second AR game environment 531) to the user 502, the wrist-wearable device 600, the VR device 710, and / or the textile-based smart garment 900 detect and coordinate one or more user inputs to allow the user 502 to interact with the AR environment.

[0104] In some embodiments, the user 502 may provide user inputs that cause actions in the corresponding AR environment via the wrist-wearable device 600, the VR device 710, and / or the textile-based smart garment 900. For example, the user 502 in the fourth AR system 500d (as Figure 5D-1 shown) raises the hand wearing the textile-based smart garment 900 to prepare to cast a spell or throw an object within the second AR game environment 531. The VR device 710 responds to the user 502 raising their (wearing the textile-based smart garment 900) hand such that the user's AR representation 522 performs a similar action (e.g., holding a virtual object or throwing a fireball 534). In some embodiments, each device uses the corresponding sensor data and / or image data to detect the user input and provide an accurate representation of the user 502's movement.

[0105] In Figure 5D-2 , the user 502 performs a throwing action while wearing the textile-based smart garment 900. The wrist-wearable device 600, the VR device 710, and / or the textile-based smart garment 900 detect the user 502's throwing action and perform a corresponding action in the second AR game environment 531. As described above, the data collected by each device is used to enhance the user's experience within the AR environment. Although not shown, the textile-based smart garment 900 may be used in combination with the AR device 710 and / or the HIPD 800.

[0106] Example AR systems, devices for interacting with such AR systems, and other computing systems have been discussed rather generally. Now, the devices and components will be discussed in more detail below. For ease of reference, some definitions of the following devices and components are defined here: These devices and components may be included in some or all of the example devices discussed below. Those skilled in the art will understand that certain types of components described below may be more suitable for a particular set of devices and less suitable for a different set of devices. However, subsequent references to the components defined herein should be considered to be covered by the provided definitions.

[0107] In some of the embodiments discussed below, a number of example devices and systems including electronic devices and systems will be discussed. Such example devices and systems are not intended to be limiting, and those skilled in the art will understand that alternative devices and systems to the example devices and systems described herein can be used to perform the operations described herein and to construct the systems and devices described herein.

[0108] As described herein, an electronic device is a device that uses electrical energy to perform a specific function. The electronic device can be any physical object that includes electronic components (such as transistors, resistors, capacitors, diodes, and integrated circuits). Examples of electronic devices include smartphones, laptop computers, digital cameras, televisions, game consoles, and music players, as well as the various example electronic devices discussed herein. As described herein, an intermediate electronic device is a device that is located between two other electronic devices and / or between a subset of the components of one or more electronic devices, and that facilitates communication, and / or data processing, and / or data transfer between the corresponding electronic devices and / or electronic components.

[0109] Example wrist-wearable device

[0110] Figure 6A and Figure 6B shows an example wrist-wearable device 600 in accordance with some embodiments. The wrist-wearable device 600 is an instance of the wrist-wearable device 104 described herein, such that the wrist-wearable device should be understood to have the features of the wrist-wearable device 600, and vice versa. Figures 1A to 3D shows a plurality of components of the wrist-wearable device 600, which can be used alone or in combination, including combinations that include other electronic devices and / or electronic components. Figure 6A

[0111] As discussed below, Figure 6A shows the wearable band 610 and the watch body 620 (or capsule) coupled to form the wrist-wearable device 600. The wrist-wearable device 600 can perform various functions and / or operations associated with navigating in a user interface and selectively launching applications, as well as the functions and / or operations described above with reference to Figures 1A to 4 described.

[0112] ​As will be described in more detail below, the operations performed by the wrist-wearable device 600 may include: (i) presenting content to the user (e.g., displaying visual content via the display 605); (ii) detecting (e.g., sensing) user input (e.g., sensing a touch on the peripheral button 623 and / or a touch at the touchscreen of the display 605, sensing a gesture detected by a sensor (e.g., a biopotential sensor)); (iii) sensing biometric data (e.g., neuromuscular signals, heart rate, temperature, or sleep) via one or more sensors 613; messaging (e.g., text, voice, or video); image capture via one or more imaging devices or cameras 625; wireless communication (e.g., cellular, near field, Wi-Fi, or personal area network); location determination; financial transactions; providing tactile feedback; alerts; notifications; biometric authentication; health monitoring; and / or sleep monitoring.

[0113] The above example functions may be performed independently in the watch body 620, independently in the wearable band 610, and / or via electronic communication between the watch body 620 and the wearable band 610. In some embodiments, when presenting an AR environment (e.g., via one of the AR systems 500a to 500d), the various functions may be performed on the wrist-wearable device 600. As those skilled in the art will understand upon reading the description provided herein, the novel wearable device described herein may be used with other types of AR environments.

[0114] The wearable band 610 may be configured to be worn by the user such that the inner (or medial) surface of the wearable structure 611 of the wearable band 610 contacts the user's skin. When worn by the user, the sensor 613 contacts the user's skin. The sensor 613 may sense biometric data, such as the user's heart rate, saturation oxygen level, temperature, sweat level, neuromuscular signal sensor, or a combination thereof. The sensor 613 may also sense data about the user's environment, including the user's movement, altitude, location, orientation, gait, acceleration, position, or a combination thereof. In some embodiments, the sensor 613 is configured to track the position and / or movement of the wearable band 610. One or more sensors 613 may include any of the sensors defined above and / or discussed below with reference to Figure 6B any of the multiple sensors discussed.

[0115] One or more sensors 613 may be distributed on the inner surface and / or the outer surface of the wearable band 610. In some embodiments, one or more sensors 613 are evenly spaced along the wearable band 610. Alternatively, in some embodiments, one or more sensors 613 are located at different points along the wearable band 610. As Figure 6AAs shown, one or more sensors 613 can be the same or different. For example, in some embodiments, one or more sensors 613 can be shaped like a pill (e.g., sensor 613a), oval, round, square, elliptical (e.g., sensor 613c), and / or any other shape that maintains contact with the user's skin (e.g., such that neuromuscular signals and / or other biometric data can be accurately measured at the user's skin). In some embodiments, one or more sensors 613 are aligned to form sensor pairs (e.g., for sensing neuromuscular signals based on differential sensing within each respective sensor). For example, sensor 613b is aligned with an adjacent sensor to form sensor pair 614a, and sensor 613d is aligned with an adjacent sensor to form sensor pair 614b. In some embodiments, the wearable band 610 does not have sensor pairs. Alternatively, in some embodiments, the wearable band 610 has a predetermined number of sensor pairs (one sensor pair, three sensor pairs, four sensor pairs, six sensor pairs, or sixteen sensor pairs).

[0116] The wearable band 610 can include any suitable number of sensors 613. In some embodiments, the number and arrangement of sensors 613 depend on the particular application for which the wearable band 610 is used. For example, a wearable band 610 configured as an armband, wristband, or chest band can include multiple sensors 613, with each use case (e.g., a medical use case) having a different number of sensors 613 and a different arrangement compared to a gaming use case or a general daily use case.

[0117] According to some embodiments, the wearable band 610 further includes an electrical ground electrode and a shielding electrode. Similar to the sensors 613, the electrical ground electrode and the shielding electrode can be distributed on the inner surface of the wearable band 610 such that they contact a portion of the user's skin. For example, the electrical ground electrode and the shielding electrode can be located at the inner surface of the coupling mechanism 616 or at the inner surface of the wearable structure 611. The electrical ground electrode and the shielding electrode can be formed and / or use the same components as the sensors 613. In some embodiments, the wearable band 610 includes more than one electrical ground electrode and more than one shielding electrode.

[0118] The sensor 613 can be formed as part of a wearable structure 611 of the wearable band 610. In some embodiments, the sensor 613 is flush or substantially flush with the wearable structure 611 such that the sensors do not extend beyond the surface of the wearable structure 611. Despite being flush with the wearable structure 611, the sensor 613 is still configured to contact the user's skin (e.g., via a skin contact surface). Alternatively, in some embodiments, the sensor 613 extends beyond the wearable structure 611 by a predetermined distance (e.g., 0.1 millimeter (mm) to 2 mm) to contact and press into the user's skin. In some embodiments, the sensor 613 is coupled to an actuator (not shown) configured to adjust the extension height of the sensor 613 (e.g., the distance from the surface of the wearable structure 611) such that the sensor 613 contacts and presses into the user's skin. In some embodiments, the actuator adjusts the extension height between 0.01 mm and 1.2 mm. This allows the user to customize the position of the sensor 613 to improve the overall comfort of the wearable band 610 when worn while still allowing the sensor 613 to contact the user's skin. In some embodiments, the sensor 613 is not distinguishable from the wearable structure 611 when worn by the user.

[0119] The wearable structure 611 can be formed of an elastic material, an elastomer, etc., configured to be stretched and adapted to be worn by the user. In some embodiments, the wearable structure 611 is a textile or a woven fabric. As described above, the sensor 613 can be formed as part of the wearable structure 611. For example, the sensor 613 can be molded into the wearable structure 611 or integrated into a woven fabric (e.g., the sensor 613 can be sewn into the fabric and mimic the flexibility of the fabric (e.g., the sensor 613 can be formed of a series of woven fabric threads)).

[0120] The wearable structure 611 can include flexible electrical connectors that interconnect the sensors 613, electrical circuits, and / or other electronic components (described below with reference to Figure 6B to be included in the wearable band 610. In some embodiments, the flexible electrical connectors are configured to interconnect the sensors 613, electrical circuits, and / or other electronic components of the wearable band 610 with corresponding sensors and / or other electronic components of another electronic device (e.g., a watch body 620). The flexible electrical connectors are configured to move with the wearable structure 611 such that adjustments to the wearable structure 611 by the user (e.g., resizing, pulling, or folding) do not stress or strain the electrical coupling of the components of the wearable band 610.

[0121] As described above, the wearable band 610 is configured to be worn by a user. Specifically, the wearable band 610 can be shaped or otherwise manipulated to be worn by the user. For example, the wearable band 610 can be shaped to have a generally circular shape such that the wearable band can be configured to be worn on the user's lower arm or wrist. Alternatively, the wearable band 610 can be shaped to be worn on another body part of the user (such as the user's upper arm (e.g., around the biceps), forearm, chest, leg, etc.). The wearable band 610 can include a retention mechanism 612 (such as a buckle or hook-and-loop fastener) for securing the wearable band 610 to the user's wrist or other body part. When the wearable band 610 is worn by the user, the sensor 613 senses data from the user's skin (referred to as sensor data). In particular, the sensor 613 of the wearable band 610 acquires (e.g., senses and records) neuromuscular signals.

[0122] The sensed data (e.g., the sensed neuromuscular signals) can be used to detect and / or determine the intention of the user to perform certain motor actions. In particular, when the user performs a muscle activation (such as a movement or gesture), the sensor 613 senses and records the neuromuscular signals from the user. The detected and / or determined motor actions (such as phalangeal (or finger) movement, wrist movement, hand movement, and / or other muscle intentions) can be used to determine control commands or control information for causing a computing device to execute one or more input commands (instructions to perform certain commands after the data is sensed). For example, the sensed neuromuscular signals can be used to control certain user interfaces displayed on the display 605 of the wrist-worn device 600, and / or can be sent to a device responsible for rendering an AR environment (such as a head-mounted display) to perform an action in the associated AR environment (such as to control the movement of a virtual device displayed to the user). Muscle activations performed by the user can include: static gestures, such as placing the user's palm down on a table; dynamic gestures, such as grasping a physical or virtual object; and covert gestures that are not perceptible to another person, such as slightly tensing a joint by co-contracting opposing muscles or using submuscular activation. Muscle activations performed by the user can include symbolic gestures (e.g., gestures that are mapped to other gestures, interactions, or commands based on a gesture vocabulary that specifies a mapping of gestures to commands).

[0123] The sensor data sensed by the sensor 613 can be used to provide the user with an enhanced interaction with physical objects (such as devices communicatively coupled to the wearable band 610) and / or virtual objects in an AR application generated by an AR system (such as user interface objects presented on the display 605 or another computing device (such as a smartphone)).

[0124] In some embodiments, the wearable band 610 includes one or more haptic devices 646 ( Figure 6B ; e.g., vibrotactile actuators), which are configured to provide haptic feedback (e.g., cutaneous sensations and / or kinesthesia) to the user's skin. The sensor 613 and / or the haptic device 646 may be configured to operate in conjunction with multiple applications, including but not limited to health monitoring, social media, gaming, and AR (e.g., applications associated with AR).

[0125] The wearable band 610 may also include a coupling mechanism 616 (e.g., the bracket or shape of the coupling mechanism may correspond to the shape of the watch body 620 of the wrist wearable device 600), which is used to detachably couple a pod (e.g., a computing unit) or the watch body 620 to the wearable band 610 (via the coupling surface of the watch body 620). In particular, the coupling mechanism 616 may be configured to receive the coupling surface of the watch body 620 near the bottom side (e.g., the side opposite to the front side where the display 605 of the watch body 620 is located), so that the user can push the watch body 620 downward into the coupling mechanism 616 to attach the watch body 620 to the coupling mechanism 616. In some embodiments, the coupling mechanism 616 may be configured to receive the top side of the watch body 620 (e.g., the side near the front side where the display 605 of the watch body 620 is located), and the watch body is pushed upward into the bracket instead of being pushed downward into the coupling mechanism 616. In some embodiments, the coupling mechanism 616 is an integrated part of the wearable band 610, so that the wearable band 610 and the coupling mechanism 616 are a single unified structure. In some embodiments, the coupling mechanism 616 is a type of frame or housing that allows the coupling surface of the watch body 620 to be held within or on the coupling mechanism 616 (e.g., a bracket, a tracking band, a support base, or a buckle) of the wearable band 610.

[0126] The coupling mechanism 616 may allow the watch body 620 to be detachably coupled to the wearable band 610 by: friction fit, magnetic coupling, rotation-based connectors, shear pin couplings, retaining springs, one or more magnets, clips, pins, hook-and-loop fasteners, or combinations thereof. The user can perform any type of action to couple the watch body 620 to the wearable band 610 and to disconnect the watch body 620 from the wearable band 610. For example, the user can twist, slide, rotate, push, pull, or rotate (or combinations thereof) the watch body 620 relative to the wearable band 610 to attach the watch body 620 to the wearable band 610 and to detach the watch body 620 from the wearable band 610. Alternatively, as discussed below, in some embodiments, the watch body 620 can be disconnected from the wearable band 610 by actuating a release mechanism 629.

[0127] The wearable band 610 can be coupled with the watch body 620 to increase the functions of the wearable band 610 (e.g., converting the wearable band 610 into a wrist wearable device 600, adding additional computing units and / or batteries to increase the computing resources and / or battery life of the wearable band 610, or adding additional sensors to improve the sensed data). As described above, the wearable band 610 (and the coupling mechanism 616) is configured to operate independently of the watch body 620 (e.g., perform functions independently of the watch body). For example, the coupling mechanism 616 can include one or more sensors 613 that contact the user's skin when the user wears the wearable band 610 and provide sensor data for determining control commands.

[0128] The user can detach the watch body 620 (or the pod) from the wearable band 610 to reduce the burden on the user by the wrist wearable device 600. For embodiments in which the watch body 620 is detachable, the watch body 620 can be referred to as a detachable structure, such that in these embodiments, the wrist wearable device 600 includes a wearable portion (e.g., the wearable band 610) and a detachable structure (the watch body 620).

[0129] Turning to the watch body 620, the watch body 620 can have a generally rectangular or circular shape. The watch body 620 is configured to be worn by the user on their wrist or another body part. More specifically, the watch body 620 is sized to be easily carried by the user, easily attached to a part of the user's clothing, and / or easily coupled to the wearable band 610 (thereby forming the wrist wearable device 600). As described above, the watch body 620 can have a shape corresponding to the coupling mechanism 616 of the wearable band 610. In some embodiments, the watch body 620 includes a single release mechanism 629 or multiple release mechanisms (e.g., two release mechanisms 629 positioned on opposite sides of the watch body 620, such as spring-loaded buttons) to decouple the watch body 620 from the wearable band 610. The release mechanism 629 can include, but is not limited to, buttons, knobs, plugs, handles, joysticks, fasteners, buckles, dials, latches, or combinations thereof.

[0130] The user can actuate the release mechanism 629 by pushing, turning, lifting, pressing, moving the release mechanism 629, or performing other actions on the release mechanism 629. Actuating the release mechanism 629 can release (e.g., decouple) the watch body 620 from the coupling mechanism 616 of the wearable band 610, thereby allowing the user to use the watch body 620 independently of the wearable band 610, and vice versa. For example, decoupling the watch body 620 from the wearable band 610 can allow the user to use the rear camera 625b to capture images. Although the coupling mechanism 616 is shown positioned at a corner of the watch body 620, the release mechanism 629 can be positioned anywhere on the watch body 620 that is convenient for the user to actuate. Additionally, in some embodiments, the wearable band 610 can also include a corresponding release mechanism for decoupling the watch body 620 from the coupling mechanism 616. In some embodiments, the release mechanism 629 is optional, and as described above, the watch body 620 can be decoupled (e.g., via twisting or rotation) from the coupling mechanism 616.

[0131] The watch body 620 can include one or more peripheral buttons 623 and 627 for performing various operations at the watch body 620. For example, the peripheral buttons 623 and 627 can be used to turn on or wake up the display 605 (e.g., transition the display 605 from a sleep state to an active state), unlock the watch body 620, increase or decrease the volume, raise or lower the brightness, interact with one or more applications, and interact with one or more user interfaces. Additionally or alternatively, in some embodiments, the display 605 acts as a touch screen and allows the user to provide one or more inputs for interacting with the watch body 620.

[0132] In some embodiments, the watch body 620 includes one or more sensors 621. The sensors 621 of the watch body 620 can be the same as or different from the sensors 613 of the wearable band 610. The sensors 621 of the watch body 620 can be distributed on the inner surface and / or outer surface of the watch body 620. In some embodiments, the sensors 621 are configured to contact the user's skin when the user is wearing the watch body 620. For example, the sensors 621 can be placed on the bottom side of the watch body 620, and the coupling mechanism 616 can be a bracket with an opening that allows the bottom side of the watch body 620 to directly contact the user's skin. Alternatively, in some embodiments, the watch body 620 does not include sensors configured to contact the user's skin (e.g., includes sensors inside and / or outside the watch body 620 that are configured to sense data of the watch body 620 and data of the surrounding environment of the watch body 620). In some embodiments, the sensors 613 are configured to track the position and / or movement of the watch body 620.

[0133] The watch body 620 and the wearable band 610 can share data using a wired communication method (e.g., Universal Asynchronous Receiver / Transmitter (UART) or USB transceiver) and / or a wireless communication method (e.g., near-field communication or Bluetooth). For example, the watch body 620 and the wearable band 610 can share data sensed by sensors 613 and 621, as well as information specific to the application and specific to the device (e.g., active and / or available applications, output devices (e.g., display or speaker) and / or input devices (e.g., touch screen, microphone or imaging sensor)).

[0134] In some embodiments, the watch body 620 may include, but is not limited to, a front camera 625a and / or a rear camera 625b, sensors 621 (e.g., biometric sensors, IMU sensors, heart rate sensors, saturation oxygen sensors, neuromuscular signal sensors, altimeter sensors, temperature sensors, bioimpedance sensors, pedometer sensors, optical sensors (e.g., Figure 6B ; imaging sensor 663), touch sensors, sweat sensors). In some embodiments, the watch body 620 may include one or more haptic devices 676( Figure 6B ; vibrotactile actuators), which are configured to provide haptic feedback (e.g., cutaneous sensation and / or kinesthesia) to the user. The sensors 621 and / or the haptic devices 676 may also be configured to operate in conjunction with multiple applications, including but not limited to health monitoring applications, social media applications, gaming applications, and AR applications (e.g., applications associated with AR).

[0135] As described above, the watch body 620 and the wearable band 610 can form a wrist wearable device 600 when coupled. The watch body 620 and the wearable band 610 act as a single device when coupled to perform the various functions described herein (e.g., operations, detections, or communications). In some embodiments, each device is provided with specific instructions for performing one or more operations of the wrist wearable device 600. For example, depending on the determination that the watch body 620 does not include a neuromuscular signal sensor, the wearable band 610 may include alternative instructions for performing the associated instructions (e.g., providing sensed neuromuscular signal data to the watch body 620 via a different electronic device). The operations of the wrist wearable device 600 can be performed by the watch body 620 alone or in combination with the wearable band 610 (e.g., via respective processors and / or hardware components), and vice versa. In some embodiments, the operations of the wrist wearable device 600, the watch body 620, and / or the wearable band 610 can be performed in conjunction with another communicatively coupled device (e.g., Figure 8A andFigure 8B is executed in combination with one or more processors and / or hardware components of the HIPD 800).

[0136] As described in the following reference Figure 6B As depicted in the block diagrams, the wearable band 610 and / or the watch body 620 may each include independent resources necessary to perform functions independently. For example, the wearable band 610 and / or the watch body 620 may each include a power source (e.g., a battery), a memory, a data memory, a processor (e.g., a CPU), communication, a light source, and / or an input / output device.

[0137] Figure 6B A block diagram showing a computing system 630 corresponding to the wearable band 610 and a computing system 660 corresponding to the watch body 620 according to some embodiments is shown. According to some embodiments, the computing system of the wrist wearable device 600 includes a combination of components of the wearable band computing system 630 and components of the watch body computing system 660.

[0138] The watch body 620 and / or the wearable band 610 may include one or more components shown in the watch body computing system 660. In some embodiments, a single integrated circuit includes all or most of the components of the watch body computing system 660, which are included in a single integrated circuit. Alternatively, in some embodiments, the components of the watch body computing system 660 are included in a plurality of integrated circuits that are communicatively coupled. In some embodiments, the watch body computing system 660 is configured to be coupled (e.g., via a wired connection or a wireless connection) to the wearable band computing system 630, which allows the two computing systems to share components, distribute tasks, and / or perform other operations described herein (either individually or as a single device).

[0139] The computing system 660 of the watch body may include one or more processors 679, a controller 677, a peripheral interface 661, a power system 695, and a memory (e.g., memory 680), each of which is defined above and described in more detail below.

[0140] The power system 695 may include a charger input 696, a power-management integrated circuit (PMIC) 697, and a battery 698, each of which is defined above. In some embodiments, the watch body 620 and the wearable band 610 may have respective charger inputs (e.g., charger inputs 696 and 657), respective batteries (e.g., batteries 698 and 659), and may share power with each other (e.g., the watch body 620 may power and / or charge the wearable band 610 and vice versa). Although the watch body 620 and / or the wearable band 610 may include respective charger inputs, a single charger input may charge both devices when the two devices are coupled. The watch body 620 and the wearable band 610 may use various techniques to receive charge. In some embodiments, the watch body 620 and the wearable band 610 may use a wired charging component (e.g., a power cord) to receive charge. Alternatively or additionally, the watch body 620 and / or the wearable band 610 may be configured for wireless charging. For example, a portable charging device may be designed to match a portion of the watch body 620 and / or a portion of the wearable band 610 and wirelessly transmit available power to the battery of the watch body 620 and / or the battery of the wearable band 610. The watch body 620 and the wearable band 610 may have independent power systems (e.g., power systems 695 and 656) to enable each to operate independently. The watch body 620 and the wearable band 610 may also share power (e.g., one may charge the other) via their respective PMICs (e.g., PMICs 697 and 658), which may share power through power conductors and ground conductors and / or through wireless charging antennas.

[0141] In some embodiments, the peripheral interface 661 may include one or more sensors 621, many of which among the one or more sensors listed below have been defined above. The sensors 621 may include one or more coupling sensors 662 for detecting when the watch body 620 is coupled to another electronic device (e.g., the wearable band 610). The sensors 621 may include an imaging sensor 663 (one or more of the camera 625 and / or a separate imaging sensor 663 (e.g., a thermal imaging sensor)). In some embodiments, the sensors 621 include one or more SpO2 sensors 664. In some embodiments, the sensors 621 include one or more biopotential signal sensors (e.g., the EMG sensor 665, which may be disposed on the user-facing portion of the watch body 620 and / or the wearable band 610). In some embodiments, the sensors 621 include one or more capacitive sensors 666. In some embodiments, the sensors 621 include one or more heart rate sensors 667. In some embodiments, the sensors 621 include one or more IMUs 668. In some embodiments, the one or more IMUs 668 may be configured to detect the movement of the user's hand or the movement of other positions where the watch body 620 is placed or held).

[0142] In some embodiments, the peripheral interface 661 includes an NFC component 669, a GPS component 670, a long-term evolution (LTE) component 671, and / or Wi-Fi and / or Bluetooth communication components 672. In some embodiments, the peripheral interface 661 includes one or more buttons 673 (e.g., Figure 6A the peripheral buttons 623 and 627 therein), and the one or more buttons, when selected by the user, cause an operation to be performed at the watch body 620. In some embodiments, the peripheral interface 661 includes one or more indicators (e.g., light-emitting diodes (LEDs)) to provide visual indicators to the user (e.g., received message, low battery level, active microphone and / or camera).

[0143] The watch body 620 may include at least one display 605 for displaying a visual representation of information or data to a user, the visual representation including user interface elements and / or three-dimensional (3D) virtual objects. The display may also include a touch screen for inputting user inputs (such as touch gestures and swipe gestures, etc.). The watch body 620 may include at least one speaker 674 and at least one microphone 675 for providing an audio signal to the user and receiving an audio input from the user. The user may provide a user input through the microphone 675 and may also receive an audio output from the speaker 674 as part of a haptic event provided by the haptic controller 678. The watch body 620 may include at least one camera 625, and the at least one camera 625 includes a front camera 625a and a rear camera 625b. The camera 625 may include an ultra-wide-angle camera, a wide-angle camera, a fish-eye camera, a spherical camera, a telephoto camera, a depth-sensing camera, or other types of cameras.

[0144] The watch body computing system 660 may include one or more haptic controllers 678 and associated components (such as, haptic devices 676), and the one or more haptic controllers and the associated components are used to provide haptic events (such as, a vibration feeling or an audio output in response to an event at the watch body 620) at the watch body 620. The haptic controller 678 may communicate with one or more haptic devices 676 (such as electroacoustic devices), and the one or more haptic devices include a speaker among one or more speakers 674 and / or other audio components and / or electromechanical devices that convert energy into linear motion (such as motors, electromagnetic coils, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (such as, components that convert an electrical signal into a haptic output on a device)). The haptic controller 678 may provide a haptic event that can be felt by a user of the watch body 620 to a corresponding haptic actuator. In some embodiments, one or more haptic controllers 678 may receive an input signal from an application in the applications 682.

[0145] In some embodiments, computer system 630 and / or computer system 660 may include a memory 680, which may be controlled by a memory controller of one or more controllers 677 and / or one or more processors 679. In some embodiments, software components stored in memory 680 include one or more applications 682 configured to perform operations at the watch body 620. In some embodiments, the one or more applications 682 include games, word processors, messaging applications, calling applications, web browsers, social media applications, media streaming applications, financial applications, calendars, clocks, and the like. In some embodiments, software components stored in memory 680 include one or more communication interface modules 683 as defined above. In some embodiments, software components stored in memory 680 include: one or more graphics modules 684 for rendering, encoding, and / or decoding audio data and / or video data; and one or more data management modules 685 for collecting, organizing, and / or providing access to data 687 stored in memory 680. In some embodiments, one or more of the applications 682 and / or one or more of the modules may work in combination with each other to perform various tasks at the watch body 620.

[0146] In some embodiments, software components stored in memory 680 may include one or more operating systems 681 (e.g., a Linux-based operating system, or an Android operating system). Memory 680 may also include data 687. Data 687 may include profile data 688A, sensor data 689A, media content data 690, and application data 691.

[0147] It should be understood that the watch body computing system 660 is an example of a computing system within the watch body 620, and the watch body 620 may have more components, fewer components, combine two or more components, and / or have different configurations and / or arrangements of these components than those shown in the watch body computing system 660. Each of the components shown in the watch body computing system 660 is implemented in hardware, software, firmware, or a combination thereof (including one or more signal processing circuits and / or application-specific integrated circuits).

[0148] Turning to wearable band computing system 630, one or more components that may be included in wearable band 610 are shown. Wearable band computing system 630 may include more components or fewer components than those shown in watch body computing system 660, may combine two or more components, and / or may have different configurations and / or arrangements of some or all of these components. In some embodiments, all or most of the components of wearable band computing system 630 are included in a single integrated circuit. Alternatively, in some embodiments, the components of wearable band computing system 630 are included in multiple integrated circuits that are communicatively coupled. As described above, in some embodiments, wearable band computing system 630 is configured to be coupled (e.g., via a wired connection or a wireless connection) to watch body computing system 660, which allows the two computing systems to share components, allocate tasks, and / or perform other operations described herein (either individually or as a single device).

[0149] Similar to watch body computing system 660, wearable band computing system 630 may include: one or more processors 649; one or more controllers 647 (including one or more haptic controllers 648); a peripheral interface 631, which may include one or more sensors 613 and other peripheral devices; a power source (e.g., power system 656); and a memory (e.g., memory 650), which includes an operating system (e.g., operating system 651), data (e.g., data 654, which includes profile data 688B and / or sensor data 689B), and one or more modules (e.g., communication interface module 652 or data management module 653).

[0150] Given the above definitions, one or more sensors 613 may be similar to sensors 621 of computer system 660. For example, sensors 613 may include one or more coupled sensors 632, one or more SpO2 sensors 634, one or more EMG sensors 635, one or more capacitive sensors 636, one or more heart rate sensors 637, and one or more IMU sensors 638.

[0151] The peripheral interface 631 may also include other components similar to those included in the peripheral interface 661 of the computer system 660, as described above with reference to the peripheral interface 661. These other components include an NFC component 639, a GPS component 640, an LTE component 641, a Wi-Fi and / or Bluetooth communication component 642, and / or one or more haptic devices 676. In some embodiments, the peripheral interface 631 includes one or more buttons 643, a display 633, a speaker 644, a microphone 645, and a camera 655. In some embodiments, the peripheral interface 631 includes one or more indicators, such as an LED.

[0152] It should be understood that the wearable band computing system 630 is an example of a computing system within the wearable band 610, and the wearable band 610 may have more components or fewer components than those shown in the wearable band computing system 630, may combine two or more components, and / or may have different configurations and / or arrangements of these components. Each of the components shown in the wearable band computing system 630 may be implemented in one of hardware, software, and firmware or a combination of hardware, software, and firmware (including one or more signal processing circuits and / or application specific integrated circuits).

[0153] Reference Figure 6A The wrist wearable device 600 of [] is an example in which the wearable band 610 and the watch body 620 are coupled. Thus, the wrist wearable device 600 will be understood to include the components shown and described for the wearable band computing system 630 and the watch body computing system 660. In some embodiments, the wrist wearable device 600 has a split architecture (e.g., a split mechanical architecture or a split electronic architecture) between the watch body 620 and the wearable band 610. In other words, all of the components shown in the wearable band computing system 630 and the watch body computing system 660 may be housed or otherwise arranged in the combined watch device 600, or may be housed or otherwise arranged in a single component within the watch body 620, the wearable band 610, and / or portions thereof (e.g., the coupling mechanism 616 of the wearable band 610).

[0154] The above techniques may be used with any device for sensing neuromuscular signals, including Figure 6A and Figure 6B the arm wearable device in [], but may also be used with other types of wearable devices for sensing neuromuscular signals (e.g., body wearable devices or head wearable devices that may have neuromuscular sensors closer to the brain or spine).

[0155] In some embodiments, the wrist-wearable device 600 can be used in combination with the head-wearable devices (e.g., AR device 700 and VR device 710) described below and / or HIPD 800, and the wrist-wearable device 600 can also be configured to allow a user to control aspects of AR (e.g., by using EMG-based gestures to control user interface objects in AR, and / or by allowing the user to interact with a touch screen on the wrist-wearable device to also control aspects of AR). In some embodiments, the wrist-wearable device 600 can also be used in combination with wearable apparel (e.g., the textile-based smart apparel 900 described below with reference to Figures 9A to 9C ). Having described an example wrist-wearable device in this manner, attention will now be turned to example head-wearable devices, such as AR device 700 and VR device 710.

[0156] Example Head-Wearable Devices

[0157] Figure 7A and Figure 7B-1 and Figure 7B-2 and Figure 7C illustrate example head-wearable devices according to some embodiments. The head-wearable device can include, but is not limited to, an AR device 700 (e.g., an AR glasses device or a smart glasses device, such as smart glasses, smart monocles, or smart contact lenses), a VR device 710 (e.g., a VR head-mounted viewer or a head-mounted display (HMD)), or other visually coupled devices. The AR device 700 and the VR device 710 are examples of the head-wearable device 204 described herein with reference to Figures 2A to 2D , such that the head-wearable device should be understood to have the characteristics of the AR device 700 and / or the VR device 710, and vice versa. The AR device 700 and the VR device 710 can perform various functions and / or operations associated with navigating in a user interface and selectively launching applications, as well as the functions and / or operations described above with reference to Figures 2A to 2D .

[0158] In some embodiments, an AR system (e.g., Figures 5A to 5D-2 ; AR systems 500a to 500d) includes an AR device 700 (as Figure 7A shown) and / or a VR device 710 (as Figure 7B-1 and Figure 7B-2 shown). In some embodiments, the AR device 700 and the VR device 710 can include one or more similar components (e.g., components for presenting an interactive AR environment, such as a processor, a memory, and / or a presenting device, the presenting device including one or more displays and / or one or more waveguides), with reference to Figure 7CSome of these components are described in more detail. The head wearable device can use a display projector (e.g., display projector assemblies 707A and 707B) and / or waveguides for projecting a representation of data to a user. Some embodiments of the head wearable device do not include a display.

[0159] Figure 7A Shows an example visual depiction of an AR device 700 (e.g., the AR device 700 may also be described herein as augmented reality glasses and / or smart glasses). The AR device 700 may work in conjunction with Figure 7A additional electronic components (such as a wearable accessory device and / or an intermediate processing device) not shown, which are in electronic communication with the AR device 700 or otherwise configured to be used in conjunction with the AR device 700. In some embodiments, the wearable accessory device and / or the intermediate processing device may be configured to be coupled to the AR device 700 via a coupling mechanism in electronic communication with a coupling sensor 724, where the coupling sensor 724 may detect when an electronic device is physically or electronically coupled to the AR device 700. In some embodiments, the AR device 700 may be configured to be coupled to a housing (e.g., a portion of the frame 704 or a portion of the temple 705), which may include one or more additional coupling mechanisms configured to couple to additional accessory devices. Figure 7A The components shown in may be implemented in hardware, software, firmware, or a combination thereof (including one or more signal processing components and / or an application-specific integrated circuit (ASIC)).

[0160] The AR device 700 includes mechanical glasses components, the mechanical glasses components including a frame 704 configured to hold one or more lenses (e.g., one or both of the lenses 706-1 and 706-2). Those of ordinary skill in the art will understand that the AR device 700 may include additional mechanical components, such as hinges configured to allow portions of the frame 704 of the AR device 700 to fold and unfold, a bridge configured to span the gap between the lenses 706-1 and 706-2 and rest on the user's nose, a nose pad configured to rest on the bridge of the nose and provide support for the AR device 700, an earpiece configured to rest on the user's ear and provide additional support for the AR device 700, and temples 705 configured to extend from the hinge to the earpiece of the AR device 700, etc. Those of ordinary skill in the art will also understand that some examples of the AR device 700 may not include the mechanical components described herein. For example, smart contact lenses configured to present AR to a user may not include any components of the AR device 700.

[0161] The lenses 706-1 and 706-2 can be separate displays or display devices (e.g., waveguides for the projected representations). The lenses 706-1 and 706-2 can work together or independently to present an image or a series of images to the user. In some embodiments, the lenses 706-1 and 706-2 can operate in conjunction with one or more display projector components 707A and 707B to present image data to the user. Although the AR device 700 includes two displays, embodiments of the present disclosure can be implemented in AR devices having a single near-eye display (NED) or more than two NEDs.

[0162] The AR device 700 includes a plurality of electronic components, many of which will be described in more detail below with reference to Figure 7C Some example electronic components are shown in Figure 7A which include sensors 723-1, 723-2, 723-3, 723-4, 723-5, and 723-6, which can be distributed along much of the frame 704 of the AR device 700. Different types of sensors are described below with reference to Figure 7C The AR device 700 also includes a left camera 739A and a right camera 739B located on different sides of the frame 704. And the glasses device includes one or more processors 748A and 748B (e.g., integrated microprocessors, such as ASICs) embedded in a portion of the frame 704.

[0163] Figure 7B-1 and Figure 7B-2 shows an example visual depiction of a VR device 710 (e.g., a head-mounted display (HMD) 712, also referred to herein as an AR head-mounted viewer, a head-wearable device, or a VR head-mounted viewer). The HMD 712 includes a front body 714 and a frame 716 (e.g., a bar or strap) shaped to fit around a user's head. In some embodiments, the front body 714 and / or the frame 716 include one or more electronic components (e.g., a display, a processor (e.g., processor 748A-1), an IMU, a tracking transmitter or detector, or a sensor) for facilitating the presentation of and / or interaction with an AR system and / or a VR system. In some embodiments, as Figure 7B-2 shown, the HMD 712 includes an output audio converter (e.g., audio converter 718-1). In some embodiments, as Figure 7B-2As shown, one or more components (e.g., one or more output audio converters 718 and the frame 716) (e.g., a part or all of the frame 716 and / or the output audio converter 718) can be configured to be attached to the HMD 712 and detached from the HMD 712 (e.g., it is detachably attached to the HMD 712). In some embodiments, coupling the detachable component to the HMD 712 enables the detachable component to enter into electronic communication with the HMD 712. The VR device 710 includes electronic components, many of which will be described in more detail below with reference to Figure 7C for a more detailed description.

[0164] Figure 7B-1 and Figure 7B-2 It is also shown that the VR device 710 has one or more cameras, such as a left camera 739A and a right camera 739B, which can be similar to the left and right cameras on the frame 704 of the AR device 700. In some embodiments, the VR device 710 includes one or more additional cameras (e.g., cameras 739C and 739D), which can be configured to enhance the image data acquired by the cameras 739A and 739B by providing more information. For example, the camera 739C can be used to provide color information not recognized by the cameras 739A and 739B. In some embodiments, one or more of the cameras 739A to 739D can include an optional IR cut-off filter, which is configured to remove infrared (IR) light received at the corresponding camera sensor.

[0165] The VR device 710 can include a housing 790, which stores one or more components of the VR device 710 and / or additional components of the VR device 710. The housing 790 can be such a modular electronic device: the modular electronic device is configured to be coupled to the VR device 710 (or the AR device 700) and supplement and / or expand the capabilities of the VR device 710 (or the AR device 700). For example, the housing 790 can include additional sensors, cameras, power supplies, and processors (e.g., processor 748A-2) to improve and / or increase the functionality of the VR device 710. Examples of different components included in the housing 790 are described below with reference to Figure 7C a description of the different components included in the housing 790.

[0166] Alternatively or additionally, in some embodiments, the head-wearable device (e.g., the VR device 710 and / or the AR device 700) includes another external device (e.g., a paired device) or is communicatively coupled to another external device, such as (hereinafter with reference to Figure 8A and Figure 8BThe described HIPD 8 and / or optional neckband. The optional neckband may be coupled to the head wearable device via one or more connectors (e.g., a wired connector or a wireless connector). The head wearable device and the neckband may operate independently without any wired or wireless connection between them. In some embodiments, the components of the head wearable device and the components of the neckband are located on one or more additional peripheral devices paired with the head wearable device, the neckband, or some combination thereof. Additionally, the neckband is intended to represent any suitable type or form of paired device. Thus, the following discussion of the neckband may also apply to various other paired devices, such as smartwatches, smartphones, wristbands, other wearable devices, handheld controllers, tablets, or laptop computers.

[0167] In some cases, pairing an external device (e.g., an intermediate processing device (e.g., HIPD device 800, optional neckband, and / or wearable accessory device)) with a head wearable device (e.g., an AR device 700 and / or a VR device 710) enables the head wearable device to achieve form factors similar to those of a pair of glasses while still providing sufficient battery power and computing power for the extended capabilities. Some or all of the battery power, computing resources, and / or additional features of the head wearable device may be provided by the paired device or shared between the paired device and the head wearable device, thereby generally reducing the weight, heat profile, and form factors of the head wearable device while allowing the head wearable device to maintain its desired functionality. For example, an intermediate processing device (e.g., HIPD 800) may allow components that would otherwise be included in the head wearable device to be included in the intermediate processing device (and / or wearable device or accessory device), thereby transferring the weight load from the user's head and neck to one or more other parts of the user's body. In some embodiments, the intermediate processing device has a larger surface area region for dissipating and dispersing heat into the surrounding environment. Thus, the intermediate processing device may allow for a larger battery capacity and computing power compared to what may otherwise be possible with a standalone head wearable device. Since the weight borne in the intermediate processing device may be less invasive to the user than the weight borne in the head wearable device, the user may be able to tolerate wearing a lighter glasses device and carrying or wearing the paired device for a longer period of time compared to tolerating wearing a heavier standalone glasses device, thereby enabling the AR environment to be more fully integrated into the user's daily activities.

[0168] In some embodiments, the intermediate processing device is communicatively coupled to the head-worn device and / or communicatively coupled to other devices. These other devices may provide certain functions to the head-worn device (e.g., tracking, positioning, depth map construction, processing, and / or storage). In some embodiments, the intermediate processing device includes a controller and a power source. In some embodiments, the sensors of the intermediate processing device are configured to sense additional data, which may be shared with the head-worn device in electronic format (analog or digital).

[0169] The controller of the intermediate processing device processes information generated by sensors on the intermediate processing device and / or on the head-worn device. The intermediate processing device (e.g., HIPD 800) may process information generated by one or more of its plurality of sensors and / or information provided by other communicatively coupled devices. For example, the head-worn device may include an IMU, and the intermediate processing device (neckband and / or HIPD 800) may perform all inertial calculations and spatial calculations based on the IMU located on the head-worn device. The following references Figure 8A and Figure 8B provide additional examples of processing performed by communicatively coupled devices (e.g., HIPD800).

[0170] The AR system can include various types of visual feedback mechanisms. For example, the display device in the AR device 700 and / or the VR device 710 can include one or more liquid-crystal displays (LCDs), light emitting diode (LED) displays, organic LED (OLED) displays, and / or any other suitable type of display screen. The AR system can include a single display screen for both eyes or can provide a display screen for each eye, which can allow for additional flexibility for zoom adjustment or for correcting refractive errors associated with the user's vision. Some AR systems also include an optical subsystem having one or more lenses (e.g., conventional concave or convex lenses, Fresnel lenses, or adjustable liquid lenses) through which the user can view the display screen. In addition to or instead of using a display screen, some AR systems include one or more projection systems. For example, the display device in the AR device 700 and / or the VR device 710 can include a micro-LED projector (e.g., using a waveguide) that projects light onto the display device, such as a transparent combination lens that allows ambient light to penetrate. The display device can refract the projected light into the user's pupil and can enable the user to view both the AR content and the real world simultaneously. The AR system can also be configured with any other suitable type or form of image projection system. As noted, some AR systems may not mix the artificial reality with the real reality, but instead substantially replace one or more of the user's perceptual perceptions of the real world with a virtual experience.

[0171] Although multiple example head-mounted devices are described herein as the AR device 700 and the VR device 710 respectively, any one or both of the multiple example head-mounted devices described herein can be configured to present a fully immersive VR scene presented in substantially the entire field of view of the user, as a supplement or alternative to a more subtle augmented reality scene presented within a portion (less than the entire field of view) of the user's field of view.

[0172] In some embodiments, the AR device 700 and / or the VR device 710 may include a haptic feedback system. The haptic feedback system may provide various types of cutaneous feedback, including vibration, force, stretch, shear stress, texture, and / or temperature. The haptic feedback system may also provide various types of kinesthetic feedback, such as movement and compliance. The haptic feedback may be implemented using motors, piezoelectric actuators, fluid systems, and / or various other types of feedback mechanisms. The haptic feedback system may be implemented independently of other AR devices, within other AR devices, and / or in combination with other AR devices (e.g., wrist-wearable devices that may be incorporated into a headset, gloves, a jumpsuit; hand-held controllers; environmental devices (e.g., a chair or a carpet); and / or any other type of device or system, such as the wrist-wearable device 600, the HIPD 800, the textile-based smart garment 900) and / or other devices described herein).

[0173] Figure 7C A computing system 720 and an optional housing 790 are shown, each of which shows components that may be included in a head-wearable device (e.g., the AR device 700 and / or the VR device 710). In some embodiments, more or fewer components may be included in the optional housing 790 depending on the actual constraints of the corresponding head-wearable device described. Additionally or alternatively, the optional housing 790 may include additional components to expand and / or enhance the functionality of the head-wearable device.

[0174] In some embodiments, the computing system 720 and / or the optional housing 790 may include one or more peripheral interfaces 722A and 722B, one or more power systems 742A and 742B (including a charger input 743, a PMIC 744, and a battery 745), one or more controllers 746A and 746B (including one or more haptic controllers 747), one or more processors 748A and 748B (as defined above, including any example from the provided plurality of examples), and memories 750A and 750B, and the above components may communicate with each other electronically. For example, one or more of the processors 748A and / or 748B may be configured to execute instructions stored in the memories 750A and / or 750B, and the instructions may cause a controller in one or more of the controllers 746A and / or 746B to cause a plurality of operations to be performed at one or more peripheral devices at the peripheral interfaces 722A and / or 722B. In some embodiments, each of the described operations may be performed based on power provided by the power systems 742A and / or 742B.

[0175] In some embodiments, the peripheral interface 722A may include one or more devices configured as part of the computing system 720, many of which have been defined above and / or described with reference to Figure 6A and Figure 6B the wrist-wearable device shown in. For example, the peripheral interface may include one or more sensors 723A. Some example sensors include one or more coupled sensors 724, one or more acoustic sensors 725, one or more imaging sensors 726, one or more EMG sensors 727, one or more capacitive sensors 728, and / or one or more IMUs 729. In some embodiments, the sensors 723A also include: a depth sensor 767; a light sensor 768; and / or any other type of sensor defined above or described with reference to any other embodiment discussed herein.

[0176] In some embodiments, the peripheral interface may include one or more additional peripheral devices, the one or more additional peripheral devices including: one or more NFC devices 730; one or more GPS devices 731; one or more LTE devices 732; one or more Wi-Fi and / or Bluetooth devices 733; one or more buttons 734 (e.g., including slidable or otherwise adjustable buttons); one or more displays 735A; one or more speakers 736A; one or more microphones 737A; one or more cameras 738A (e.g., including a first camera 739-1 to an nth camera 739-n, the first camera 739-1 to the nth camera 739-n being similar to the left camera 739A and / or the right camera 739B); one or more haptic devices 740; and / or any other type of peripheral device defined above or described with reference to any other embodiment discussed herein.

[0177] A head-mounted device may include various types of visual feedback mechanisms (e.g., display devices). For example, the display devices in AR device 700 and / or VR device 710 may include one or more liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic LED (OLED) displays, micro LEDs, and / or any other suitable type of display screen. The head-mounted device may include a single display screen (e.g., configured to be viewed by both eyes), and / or may provide a separate display screen for each eye, which may allow for additional flexibility for zoom adjustment and / or for correcting refractive errors associated with the user's vision. Some embodiments of the head-mounted device also include an optical subsystem having one or more lenses (e.g., conventional concave or convex lenses, Fresnel lenses, or adjustable liquid lenses) through which the user may view the display screen. For example, the corresponding display 735A may be coupled to each of lenses 706-1 and 706-2 of AR device 700. The displays 735A coupled to each of lenses 706-1 and 706-2 may work together or independently to present an image or a series of images to the user. In some embodiments, AR device 700 and / or VR device 710 includes a single display 735A (e.g., a near-eye display) or more than two displays 735A.

[0178] In some embodiments, one or more displays 735A of a first group may be used to present an augmented reality environment, and one or more display devices 735A of a second group may be used to present a VR environment. In some embodiments, one or more waveguides (e.g., as a means of transmitting light from a display projector assembly and / or one or more displays 735A to the user's eyes) are used in combination with presenting AR content to a user of AR device 700 and / or VR device 710. In some embodiments, one or more waveguides are fully or partially integrated into AR device 700 and / or VR device 710. As a supplement or alternative to the display screen, some AR systems include one or more projection systems. For example, the display devices in AR device 700 and / or VR device 710 may include (e.g., using waveguides) a micro LED projector that projects light onto the display device, such as a transparent combination lens that allows ambient light to penetrate. The display device may refract the projected light into the user's pupil and may enable the user to view both artificial reality content and the real world simultaneously. The head-mounted device may also be configured with any other suitable type or form of image projection system. In some embodiments, one or more waveguides are provided as a supplement or alternative to one or more displays 735A.

[0179] In some embodiments of a head-worn device, ambient light and / or a real-world live view (e.g., a live feed of the surrounding environment that a user would normally see) can be made to penetrate a display element of the corresponding head-worn device that is presenting aspects of an AR system. In some embodiments, ambient light and / or a real-world live view can be made to penetrate a portion (less than the entire AR environment) of the AR environment presented within the user's field of view (e.g., a portion of the AR environment that is in the same location as a physical object within the user's real-world environment that is located within a specified boundary (e.g., a guardian boundary), the specified boundary being configured to be used by the user while they are interacting with the AR environment). For example, a visual user interface element (e.g., a notification user interface element) can be presented at the head-worn device, and a certain amount of ambient light and / or a real-world live view (e.g., 15% to 50% of the ambient light and / or real-world live view) can be made to penetrate the user interface element such that the user can distinguish at least a portion of the physical environment on which the user interface element is being displayed.

[0180] The head-worn device can include one or more external displays 735A for presenting information to the user. For example, the external display 735A can be used to display the current battery level, network activity (e.g., connection, disconnection), current activity (e.g., playing a game, making a call, attending a meeting, or watching a movie), and / or other relevant information. In some embodiments, the external display 735A can be used to communicate with other people. For example, a user of the head-worn device can cause the external display 735A to present a "Do Not Disturb" notification. The user can also use the external display 735A to share any information collected by one or more components of the peripheral interface 722A and / or generated by the head-worn device (e.g., during the operation and / or execution of one or more applications).

[0181] The memory 750A can include instructions and / or data executable by a memory controller of one or more processors 748A (and / or the processor 748B of the housing 790) and / or one or more controllers 746A (and / or the controller 746B of the housing 790). The memory 750A can include: one or more operating systems 751; one or more applications 752, one or more communication interface modules 753A, one or more graphics modules 754A, one or more AR processing modules 755A; and / or any other type of module or component defined above or described with reference to any other embodiment discussed herein.

[0182] The data 760 stored in the memory 750A can be used in combination with one or more of the applications and / or programs discussed above. The data 760 can include: profile data 761; sensor data 762; media content data 763; AR application data 764; and / or any other type of data defined above or described with reference to any other embodiments discussed herein.

[0183] In some embodiments, the controller 746A of the head-wearable device processes information generated by sensors 723A on the head-wearable device, and / or another component of the head-wearable device, and / or another component communicatively coupled to the head-wearable device (e.g., a component of the housing 790, such as a component of the peripheral interface 722B). For example, the controller 746A can process information from the acoustic sensor 725 and / or the image sensor 726. For each detected sound, the controller 746A can perform a direction of arrival (DOA) estimation to estimate the direction in which the detected sound arrives at the head-wearable device. When one or more of the plurality of acoustic sensors 725 detect a sound, the controller 746A can populate an audio data set with information (e.g., represented by the sensor data 762).

[0184] In some embodiments, a physical electronic connector can transfer information between the head-wearable device and another electronic device, and / or between one or more processors 748A and the controller 746A of the head-wearable device. The information can be in the form of: optical data; electrical data; wireless data; or any other transmittable data form. Shifting the processing of the information generated by the head-wearable device to an intermediate processing device can reduce the weight and heat of the glasses device, making it more comfortable and safer for the user. In some embodiments, an optional accessory device (e.g., an electronic neckband or HIPD 800) is coupled to the head-wearable device via one or more connectors. Each connector can be a wired connector or a wireless connector, and can include electronic components and / or non-electronic (e.g., structural) components. In some embodiments, the head-wearable device and the accessory device can operate independently without any wired or wireless connection between them.

[0185] A head-wearable device may include various types of computer vision components and subsystems. For example, an AR device 700 and / or a VR device 710 may include one or more optical sensors, such as a two-dimensional (2D) camera or a three-dimensional (3D) camera, a time-of-flight depth sensor, a single-beam rangefinder or a scanning lidar rangefinder, a 3D lidar (LiDAR) sensor, and / or any other suitable type or form of optical sensor. The head-wearable device may process data from one or more of these sensors to identify the position of the user and / or various aspects of the user's real-world physical environment (including the position of real-world objects in the real-world physical environment). In some embodiments, the methods described herein are used to map the real world to provide context to the user regarding the real-world environment and / or to perform various other functions such as generating interactive virtual objects (which may be copies or digital twins of real-world objects that can interact with the AR environment). For example, Figure 7B-1 and Figure 7B-2 shows a VR device 710 having cameras 739A to 739D, which can be used to provide depth information for creating a voxel field and a 2D mesh to provide object information to the user to avoid collisions.

[0186] The optional housing 790 may include components similar to those described above with reference to the computing system 720. For example, the optional housing 790 may include a corresponding peripheral interface 722B, which includes more or fewer components than those described above with reference to the peripheral interface 722A. As described above, the components of the optional housing 790 may be used to enhance and / or extend the functionality of the head-wearable device. For example, the optional housing 790 may include corresponding sensors 723B, speakers 736B, displays 735B, microphones 737B, cameras 738B, and / or other components for collecting and / or presenting data. Similarly, the optional housing 790 may include one or more processors 748B, controllers 746B, and / or memories 750B (including corresponding communication interface modules 753B, one or more graphics modules 754B, one or more AR processing modules 755B), and the above components may be used alone and / or in combination with the components of the computing system 720.

[0187] The techniques described above in Figures 7A to 7C can be used with different head-wearable devices. In some embodiments, a head-wearable device (e.g., an AR device 700 and / or a VR device 710) may be used in combination with one or more wearable devices, such as a wrist-wearable device 600 (or its components) and / or a textile-based smart garment 900( Figures 9A to 9C)and HIPD 800.

[0188] The example head wearable device has been described in this manner. Now, attention will be turned to an example handheld intermediate processing device, such as HIPD 800.

[0189] Example handheld intermediate processing device

[0190] Figure 8A and Figure 8B illustrates an example handheld intermediate processing device (HIPD) 800 according to some embodiments. HIPD 800 can perform various functions and / or operations associated with navigating in a user interface and selectively launching applications.

[0191] Figure 8A Shows a top view 805 and a side view 825 of HIPD 800. HIPD 800 is configured to be communicatively coupled with one or more wearable devices (or other electronic devices) associated with a user. For example, HIPD 800 is configured to be communicatively coupled with the user's wrist wearable device 600 (or its components, such as the watch body 620 and the wearable band 610), the AR device 700, and / or the VR device 710. HIPD 800 can be configured to be held by a user (e.g., as a handheld controller), carried by the user (e.g., in their pocket or in their bag), placed near the user (e.g., on their table when the user is sitting at a table, or on a charging dock), and / or placed at or within a predetermined distance from the wearable device or other electronic device (e.g., where, in some embodiments, the predetermined distance is the maximum distance at which HIPD 800 can successfully communicatively couple with an electronic device (e.g., a wearable device) (e.g., 10 meters)).

[0192] The HIPD 800 can perform various functions independently and / or in combination with one or more wearable devices (e.g., wrist wearable device 600, AR device 700, and / or VR device 710). The HIPD 800 is configured to enhance and / or improve the functionality of devices that are communicatively coupled (e.g., wearable devices). The HIPD 800 is configured to perform one or more functions or operations associated with: interacting with the user interfaces and applications of communicatively coupled devices, interacting with an AR environment, interacting with a VR environment, and / or serving as a human-machine interface controller. Additionally, as will be described in more detail below, the functions and / or operations of the HIPD 800 may include, but are not limited to: task transfer and / or conveyance; heat transfer and / or conveyance; six degrees of freedom (6DoF) ray casting and / or gaming (e.g., using imaging devices or cameras 814A and 814B, which may be used for simultaneous localization and mapping (SLAM) and / or in conjunction with other image processing techniques); portable charging; messaging; image acquisition via one or more imaging devices or cameras (e.g., cameras 822A and 822B); sensing user input (e.g., sensing touches on the multi-touch input surface 802); wireless communication and / or interconnectivity (e.g., cellular, near field, Wi-Fi, or personal area network); location determination; financial transactions; providing haptic feedback; alerts; notifications; biometric authentication; health monitoring; sleep monitoring, etc. The above example functions may be performed independently in the HIPD 800 and / or in the communication between the HIPD 800 and another wearable device described herein. In some embodiments, the functions may be performed in conjunction with an AR environment on the HIPD 800. As those skilled in the art will recognize upon reading the various descriptions provided herein, the novel HIPD 800 described herein may be used with any type of suitable AR environment.

[0193] When the HIPD 800 is communicatively coupled to a wearable device and / or other electronic device, the HIPD 800 is configured to perform one or more operations initiated at the wearable device and / or the other electronic device. In particular, one or more operations of the wearable device and / or other electronic device can be transferred to the HIPD 800 for execution. The HIPD 800 performs one or more operations of the wearable device and / or other electronic device and provides data corresponding to the completed operations to the wearable device and / or the other electronic device. For example, a user can initiate a video stream using the AR device 700, and the backend tasks associated with executing the video stream (e.g., video rendering) can be transferred to the HIPD 800, which executes the backend tasks and provides the corresponding data to the AR device 700 to perform the remaining frontend tasks associated with the video stream (e.g., presenting the rendered video data via the display of the AR device 700). In this way, the HIPD 800, which has more computing resources and a larger thermal margin than the wearable device, can perform computationally intensive tasks for the wearable device, thereby improving the performance of the operations performed by the wearable device.

[0194] The HIPD 800 includes a multi-touch input surface 802 on a first side (e.g., the front surface), which is configured to detect one or more user inputs. In particular, the multi-touch input surface 802 can detect single-tap inputs, multi-tap inputs, swipe gestures and / or inputs, force- and / or pressure-based touch inputs, and hold-and-click, etc. The multi-touch input surface 802 is configured to detect capacitive touch inputs and / or force (and / or pressure) touch inputs. The multi-touch input surface 802 includes a first touch input surface 804 defined by a surface depression and a second touch input surface 806 defined by a substantially flat portion. The first touch input surface 804 can be disposed adjacent to the second touch input surface 806. In some embodiments, the first touch input surface 804 and the second touch input surface 806 can be of different sizes, shapes, and / or can cover different portions of the multi-touch input surface 802. For example, the first touch input surface 804 can be generally circular, and the second touch input surface 806 is generally rectangular. In some embodiments, the surface depression of the multi-touch input surface 802 is configured to guide the user's manipulation of the HIPD 800. In particular, the surface depression is configured such that the user holds the HIPD 800 upright when holding it in one hand (e.g., such that the imaging devices or cameras 814A and 814B point towards the ceiling or sky). Additionally, the surface depression is configured such that the user's thumb is located within the first touch input surface 804.

[0195] In some embodiments, different touch input surfaces include multiple touch input regions. For example, the second touch input surface 806 includes at least a first touch input region 808 within the second touch input region 806, and a third touch input region 810 within the first touch input region 808. In some embodiments, one or more of the multiple touch input regions are optional and / or user-defined (e.g., a user can specify a touch input region based on their preference). In some embodiments, each touch input surface and / or touch input region is associated with a predetermined set of commands. For example, a user input detected within the first touch input region 808 causes the HIPD 800 to execute a first command, and a user input detected within the second touch input region 806 causes the HIPD 800 to execute a second command different from the first command. In some embodiments, different touch input surfaces and / or touch input regions are configured to detect one or more types of user input. Different touch input surfaces and / or touch input regions can be configured to detect the same type or different types of user input. For example, the first touch input region 808 can be configured to detect force touch input (e.g., the magnitude of a user press) and capacitive touch input, and the second touch input region 806 can be configured to detect capacitive touch input.

[0196] The HIPD 800 includes one or more sensors 851 for sensing data used in performing one or more operations and / or functions. For example, the HIPD 800 can include an IMU that is used in conjunction with a camera 814 for 3D object manipulation in an AR environment or a VR environment (e.g., zooming in on, moving, or destroying an object). Non-limiting examples of the sensors 851 included in the HIPD 800 include light sensors, magnetometers, depth sensors, pressure sensors, and force sensors. Additional examples of the sensors 851 are provided below with reference to Figure 8B Provide additional examples of the sensors 851.

[0197] The HIPD 800 may include one or more light indicators 812 to provide one or more notifications to the user. In some embodiments, the light indicator is an LED or other type of lighting device. The light indicator 812 may act as a privacy light to notify the user and / or others near the user that the imaging device and / or microphone is active. In some embodiments, the light indicator is positioned near one or more touch input surfaces. For example, the light indicator may be positioned around the first touch input surface 804. The light indicator may illuminate in different colors and / or patterns to provide one or more notifications and / or information to the user. For example, the light indicator positioned around the first touch input surface 804 may blink when the user receives a notification (e.g., a message), turn red when the HIPD 800 loses power, act as a progress bar (e.g., a halo that closes when a task is completed (e.g., 0% to 100%)), or act as a volume indicator.

[0198] In some embodiments, the HIPD 800 includes one or more additional sensors on another surface. For example, as Figure 8A shown, the HIPD 800 includes a group of one or more sensors (e.g., sensor group 820) on the edge of the HIPD 800. When positioned on the edge of the HIPD 800, the sensor group 820 may be pre-positioned at a predetermined tilt angle (e.g., 26 degrees), which allows the sensor group 820 to tilt towards the user when placed on a table or other flat surface. Alternatively, in some embodiments, the sensor group 820 is positioned on the surface opposite the multi-touch input surface 802 (e.g., the back). One or more sensors in the sensor group 820 are discussed in detail below.

[0199] A side view 825 of the HIPD 800 shows the sensor group 820 and the camera 814B. The sensor group 820 includes one or more cameras 822A and 822B, a depth projector 824, an ambient light sensor 828, and a depth receiver 830. In some embodiments, the sensor group 820 includes a light indicator 826. The light indicator 826 can act as a privacy indicator to let the user and / or people around the user know that the camera and / or microphone is active. The sensor group 820 is configured to capture the user's facial expressions so that the user can manipulate a customized avatar (e.g., display emotions (such as smiling or laughing) on the user's avatar or digital representation). The sensor group 820 can be configured as a side stereo red, green, blue (RGB) system, a rear indirect Time-of-Flight (iToF) system, or a rear stereo RGB system. As those skilled in the art will understand when reading the description provided herein, the HIPD 800 described herein can use different sensor group 820 configurations and / or sensor group 820 arrangements.

[0200] In some embodiments, the HIPD 800 includes one or more haptic devices 871 ( Figure 8B ; e.g., vibrotactile actuators), and the one or more haptic devices are configured to provide haptic feedback (e.g., kinesthetic). The sensors 851 and / or the haptic devices 871 can be configured to operate in conjunction with a plurality of applications and / or communication-coupled devices, and the plurality of applications and / or the communication-coupled devices include, but are not limited to, wearable devices, health monitoring applications, social media applications, gaming applications, and AR applications (e.g., applications associated with artificial reality).

[0201] The HIPD 800 is configured to operate without a display. However, in an alternative embodiment, the HIPD 800 can include a display 868 ( Figure 8B ). The HIPD 800 can also include one or more optional peripheral buttons 867 ( Figure 8B ). For example, the peripheral button 867 can be used to turn the HIPD 800 on or off. Additionally, the housing of the HIPD 800 can be formed of a polymer and / or an elastomer. The HIPD 800 can be configured to have a non-slip surface to allow the HIPD 800 to be placed on a surface without the need for user supervision of the HIPD 800. In other words, the HIPD 800 is designed so that it will not easily slip off a surface. In some embodiments, the HIPD 800 includes one or more magnets for coupling the HIPD 800 to another surface. This allows the user to place the HIPD 800 on different surfaces and provides the user with greater flexibility in using the HIPD 800.

[0202] As described above, the HIPD 800 can distribute and / or provide instructions for performing one or more tasks at the HIPD 800 and / or a communicatively coupled device. For example, the HIPD 800 can identify one or more backend tasks to be performed by the HIPD 800 and one or more frontend tasks to be performed by a communicatively coupled device. Although the HIPD 800 is configured to transfer and / or convey tasks of a communicatively coupled device, the HIPD 800 can (e.g., Figure 8B ; via one or more processors, such as the CPU 877; Figure 8B ) perform both backend tasks and frontend tasks. The HIPD 800 can be used to perform, but is not limited to: enhanced calling (e.g., receiving and / or sending 3D or 2.5D real-time volumetric calls, real-time digital human representation calls, and / or avatar calls), discreet messaging, 6DoF portrait / landscape gaming, AR / VR object manipulation, AR / VR content display (e.g., presenting content via a virtual display), and / or other AR / VR interactions. The HIPD 800 can perform the above operations alone or in combination with a wearable device (or other communicatively coupled electronic device).

[0203] Figure 8B FIG. shows a block diagram of the HIPD computing system 840 of the HIPD 800 according to some embodiments. The HIPD 800 described in detail above can include one or more components shown in the HIPD computing system 840. The HIPD 800 will be understood to include the various components shown and described below for the HIPD computing system 840. In some embodiments, all or most of the components of the HIPD computing system 840 are included in a single integrated circuit. Alternatively, in some embodiments, the components of the HIPD computing system 840 are included in a plurality of communicatively coupled integrated circuits.

[0204] The HIPD computing system 840 may include: a processor (e.g., CPU 877, GPU, and / or a CPU with integrated graphics); a controller 875; a peripheral interface 850 that includes one or more sensors 851 and other peripheral devices; a power supply (e.g., power system 895); and a memory (e.g., memory 878) that includes an operating system (e.g., operating system 879), data (e.g., data 888), one or more applications (e.g., application 880), and one or more modules (e.g., communication interface module 881, graphics module 882, task and process management module 883, interoperability module 884, AR processing module 885, and / or data management module 886). The HIPD computing system 840 also includes a power system 895 that includes a charger input and output 896, a PMIC 897, and a battery 898, all of the above components being defined above.

[0205] In some embodiments, the peripheral interface 850 may include one or more sensors 851. The sensors 851 may include sensors similar to those described above with reference to Figure 6B For example, the sensors 851 may include an imaging sensor 854, an (optional) EMG sensor 856, an IMU 858, and a capacitive sensor 860. In some embodiments, the sensors 851 may include: one or more pressure sensors 852 for sensing pressure data; an altimeter 853 for sensing the height of the HIPD 800; a magnetometer 855 for sensing magnetic fields; a depth sensor 857 (or time-of-flight sensor) for determining the difference between the camera and an object in an image; a position sensor 859 (e.g., a flexible position sensor) for sensing relative displacement or change in position of a part of the HIPD 800; a force sensor 861 for sensing the force applied to a part of the HIPD 800; and a light sensor 862 (e.g., an ambient light sensor) for detecting the amount of light. The sensors 851 may include Figure 8B one or more sensors not shown in

[0206] Similar to the peripheral devices described above with reference to Figure 6B The peripheral interface 850 may also include an NFC component 863, a GPS component 864, an LTE component 865, Wi-Fi and / or Bluetooth communication components 866, a speaker 869, a haptic device 871, and a microphone 873. As described above with reference to Figure 8A The HIPD 800 may optionally include a display 868 and / or one or more buttons 867. The peripheral interface 850 may also include one or more cameras 870, a touch surface 872, and / or one or more light emitters 874. As described above with reference toFigure 8A The described multi-touch input surface 802 is an example of a touch surface 872. The light emitter 874 can be one or more LEDs, lasers, etc., and can be used to project or present information to the user. For example, the light emitter 874 can include the light indicators 812 and 826 described above with reference to Figure 8A The camera 870 (e.g., the cameras 814A, 814B, and 822 described above in Figure 8A ) can include one or more wide-angle cameras, fish-eye cameras, spherical cameras, compound-eye cameras (e.g., stereo and multi-camera), depth cameras, RGB cameras, time-of-flight (ToF) cameras, RGB-D cameras (depth and ToF cameras), and / or other available cameras. The camera 870 can be used for: SLAM; 6DoF ray casting, gaming, object manipulation, and / or other rendering; face recognition and facial expression recognition, etc.

[0207] Similar to the watch body computing system 660 and the band computing system 630 described above with reference to Figure 6B , the HIPD computing system 840 can include one or more haptic controllers 876 and associated components (e.g., haptic devices 871), which are used to provide haptic events at the HIPD 800.

[0208] The memory 878 can include high-speed random access memory and / or non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Access to the memory 878 by other components of the HIPD 800 (e.g., one or more processors and the peripheral interface 850) can be controlled by the memory controller in the controller 875.

[0209] In some embodiments, the software components stored in the memory 878 include one or more operating systems 879, one or more applications 880, one or more communication interface modules 881, one or more graphics modules 882, and one or more data management modules 885, and these software components are similar to the software components described above with reference to Figure 6B .

[0210] In some embodiments, the software components stored in memory 878 include a task and process management module 883 that is configured to: identify one or more front-end tasks and back-end tasks associated with an operation performed by a user; execute one or more front-end tasks and / or back-end tasks; and / or provide instructions to one or more communicatively coupled devices to cause the execution of the one or more front-end tasks and / or back-end tasks. In some embodiments, the task and process management module 883 uses data 888 (e.g., device data 890) to distribute one or more front-end tasks and / or back-end tasks based on: the computing resources of the communicatively coupled devices, available power, thermal margin, ongoing operations, and / or other factors. For example, the task and process management module 883 may cause one or more back-end tasks (of an operation performed at the communicatively coupled AR device 700) to be executed at the HIPD 800 based on determining that the operation is utilizing a predetermined amount (e.g., at least 70%) of the computing resources available at the AR device 700.

[0211] In some embodiments, the software components stored in memory 878 include an interoperability module 884 that is configured to exchange and utilize information received and / or provided to different communicatively coupled devices. The interoperability module 884 allows different systems, devices, and / or applications to connect and communicate in a coordinated manner without user input. In some embodiments, the software components stored in memory 878 include an AR module 885 that is configured to process signals based at least on sensor data for use in an AR and / or VR environment. For example, the AR processing module 885 may be used for 3D object manipulation, gesture recognition, face recognition, and facial expression recognition.

[0212] Memory 878 may also include data 887 that includes structured data. In some embodiments, data 887 may include profile data 889, device data 889 (including device data of one or more devices communicatively coupled to the HIPD 800, such as device type, hardware, software, and configuration), sensor data 891, media content data 892, and application data 893.

[0213] It should be understood that the HIPD computing system 840 is an example of a computing system within the HIPD 800, and the HIPD 800 may have more components or fewer components than those shown in the HIPD computing system 840, may combine two or more components, and / or may have different configurations and / or arrangements of these components. Each of the components shown in the HIPD computing system 840 is implemented in hardware, software, firmware, or a combination thereof (including one or more signal processing circuits and / or application specific integrated circuits).

[0214] As described above in Figure 8A and Figure 8B The various techniques described can be used with any device used as a human-machine interface controller. In some embodiments, the HIPD 800 can be used in combination with one or more wearable devices (such as head-wearable devices (such as, AR device 700 and VR device 710) and / or wrist-wearable devices 600 (or their components)). In some embodiments, the HIPD 800 can also be used in combination with wearable clothing (such as textile-based smart clothing 900( Figures 9A to 9C ))). The example HIPD 800 has been described in this way. Now, attention will be turned to an example feedback device (such as textile-based smart clothing 900).

[0215] Example textile-based smart clothing

[0216] Figure 9A and Figure 9B show an example textile-based smart clothing according to some embodiments. The textile-based smart clothing 900 (such as, wearable gloves, shirts, headbands, wristbands, and / or socks) is configured to be communicatively coupled with one or more electronic devices (such as wrist-wearable devices 600, head-wearable devices, HIPD 800, laptop computers, tablet computers, and / or other computing devices). The textile-based smart clothing 900 can perform various functions and / or operations associated with navigating in a user interface and selectively launching applications, as well as the functions and / or operations described above with reference to Figures 1A to 3D (additionally or alternatively, with reference to the wrist-wearable device 104).

[0217] The textile-based smart clothing 900 can be part of an AR system (such as, the AR system 500d described above with reference to Figure 5D-1 and Figure 5D-2 ). The textile-based smart clothing 900 is also configured to provide feedback (such as, tactile feedback or other haptic feedback) to a user based on the user's interaction with a computing system (such as, navigating a user interface, operating an application (such as, game vibration, or media-responsive touch), device notifications) and / or the user's interaction within an AR environment. In some embodiments, the textile-based smart clothing 900 receives instructions from communicatively coupled devices (such as, wrist-wearable devices 600, head-wearable devices, and HIPD 800) to cause a feedback response to be executed. Alternatively or additionally, in some embodiments, the textile-based smart clothing 900 determines one or more feedback responses to be provided to the user. The textile-based smart clothing 900 can be based on one or more sensors among its multiple sensors (such as, Figure 9C;sensor data collected by a sensor 951) or a communicatively coupled sensor (e.g., a sensor of the wrist-wearable device 600, a sensor of the head-wearable device, a sensor of the HIPD 800, and / or a sensor of other computing devices) to determine one or more feedback responses.

[0218] Non-limiting examples of feedback determined by the textile-based smart garment 900 and / or communicatively coupled devices include visual feedback, audio feedback, tactile (e.g., haptic or kinesthetic) feedback, thermal or temperature feedback, and / or other sensorially perceivable feedback. The textile-based smart garment 900 may include corresponding feedback devices (e.g., the tactile device or component 962 or other feedback devices or components) for providing feedback responses to the user. Similarly, the textile-based smart garment 900 may be communicatively coupled to another device (and / or the feedback device of another device) to coordinate the feedback provided to the user. For example, the VR device 710 may present an AR environment to the user, and when the user interacts with an object (e.g., a virtual cup) within the AR environment, the textile-based smart garment 900 provides a corresponding response to the user. In particular, the textile-based smart garment 900 may provide haptic feedback that prevents one or more fingers of the user from bending beyond a certain point (or at least impedes / prevents the movement of one or more fingers of the user) to simulate the feeling of touching a solid cup, and / or provide thermal feedback that simulates the feeling of a cold or hot beverage.

[0219] Additionally or alternatively, in some embodiments, the textile-based smart garment 900 is configured to act as a controller that is configured to perform one or more functions or operations associated with: interacting with the user interfaces and applications of communicatively coupled devices, interacting with an AR environment, interacting with a VR environment, and / or acting as a human-machine interface controller.

[0220] Figure 9A shows one or more tactile components 962 (e.g., the first tactile component 962-1 to the fourth tactile component 962-4) on a portion of the palm side of the textile-based smart garment 900 adjacent to the user's hand, and Figure 9BShows an additional tactile component (e.g., the fifth tactile component 962-5) on a portion of the back of the hand side of the textile-based smart garment 900 adjacent to the user's hand. In some embodiments, the tactile component 962 includes such an mechanism: the mechanism provides resistance at least when the corresponding tactile component 962 transitions from a first state (e.g., a first pressurized state (e.g., at atmospheric pressure or deflated)) to a second state (e.g., a second pressurized state (e.g., inflated to a threshold pressure)). In other words, the described tactile component 962 can transition between a first pressurized state and a second pressurized state to provide tactile feedback to the user. The structure of the tactile component 962 can be integrated into various devices configured to contact or be close to the user's skin, including but not limited to devices such as glove-wearing devices, body-worn garment devices, and head-mounted viewer devices. Each of the tactile components 962 can be included in or physically coupled to a garment component 904 of the textile-based smart garment 900. For example, each of the tactile components 962-1, 962-2, 962-3, ……, 962-N is physically coupled to the garment 904 and is configured to contact the corresponding phalanges of the user's thumb and fingers.

[0221] Due to the ever-changing nature of artificial reality, the tactile component 962 may need to transition between multiple states hundreds or possibly thousands of times during a single use. Therefore, the tactile component 962 described herein is durable and is designed to quickly transition between states. To provide some context, in the first pressurized state, the tactile component 962 does not impede the free movement of a part of the wearer's body. For example, one or more tactile components 962 incorporated into a glove are made of a flexible material (e.g., an electrostatic zipper actuator) that does not impede the free movement of the wearer's hand and fingers. The tactile component 962 is configured to conform to the shape of a part of the wearer's body when in the first pressurized state. However, when in the second pressurized state, the tactile component 962 can be configured to restrict and / or impede the free movement of a part of the wearer's body (e.g., an appendage of the user's hand). For example, the corresponding tactile component 962 (or corresponding tactile components) can restrict the movement of the wearer's finger (e.g., prevent the finger from bending or extending) when the tactile component 962 is in the second pressurized state. Additionally, when in the second pressurized state, the tactile component 962 can adopt different shapes, some of which are configured to adopt a planar, rigid (e.g., flat and rigid) shape, while some other tactile components 962 are configured to be at least partially curved or bent.

[0222] The textile-based smart garment 900 can be an AR system (e.g., Figures 5A to 5D-2One of the multiple devices in the AR system). For example, the user can wear a pair of gloves (e.g., the first type of textile-based smart clothing 900), wear a wrist-wearable device 600( Figure 6A and Figure 6B ), the tactile component of), wear a headband (e.g., the second type of textile-based smart clothing 900), or hold a HIPD 800. As explained above, the tactile component 962 is configured to provide a tactile simulation to the wearer of the textile-based smart clothing 900. The clothing 904 of each textile-based smart clothing 900 can be one of various clothing items (e.g., gloves, socks, shirts, and / or pants). Thus, the user can wear multiple textile-based smart clothing 900s, each configured to provide tactile stimulation to the corresponding part of the body wearing the textile-based smart clothing 900. Although the textile-based smart clothing 900 is described as a separate device, in some embodiments, the textile-based smart clothing 900 can be combined with other wearable devices described herein. For example, the textile-based smart clothing 900 can form part of a VR device 710 (e.g., the headband part).

[0223] Figure 9C Shows a block diagram of a computing system 940 of a tactile component 962 according to some embodiments. The computing system 940 can include one or more peripheral interfaces 950, one or more power systems 995 (including a charger input 996, a PMIC 997, and a battery 998), one or more controllers 975 (including one or more tactile controllers 976), one or more processors 977 (as defined above, including any of the provided examples), and a memory 978, and these components can all communicate electronically with each other. For example, one or more processors 977 can be configured to execute instructions stored in the memory 978, and the instructions can cause a controller in one or more controllers 975 to perform multiple operations at one or more peripheral devices of the peripheral interface 950. In some embodiments, each of the described operations can be performed based on the power provided by the power system 995.

[0224] In some embodiments, the peripheral interface 950 can include one or more devices configured as part of the computing system 940, and many of the one or more devices have been defined and / or referenced above Figures 6A to 8BThe wrist-wearable device shown in [the figure] is described. For example, the peripheral interface 950 may include one or more sensors 951, such as one or more pressure sensors 952, one or more EMG sensors 956, one or more IMUs 958, one or more position sensors 959, one or more capacitive sensors 960, one or more force sensors 961; and / or any other type of sensor defined above or described with reference to any other embodiment discussed herein. In some embodiments, the peripheral interface may include one or more additional peripheral devices, the one or more additional peripheral devices including: one or more Wi-Fi and / or Bluetooth devices 968; an LTE component 969; a GPS component 970; a microphone 971; one or more haptic components 962; and one or more support structures 963 (the one or more support structures 963 may include one or more bladders 964; one or more manifolds 965; one or more pressure-changing devices 967; one or more displays 972; one or more buttons 973; one or more speakers 974; and / or any other type of peripheral device defined above or described with reference to any other embodiment discussed herein). In some embodiments, the computing system 940 includes more components or fewer components than Figure 9C shown in [the figure].

[0225] In some embodiments, each haptic component 962 includes a support structure 963 and at least one bladder 964. The bladder 964 (e.g., a membrane) is a sealed, expandable pocket made of a durable and puncture-resistant material such as thermoplastic polyurethane (TPU) or a flexible polymer. The bladder 964 contains a medium (e.g., a fluid such as air, an inert gas, or even a liquid) that can be added to or removed from the bladder 964 to change the pressure (e.g., fluid pressure) inside the bladder 964. The support structure 963 is made of a material that is stronger and stiffer than the material of the bladder 964. The respective support structure 963 coupled to the respective bladder 964 is configured to reinforce the respective bladder 964 as the respective bladder changes shape and size in response to a change in pressure (e.g., fluid pressure) within the bladder. The haptic component 962 example above is non-limiting. The haptic component 962 may include an eccentric rotating mass (ERM), a linear resonant actuator (LRA), a voice coil motor (VCM), a piezoelectric haptic actuator, a thermoelectric device, an electromagnetic coil actuator, an ultrasonic transducer, a resistive heater, a Peltier device, and / or other devices configured to generate a perceptible response.

[0226] The textile-based smart garment 900 also includes a haptic controller 976 and a pressure-changing device 967. Alternatively, in some embodiments, the computing system 940 is communicatively coupled to the haptic controller 976 and / or the pressure-changing device 967 (e.g., in electronic communication with one or more processors 977 of the computing system 940). The haptic controller 976 is configured to control the operation of the pressure-changing device 967 and, in turn, control the operation of the textile-based smart garment 900. For example, the haptic controller 976 sends one or more signals to the pressure-changing device 967 to activate the pressure-changing device 967 (e.g., turn the pressure-changing device on and off). The one or more signals can specify the desired pressure (e.g., pounds per square inch) to be output by the pressure-changing device 967. The generation of the one or more signals and, in turn, the pressure output by the pressure-changing device 967 can be based on information collected by sensors 951 of the textile-based smart garment 900 and / or other communicatively coupled devices. For example, the haptic controller 976 can provide one or more signals based on the collected sensor data to cause the pressure-changing device 967 to increase the pressure (e.g., fluid pressure) inside the first haptic component 962 at a first time; and the haptic controller 976 can provide one or more additional signals to the pressure-changing device 967 based on additional sensor data to cause the pressure-changing device 967 to further increase the pressure inside the second haptic component 962 at a second time after the first time. Additionally, the haptic controller 976 can provide one or more signals to cause the pressure-changing device 967 to inflate one or more bladders 964 located in a first portion (e.g., a first finger) of the textile-based smart garment 900 while one or more bladders 964 located in a second portion (e.g., a second finger) of the textile-based smart garment 900 remain unchanged. Further, the haptic controller 976 can provide one or more signals to cause the pressure-changing device 967 to inflate one or more bladders 964 located in the first textile-based smart garment 900 to a first pressure and inflate one or more other bladders 964 located in the first textile-based smart garment 900 to a second pressure different from the first pressure. Depending on the number of textile-based smart garments 900 served by the pressure-changing device 967 and the number of bladders in these textile-based smart garments, many different inflation configurations can be achieved via one or more signals, and the above examples are not meant to be limiting.

[0227] The textile-based smart garment 900 can include an optional manifold 965 that is located between the pressure-changing device 967, the haptic component 962, and / or other parts of the textile-based smart garment 900. The manifold 965 can include one or more valves (not shown) that pneumatically couple each of the plurality of haptic components 962 to the pressure-changing device 967 via conduits. In some embodiments, the manifold 965 communicates with the controller 975, and the controller 975 controls one or more valves of the manifold 965 (e.g., the controller generates one or more control signals). The manifold 965 is configured to switchably couple the pressure-changing device 967 to one or more haptic components 962 of the textile-based smart garment 900. In some embodiments, one or more textile-based smart garments 900 or other haptic devices can be coupled in a network of haptic devices, and the manifold 965 can distribute fluid among the plurality of coupled textile-based smart garments 900.

[0228] In some embodiments, instead of using the manifold 965 to pneumatically couple the pressure-changing device 967 to the haptic component 962, the textile-based smart garment 900 can include a plurality of pressure-changing devices 967, where each pressure-changing device 967 is directly pneumatically coupled to a single (or multiple) haptic component 962. In some embodiments, the pressure-changing device 967 and the optional manifold 965 can be configured as part of one or more textile-based smart garments 900 (not shown), while in other embodiments, the pressure-changing device 967 and the optional manifold 965 can be configured to be external to the textile-based smart garment 900. In some embodiments, a single pressure-changing device 967 can be shared by a plurality of textile-based smart garments 900 or other haptic devices. In some embodiments, the pressure-changing device 967 is a pneumatic device, a hydraulic device, a pneumatic-hydraulic device, or some other device capable of adding a medium (e.g., fluid, liquid, or gas) to and removing the medium from one or more haptic components 962.

[0229] The memory 978 includes instructions and data, some or all of which can be stored in the memory 978 as a non-transitory computer-readable storage medium. For example, the memory 978 can include: one or more operating systems 979; one or more communication interface applications 981; one or more interoperability modules 984; one or more AR processing applications 985; one or more data management modules 986; and / or any other type of data defined above or referenced Figures 6A to 8B and described herein.

[0230] Memory 978 also includes data 988, which can be used in conjunction with one or more of the multiple applications discussed above. The data 988 can include: device data 990; sensor data 991; and / or any other type of data defined or referenced above Figures 6A to 8B and described.

[0231] Figures 9A to 9C The different components of the computing system 940 (and the textile-based smart garment 900) shown in can be coupled via a wired connection (e.g., via a bus). Alternatively, Figures 9A to 9C one or more of the multiple devices shown in can be wirelessly connected (e.g., via a short-range communication signal).

[0232] Example embodiments

[0233] Now turning to some example embodiments of the methods, devices, systems, and computer-readable storage media described above.

[0234] (A1) In one aspect, some embodiments include a method of providing a force dimension to an interface element (e.g., method 400). In some embodiments, the method is performed at a wearable device (e.g., a wrist wearable device 600, an AR device 700, and / or a textile-based garment 900) having a memory (e.g., memory 680) and one or more processors (e.g., one or more processors 679). The method includes: (i) communicatively coupling the wearable device (e.g., the wrist wearable device 104) to an activatable device (e.g., an electronic device 202), the wearable device including one or more neuromuscular signal sensors (e.g., an EMG sensor 665 and / or an EMG sensor 956), the activatable device including a mechanical user interface element (e.g., a button 205); (ii) obtaining data from the one or more neuromuscular signal sensors in response to activation of the mechanical user interface element (e.g., Figure 1C; a voltage signal 116), the data corresponding to the activation; (iii) based on data from the one or more neuromuscular signal sensors, determining whether the activation of the mechanical user interface element includes an activation force that meets one or more predefined criteria (e.g., determining whether signal 116 meets or exceeds a threshold f0); (iv) in accordance with determining that the activation of the mechanical user interface element includes the activation force that meets one or more predefined criteria, causing a first function corresponding to the mechanical user interface element to be executed (e.g., a digital assistant function); and (v) in accordance with determining that the activation of the mechanical user interface element does not include the activation force that meets one or more predefined criteria, causing a second function corresponding to the mechanical user interface element to be executed (e.g., a playback function), wherein the second function is different from the first function. For example, the activatable device is a light switch, a speaker, or a digital assistant device, etc. In some embodiments, the one or more predefined criteria include a force threshold and / or a voltage threshold. In some embodiments, the communication coupling includes Bluetooth coupling or Bluetooth Low Energy (BLE) coupling.

[0235] (A2) In some embodiments of A1, the mechanical user interface element is a switch, a joystick, or a button. For example, the mechanical user interface element is a button or a lever on a controller. In some embodiments, the activatable device includes one or more capacitive (and / or inductive) user interface elements (e.g., without associated force sensing capabilities).

[0236] (A3) In some embodiments of A1 or A2, the activatable device does not include a display. For example, the activatable device does not include a touch screen, a liquid crystal display (LCD), or a light emitting diode (LED) display.

[0237] (A4) In some embodiments of any of A1 to A3: (i) the mechanical user interface element is a first mechanical user interface element; (ii) the activatable device includes a second mechanical user interface element; (iii) the second mechanical user interface element is associated with one or more predefined force criteria that are different from the one or more predefined criteria associated with the first mechanical user interface element. For example, the electronic device 202 may include a button 205 and one or more additional physical user interface elements (e.g., buttons, joysticks, and / or switches). In this example, the additional physical user interface elements (e.g., based on the type of the user interface element and / or the material of the user interface element) have different force criteria.

[0238] (A5)In some embodiments of any one of A1 to A4, the method further includes: receiving, at the wearable device, a request for force data from an activatable device, wherein the wearable device is communicatively coupled to the activatable device in response to the request. In some embodiments, in response to the request, the wearable device and the activatable device automatically handshake and / or pair (e.g., establish a Bluetooth connection or a Wi-Fi connection). In some embodiments, the request is broadcast to devices within the communication range of the activatable device.

[0239] (A6)In some embodiments of any one of A1 to A5, the method further includes: receiving, at the activatable device, a broadcast communication from the wearable device that identifies the force capabilities of the wearable device, wherein the wearable device is communicatively coupled to the activatable device in response to the broadcast communication. In some embodiments, in response to the broadcast communication, the wearable device and the activatable device automatically handshake and / or pair (e.g., establish a Bluetooth connection or a Wi-Fi connection).

[0240] (A7)In some embodiments of any one of A1 to A6, the wearable device is or includes a wrist wearable device (e.g., a smartwatch, an armband, or a bracelet). In some embodiments, the wearable device is a glove, a sleeve, or other apparel (e.g., textile-based apparel 900).

[0241] (A8)In some embodiments of any one of A1 to A7, the method further includes: providing a notification (e.g., notification 314) to a user of the wearable device, the notification including information about being communicatively coupled to the activatable device and / or information about providing data from the one or more neuromuscular signal sensors. For example, the notification is an audio notification and / or a visual notification provided at the wearable device. In some embodiments, the notification includes information about interaction options available based on sensors communicatively coupled to the activatable device.

[0242] (A9)In some embodiments of any one of A1 to A8, the method further includes: providing information about a first function and a second function to the user before detecting an activation of a mechanical user interface element (e.g., providing interaction options 206 or 308). In some embodiments, the information is provided via a display coupled to the activatable device (e.g., a separate display device and / or a display component of the activatable device and / or the wearable device).

[0243] (A10)In some embodiments of A9, the information about the first function and the second function is displayed to the user via a head wearable device (e.g., AR glasses). For example, interaction option 206 is displayed to the user via head wearable device 204.

[0244] (A11)In some embodiments of any one of A1 to A10, the wearable device is communicatively coupled to the activatable device based on the wearable device being in the vicinity of the activatable device. For example, the wearable device is coupled based on the user being close enough to view and / or interact with the activatable device. As another example, the wearable device is coupled to the activatable device based on the wearable device being within the communication range of the activatable device.

[0245] (A12)In some embodiments of any one of A1 to A11, the method further comprises: (i) communicatively coupling the wearable device to a second activatable device, the second activatable device including a second mechanical user interface element; (ii) obtaining additional data from the one or more neuromuscular signal sensors corresponding to an activation of the second mechanical user interface element; (iii) based on the additional data from the one or more neuromuscular signal sensors, determining whether the activation of the second mechanical user interface element includes a corresponding force that meets one or more predefined criteria; (iv) in response to determining that the activation of the second mechanical user interface element includes the corresponding force that meets the one or more predefined criteria, causing a first function corresponding to the second mechanical user interface element to be executed; and (v) in response to determining that the activation of the second mechanical user interface element does not include the corresponding force that meets the one or more predefined criteria, causing a second function corresponding to the second mechanical user interface element to be executed. In some embodiments, the force criteria associated with the second activatable device are different from the force criteria associated with the activatable device. In some embodiments, the wearable device provides force data to the closest activatable device communicatively coupled to the wearable device. In some embodiments, the wearable device provides force data to all activatable devices communicatively coupled to the wearable device.

[0246] (B1)In another aspect, some embodiments include a method of providing a force dimension to an interface element. In some embodiments, the method is performed at a wearable device (e.g., wrist wearable device 600, AR device 700, and / or textile-based garment 900) having a memory (e.g., memory 680) and one or more processors (e.g., one or more processors 679). The method comprises: (i) communicatively coupling the wearable device (e.g., wrist wearable device 104) to an activatable device, the wearable device including one or more neuromuscular signal sensors (e.g., EMG sensor 665), the activatable device including a physical user interface element (e.g., button 205) but not including a display (e.g., not including a screen); (ii) obtaining data from the one or more neuromuscular signal sensors in response to an activation of the physical user interface element (e.g., as Figure 1F(as shown), the data corresponds to the activation; (iii) based on data from the one or more neuromuscular signal sensors, determine whether the activation of the physical user interface element includes an activation force that meets one or more predefined criteria (e.g., determine whether the activation force exceeds a force threshold and / or a voltage threshold); (iv) according to the determination that the activation of the physical user interface element includes the activation force that meets one or more predefined criteria, cause a first function corresponding to the physical user interface element to be executed (e.g., switch to turn on / off a group of lights); and (v) according to the determination that the activation of the physical user interface element does not include the activation force that meets one or more predefined criteria, cause a second function corresponding to the physical user interface element to be executed, where the second function (e.g., switch to turn on / off the nearest light) is different from the first function.

[0247] In some embodiments, instead of using or in addition to using the one or more neuromuscular signal sensors, one or more of the postures described above (e.g., refer to Figures 1A to 3D ) are detected using an optical sensor (e.g., a camera) or a sensor associated with an inertial measurement unit (IMU). In some embodiments, one or more of the postures described above are replaced by postures performed by other parts of the user's body (e.g., head postures, leg postures, or torso postures). As an example, one or more neuromuscular signal sensors, data from an IMU, and a camera can be used to detect the posture of pressing a button.

[0248] In some embodiments, the wearable device detects neuromuscular signals transmitted through the user's neck or back. In some example embodiments, this can be done using a neuromuscular signal sensor coupled to a VR headset or AR glasses. In some embodiments, one or more of the gestures described above are replaced by (or performed using) air gestures on a controller (e.g., a handheld controller or a foot controller).

[0249] In another aspect, some embodiments include a computing system that includes one or more processors and a memory coupled to the one or more processors. The memory stores one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for performing any of the multiple methods described herein (e.g., the above method 400, and A1 to A12 and B1).

[0250] In another aspect, some embodiments include a non-transitory computer-readable storage medium storing one or more programs for execution by one or more processors of a computing system, the one or more programs including instructions for performing any of the various methods described herein (e.g., method 400 above, and A1 to A12 and B1).

[0251] Any data collection performed by the various devices described herein and / or any device configured to perform or cause to be performed the different embodiments described above with reference to any of the figures (hereinafter referred to as "devices") is performed with the consent of the user and in a manner that complies with all applicable privacy laws. The user is provided with options to allow the device to collect data and options to limit or deny the device from collecting data. The user can choose to enable or disable any data collection at any time. In addition, the user is provided with an option to request deletion of any collected data.

[0252] It will be understood that although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another.

[0253] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the claims. As used in the description of the embodiments and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms as well. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms "comprises" and / or "comprising", when used in this specification, specify the presence of the stated features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0254] As used herein, depending on the context, the term "if" can be interpreted to mean: "when the stated condition precedent is true"; or "once" the stated condition precedent is true; or "in response to determining" the stated condition precedent is true; or "in accordance with determining" the stated condition precedent is true; or "in response to detecting" the stated condition precedent is true. Similarly, depending on the context, the phrase "if it is determined that [the stated condition precedent is true]" or "if [the stated condition precedent is true]" or "when [the stated condition precedent is true]" can be interpreted to mean: "once it is determined that" the stated condition precedent is true; or "in response to determining" the stated condition precedent is true; or "in accordance with determining" the stated condition precedent is true; or "once it is detected that" the stated condition precedent is true; or "in response to detecting that" the stated condition precedent is true.

[0255] For purposes of explanation, the foregoing description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the claims to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments were chosen and described in order to best explain the principles of operation and the practical application, thereby enabling others skilled in the art to implement.

Claims

1. A method for providing a force dimension to an interface element, the method comprising: communicatively coupling a wearable device to an activatable device, the wearable device including one or more neuromuscular signal sensors, and the activatable device including a mechanical user interface element; obtaining data from the one or more neuromuscular signal sensors in accordance with an activation of the mechanical user interface element, the data corresponding to the activation; determining, based on the data from the one or more neuromuscular signal sensors, whether the activation of the mechanical user interface element includes an activation force that meets one or more predefined criteria; causing a first function corresponding to the mechanical user interface element to be performed in accordance with determining that the activation of the mechanical user interface element includes the activation force that meets the one or more predefined criteria; and causing a second function corresponding to the mechanical user interface element to be performed in accordance with determining that the activation of the mechanical user interface element does not include the activation force that meets the one or more predefined criteria, wherein the second function is different from the first function.

2. The method according to claim 1, wherein The mechanical user interface element is a switch, a joystick, or a button.

3. The method according to claim 1 or 2, wherein The activatable device does not include a display.

4. The method according to claim 1, 2, or 3, wherein: the mechanical user interface element is a first mechanical user interface element; the activatable device includes a second mechanical user interface element; and the second mechanical user interface element is associated with one or more predefined force criteria that are different from the one or more predefined criteria associated with the first mechanical user interface element.

5. The method according to any one of the preceding claims, further comprising one or more of the following: i. Receive, at the wearable device, a request for force data from the activatable device, where the wearable device is communicatively coupled to the activatable device in response to the request; and / or ii. receiving, at the activatable device, broadcast communication from the wearable device that identifies a force capability of the wearable device, wherein, in response to the broadcast communication, the wearable device is communicatively coupled to the activatable device.

6. The method according to any one of the preceding claims, wherein, The wearable device includes a wrist wearable device.

7. The method according to any one of the preceding claims, further comprising one or more of the following: i. providing a notification to a user of the wearable device, the notification including information about being communicatively coupled to the activatable device and / or information about providing data from the one or more neuromuscular signal sensors; and / or ii. Provide information about the first function and the second function to the user before detecting activation of the mechanical user interface element; and preferably, wherein, information about the first function and the second function is displayed to the user via a head wearable device.

8. The method according to any one of the preceding claims, wherein, The wearable device is communicatively coupled to the activatable device based on the wearable device being in the vicinity of the activatable device.

9. The method according to any one of the preceding claims, further comprising: communicatively coupling the wearable device to a second activatable device, the second activatable device including a second mechanical user interface element; Obtain additional data from the one or more neuromuscular signal sensors based on the activation of the second mechanical user interface element, the additional data corresponding to the activation of the second mechanical user interface element; Based on the additional data from the one or more neuromuscular signal sensors, determine whether the activation of the second mechanical user interface element includes a stress force that meets one or more predefined criteria; Based on determining that the activation of the second mechanical user interface element includes the stress force that meets one or more predefined criteria, cause a third function corresponding to the second mechanical user interface element to be executed; and Based on determining that the activation of the second mechanical user interface element does not include the stress force that meets one or more predefined criteria, cause a fourth function corresponding to the second mechanical user interface element to be executed.

10. A method of providing a force dimension to an interface element, the method comprising: Communicatively couple a wearable device to an activatable device, the wearable device including one or more neuromuscular signal sensors, the activatable device including a physical user interface element but not including a display; Obtain data from the one or more neuromuscular signal sensors based on the activation of the physical user interface element, the data corresponding to the activation; Based on the data from the one or more neuromuscular signal sensors, determine whether the activation of the physical user interface element includes a stress force that meets one or more predefined criteria; Based on determining that the activation of the physical user interface element includes the stress force that meets one or more predefined criteria, cause a first function corresponding to the physical user interface element to be executed; and Based on determining that the activation of the physical user interface element does not include the stress force that meets one or more predefined criteria, cause a second function corresponding to the physical user interface element to be executed, wherein the second function is different from the first function.

11. A wearable device, comprising: One or more neuromuscular signal sensors; Control circuitry; A memory; and One or more sets of instructions stored in the memory and configured to be executed by the control circuitry, the one or more sets of instructions including instructions for: Communicatively coupling to an activatable device that includes a mechanical user interface element; Obtain data from the one or more neuromuscular signal sensors based on the activation of the mechanical user interface element, the data corresponding to the activation; Based on the data from the one or more neuromuscular signal sensors, determine whether the activation of the mechanical user interface element includes a stress force that meets one or more predefined criteria; Based on determining that the activation of the mechanical user interface element includes the stress force that meets one or more predefined criteria, cause a first function corresponding to the mechanical user interface element to be executed; and Determining that activation of the mechanical user interface element does not include an activation force that meets one or more predefined criteria, such that a second function corresponding to the mechanical user interface element is performed, where the second function is different from the first function.

12. The wearable device according to claim 11, wherein, The mechanical user interface element is a switch, joystick, or button; and / or preferably, wherein the activatable device does not include a display.

13. The wearable device according to claim 11 or 12, wherein: The mechanical user interface element is a first mechanical user interface element; The activatable device includes a second mechanical user interface element; and The second mechanical user interface element is associated with one or more predefined force criteria that are different from the one or more predefined criteria associated with the first mechanical user interface element.

14. The wearable device according to claim 11, 12 or 13, wherein, The set or sets of instructions further include one or more of the following: i. Instructions for receiving a request for force data from the activatable device, wherein the wearable device is communicatively coupled to the activatable device in response to the request; and / or ii. Instructions for providing a notification to a user of the wearable device, the notification including information about being communicatively coupled to the activatable device and / or information about providing data from the one or more neuromuscular signal sensors.

15. The wearable device according to any one of claims 11 to 14, wherein, The wearable device includes a wrist-worn device.

Images