Multi-modal input for wearable devices and systems and methods of use thereof

Abstract

Multi-modal inputs for wearable devices and systems and methods of use thereof are provided. A method of performing an input gesture to control a user interface element is described. The method includes detecting, at a first point in time, a first input via an input device of at least two input devices associated with the artificial reality headset while a multi-modal gesture is available via the at least two input devices, and performing an action at the artificial reality headset. In response to detecting a second input via another of the at least two input devices, another action is performed within a user interface presented via the artificial reality headset. The method also includes detecting a first input via the input device of the at least two input devices at a second time point after the first time point and when the multi-modal gesture is no longer available via the at least two input devices associated with the artificial reality head-mounted device.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 591,111, filed on October 17, 2023, entitled “Multi-Modal Inputs for Wearable Devices, and Systems and Methods of Use thereof,” and priority to U.S. Non-Provisional Application No. 18 / 909,860, filed on October 8, 2024; the entire contents of both applications are incorporated herein by reference. Technical Field

[0003] This generally relates to multimodal gestures detected at an artificial reality head-mounted device (headset), including but not limited to techniques for performing one or more operations in a user interface based on one or more input levels of the multimodal gesture. Background Art

[0004] A user of an artificial reality has a limited ability to remember input gestures performed on one or more devices owned by the user. For example, a user may remember a few gestures without difficulty, and may learn a few additional gestures without difficulty. However, beyond a certain number of gestures, it becomes difficult for the user to learn and remember the gestures. Therefore, there is a need to simplify user interface interaction options for the user, but still allow the user to perform a wide variety of operations on the user interface.

[0005] Therefore, there is a need to address one or more of the challenges identified above. A brief overview of solutions to the above mentioned problems is described below. Summary of the invention

[0006] The methods, systems, and devices described herein allow a user wearing an augmented reality head mounted device coupled to another device to perform multimodal gestures when multimodal gestures are enabled on both devices. Creating a multi-level input gesture hierarchy allows the user to learn fewer gestures that can perform multiple operations depending on which part of the hierarchy they are in. For example, a user can perform a first gesture when multimodal gestures are enabled, which performs a first operation in a user interface. The user can then perform the same first gesture when multimodal gestures are disabled, which performs a different operation on the head wearable device that is not associated with the user interface (e.g., an audio control gesture such as changing the volume). In this example, the user only needs to know one input gesture, but can perform multiple operations depending on whether the multimodal operation is available. Additional examples are provided below.

[0007] A method for controlling one or more user interface (UI) elements via gestures performed at one or more input devices is described below. The method includes: an artificial reality head mounted device worn by a user at a first time point, a multimodal gesture is available via at least two input devices associated with the artificial reality head mounted device. The method also includes: in response to detecting a first input via a first input device of the at least two input devices, performing a first action at the artificial reality head mounted device. The method also includes: in response to detecting a second input via a second input device of the at least two input devices, performing a second action within a user interface presented via the artificial reality head mounted device. The method also includes: at a second time point after the first time point, the multimodal gesture is no longer available via the at least two input devices associated with the artificial reality head mounted device. The method also includes: in response to detecting the first input via the first input device of the at least two input devices, performing a second action within the user interface presented via the artificial reality head mounted device.

[0008] The features and advantages described in the specification are not necessarily all inclusive, and in particular, certain additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, the specification and the claims. In addition, it should be noted that the terms used in this specification are selected primarily for readability and guidance purposes.

[0009] Having summarized the above example aspects, a brief description of the drawings will now be presented. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] For a better understanding of the various embodiments described, reference should be made to the following detailed description taken in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout the figures.

[0011] FIG. 1A to FIG. 1T A system configured to detect multimodal input gestures at a plurality of communicatively coupled devices is shown in accordance with some embodiments.

[0012] FIG. 2A to FIG. 2D Examples of one or more gestures performed at a head wearable device and a wrist wearable device are shown in accordance with some embodiments.

[0013] FIG. 3A to FIG. 3C Examples of one or more gestures performed at a head wearable device are shown in accordance with some embodiments.

[0014] Figure 4 An example method flow chart for controlling a UI using one or more actions in accordance with some embodiments is shown.

[0015] Figure 5A and Figure 5B A flow diagram for controlling a UI using one or more actions is shown in accordance with some embodiments.

[0016] Fig. 6A , Figure 6B , Figure 6C-1 and Figure 6C-2 An example artificial reality system is shown in accordance with some embodiments.

[0017] Fig. 7A and Figure 7B An example wrist wearable device is shown in accordance with some embodiments.

[0018] Fig. 8A , Figure 8B-1 , Figure 8B-2 and Figure 8C An example head wearable device is shown in accordance with some embodiments.

[0019] Fig. 9A and Fig. 9B An example handheld intermediate processing device is shown in accordance with some embodiments.

[0020] As a matter of practice, the various features shown in the drawings may not be drawn to scale. Accordingly, the dimensions of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method, or device. Finally, similar reference numerals may be used throughout the specification and drawings to represent similar features. DETAILED DESCRIPTION

[0021] Numerous details are described herein to provide a thorough understanding of the example embodiments shown in the various figures. However, some embodiments may be practiced without many of these specific details, and the scope of the claims is limited only by those features and aspects specifically recited in the claims. In addition, it is not necessary to describe well-known processes, components, and materials in full detail to avoid obscuring the relevant aspects of the embodiments described herein.

[0022] The various embodiments of the present disclosure may include various types or embodiments of artificial reality systems, or may be implemented in combination with several types or embodiments of artificial reality systems. As described herein, artificial reality (AR) is a presentation provided by an artificial reality system within the user's physical environment, any superimposed functionality and / or sensory detectable. Such artificial reality may include and / or represent virtual reality (VR), augmented reality, mixed artificial-reality (MAR), or some combination of these and / or a variation of one of these. For example, a user may perform an air swipe gesture to skip a song through a song providing an application programming interface (API): the song providing API provides playback at, for example, a home speaker. As described herein, AR environments include, but are not limited to: VR environments (including non-immersive, semi-immersive, and fully immersive VR environments); augmented reality environments (including marker-based augmented reality environments, markerless augmented reality environments, location-based augmented reality environments, and projection-based augmented reality environments); hybrid reality environments; and other types of hybrid reality environments.

[0023] Artificial reality content may include fully generated content or generated content combined with collected (e.g., real-world) content. Artificial reality content may include video, audio, tactile events, or some combination thereof, any of which may be presented in a single channel or in multiple channels (e.g., stereoscopic video that gives the viewer a three-dimensional effect). In addition, in some embodiments, artificial reality may also be associated with an application, product, accessory, service, or some combination thereof, which is used, for example, to create content in artificial reality and / or otherwise used in artificial reality (e.g., to perform activities in artificial reality).

[0024] As described herein, gestures may include air gestures, surface contact gestures, and / or other gestures that may be detected and determined based on movement of a single hand (e.g., a single hand gesture performed using a user's hand, the single hand gesture being detected by one or more sensors of a wearable device (e.g., an electromyography (EMG) and / or an inertial measurement unit (IMU) of a wrist wearable device), and / or detected via image data collected by an imaging device of the wearable device (e.g., a camera of a head wearable device)) or a combination of movements of the user's hands. In some embodiments, air means that the user's hand is not in contact with a surface, object, or portion of an electronic device (e.g., a head wearable device or other communicatively coupled device, such as a wrist wearable device), in other words, the gesture is performed in an open space in a three-dimensional (3D) space without contacting a surface, object, or electronic device. More generally, surface contact gestures (contact at a surface, an object, a body part of a user, or an electronic device) are also contemplated, in which contact (or an attempt to contact) is detected at a surface (e.g., a single or double-finger tap on a table, on a hand or another finger of a user, on a user's leg, a sofa, a steering wheel, etc.). The different gestures disclosed herein may be detected using image data and / or sensor data (e.g., neuromuscular signals sensed by one or more biopotential sensors (e.g., EMG sensors) or other types of data from other sensors (e.g., proximity sensors, time-of-flight (ToF) sensors, sensors of an inertial measurement unit, etc.), which are detected by a wearable device worn by a user and / or other electronic devices owned by the user (e.g., a smartphone, a laptop computer, an imaging device, an intermediate device, and / or other devices described herein).

[0025] FIG. 1A to FIG. 1T A system configured to perform multimodal input at multiple communicatively coupled devices according to some embodiments is shown. The system includes at least two communicatively coupled devices, such as a wrist wearable device 110 and a head wearable device 120. Each device has a respective input modality that allows a user 105 to perform one or more specific operations and / or specific controls via the device that detects the input. For example, the head wearable device 120 responds to detecting an input performed by the user 105 at the head wearable device 120 (e.g., input performed on a touch input surface, voice command, gesture recognition, etc.) so that an operation (e.g., image acquisition) is performed at the head wearable device 120. FIG. 1A to FIG. 1TThe illustrated system extends the capabilities of the wrist wearable device 110 and the head wearable device 120 by enabling multimodal input, such that the respective input modality of each device supplements the operations performed by the at least two communicatively coupled devices.

[0026] Figure 1A A user 105 wearing a wrist wearable device 110 and a head wearable device 120 is shown performing an input gesture at the head wearable device 120. The head wearable device 120 is configured to, in response to detecting the input gesture, cause a specific command and / or operation associated with the input gesture to be performed. The specific command and / or operation is based on the availability of the multimodal gesture. Specifically, if the multimodal gesture is available at the head wearable device 120, the input gesture causes the head wearable device 120 to execute a first command when detected; however, if the multimodal gesture is not available at the head wearable device 120, the input gesture causes the head wearable device 120 to execute a second command different from the first command when detected. Refer to the following Figures 2A to 3C Example input gestures and associated commands are discussed.

[0027] When at least two electronic devices are communicatively coupled, multimodal gestures are available. Figure 1A In some embodiments, when the head wearable device 120 is communicatively coupled to the wrist wearable device 110, the multimodal gesture is available at the two devices. Alternatively or additionally, in some embodiments, the multimodal gesture is available based on determining that at least two electronic devices are running the same application (e.g., a music application, a running application, a camera application, etc.), and / or based on determining that at least two electronic devices are close to each other (e.g., within 1 meter, in the same room, in the same building), and / or are associated with the same one or more users, and / or share the same application information (e.g., user account, family account, etc.). In some embodiments, the multimodal gesture is available based on determining that at least two electronic devices share device profile information (e.g., device user account, email address, etc.) and / or share a network (e.g., a private home network).

[0028] return Figure 1A , the user 105 is walking in the museum while listening to music through a speaker (e.g., speaker 836A ( Figure 8C), headphones, etc. to listen to music. The wrist wearable device 110 and the head wearable device 120 worn by the user 105 are communicatively coupled, which enables multimodal gestures, so that a first input gesture performed by the user 105 at the first part 120a of the head wearable device 120 (e.g., a sliding gesture at the touch input surface of the head wearable device 120) causes the head wearable device 120 to perform a command and / or operation associated with the first input gesture of the multimodal gesture in response to detecting the first input gesture. For example, when the user 105 performs the first input gesture at the first part 120a of the head wearable device 120 when the multimodal gesture is available, the head wearable device 120 causes the volume of the ongoing media (e.g., music) to change. Alternatively, if the user 105 has performed the first input gesture at the first part 120a of the head wearable device 120 when the multimodal gesture is not available (as described below with reference to Fig.1O As described), the head wearable device 120 will cause a user interface (UI) element to select a portion of the presented media application (e.g., a directional-pad of the UI element is aligned).

[0029] In some embodiments, the head wearable device 120 detects one or more input gestures via a touch input surface (e.g., input detected by one or more capacitive sensors 828), via a button 834 coupled to a frame 804 of the head wearable device 120, and / or by input gesture recognition using one or more cameras 839A to 839D communicatively coupled to the head wearable device 120. The wrist wearable device 110 detects one or more input gestures via input gesture recognition by one or more biopotential sensors (e.g., EMG sensor 765; Figure 7B ) detected by the wrist wearable device 110, input at buttons 773 and / or 743 coupled to the wrist wearable device 110, and / or touch input performed at the display 705 coupled to the wrist wearable device 110 to detect one or more input gestures.

[0030] In some embodiments, the first portion 120a of the head wearable device 120 is a temple 805 (eg, a right temple 805 or a left temple 805; Fig. 8A ) or side portions. The temple 805 may include one or more sensors 823A (eg, as described below with reference to FIG. 1 ) for detecting contact between the user 105 and the temple 805. FIG. 8A to FIG. 8CThe one or more capacitive sensors 828, proximity sensors 869, and / or other sensors described herein) may be used to enable the temples to be used as touch input surfaces. The touch input surface of the head wearable device 120 may be a first example input modality of the head wearable device 120. The head wearable device 120 may include more than one input modality. For example, the head wearable device 120 may also include one or more of the following input modalities: based on one or more cameras (e.g., left camera 839A and right camera 839B) that are communicatively coupled to the head wearable device 120 and / or integrated into the head wearable device 120; Fig. 8A ) gesture recognition based on image data collected by the user; voice input (including voice commands to support virtual assistants); gestures based on gaze detection; and / or input at an actuation device (e.g., a button, switch, etc.). FIG. 8A to FIG. 8C Additional input modalities of the head wearable device 120 are discussed in .

[0031] Figure 1B A portion of the field of view of the user 105 is shown, the portion including a first scene 108a through one or more lenses of the head wearable device 120. The head wearable device 120 presents within the portion of the field of view of the user 105: a first visual cue 132a indicating that a song is currently playing (e.g., Figure 1B The visual prompt in 134 indicates "Song 2 is now playing"); and a second visual prompt 134 (e.g., a volume indicator). Figure 1B As shown, the second visual prompt 134 is in response to the user 105 performing a first input gesture (eg, Figure 1A Specifically, in response to a first input gesture performed at the first portion 120a of the head wearable device 120, the volume of audio data presented by a speaker communicatively coupled to the head wearable device 120 is manipulated. For example, Figure 1B As shown, the user 105 slides his index finger along a first direction (e.g., from right to left, or from left to right) at the first portion 120a of the head wearable device 120 so that the head wearable device reduces the volume of music played by a speaker communicatively coupled to the head wearable device 120. In some embodiments, the user 105 slides his finger along a second direction different from the first direction (e.g., opposite to the first direction) so that the volume of the presented music increases.

[0032] Figure 1CThe user 105 is shown performing a second input gesture (e.g., a double-click input gesture with his finger (e.g., the user's index finger)) at the first portion 120a of the head wearable device 120 at a second point in time. At the second point in time, the multimodal gesture is available, and the second input gesture at the first portion 120a of the head wearable device 120 causes the head wearable device 120 to perform a command and / or operation specific to the multimodal. For example, Figure 1C and Figure 1D As shown, the second input gesture at the first portion 120a of the head wearable device 120 causes the music application presented by the head wearable device 120 to skip to the next song.

[0033] Figure 1D FIG. 1 shows a portion of the field of view of user 105, the portion including second scene 108 b. In response to detecting the second input gesture, head wearable device 120 presents an updated first visual cue 132 b within the portion of the field of view of user 105. Specifically, updated first visual cue 132 b indicates that the song has been played from Figure 1B The "Song 2" shown in the figure changes to "Song 3".

[0034] Figure 1E The user 105 is shown performing a third input gesture at the wrist wearable device 110 at a third point in time. At the third point in time, multimodal gestures are available, and the third input gesture at the wrist wearable device 110 causes the head wearable device 120 to perform a multimodal-specific command and / or operation (e.g., a navigation command). Figure 1E , the wrist wearable device 110 uses one or more biopotential sensors (e.g., EMG sensors) to detect a third input gesture performed by the user 105. Using the biopotential sensor to detect input gestures can be a first input modality of the wrist wearable device. Additional input modalities of the wrist wearable device 110 may include: detecting input gestures performed by a user at a touch display coupled to the wrist wearable device 110 and / or integrated into the wrist wearable device 110; detecting actuation of one or more buttons of the wrist wearable device 110; and / or voice input detected via a microphone of the wrist wearable device 110. The above example input gesture modalities are not exhaustive; refer to the following for more information. Fig. 7A and Figure 7B Additional input gesture modalities are discussed.

[0035] Figure 1EThe user 105 is also shown performing a third input gesture on the wrist wearable device using a first phalanx (e.g., the user's thumb) and a second phalanx (e.g., the user's index finger). The third input gesture is a sliding gesture performed on the directional pad, such that the movement of the first phalanx is aligned with a cursor 136 (e.g., Figure 1F ). By performing a third input gesture at the wrist wearable device 110 to control the UI presented at the head wearable device 120, the user 105 is able to quickly select one or more UI elements and / or control applications connected to the two devices. Specifically, the user 105 can perform quick gestures to navigate and / or control the devices in the presented UI without having to move his hand to one of the devices or retrieve another device (e.g., a smartphone) to control the UI.

[0036] Figure 1F FIG. 1 shows a portion of the field of view of the user 105, which includes the third scene 108c. The head wearable device 120 responds to detecting the third input gesture (e.g., Figure 1E ), the updated position of cursor 136 is presented within the portion of the user's 105 field of view. Figure 1F , user 105 is navigating to a UI element that, if selected, takes user 105 to a screen where he or she can select one or more additional songs. Specifically, Figure 1F The user 105 is shown moving the cursor 136 to the music UI element 138 (eg, a music icon that returns the user 105 to the song selection page). Figure 1F It also shows that Figure 1B and Figure 1DIn some embodiments, when an input gesture is performed at another device coupled to the head wearable device 120, additional features are displayed in the active UI. For example, when multimodal gestures are enabled at the system and the user 105 performs an input gesture at the wrist wearable device 110, the music application user interface 142 (e.g., the active UI) also includes additional UI elements (e.g., audio controls 140a to 140g) that can be selected using input gestures performed at the wrist wearable device 110 and / or the head wearable device 120. In some embodiments, audio controls 140a to 140g perform the following audio controls: (i) skip back 140a, skip backward one song; (ii) fast rewind 140b, quickly skip a portion of a song in reverse at a first interval; (iii) pause 140c, pause the song but save the position where the song was paused; (iv) play 140d, start the song; (v) stop 140e, stop the song but do not save the position where the song was stopped; (vi) fast forward 140f, quickly skip a portion of a song at a second interval; and (vii) skip forward 140g, skip forward one song.

[0037] Figure 1G Also shown is user 105 performing a fourth input gesture (e.g., a pinch gesture) at wrist wearable device 110 at a fourth point in time. At the fourth point in time, multimodal gestures are available, and the fourth input gesture at wrist wearable device 110 causes head wearable device 120 to perform multimodal-specific commands and / or operations (e.g., navigation commands). Figure 1G In some embodiments, the fourth input gesture is performed when the user 105 presses the first phalanx (e.g., the user's thumb) and the second phalanx (e.g., the user's index finger) together (e.g., a pinch gesture). In some embodiments, the pinch gesture indicates a selection of the selected user interface element. For example, Figure 1H As shown, when cursor 136 is over music UI element 138 and based on a determination that user 105 has performed a pinch gesture, this causes music UI element 138 to be selected.

[0038] Figure 1H A portion of the field of view of the user 105 is shown, the portion including viewing the fourth scene 108d. In response to detecting the fourth input gesture (e.g., a navigation gesture), the head wearable device 120 presents an updated position on the cursor 136 within the portion of the field of view of the user 105. As described above, Figure 1F The user 105 is shown moving the cursor 136 to the music UI element 138 (e.g., a music icon that returns the user 105 to the song selection page). Specifically, Figure 1H A cursor 136 on a music UI element 138 is shown.

[0039] Fig. 1I The user 105 is shown performing a fifth input gesture (e.g., a pinch and drag input gesture; an additional navigation gesture) at the wrist wearable device 110 at a fifth point in time. At the fifth point in time, multimodal gestures are available, and the fifth input gesture at the wrist wearable device 110 causes the head wearable device 120 to perform a command and / or operation specific to multimodality (e.g., a navigation command). Fig. 1I In response to the user performing the fifth gesture, the user interface scrolls multiple UI elements (e.g., multiple songs in a playlist). To perform the fifth input gesture (e.g., the pinch and drag input gesture), the user 105 first holds the first phalanx and the second phalanx together by continuously pressing the first phalanx and the second phalanx together. Figure 1G While maintaining the fourth input gesture (e.g., pinch gesture), user 105 simultaneously moves his hand in a first direction (e.g., away from the user's body to the left) or in a second direction (e.g., away from the user's body to the right) to scroll one or more UI elements (e.g., Figure 1J ). For example, in Fig. 1I , the user 105 moves his hand in a first direction to scroll up through the multiple songs, thereby going from "Song 2" to "Song 6".

[0040] Figure 1J FIG. 1 shows a portion of the field of view of the user 105, which includes the fifth scene 108e. The head wearable device 120 responds to the user 105 in Fig. 1I , the user 105 is presented with the plurality of UI elements (e.g., a plurality of songs) actively scrolled by the user 105 within the portion of the field of view of the user 105. In some embodiments, when the user 105 performs the fifth input gesture (e.g., the pinch and hold input gesture), the cursor 136 highlights the currently selected object, thereby providing the user 105 with a reference point for the start of scrolling. Figure 1J In the example embodiment, when the user 105 moves the input gesture along the first direction, the cursor will continue to move downward among the multiple UI elements until one UI element is selected.

[0041] Figure 1K The user 105 is shown performing a sixth input gesture (e.g., releasing the pinch and drag input gesture) at the wrist wearable device 110 at a sixth point in time. At the sixth point in time, multimodal gestures are available, and the sixth input gesture at the wrist wearable device 110 causes the head wearable device 120 to perform a multimodal-specific command and / or operation (e.g., a navigation command). Specifically, Figure 1KIt is shown that according to determining the sixth input gesture of the user 105, the UI of the head wearable device 120 selects one UI element among multiple UI elements (for example, multiple songs in a playlist).

[0042] Figure 1L A portion of the field of view of the user 105 is shown, which includes the sixth scene 108f. Figure 1L As shown, the head wearable device 120 presents a selected one of the multiple UI elements within the portion of the field of view of the user 105 based on determining that the sixth input gesture is performed (for example, when the user 105 releases the input gesture). In some embodiments, the cursor 136 becomes a solid line to indicate that the UI element is selected and the sixth input gesture has ended.

[0043] Figure 1M It is shown that at the seventh time point, the battery of the wrist wearable device 110 is too low to continue receiving input gestures from the user 105. At the seventh time point, based on a determination that no eligible devices (e.g., not enabled for multimodal gestures) are running the same application or are near the head wearable device 120 (e.g., the previously coupled device (e.g., wrist wearable device 110) runs out of battery), multimodal gestures are no longer available. Therefore, Figures 1M to 1T Multiple scenarios are shown when the multimodal gesture is not available. Based on determining that the multimodal gesture is not available, in response to detecting the seventh input gesture, executing commands and / or operations associated with the head wearable device 120. In some embodiments, the user 105 can connect an additional device (e.g., a smartphone, an additional wrist wearable device, etc.) to reactivate the multimodal gesture. However, Figures 1M to 1T A system is shown in which multimodal gestures are no longer available at the system and input gestures are performed at the head wearable device 120 .

[0044] Figure 1N A portion of the field of view of the user 105 is shown, the portion including the seventh scene 108g. In response to performing the sixth input gesture (e.g., the user 105 releases the input gesture), the head wearable device 120 presents a selection of the UI element (e.g., Song 6). The cursor 136 displays a solid line indicating that the play 140d button is selected and Song 6 is now playing. This is also indicated by the third visual cue 132c indicating "Now Playing".

[0045] Fig.1OThe user 105 is shown performing a seventh input gesture (e.g., a navigation input gesture) at an eighth point in time at the head wearable device 120. At the eighth point in time, the multimodal gesture is unavailable, and the seventh input gesture at the head wearable device 120 causes the head wearable device 120 to perform a non-multimodal-specific command and / or operation. Fig.1O It is also shown that when the user 105 uses a first phalanx (e.g., a thumb of the user) at the first portion 120a of the head wearable device 120 (e.g., at the touch input surface), the seventh input gesture is performed as a directional slide key, so that the movement of the first phalanx is related to the movement of the cursor 136 on the display of the head wearable device 120. The movement of the directional slide key is similar to Figure 1E The movement of the slide key in the direction shown in Fig.1O The method is performed at the head wearable device 120 in the embodiment of the present invention.

[0046] Figure 1P A portion of the user's 105 field of view is shown, which includes the eighth scene 108h. The head wearable device 120 presents within the portion of the user's 105 field of view that the user 105 is listening to song 6 and the cursor 136 surrounds the play 140d button with a dotted line, which indicates that the cursor 136 can be moved to the next UI element. Specifically, Figure 1P The user 105 is shown performing a seventh input gesture (eg, a directional key slide gesture at the first portion 120a of the head wearable device 120) such that the cursor moves from the play 140d button to the skip forward 140g button.

[0047] Figure 1Q The figure shows that the user 105 performs an eighth input gesture (e.g., a single-click input gesture with his finger (e.g., the user's index finger)) at the first portion 120a of the head wearable device 120 at a ninth time point. At the ninth time point, a non-multimodal gesture is performed at the head wearable device 120, and the non-multimodal gesture causes the head wearable device 120 to perform a specific command / operation (e.g., a navigation gesture). Specifically, the eighth input gesture causes the head wearable device 120 to perform the input gesture of the user 105 at the first portion 120a of the head wearable device 120. Fig.1O The moving cursor 136 highlights the selection of the audio control UI element.

[0048] Figure 1R A portion of the user's 105 field of view is shown, the portion including the ninth scene 108i. The head wearable device 120 presents, within the portion of the user's 105 field of view, a selection of a skip ahead 140g audio control UI element that causes a song change. Figure 1R The cursor 136 is shown with a solid line indicating that the audio control UI element is selected.

[0049] Figure 1S It is shown that when the user 105 has completed the execution input gesture at the tenth time point, the user's hand is in a relaxed state.

[0050] Figure 1T A portion of the field of view of the user 105 is shown, the portion including the tenth scene 108j. The head wearable device 120 presents within the portion of the field of view of the user 105 that song 6 has been skipped forward one song to song 7. Figure 1T It is also shown that song 7 is now playing, as indicated by the fourth visual cue 132d indicating "Now Playing." In addition, the music UI is minimized to a size that minimally interferes with the user's field of vision through the head wearable device 120.

[0051] As the following reference Figures 2A to 3C As described, depending on the availability of multimodal gestures, the user 105 can control other audio functions (e.g., play / pause a song, switch to the next or previous song, and / or launch a virtual assistant) and other control functions (e.g., UI navigation and / or system control functions) by performing one or more input gestures at the first portion 120a of the head wearable device 120.

[0052] FIG. 2A to FIG. 2D Example multimodal gestures available at a head wearable device and a wrist wearable device according to some embodiments are shown. Figures 1A to 1L As described, multimodal gestures are available when communicatively coupled devices work together to control and / or operate one or more applications and / or processes running on at least one communicatively coupled device. Specifically, wearable devices such as wrist wearable device 110 and head wearable device 120 can detect corresponding user input when communicatively coupled, and cause one or more operations and / or commands to be performed at the application and / or process running on a specific wearable device or each of the wearable devices. In some embodiments, the one or more operations and / or commands are based on the communicatively coupled device and / or the ongoing application and / or process that detects the user input.

[0053] Figure 2A 1 shows a first set of gestures available at the wrist wearable device 110 when the wrist wearable device 110 is communicatively coupled to the head wearable device 120 and multimodal gestures are available. The first set of gestures available at the wrist wearable device 110 includes gestures via a biopotential sensor (e.g., EMG sensor 765) of the wrist wearable device 110; Figure 7B) detected by a user. The one or more biopotential-based gestures may be gestures performed by a user, such as surface contact gestures, air gestures, and other gestures described above. The biopotential-based gestures may include one or more input levels. For example, the first input level of the one or more biopotential-based gestures may include contact between a first phalanx (e.g., a portion of the thumb 230 of the hand 210) and a second phalanx (e.g., a portion of the index finger 235 of the hand 210) of the user, and the second input level of the one or more biopotential-based gestures may include contact between the first phalanx and the third phalanx (e.g., a portion of the middle finger 240 of the hand 210) of the user. In some embodiments, one or more phalanges may be associated with corresponding commands and / or operations. For example, in some embodiments, the second phalanx may be associated with navigation-based commands and / or operations (e.g., commands for navigating a user interface), and the third phalanx may be associated with system-based commands and / or operations (e.g., commands for operating an electronic device).

[0054] The above examples are non-limiting; those skilled in the art will recognize after reading this description that biopotential-based gestures may include contact between any two phalanges of a user or no contact between the phalanges of a user's hand (e.g., waving, wrist rotation, finger pointing, etc.). Figure 2D As described, each input level's biopotential-based gestures may be associated with one or more corresponding commands and / or operations.

[0055] Figure 2B 10. A second set of gestures available at the head wearable device 120 when the head wearable device 120 is communicatively coupled to the wrist wearable device 110 and multimodal gestures are available is shown. The second set of gestures available at the head wearable device 120 includes gestures via one or more sensors 823A of the head wearable device 120 (e.g., capacitive sensors 828; Figure 8C ) detected by the user. The one or more touch input gestures may be hand contact patterns (e.g., single click, double click, etc.) performed by the user at a portion of the head wearable device 120. For example, the user may use his second phalanx (e.g., a portion of the index finger 235) to contact a portion of the head wearable device 120 and cause the head wearable device 120 to perform a command or operation. Figure 2C As described, touch input gestures performed at the head wearable device 120 can be associated with one or more corresponding commands and / or operations.

[0056] Figure 2C1 shows example multimodal gesture commands associated with a second set of gestures available at the head wearable device 120 when the head wearable device 120 is communicatively coupled to the wrist wearable device 110 and an application (e.g., a media streaming application) is in progress. Specifically, Figure 2C It is shown that: when the user wears a portion of the head wearable device 120 (for example, the first portion 120a; FIG. 1A to FIG. 1T );A first input gesture performed when the user slides a portion of his index finger 235 on a portion of the head wearable device 120 with a portion of his index finger 235;A second input gesture performed when the user performs a single click on a portion of the head wearable device 120 with a portion of his index finger 235;A third input gesture performed when the user performs a double click on a portion of the head wearable device 120 with a portion of his index finger 235;A fourth input gesture performed when the user performs a triple click on a portion of the head wearable device 120 with a portion of his index finger 235;A fifth input gesture performed when the user taps and holds on a portion of the head wearable device 120 with a portion of his index finger 235. Each gesture is associated with a corresponding command. For example, the first input gesture is associated with a command for adjusting the volume of the presented audio data, the second input gesture is associated with a command for pausing or playing an audio track of a media streaming application, the third input gesture is associated with a command for skipping to the next song of the media streaming application, the fourth input gesture is associated with a command for returning to the previous song of the media streaming application, and the fifth input gesture is associated with a command for launching a virtual assistant (or launching an audio input modality).

[0057] Figure 2D 1 shows example multimodal gesture commands associated with a first set of gestures available at the wrist wearable device 110 when the wrist wearable device 110 is communicatively coupled to the head wearable device 120 and an application (e.g., a media streaming application) is in progress. In addition, Figure 2D The input gestures shown in are detected by one or more sensors 823A coupled to the wrist wearable device 110, and each gesture is associated with a corresponding command. Specifically, Figure 2DShown are: a first input gesture performed when the user slides a portion of his thumb 230 over a portion of his index finger 235, enabling the index finger 235 to be used as a directional sliding key; a second input gesture performed when the user performs a pinch with a portion of his index finger 235; a third input gesture performed when the user performs a pinch gesture, holds and moves (e.g., drags on a table) his hand 210; a fourth input gesture performed when the user performs a pinch gesture with his thumb 230 and middle finger 240, and performs a single-click with his index finger 235 (the single-click is a tap near the location where the pinch gesture is performed); a fifth input gesture performed when the user performs a pinch gesture with his thumb 230 and middle finger 240, and performs a double-click with his index finger 235 (the double-click is a tap near the location where the pinch gesture is performed); and a sixth input gesture performed when the user performs a pinch gesture with his thumb 230 and middle finger 240, and holds the pinch gesture until the corresponding operation is completed.

[0058] Each gesture is associated with a corresponding command. For example, a first input gesture is associated with a swipe gesture for aiming at a UI element (e.g., Figure 1E ), and the second input gesture is associated with a command for selecting a UI element (e.g., Figure 1G The first input gesture is associated with a command for waking up the head wearable device 120 or a communication coupled device, the second input gesture is associated with a command for putting the head wearable device 120 or a communication coupled device to sleep, and the third input gesture is associated with a command for scrolling the UI element, the fourth input gesture is associated with a command for returning in the UI, the fifth input gesture is associated with a command for waking up the head wearable device 120 or a communication coupled device, and the sixth input gesture is associated with a command for putting the head wearable device 120 or the communication coupled device to sleep.

[0059] FIG. 3A to FIG. 3C 1 shows example input gestures available at the head wearable device 120 when multimodal gestures are not available according to some embodiments. Figure 1M to Figure 1T As described, when the head wearable device 120 is not communicatively coupled to another device that is configured to work with the head wearable device 120 to enable multimodal gestures, the multimodal gestures are unavailable. The multimodal gestures can be available and unavailable at different points in time, and the communicatively coupled devices can seamlessly transition between different modes. For example, when a device that was previously communicatively coupled to the head wearable device 120 that enabled multimodal gestures is unavailable for various reasons (e.g., Figure 1M to Figure 1TWhen the multimodal gesture is disconnected due to battery power loss as shown, moving out of the range of the head wearable device 120, etc., the multimodal gesture is disabled, and the head wearable device 120 detects input based on the predefined input gesture. Specifically, when the multimodal gesture is not available, the head wearable device 120 can detect the corresponding user input and cause one or more operations and / or commands to be performed at the application and / or process running at the head wearable device 120.

[0060] Figure 3A and Figure 3B 1 and 2 respectively illustrate a third set of gestures and a fourth set of gestures available at the head wearable device 120 when the multimodal gesture is not available. The third set of gestures and the fourth set of gestures available at the head wearable device 120 include one or more sensors 823A (e.g., capacitive sensors 828; Figure 8C ) detects one or more touch input gestures (or surface contact gestures). Figure 3C As described, touch input gestures performed at the head wearable device 120 can be associated with one or more corresponding commands and / or operations.

[0061] Figure 3A A third set of gestures available at the head wearable device 120 is shown. One or more touch input gestures available in the third set of gestures can be a hand contact pattern (e.g., single click, double click, slide, etc.) performed by the user 105 at a portion of the head wearable device 120. For example, the user can use his second phalanx (e.g., a portion of the index finger 235) to contact a portion of the head wearable device 120 and cause the head wearable device 120 to perform a command or operation (e.g., similar to Figure 2B However, as described in the following reference Figure 3C The commands / operations described as a result of the third set of gestures are different from Figure 2B those commands / operations described in .

[0062] Figure 3B A fourth set of gestures available at the head wearable device 120 is shown. One or more touch input gestures available in the fourth set of gestures can be a hand contact pattern (e.g., single click, double click, swipe, etc.) performed by the user 105 at a portion of the head wearable device 120 using at least two phalanges (e.g., a portion of the index finger 235 of the hand 210 and a portion of the middle finger 240 of the hand 210). As described below with reference to Figure 3C As described, touch input gestures performed at the head wearable device 120 can be associated with one or more corresponding commands and / or operations (eg, system-level operations).

[0063] Figure 3C 1 shows example commands associated with a third set of gestures and a fourth set of gestures available at the head wearable device 120 when an application (e.g., a media streaming application) is in progress. Figure 1M to Figure 1T It is shown in Figure 3C Examples of several commands shown in Fig.1O Slide gestures and direction key gestures in the Figure 1Q , etc.). Specifically, Figure 3C Shown are: a first input gesture performed when a user slides a portion of their index finger 235 on a portion of the head wearable device 120; a second input gesture performed when the user performs a single click on a portion of the head wearable device 120 with a portion of their index finger 235; a third input gesture performed when the user performs a single click on a portion of the head wearable device 120 with a portion of their index finger 235 and a portion of their middle finger 240; a fourth input gesture performed when the user performs a double click on the first portion 120a of the head wearable device 120 with a portion of their index finger 235 and a portion of their middle finger 240; a fifth input gesture performed when the user performs a tap and hold on a portion of the head wearable device 120 with a portion of their index finger 235 and a portion of their middle finger 240; and a sixth input gesture performed when the user performs a slide on a portion of the head wearable device 120 with a portion of their index finger 235 and a portion of their middle finger 240. Each gesture is associated with a corresponding command. For example, the first input gesture is associated with a command for aligning a UI element (e.g., such as Fig.1O ), and the second input gesture is associated with selecting the targeted UI element (e.g., Figure 1Q ), a third input gesture is associated with a command for returning in the UI, a fourth input gesture is associated with a command for waking up the head wearable device 120, a fifth input gesture is associated with a command for putting the head wearable device 120 to sleep, and a sixth input gesture is associated with a command for launching a virtual assistant (or launching an audio input modality).

[0064] Although the above reference Figures 1A to 3C The examples provided describe audio control commands at a music streaming application, but the one or more input gestures may be associated with different commands and / or operations depending on the application running on the head wearable device, wrist wearable device, and / or other communicatively coupled devices.

[0065] Figures 4 to 5BA flowchart and a flow chart of a method for executing one or more commands at a head wearable device communicatively coupled to at least one other device using one or more input gestures according to some embodiments are shown. The operations (e.g., steps) of methods 400 and 500 can be performed by one or more processors (e.g., a central processing unit and / or a microcontroller unit (MCU)) of a system including one or more communicatively coupled devices, such as head wearable device 120 and wrist wearable device 110. Figures 4 to 5B At least some of the operations shown in correspond to instructions stored in a computer memory or computer-readable storage medium (e.g., storage device, random access memory (RAM) and / or memory) of the head wearable device 120 and / or the wrist wearable device 110. The operations of methods 400 and 500 may be performed by a single device alone or in conjunction with one or more processors and / or hardware components of another communicatively coupled device (e.g., head wearable device 120, wrist wearable device 110, etc.), and / or instructions stored in a memory or computer-readable medium of another device communicatively coupled to the system. In some embodiments, the various operations of the methods described herein are interchangeable and / or optional, and the corresponding operations of these methods are performed by any of the aforementioned devices, systems, or combinations of devices and / or systems. For convenience, the method operations will be described below as being performed by a specific component or device, but in all embodiments should not be interpreted as limiting the execution of the operations to that specific device.

[0066] (A1) Figure 4 In accordance with some embodiments, one or more input gestures are shown for communicating with at least one other device (e.g., Fig. 7A and Figure 7B A wrist wearable device described herein) is communicatively coupled to an artificial reality head mounted device (e.g., the following reference FIG. 8A to FIG. 8C 1. The method 400 is performed at a head wearable device 120 and / or a wrist wearable device 110 worn by a user. Each of the head wearable device 120 and the wrist wearable device includes one or more input modalities (e.g., a touch input surface, an EMG sensor 765 ( Figure 7B ) etc.). In some embodiments, method 400 is further performed when the user wears an artificial reality head mounted device (402).

[0067] The method 400 includes: at a first point in time, when a multimodal gesture is available via at least two input devices associated with an artificial reality head mounted device (404): i) (406) in response to a first input device (e.g., including one or more capacitive sensors 828 ( Figure 8C )'s touch input surface), performing a first action (e.g., an audio control action) at the artificial reality head mounted device; and ii) (408) responding to a second input device (e.g., one or more EMG sensors 765; Figure 7B ) detects a second input, performs a second action (e.g., a user interface control action, such as a second input) within a user interface presented via the artificial reality head mounted device. Figure 1E and Figure 1F ). Figures 1A to 1L One or more example scenarios showing that multimodal gestures are available are provided in FIG. Figure 1A The user 105 is shown performing a sliding input gesture at the head wearable device 120, causing the head mounted device to perform an audio control action (e.g., change the volume). Figure 1E and Figure 1F The user 105 is shown performing a pinch gesture detected by the EMG sensor 765 coupled to the wrist wearable device 110, which produces a selection command at the user interface.

[0068] The method also includes, at a second time point after the first time point, when the multimodal gesture is no longer available via the at least two input devices associated with the artificial reality head mounted device (410), in response to detecting the first input via a first input device of the at least two input devices, performing (412) a second action within a user interface presented via the artificial reality head mounted device. Fig.1O and Figure 1P In , when multimodal gestures are not available, when the user performs a Figure 1A , when the same input gesture is executed in , the same input gesture, when executed by user 105 at the head wearable device, generates a selection command at the user interface. In other words, the same input gesture can execute different commands and / or operations when the multimodal gesture is available or not.

[0069] (A2) In some embodiments of A1, the first action includes a media control command (e.g., volume adjustment, play / pause command, skipping the current track, going to the previous track, and / or launching a virtual assistant). Figure 1AAs shown and described in , the user 105 performs a first action (eg, a first input gesture) at the head wearable device 120, which causes the volume of the currently playing media to be adjusted (eg, Figure 1A shows volume down).

[0070] (A3) In some embodiments of A1 and A2, the first input device is a touch input surface, and the first input is a surface contact gesture at a portion of the artificial reality head mounted device. In some embodiments, the portion of the artificial reality head mounted device on which the surface contact gesture is performed includes the touch input surface that detects user input using the capacitive sensor 828 and / or the proximity sensor 869. For example, Figure 1A As shown, the surface contact gesture is a sliding gesture performed at a touch input surface (eg, at a first input) of a head wearable device 120 (eg, an artificial reality head mounted device).

[0071] (A3.5) In some embodiments of A1 and A2, the first input device is a capacitive sensor and the first input is a surface contact gesture at a portion of the artificial reality head mounted device. In some embodiments, the portion of the artificial reality head mounted device on which the surface contact gesture is performed includes a touch input surface having a capacitive sensor 828. For example, Figure 1A As shown, the surface contact gesture is a sliding gesture performed at a touch input surface (eg, at a first input) of a head wearable device 120 (eg, an artificial reality head mounted device).

[0072] (A4) In some embodiments of A1 to A3, the first input is an input in a first input level group, the first input level group including a first input level and a second input level. The first input level of the first input level group includes one or more inputs associated with navigation control commands, and the second input level of the first input level group includes one or more inputs associated with system control commands for an artificial reality head mounted device. For example, as described above with reference to Figure 3C As described, the head wearable device can detect single phalange input and double phalange input, which are associated with one or more corresponding commands or operations. In some embodiments, one or more inputs associated with the first input level include target selection using a directional key sliding gesture and a selection operation using a single click gesture. For example, Fig.1O , where a user is navigating a UI at the head wearable device 120 using a swipe motion performed at a first portion of the head wearable device 120. Figure 1QIn some embodiments, when the user 105 selects an audio control on the UI of the head wearable device 120, the user performs the selection operation using a single-tap gesture. In some embodiments, the one or more inputs associated with the second input level include: (i) navigating back in the UI using a single-tap; (ii) performing a wake-up command using a double-tap; (iii) performing a sleep operation by performing a tap and hold; and (iv) adjusting the volume of the currently playing media by performing a two-finger left-right swipe.

[0073] (A5) In some embodiments of A4, a first input level of the first input level group is associated with input performed with a single phalange of a user's hand (e.g., a portion of an index finger), and a second input level of the first input level group is associated with input performed with at least two phalanges of the user's hand (e.g., a portion of an index finger and a portion of a middle finger). For example, the above reference Figure 3C One or more input gestures performed with a single phalanx and one or more input gestures performed with at least two phalanxes are described.

[0074] (A6) In some embodiments of A4 and A5, the first input level group includes a third input level, and the third input level of the first input level group includes one or more inputs associated with the media control command. Figure 3C As described, a user may use at least two finger bones to perform a sliding gesture that causes a head wearable device to adjust the volume.

[0075] (A7) In some embodiments of A6, the third input level of the first input level group is associated with a sliding gesture performed with at least two fingers of a user's hand. For example, Figure 3C A two-finger swipe gesture is shown that causes the volume of currently playing media to increase or decrease.

[0076] (A8) In some embodiments of A1 to A7, the second action includes a navigation control command and / or a system control command for the artificial reality head mounted device. In some embodiments, the navigation control command is performed using an input gesture sensed at the wrist wearable device 110, wherein the input gesture includes: Figure 1E In some embodiments, system control commands are executed using input gestures also sensed at wrist wearable device 110, including input gestures such as Figure 1G A pinch gesture is shown for confirming selection of a UI element. Additional command gestures include putting the device to sleep, waking it up, and adjusting the volume of the current media playing. Figure 2D A number of input gestures are shown including navigation control commands and system control commands.

[0077] (A9) In some embodiments of A1 to A8, the second input device is a biopotential signal sensing device, and the second input is an air gesture performed or intended to be performed by the user. Figure 1G As shown, the second input device is the wrist wearable device 110, and the performed air gesture is a pinch gesture to confirm selection of the UI element. Figure 2D Additional mid-air gestures that may be performed by user 105 and sensed by wrist wearable device 110 are shown, such as a pinch, hold, and drag gesture, a pinch and single-tap gesture, a pinch and double-tap gesture, and a pinch and hold gesture.

[0078] (A10) In some embodiments of A1 to A9, the second input is an input in a second input level group, which includes a first input level and a second input level. In some embodiments, the second input level group includes a gesture sensed using a biopotential sensor at the wrist wearable device 110. The first input level of the second input level group includes one or more inputs associated with navigation control commands. In some embodiments, the navigation control commands associated with the first input level include directional key navigation and object selection. For example, Figure 1E The user 105 is shown performing a directional key swipe gesture to navigate the music application user interface 142. The second input level of the second input level group includes one or more inputs associated with system control commands for the artificial reality head mounted device. In some embodiments, the system control commands associated with the second input level include waking up the head wearable device 120, sleeping the head wearable device 120, or navigating backward in the user interface at the head wearable device 120.

[0079] (A11) In some embodiments of A10, the first input level of the second input level group is associated with a gesture performed using a first phalanx (e.g., a portion of a thumb of the user) and a second phalanx (e.g., a portion of an index finger of the user), and the second input level of the second input level group is associated with a gesture performed using a first phalanx (e.g., a portion of a thumb of the user) and a third phalanx (e.g., a portion of a middle finger) of the user. In some embodiments, the first input level gesture performed using the first and second phalanx of the user is: a directional key gesture (for aiming); a pinch gesture (for selection); a pinch, hold, and drag gesture (for scrolling); a grab gesture; a single click, a double click, and the like. In some embodiments, the second input level gesture performed using the first and third phalanx of the user is: a pinch and single click (for performing a system operation or a device back operation); a pinch and double click (for waking up the system or device); a pinch and hold (for putting the system or device to sleep), and the like. Figure 2DExamples of one or more input gestures in the second input hierarchy group performed with first, second, and third phalanges of a hand of a user are shown.

[0080] (A12) In some embodiments of A10 and A11, the second input level group includes a third input level, and the third input level of the second input level group includes one or more inputs associated with media control commands. In some embodiments, the third input level includes a gesture sensed by the wrist wearable device 110, which causes a volume adjustment at the head wearable device 120.

[0081] (A13) In some embodiments of A12, the third input level of the second input level group is associated with gestures performed using a first phalanx (e.g., a portion of a thumb of the user) and a fourth phalanx (e.g., a portion of a ring finger of the user). In some embodiments, one or more of the following input gestures are performed using the first phalanx and the fourth phalanx of the user: a directional key gesture; a pinch gesture; a pinch and hold gesture; a grab gesture; a single click; a double click, etc. The commands and / or operations associated with the third input level of the second input level group are different from the first input level and the second input level.

[0082] (A14) In some embodiments of A1 to A13, when the user is performing or intending to perform the first input (e.g., the first gesture or the input gesture on the touch input surface), the second input device is caused to stop detecting one or more inputs. For example, in some embodiments, when the user 105 performs the first input at the head wearable device 120, one or more biopotential sensors (e.g., EMG sensors and / or IMU sensors) coupled to the wrist wearable device are disabled. This prevents false positives from being detected between the two devices.

[0083] (A15) In some embodiments of A1 to A14, when the user is performing or intends to perform the second input, the first input device (e.g., a touch input surface at the head wearable device) stops detecting one or more inputs (e.g., one or more input gestures performed by the user 105). In some embodiments, the second input is an air gesture sensed by one or more biopotential sensors coupled to the wrist wearable device 110. For example, in some embodiments, when the user 105 is performing the first input at the wrist wearable device 110, the sensor coupled to the head wearable device 120 (e.g., a touch input surface that senses one or more input gestures, a camera, etc.) is disabled.

[0084] (A16) In some embodiments of A1 to A15, when a second input device of at least two input devices associated with the artificial reality head-mounted device (e.g., a biopotential sensor coupled to the wrist wearable device 110) is no longer communicatively coupled or detectable by the artificial reality head-mounted device, it is determined that the multimodal gesture is no longer available via the at least two input devices. In other words, when the wrist wearable device 110 or other sensing device is no longer present / coupled to the head wearable device 120, the multimodal gesture is no longer available. As described above, when the multimodal gesture is no longer available, a different set of input gestures is now available, which allows the user to provide one or more different inputs using a single input device (e.g., a touch input sensor at the head wearable device 120). For example, the touch input sensor at the head wearable device 120 (e.g., a capacitive sensor or a proximity sensor) is able to distinguish between a single phalanx gesture, a two-phalanx gesture, and the like. FIG. 3A to FIG. 3C One or more input gestures that are available when the multimodal gesture is no longer available are shown. Figure 1M to Figure 1T Also shown are one or more input gestures performed at the head wearable device 120 when the multimodal gesture is no longer available.

[0085] (A17) In some embodiments of A1 to A16, at the second time point, the first input device is configured to detect one or more inputs in the first input hierarchy group. In some embodiments, after the wrist wearable device 110 is no longer communicatively coupled to the head wearable device 120, the head wearable device 120 is configured to detect different input hierarchies. For example, the head wearable device 120 is configured to classify the input gesture performed by the user as part of a specific input hierarchy, such as a first input hierarchy corresponding to an input gesture performed using a single phalange gesture, and a second input hierarchy corresponding to a gesture performed using a two-phalange gesture. Figure 2C and Figure 3C Different input gestures available at the head wearable device are shown.

[0086] (A18) In some embodiments of A17, the first input level group includes a first input level, and the first input level includes one or more inputs associated with the navigation control command. Fig.1O As shown, user 105 performs a directional key slide gesture (e.g., from the first input level) to cause the UI to navigate to the audio control 140 element (e.g., audio controls 140a to 140g). The first input level group also includes: a second input level that includes one or more inputs associated with system control commands for the artificial reality head mounted device; and a third input level that includes one or more inputs associated with media control commands. Figure 3COne or more input gestures associated with system control commands (e.g., navigating back in the UI via a single-click input gesture, waking up the head wearable device 120 via a double-click input gesture, and putting the head wearable device 120 to sleep via a tap and hold input gesture) and one or more input gestures associated with media control commands (e.g., a two-finger slide gesture to control the volume) are shown.

[0087] (A19) In some embodiments of A1 to A18, the first input device is a touch input surface coupled to the artificial reality head mounted device, and the second input device is a biopotential signal sensing device communicatively coupled to the artificial reality head mounted device. In some embodiments, the touch input surface includes a capacitive sensor and / or a proximity sensor that detects one or more input gestures performed by the user 105. For example, Figure 8C One or more capacitive sensors 828 are depicted coupled to the head wearable device 120. For example, Figure 7B Several biopotential signal sensing sensors (eg, EMG sensor 765 ) coupled to wrist wearable device 110 are described.

[0088] (A20) In some embodiments of A1 to A18, the first input device is a touch input sensor coupled to the artificial reality head mounted device, and the second input device is an imaging device communicatively coupled to the artificial reality head mounted device. In some embodiments, the touch input surface includes a capacitive sensor and / or a proximity sensor that detects one or more input gestures performed by the user 105. Figure 8C As shown, the capacitive sensor 828 and the one or more imaging sensors 826 are coupled to the peripheral interface 822A at the head wearable device 120. In some embodiments, the user 105 performs an air gesture detected by one or more imaging sensors (e.g., a webcam, a selfie camera, an imaging device of the head wearable device) that are communicatively coupled to the head wearable device 120, so that one or more operations (e.g., volume control, UI element selection, etc.) are performed at the head wearable device 120. In some embodiments, the first input device and the second input device detect different input gestures simultaneously.

[0089] (A21) In some embodiments of A1 to A18, the first input device is a touch input sensor (e.g., a capacitive sensor 828) coupled to the artificial reality head-mounted device, and the second input device is an inertial measurement unit (e.g., an IMU sensor 829) communicatively coupled to the artificial reality head-mounted device.

[0090] (A22) In some embodiments of A1 to A18, the first input device is an imaging device (e.g., imaging sensor 826, camera 838A, etc.) communicatively coupled to the artificial reality head-mounted device, and the second input device is a biopotential signal sensing device (e.g., EMG sensor 827, EMG sensor 735 and / or EMG sensor 765 coupled to a wrist wearable device, etc.) communicatively coupled to the artificial reality head-mounted device.

[0091] (A23) In some embodiments of A1 to A18, the first input device is an imaging device (e.g., imaging sensor 826, camera 838A, etc.) communicatively coupled to the artificial reality head-mounted device, and the second input device is an inertial measurement unit (e.g., IMU sensor 829) communicatively coupled to the artificial reality head-mounted device.

[0092] (A24) In some embodiments of A1 to A23, performing a first action (e.g., an audio control action) at an artificial reality head mounted device includes causing the artificial reality head mounted device to display a notification hub that includes one or more user interface elements associated with the first action. For example, Figure 1A The user 105 is shown performing an input gesture at the head wearable device that causes volume manipulation, and Figure 1B A second visual cue 134 (e.g., a notification center) is shown that actively displays to the user 105 that the volume is changing as a result of the user's input gesture. In some embodiments, the wrist wearable device 110 displays a notification center that includes one or more user interface elements associated with the one or more input gestures performed by the user 105. In addition, the one or more input gestures performed by the user cause the head wearable device and the wrist wearable device to display different centers when additional input gestures are detected (e.g., at the head wearable device 120 or the wrist wearable device 110).

[0093] (A25) In some embodiments of A24, the notification center is an audio control center, and the first action is a media control command. Figure 1B As shown, second visual prompt 134 shows the result of user 105 changing the audio control characteristic, as well as the result of the user changing the audio control characteristic. Figure 1A An input gesture (eg, a first action) performed in the example embodiment of the present invention is configured to control a media property (eg, change the volume).

[0094] (A26) In some embodiments of A25, an audio control center is presented in place of a system information center. In some embodiments, one or more system information notifications are displayed to the user 105 along with the audio control data, such as battery life, signal strength, and notifications.

[0095] (A27) In some embodiments of A1 to A26, a first action (e.g., an audio control action) is performed at the artificial reality head mounted device; Figure 1A The slide gesture in the first action includes causing the artificial reality head mounted device (or wrist wearable device) to present an audio notification based on the first action. In some embodiments, the user 105 performs a slide gesture at the head wearable device 120 to change the volume, so that the system notifies the user 105 through a virtual assistant or one or more sound effects of the volume level. For example, the one or more sound effects may include a discernible sound when the user 105 is at the lowest volume level, and a different discernible sound when the user 105 is at the highest volume level. In some embodiments, the audio prompt is available when the display is turned off and / or disabled. For example, if the display is inactive, the user 105 is able to get feedback about the volume level via an audio prompt, rather than just relying on visual prompts.

[0096] (A28) In some embodiments of A1 to A27, wherein causing the artificial reality head mounted device to execute a command specific to the artificial reality head mounted device includes activating a virtual assistant; and the method further includes: activating a microphone of the artificial reality head mounted device and / or a second input device (e.g., wrist wearable device 110), obtaining audio data via the microphone, and in response to detecting one or more user input commands from the audio data, causing the second input device and / or the artificial reality head mounted device to execute the one or more user input commands. In some embodiments, the user 105 performs an input gesture to activate the virtual assistant, thereby enabling the voice command to cause the artificial reality head mounted device to perform one or more operations. FIG. 1A to FIG. 1T All operations performed in the app can be performed via voice commands.

[0097] (B1) According to some embodiments, a system includes one or more of a wrist wearable device, a head wearable device and / or a handheld intermediate processing device, and the system is configured to perform operations corresponding to any one of A1 to A28.

[0098] (C1) According to some embodiments, a non-transitory computer-readable storage medium includes instructions that, when executed by a computing device that communicates with a wrist wearable device, a head wearable device, and / or a handheld intermediate processing device, cause the computing device to perform operations corresponding to any one of A1 to A28.

[0099] (D1) According to some embodiments, an apparatus on a wrist wearable device, a head wearable device, and / or a handheld intermediate processing device is used to perform or cause the performance of any of the methods A1 to A28.

[0100] Figure 5A and Figure 5B According to some embodiments, one or more input gestures are used to communicate with at least one other device (e.g., the following reference Fig. 7A and Figure 7B A wrist wearable device described herein) is communicatively coupled to an artificial reality head mounted device (e.g., the following reference FIG. 8A to FIG. 8C Flowchart of a second method 500 for executing one or more commands at an instance of a head wearable device described in the description. The method 500 includes: when a user is wearing an artificial reality head mounted device (e.g., head wearable device 120) (502), determining (504) whether at least two input devices associated with the artificial reality head mounted device are available. The method 500 includes: based on determining that the at least two input devices associated with the artificial reality head mounted device are available ("yes" at operation 504), detecting (506) user input. For example, as Figures 1A to 1L As shown, the head wearable device 120 and the wrist wearable device 110 are at least two input devices that are communicatively coupled and can be used to detect an input gesture performed by the user 105.

[0101] The second method 500 also includes determining (508) whether the user input is performed at a first input device associated with the artificial reality head mounted device. The second method 500 includes: based on determining that the user input is performed at the first input device associated with the artificial reality head mounted device ("yes" at operation 508), performing (510) a first action at the artificial reality head mounted device, and returning to operation 502. For example, if Figure 1A As shown, the user 105 performs a sliding gesture (eg, user input) at a touch input surface (eg, a first input device) on the head wearable device, causing the volume of the currently playing media to decrease.

[0102] Alternatively, the second method 500 includes, based on determining that the user input was not performed at a first input device associated with the artificial reality head mounted device (“No” at operation 508), determining (516) whether the user input was performed at a wearable second input device associated with the artificial reality head mounted device. For example, Figures 1A to 1L As shown, a biopotential sensor (e.g., EMG sensor 765; Figure 7B) is a second input device associated with the artificial reality head mounted device. The second method 500 includes: based on determining that the user input is performed at a wearable second input device associated with the artificial reality head mounted device ("yes" at operation 516), performing (518) a second action within a user interface presented at the artificial reality head mounted device, and returning to operation 502. For example, Fig. 1I and Figure 1J The user is shown performing a pinch, hold, and drag input gesture detected by one or more EMG sensors 765 at the wrist wearable device 110, which causes a scrolling action (e.g., an example of a second action) at the UI of the head wearable device 120 to allow the user 105 to scroll through multiple songs.

[0103] Returning to operation (516), the second method 500 includes: based on determining that the user input is not performed at a wearable second input device associated with the artificial reality head mounted device ("No" at operation 516), determining (520) whether the user input is performed at a third input device associated with the artificial reality head mounted device that is different from the first input device and the second input device. The second method 500 includes: based on determining that the user input is performed at a third input device associated with the artificial reality head mounted device that is different from the first input device and the second input device ("Yes" at operation 520), performing (522) a third action within a user interface presented at the artificial reality head mounted device, and returning to operation 502. In some embodiments, a third device (e.g., a smartphone or other communicatively coupled device) detects a user input such as pressing a button on the device, thereby causing an action such as volume adjustment at the head wearable device 120. Alternatively, the second method 500 also includes: returning to operation 502 based on determining that the user input is not performed at a third input device associated with the artificial reality head-mounted device and different from the first input device and the second input device ("No" at operation 520).

[0104] Returning to operation (504), the second method 500 includes: based on determining that the at least two input devices are not associated with the artificial reality head mounted device ("No" at operation 504), the second method 500 also includes detecting (512) a user input performed at a first input device associated with the artificial reality head mounted device. For example, Figures 1M to 1T It is shown that when the multimodal gesture is not available, the input device at the head wearable device 120 is available.

[0105] The second method 500 also includes determining (514) whether the user input is associated with an action at the artificial reality head mounted device. The second method 500 includes, based on determining that the user input is associated with an action at the artificial reality head mounted device ("yes" at operation 514), performing (510) a first action at the artificial reality head mounted device, and returning to operation 502. Fig.1O The user 105 is shown performing a slide input gesture on the touch input surface at the head wearable device 120 that causes a navigation gesture (eg, a first action) that moves focus in the UI from the play 140d button to the skip forward 140g button.

[0106] Alternatively, the second method 500 includes, based on determining that the user input is not associated with an action at the artificial reality head mounted device (“No” at operation 514), performing (518) a second action within a user interface presented at the artificial reality head mounted device, and returning to operation 502. For example, Figure 1Q The user is shown performing a single tap input gesture at the touch input surface at the head wearable device 120 , which causes a selection operation presented in the UI at the head wearable device 120 .

[0107] The above-mentioned devices are described in more detail below, including systems, wrist wearable devices, head-mounted devices, and smart textile-based clothing. The above-mentioned specific operations may occur as a result of specific hardware, which is described in more detail below. The devices described below are not restrictive, and features on these devices may be removed or additional features may be added to these devices. Different devices may include one or more similar hardware components. For the sake of brevity, similar devices and components are described below. Any differences between the devices and components are described in their respective sections below.

[0108] As described herein, a processor (e.g., a central processing unit (CPU) or a microcontroller unit (MCU)) is an electronic component responsible for executing instructions and controlling the operation of an electronic device (e.g., a wrist wearable device 700, a head wearable device, a handheld intermediate processing device (HIPD) 900, a smart textile-based garment, or other computer system). There are various types of processors that can be used interchangeably or are specifically required by the various embodiments described herein. For example, the processor may be: (i) a general-purpose processor designed to perform a variety of tasks, such as running software applications, managing operating systems, and performing arithmetic and logical operations; (ii) a microcontroller designed for specific tasks, such as controlling electronic devices, sensors, and motors; (iii) a graphics processing unit (GPU) designed to accelerate the creation and rendering of images, videos, and animations (e.g., virtual reality animations, such as three-dimensional modeling); (iv) a field-programmable gate array (FPGA) that can be programmed and reconfigured after manufacturing and / or customized to perform specific tasks, such as signal processing, encryption, and machine learning; (v) a digital signal processor (DSP) designed to perform mathematical operations on signals (e.g., audio, video, and radio waves). It will be understood by those skilled in the art that one or more processors of one or more electronic devices may be used in the various embodiments described herein.

[0109] As described herein, a controller is an electronic component that manages and coordinates the operation of other components within an electronic device (e.g., controlling inputs, processing data, and / or generating outputs). Examples of controllers may include: (i) a microcontroller, which includes small, low-power controllers commonly used in embedded systems and Internet of Things (IoT) devices; (ii) a programmable logic controller (PLC), which may be configured for use in industrial automation systems to control and monitor manufacturing processes; (iii) a system-on-a-chip (SoC) controller, which integrates multiple components such as a processor, memory, input / output (I / O) interfaces, and other peripherals into a single chip; and / or a DSP. As described herein, a graphics module is a component or software module designed to handle graphics operations and / or graphics processes, and the graphics module may include a hardware module and / or a software module.

[0110] As described herein, memory refers to an electronic component in a computer or electronic device that stores data and instructions for access and manipulation by a processor. The devices described herein may include volatile memory and non-volatile memory. Examples of memory may include: (i) random access memory (RAM) configured to temporarily store data and instructions, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM) or other random access solid-state storage devices; (ii) read-only memory (ROM) configured to permanently store data and instructions (e.g., one or more parts in system firmware and / or in a boot loader); (iii) flash memory, magnetic disk storage devices, optical disk storage devices, other non-volatile solid-state storage devices (e.g., universal serial bus (USB) drives, memory cards and / or solid-state drives (SSD)) that can be configured to store data in electronic devices; and (iv) a cache memory configured to temporarily store frequently accessed data and instructions. As described herein, the memory may include structured data (e.g., a structured query language (SQL) database, a MongoDB database, GraphQL data, or JSON data). Other examples of memory may include: (i) profile data, which includes user account data, user settings, and / or other user data stored by the user; (ii) sensor data detected by one or more sensors and / or otherwise acquired; (iii) media content data, which includes stored image data, audio data, and documents, etc.; (iv) application data, which may include data collected and / or otherwise acquired and stored during use of the application; and / or any other type of data described herein.

[0111] As described herein, a power system of an electronic device is configured to convert input power into a form that can be used to operate the device. The power system can include various components, including: (i) a power source, which can be an alternating current (AC) adapter power source or a direct current (DC) adapter power source; (ii) a charger input, which can be configured to use a wired connection and / or a wireless connection (the charger input can be part of a peripheral interface, such as a USB, a micro USB interface, near-field magnetic coupling, magnetic induction and magnetic resonance charging, and / or radio frequency (RF) charging); (iii) a power management integrated circuit, which is configured to distribute power to various components of the device and ensure that the device operates within safe limits (e.g., regulating voltage, controlling current and / or managing heat dissipation); and / or (iv) a battery, which is configured to store power to provide usable power to one or more components of the electronic device.

[0112] As described herein, a peripheral interface is an electronic component (e.g., an electronic component of an electronic device) that allows the electronic device to communicate with other devices or peripherals and can provide a means for inputting and outputting data and signals. Examples of peripheral interfaces may include: (i) a USB interface and / or a micro USB interface configured to connect a device to an electronic device; (ii) a Bluetooth interface configured to allow multiple devices to communicate with each other, including Bluetooth Low Energy (BLE); (iii) a near-field communication (NFC) interface configured as a short-range wireless interface for operations such as access control; (iv) a pogo pin, which can be a small, spring-loaded pin configured to provide a charging interface; (v) a wireless charging interface; (vi) a global-positioning system (GPS) interface; (vii) a Wi-Fi interface used to provide a connection between a device and a wireless network; and (viii) a sensor interface.

[0113] As described herein, a sensor is an electronic component (e.g., an electronic component in an electronic device (e.g., a wearable device) and / or an electronic component that otherwise communicates electronically with an electronic device) that is configured to detect physical changes and environmental changes and generate an electrical signal. Examples of sensors may include: (i) an imaging sensor for collecting imaging data (e.g., including one or more cameras disposed on a corresponding electronic device); (ii) a biopotential signal sensor; (iii) an inertial measurement unit (e.g., an IMU) for detecting, for example, angular velocity, force, magnetic field, and / or acceleration changes; (iv) a heart rate sensor for measuring a user's heart rate; (v) a blood oxygen saturation (SpO2) sensor for measuring a user's blood oxygen saturation and / or other biometric data; (vi) a capacitive sensor for detecting a change in electrical potential at a portion of a user's body (e.g., a sensor-skin interface) and / or the proximity of other devices or objects; and (vii) a light sensor (e.g., a ToF sensor, an infrared light sensor, or a visible light sensor); and / or a sensor for sensing data from a user or the user's environment. As described herein, a biopotential signal sensing component is a device (e.g., a biopotential signal sensor) for measuring electrical activity within the body. Some types of biopotential signal sensors include: (i) an electroencephalography (EEG) sensor configured to measure electrical activity in the brain to diagnose neurological diseases; (ii) an electrocardiography (ECG or EKG) sensor configured to measure the electrical activity of the heart to diagnose heart problems; (iii) an electromyography (EMG) sensor configured to measure the electrical activity of muscles and diagnose neuromuscular disorders; (iv) an electrooculography (EOG) sensor configured to measure the electrical activity of eye muscles to detect eye movements and diagnose eye disorders.

[0114] As described herein, an application (e.g., software) stored in a memory of an electronic device includes instructions stored in the memory. Examples of such applications include: (i) games; (ii) word processors; (iii) messaging applications; (iv) media streaming applications; (v) financial applications; (vi) calendars; (vii) clocks; (viii) web browsers; (ix) social media applications; (x) camera applications; (xi) web-based applications; (xii) health applications; (xiii) artificial reality (AR) applications; and / or any other application that may be stored in the memory. An application may operate in conjunction with data and / or one or more components of a device or multiple communicatively coupled devices to perform one or more operations and / or functions.

[0115] As described herein, the communication interface module may include hardware and / or software capable of communicating data using any of the following protocols: a custom wireless protocol or a standard wireless protocol (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.11a, WirelessHART, or MiWi); a custom wired protocol or a standard wired protocol (e.g., Ethernet or HomePlug); and / or any other suitable communication protocol (including communication protocols that have not been developed as of the filing date of this document). A communication interface is a mechanism that enables different systems or devices to exchange information and data with each other, including hardware, software, or a combination of both hardware and software. For example, a communication interface may refer to a physical connector and / or port (e.g., USB, Ethernet, High Definition Multimedia Interface (HDMI), or Bluetooth) on a device that enables communication with other devices. In some embodiments, a communication interface may refer to a software layer that enables different software programs to communicate with each other (e.g., an application programming interface (API), and protocols such as Hypertext Transfer Protocol (HTTP) and Transmission Control Protocol / Internet Protocol (TCP / IP)).

[0116] As described herein, a graphics module is a component or software module designed to process graphics operations and / or graphics processes, and the graphics module may include a hardware module and / or a software module.

[0117] As described herein, a non-transitory computer-readable storage medium is a physical device or storage medium that can be used to store electronic data in a non-transitory form (eg, such that the data is permanently stored until the data is intentionally deleted or modified).

[0118] Example AR Systems 600a-600c

[0119] Fig. 6A , Figure 6B , Figure 6C-1 and Figure 6C-2 An example AR system is shown in accordance with some embodiments. Fig. 6A A first AR system 600a and a first example user interaction using a wrist wearable device 700 , a head wearable device (eg, an AR device 800 ), and / or a handheld intermediary processing device (HIPD) 900 are shown. Figure 6B A second AR system 600 b is shown, along with a second example user interaction using the wrist wearable device 700 , the AR device 800 , and / or the HIPD 900 . Figure 6C-1 and Figure 6C-2 A third AR system 600 c and a third example user interaction using a wrist wearable device 700 , a head wearable device (eg, a virtual reality (VR) device 810 ), and / or a HIPD 900 are shown.

[0120] The following references Fig. 7A and Figure 7B Describes the wrist wearable device 700 and its constituent components; the following reference FIG. 8A to FIG. 8C Describes a head wearable device and its components; and the following references Fig. 9A and Fig. 9B The HIPD 900 and its constituent components are described. The wrist wearable device 700, the head wearable device, and / or the HIPD 900 can be communicatively coupled via a network 625 (e.g., cellular, near-field, Wi-Fi, a personal area network, or a wireless local area network (LAN)). In addition, the wrist wearable device 700, the head wearable device, and / or the HIPD 900 can also be communicatively coupled to one or more servers 630, computers 640 (e.g., laptops or computers), mobile devices 650 (e.g., smartphones or tablets), and / or other electronic devices via a network 625 (e.g., cellular, near-field, Wi-Fi, a personal area network, or a wireless LAN).

[0121] Go to Fig. 6A , showing a user 602 wearing a wrist wearable device 700 and an AR device 800, and the user places a HIPD 900 on his desk. The wrist wearable device 700, the AR device 800, and the HIPD 900 facilitate the user's interaction with the AR environment. Specifically, as shown in the first AR system 600a, the wrist wearable device 700, the AR device 800, and / or the HIPD 900 enable the presentation of one or more avatars 604, digital representations 606 of contacts, and virtual objects 608. As discussed below, the user 602 can interact with one or more avatars 604, digital representations 606 of contacts, and virtual objects 608 via the wrist wearable device 700, the AR device 800, and / or the HIPD 900.

[0122] The user 602 may provide user input using any of the wrist wearable device 700, the AR device 800, and / or the HIPD 900. For example, the user 602 may perform one or more gestures to provide user input, which are performed by the wrist wearable device 700 (e.g., using the following reference Fig. 7A and Figure 7B One or more EMG sensors and / or IMUs described herein) and / or detected by the AR device 800 (e.g., using the following reference Fig. 8A and Figure 8B-1 as well as Figure 8B-2 Alternatively or additionally, user 602 may provide user input via one or more touch surfaces of wrist wearable device 700, AR device 800, and / or HIPD 900, and / or voice commands collected by microphones of wrist wearable device 700, AR device 800, and / or HIPD 900. In some embodiments, wrist wearable device 700, AR device 800, and / or HIPD 900 include a digital assistant for helping the user provide user input (e.g., completing a series of operations, recommending different operations or commands, providing reminders, or confirming commands). In some embodiments, user 602 may provide user input via one or more facial gestures and / or facial expressions. For example, a camera of wrist wearable device 700, AR device 800, and / or HIPD 900 may track the eyes of user 602 for navigating the user interface.

[0123] The wrist wearable device 700, the AR device 800, and / or the HIPD 900 may operate alone or in combination to allow the user 602 to interact with the AR environment. In some embodiments, the HIPD 900 is configured to operate as a central hub or control center for the wrist wearable device 700, the AR device 800, and / or another communicatively coupled device. For example, the user 602 may provide input for interacting with the AR environment at any of the wrist wearable device 700, the AR device 800, and / or the HIPD 900, and the HIPD 900 may identify one or more back-end tasks and front-end tasks to perform the requested interaction, and distribute instructions to perform the one or more back-end tasks and front-end tasks at the wrist wearable device 700, the AR device 800, and / or the HIPD 900. In some embodiments, the back-end tasks are background processing tasks that are not perceptible to the user (e.g., rendering content, decompression, or compression), while the front-end tasks are user-oriented tasks that are perceptible to the user (e.g., presenting information to the user or providing feedback to the user). As described below with reference to Fig. 9A and Fig. 9BAs described, the HIPD 900 can perform backend tasks and provide the wrist wearable device 700 and / or the AR device 800 with operation data corresponding to the performed backend tasks, so that the wrist wearable device 700 and / or the AR device 800 can perform frontend tasks. In this way, the HIPD 900 (which has more computing resources and greater thermal headroom than the wrist wearable device 700 and / or the AR device 800) performs computing-intensive tasks and reduces the computer resource utilization and / or power usage of the wrist wearable device 700 and / or the AR device 800.

[0124] In the example shown in the first AR system 600a, the HIPD 900 identifies one or more back-end tasks and front-end tasks associated with a user request to initiate an AR video call with one or more other users (represented by avatars 604 and digital representations 606 of contacts), and distributes instructions to cause the execution of the one or more back-end tasks and front-end tasks. Specifically, the HIPD 900 executes a back-end task for processing and / or rendering image data (and other data) associated with the AR video call, and provides the AR device 800 with operation data associated with the executed back-end task, so that the AR device 800 executes a front-end task for presenting the AR video call (e.g., presenting the avatar 604 and the digital representation 606 of the contact).

[0125] In some embodiments, the HIPD 900 can be operated as a focus or anchor point for information presentation. This allows the user 602 to know generally where the information is presented. For example, as shown in the first AR system 600a, the avatar 604 and the digital representation 606 of the contact are presented above the HIPD 900. Specifically, the HIPD 900 and the AR device 800 operate in combination to determine the location for presenting the avatar 604 and the digital representation 606 of the contact. In some embodiments, information can be presented within a predetermined distance (e.g., within five meters) from the HIPD 900. For example, as shown in the first AR system 600a, the virtual object 608 is presented on the table at a distance from the HIPD 900. Similar to the above example, the HIPD 900 and the AR device 800 can be combined to operate to determine the location for presenting the virtual object 608. Alternatively, in some embodiments, the presentation of information is not constrained by the HIPD 900. More specifically, the avatar 604 , the digital representation of the contact 606 , and the virtual object 608 do not have to be presented within a predetermined distance of the HIPD 900 .

[0126] User input provided at the wrist wearable device 700, the AR device 800, and / or the HIPD 900 is coordinated so that the user can use any device to initiate, continue, and / or complete an operation. For example, the user 602 can provide user input to the AR device 800 to cause the AR device 800 to present a virtual object 608, and when the AR device 800 presents the virtual object 608, the user 602 can provide one or more gestures via the wrist wearable device 700 to interact with and / or manipulate the virtual object 608.

[0127] Figure 6B The user 602 is shown wearing the wrist wearable device 700 and the AR device 800, and holding the HIPD 900. In the second AR system 600b, the wrist wearable device 700, the AR device 800, and / or the HIPD 900 are used to receive one or more messages and / or provide one or more messages to contacts of the user 602. Specifically, the wrist wearable device 700, the AR device 800, and / or the HIPD 900 detect and coordinate one or more user inputs to launch a messaging application, and prepare a response to the received message via the messaging application.

[0128] In some embodiments, the user 602 launches an application on the wrist wearable device 700, the AR device 800, and / or the HIPD 900 via user input, which causes the application to be launched on at least one device. For example, in the second AR system 600b, the user 602 performs a gesture associated with a command for launching a messaging application (represented by the messaging user interface 612); the wrist wearable device 700 detects the gesture; and the wrist wearable device 700, based on determining that the user 602 is wearing the AR device 800, causes the AR device 800 to present the messaging user interface 612 of the messaging application. The AR device 800 can present the messaging user interface 612 to the user 602 via its display (e.g., as shown in the field of view 610 of the user 602). In some embodiments, the application is launched and can be run on the device (e.g., the wrist wearable device 700, the AR device 800, and / or the HIPD 900) that detects the user input for launching the application, and the device provides operation data to another device to cause the messaging application to be presented. For example, the wrist wearable device 700 can detect user input for launching a messaging application; launch and run the messaging application; and provide operational data to the AR device 800 and / or the HIPD 900 to cause the messaging application to be presented. Alternatively, the application can be launched and run on a device different from the device that detects the user input. For example, the wrist wearable device 700 can detect a gesture associated with launching a messaging application, and can cause the HIPD 900 to run the messaging application and coordinate the presentation of the messaging application.

[0129] In addition, the user 602 can provide user input provided at the wrist wearable device 700, the AR device 800, and / or the HIPD 900 to continue and / or complete an operation initiated at another device. For example, after launching a messaging application via the wrist wearable device 700 and when the AR device 800 presents the messaging user interface 612, the user 602 can provide input at the HIPD 900 to prepare a response (e.g., as shown by a sliding gesture performed on the HIPD 900). The gesture performed by the user 602 on the HIPD 900 can be provided and / or displayed on another device. For example, the sliding gesture performed by the user 602 on the HIPD 900 is displayed on the virtual keyboard of the messaging user interface 612 displayed by the AR device 800.

[0130] In some embodiments, the wrist wearable device 700, the AR device 800, the HIPD 900, and / or other communicatively coupled devices may present one or more notifications to the user 602. The notification may be an indication of a new message, an incoming call, an application update, a status update, and the like. The user 602 may select a notification via the wrist wearable device 700, the AR device 800, or the HIPD 900, and cause an application or operation associated with the notification to be presented on at least one device. For example, the user 602 may receive a notification that a message has been received at the wrist wearable device 700, the AR device 800, the HIPD 900, and / or other communicatively coupled devices, and provide user input at the wrist wearable device 700, the AR device 800, and / or the HIPD 900 to view the notification, and the device that detects the user input may cause an application associated with the notification to be launched and / or presented at the wrist wearable device 700, the AR device 800, and / or the HIPD 900.

[0131] Although the above example describes coordinated input for interacting with a messaging application, a person skilled in the art will recognize upon reading this description that user input can be coordinated to interact with any number of applications, including but not limited to gaming applications, social media applications, camera applications, web-based applications, and financial applications, etc. For example, the AR device 800 can present gaming application data to the user 602, and the HIPD 900 can use a controller to provide input to the game. Similarly, the user 602 can use the wrist wearable device 700 to activate the camera of the AR device 800, and the user can use the wrist wearable device 700, the AR device 800, and / or the HIPD 900 to manipulate image acquisition (e.g., zoom in or out, or apply filters) and acquire image data.

[0132] Go to Figure 6C-1 and Figure 6C-2 , showing a user 602 wearing a wrist wearable device 700 and a VR device 810, and holding a HIPD 900. In the third AR system 600c, the wrist wearable device 700, the VR device 810, and / or the HIPD 900 are used to interact within an AR environment (e.g., a VR game or other AR application). When the VR device 810 presents a representation of a VR game (e.g., a first AR game environment 620) to the user 602, the wrist wearable device 700, the VR device 810, and / or the HIPD 900 detect and coordinate one or more user inputs to allow the user 602 to interact with the VR game.

[0133] In some embodiments, the user 602 may provide user input that causes an action in the corresponding AR environment via the wrist wearable device 700, the VR device 810, and / or the HIPD 900. Figure 6C-1 602) raises the HIPD 900 in preparation for swinging in the first AR gaming environment 620. The VR device 810 causes the AR representation 622 of the user to perform a similar action (e.g., raise a virtual object, such as a virtual sword 624) in response to the user 602 raising the HIPD 900. In some embodiments, each device uses corresponding sensor data and / or image data to detect user input and provide an accurate representation of the user's 602 movement. For example, the imaging sensor 954 of the HIPD 900 (e.g., as shown below) Fig. 9A and Fig. 9B The simultaneous localization and mapping (SLAM) camera or other camera discussed in the specification can be used to detect the position of the HIPD 900 relative to the body of the user 602 so that the virtual objects can be appropriately positioned within the first AR game environment 620; the sensor data from the wrist wearable device 700 can be used to detect the speed at which the user 602 raises the HIPD 900 so that the user's AR representation 622 and virtual sword 624 are synchronized with the movement of the user 602; and the image sensor 826 ( FIG. 8A to FIG. 8C ) can be used to represent the body of the user 602, boundary conditions, or real-world objects within the first AR game environment 620.

[0134] exist Figure 6C-2 , the user 602 performs a downward swing while holding the HIPD 900. The wrist wearable device 700, the VR device 810, and / or the HIPD 900 detect the downward swing of the user 602, and perform a corresponding action in the first AR game environment 620. In some embodiments, the data collected by each device is used to improve the user's experience in the AR environment. For example, the sensor data of the wrist wearable device 700 can be used to determine the speed and / or force of performing a downward swing, and the image sensor of the HIPD 900 and / or the VR device 810 can be used to determine the location of the swing and how the swing should be represented in the first AR game environment 620, which in turn can be used as an input to the AR environment (e.g., a game mechanism that can use the detected speed, force, position, and / or various aspects of the user 602's action to classify the user's input (e.g., the user performs a light strike, a heavy strike, a critical strike, a glancing strike, a miss) or can calculate an output (e.g., an amount of damage)).

[0135] Although the wrist wearable device 700, VR device 810, and / or HIPD 900 are described as detecting user input, in some embodiments, the user input is detected at a single device (where the single device is responsible for distributing the signal to other devices for performing the user input). For example, the HIPD 900 can run an application for generating the first AR game environment 620, and provide the VR device 810 with corresponding data for rendering the first AR game environment 620, and detect the movement of the user 602 (when holding the HIPD 900) so that the corresponding action is performed within the first AR game environment 620. Additionally or alternatively, in some embodiments, the operation data (e.g., sensor data, image data, application data, device data, and / or other data) of one or more devices is provided to a single device (e.g., HIPD 900) to process the operation data and cause the corresponding device to perform an action associated with the processed operation data.

[0136] Having discussed example AR systems, devices for interacting with such AR systems, and other computing systems in a relatively general manner, each device and each component will now be discussed in more detail below. For ease of reference, some definitions of the following devices and components are defined herein: these devices and components may be included in some or all of the multiple example devices discussed below. Those skilled in the art will recognize that certain types of components described below may be more suitable for a particular set of devices and less suitable for a different set of devices. However, subsequent references to the components defined herein should be deemed to be covered by the definitions provided.

[0137] In some embodiments discussed below, multiple example devices and systems including electronic devices and systems will be discussed. Such example devices and systems are not intended to be limiting, and those skilled in the art will appreciate that alternative devices and systems to the example devices and systems described herein may be used to perform the operations and construct the systems and devices described herein.

[0138] As described herein, an electronic device is a device that uses electrical energy to perform a specific function. The electronic device can be any physical object containing electronic components (e.g., transistors, resistors, capacitors, diodes, and integrated circuits). Examples of electronic devices include smart phones, laptop computers, digital cameras, televisions, game consoles, and music players, as well as example electronic devices discussed herein. As described herein, an intermediate electronic device is a device that is located between two other electronic devices and / or between a subset of multiple components of one or more electronic devices, and that facilitates communication, and / or data processing, and / or data transmission between corresponding electronic devices and / or electronic components.

[0139] Example wrist wearable device

[0140] Fig. 7A and Figure 7B An example wrist wearable device 700 is shown in accordance with some embodiments. The wrist wearable device 700 is a device described herein with reference to Figures 1A to 3C The example of wrist-wearable device 110 is described such that wrist-wearable device 700 should be understood to have features of wrist-wearable device 110 , and vice versa. Fig. 7A Various components of wrist wearable device 700 are shown, which can be used alone or in combination, including combinations that include other electronic devices and / or electronic components.

[0141] As discussed below, Fig. 7A Wearable band 710 and watch body 720 (or capsule) are shown coupled to form wrist wearable device 700. Wrist wearable device 700 can perform various functions and / or operations associated with navigating in a user interface and selectively launching applications, as well as the above referenced Figures 1A to 3C Describes the functionality and / or operation.

[0142] As will be described in more detail below, operations performed by wrist wearable device 700 may include: (i) presenting content to a user (e.g., displaying visual content via display 705); (ii) detecting (e.g., sensing) user input (e.g., sensing a touch on peripheral button 723 and / or a touch at a touch screen of display 705, sensing a gesture detected by a sensor (e.g., a biopotential sensor)); (iii) sensing biometric data (e.g., neuromuscular signals, heart rate, temperature, or sleep) via one or more sensors 713; messaging (e.g., text, voice, or video); image acquisition via one or more imaging devices or cameras 725; wireless communications (e.g., cellular, near-field, Wi-Fi, or personal area network); location determination; financial transactions; providing tactile feedback; alarms; notifications; biometric authentication; health monitoring; and / or sleep monitoring.

[0143] The above example functions can be performed independently in the watch body 720, independently in the wearable band 710, and / or via electronic communication between the watch body 720 and the wearable band 710. In some embodiments, when an AR environment is presented (e.g., via one of the AR systems 600a to 600c), the functions can be performed on the wrist wearable device 700. As will be appreciated by those skilled in the art upon reading the description provided herein, the novel wearable device described herein can be used with other types of AR environments.

[0144] The wearable band 710 can be configured to be worn by a user so that the inner (or inner side) surface of the wearable structure 711 of the wearable band 710 contacts the user's skin. The sensor 713 contacts the user's skin when worn by the user. The sensor 713 can sense biometric data, such as the user's heart rate, saturated oxygen level, temperature, sweat level, neuromuscular signals, or a combination thereof. The sensor 713 can also sense data about the user's environment, including the user's movement, altitude, location, orientation, gait, acceleration, position, or a combination thereof. In some embodiments, the sensor 713 is configured to track the position and / or movement of the wearable band 710. One or more sensors 713 may include those defined above and / or the following regarding Figure 7B Any sensor of the plurality of sensors discussed.

[0145] One or more sensors 713 may be distributed on the inner surface and / or outer surface of the wearable band 710. In some embodiments, one or more sensors 713 are evenly spaced along the wearable band 710. Alternatively, in some embodiments, one or more sensors 713 are located at different points along the wearable band 710. Fig. 7A As shown, one or more sensors 713 may be the same or different. For example, in some embodiments, one or more sensors 713 may be shaped as a pill (e.g., sensor 713a), an oval, a circle, a square, an ellipse (e.g., sensor 713c) and / or any other shape that maintains contact with the user's skin (e.g., so that neuromuscular signals and / or other biometric data can be accurately measured at the user's skin). In some embodiments, one or more sensors 713 are aligned to form a sensor pair (e.g., for sensing neuromuscular signals based on differential sensing within each corresponding sensor). For example, sensor 713b is aligned with an adjacent sensor to form sensor pair 714a, and sensor 713d is aligned with an adjacent sensor to form sensor pair 714b. In some embodiments, the wearable band 710 does not have a sensor pair. Alternatively, in some embodiments, the wearable band 710 has a predetermined number of sensor pairs (one sensor pair, three sensor pairs, four sensor pairs, six sensor pairs, or sixteen sensor pairs).

[0146] The wearable band 710 may include any suitable number of sensors 713. In some embodiments, the number and arrangement of sensors 713 depends on the specific application in which the wearable band 710 is used. For example, a wearable band 710 configured as an armband, wristband, or chest band may include multiple sensors 713, wherein various use cases (e.g., a medical use case versus a gaming use case or a general daily use case) have a different number of sensors 713 and a different arrangement.

[0147] According to some embodiments, the wearable band 710 also includes an electrical ground electrode and a shield electrode. Similar to the sensor 713, the electrical ground electrode and the shield electrode can be distributed on the inner surface of the wearable band 710 so that they contact a portion of the user's skin. For example, the electrical ground electrode and the shield electrode can be located at the inner surface of the coupling mechanism 716 or at the inner surface of the wearable structure 711. The electrical ground electrode and the shield electrode can be formed by the same components as the sensor 713 and / or use the same components as the sensor 713. In some embodiments, the wearable band 710 includes more than one electrical ground electrode and more than one shield electrode.

[0148] The sensor 713 can be formed as part of the wearable structure 711 of the wearable band 710. In some embodiments, the sensor 713 is flush or substantially flush with the wearable structure 711 so that the sensors do not extend beyond the surface of the wearable structure 711. Despite being flush with the wearable structure 711, the sensor 713 is still configured to contact the user's skin (e.g., via a skin contact surface). Alternatively, in some embodiments, the sensor 713 extends beyond the wearable structure 711 by a predetermined distance (e.g., 0.1 millimeters (mm) to 2 mm) to contact and press into the user's skin. In some embodiments, the sensor 713 is coupled to an actuator (not shown) that is configured to adjust the extension height of the sensor 713 (e.g., the distance from the surface of the wearable structure 711) so that the sensor 713 contacts and presses into the user's skin. In some embodiments, the actuator adjusts the extension height between 0.01 mm and 1.2 mm. This allows the user to customize the position of the sensor 713 to improve the overall comfort of the wearable band 710 when being worn, while still allowing the sensor 713 to contact the user's skin. In some embodiments, the sensor 713 is not distinguishable from the wearable structure 711 when being worn by the user.

[0149] The wearable structure 711 can be formed of an elastic material, an elastomer, etc., which is configured to be stretched and suitable for being worn by a user. In some embodiments, the wearable structure 711 is a textile or a woven fabric. As described above, the sensor 713 can be formed as a part of the wearable structure 711. For example, the sensor 713 can be molded into the wearable structure 711 or integrated into the woven fabric (for example, the sensor 713 can be sewn into the fabric and simulate the flexibility of the fabric (for example, the sensor 713 can be composed of a series of woven fabric threads)).

[0150] The wearable structure 711 may include sensors 713, electronic circuits, and / or other electronic components to be included in the wearable band 710 (hereinafter referred to as Figure 7BIn some embodiments, the flexible electronic connector is configured to interconnect the sensor 713, electronic circuits, and / or other electronic components of the wearable band 710 with corresponding sensors and / or other electronic components of another electronic device (e.g., the watch body 720). The flexible electronic connector is configured to move with the wearable structure 711 so that adjustments (e.g., resizing, pulling, or folding) of the wearable structure 711 by the user do not cause stress or strain on the electrical coupling of the components of the wearable band 710.

[0151] As described above, the wearable band 710 is configured to be worn by a user. Specifically, the wearable band 710 can be shaped or otherwise manipulated to be worn by a user. For example, the wearable band 710 can be shaped to have a generally circular shape so that the wearable band can be configured to be worn on the user's lower arm or wrist. Alternatively, the wearable band 710 can be shaped to be worn on another body part of the user (e.g., the user's upper arm (e.g., around the biceps), forearm, chest, leg, etc.). The wearable band 710 may include a retaining mechanism 712 (e.g., a buckle or hook-and-loop fastener) that is used to secure the wearable band 710 to the user's wrist or other body part. When the wearable band 710 is worn by the user, the sensor 713 senses data from the user's skin (referred to as sensor data). In particular, the sensor 713 of the wearable band 710 acquires (e.g., senses and records) neuromuscular signals.

[0152] The sensed data (e.g., sensed neuromuscular signals) can be used to detect and / or determine the user's intention to perform certain motor actions. In particular, when the user performs muscle activation (e.g., movement or gesture), the sensor 713 senses and records neuromuscular signals from the user. The detected and / or determined motor actions (e.g., phalanges (or fingers) movements, wrist movements, hand movements and / or other muscle intentions) can be used to determine control commands or control information (instructions for executing certain commands after sensing data) for causing the computing device to execute one or more input commands. For example, the sensed neuromuscular signals can be used to control certain user interfaces displayed on the display 705 of the wrist wearable device 700, and / or can be sent to a device (e.g., head-mounted display) responsible for rendering the AR environment to perform actions in the associated AR environment (e.g., to control the movement of a virtual device displayed to the user). The muscle activations performed by the user may include static gestures, such as placing the user's palm down on a table; dynamic gestures, such as grasping a physical or virtual object; and covert gestures that are imperceptible to another person, such as slightly tensing a joint by co-contracting opposing muscles or using submuscular activation. The muscle activations performed by the user may include symbolic gestures (e.g., gestures that are mapped to other gestures, interactions, or commands, such as based on a gesture vocabulary that specifies a mapping of gestures to commands).

[0153] The sensor data sensed by sensor 713 can be used to provide the user with enhanced interaction with physical objects (e.g., devices communicatively coupled to wearable band 710) and / or virtual objects generated by the AR system in an AR application (e.g., user interface objects presented on display 705 or another computing device (e.g., a smartphone)).

[0154] In some embodiments, the wearable band 710 includes one or more haptic devices 746 ( Figure 7B ; for example, a vibrotactile actuator), the one or more tactile devices being configured to provide tactile feedback (e.g., skin sensation and / or kinesthetic perception) to the user's skin. Sensor 713 and / or tactile device 746 can be configured to operate in conjunction with a plurality of applications, including but not limited to health monitoring, social media, gaming, and AR (e.g., applications associated with AR).

[0155] The wearable band 710 may also include a coupling mechanism 716 (e.g., a bracket or shape of the coupling mechanism may correspond to the shape of the watch body 720 of the wrist wearable device 700) for removably coupling the capsule (e.g., computing unit) or the watch body 720 to the wearable band 710 (via a coupling surface of the watch body 720). In particular, the coupling mechanism 716 may be configured to receive a coupling surface of the watch body 720 near a bottom side (e.g., a side opposite to the front side of the watch body 720 where the display 705 is located) so that a user can push the watch body 720 downward into the coupling mechanism 716 to attach the watch body 720 to the coupling mechanism 716. In some embodiments, the coupling mechanism 716 may be configured to receive a top side of the watch body 720 (e.g., a side near the front side where the display 705 of the watch body 720 is located) that is pushed upward into the bracket instead of being pushed downward into the coupling mechanism 716. In some embodiments, coupling mechanism 716 is an integral component of wearable band 710, such that wearable band 710 and coupling mechanism 716 are a single unified structure. In some embodiments, coupling mechanism 716 is a type of frame or housing that allows the coupling surface of watch body 720 to be retained within or on coupling mechanism 716 (e.g., a cradle, tracking band, support base, or buckle) of wearable band 710.

[0156] The coupling mechanism 716 can allow the watch body 720 to be removably coupled to the wearable band 710 by friction fit, magnetic coupling, a rotation-based connector, a scissor pin coupling, a retaining spring, one or more magnets, a clip, a pin, a hook and loop fastener, or a combination thereof. The user can perform any type of action to couple the watch body 720 to the wearable band 710 and detach the watch body 720 from the wearable band 710. For example, the user can twist, slide, turn, push, pull, or rotate (or a combination thereof) the watch body 720 relative to the wearable band 710 to attach the watch body 720 to the wearable band 710 and detach the watch body 720 from the wearable band 710. Alternatively, as discussed below, in some embodiments, the watch body 720 can be detached from the wearable band 710 by actuation of the release mechanism 729.

[0157] The wearable band 710 can be coupled with the watch body 720 to increase the functionality of the wearable band 710 (e.g., converting the wearable band 710 into a wrist wearable device 700, adding additional computing units and / or batteries to increase the computing resources and / or battery life of the wearable band 710, or adding additional sensors to improve the sensed data). As described above, the wearable band 710 (and the coupling mechanism 716) are configured to operate independently of the watch body 720 (e.g., perform functions independently of the watch body). For example, the coupling mechanism 716 may include one or more sensors 713 that contact the user's skin when the user wears the wearable band 710 and provide sensor data for determining control commands.

[0158] The user can remove the watch body 720 (or the capsule) from the wearable band 710 to reduce the burden on the user of the wrist wearable device 700. For embodiments in which the watch body 720 is detachable, the watch body 720 can be referred to as a detachable structure, so that in these embodiments, the wrist wearable device 700 includes a wearable portion (e.g., the wearable band 710) and a detachable structure (watch body 720).

[0159] Turning to the watch body 720, the watch body 720 may have a generally rectangular or circular shape. The watch body 720 is configured to be worn by a user on their wrist or on another body part. More specifically, the watch body 720 is sized to be easily carried by a user, easily attached to a portion of a user's clothing, and / or easily coupled to the wearable band 710 (thereby forming a wrist wearable device 700). As described above, the watch body 720 may have a shape corresponding to the coupling mechanism 716 of the wearable band 710. In some embodiments, the watch body 720 includes a single release mechanism 729 or multiple release mechanisms (e.g., two release mechanisms 729 positioned on opposite sides of the watch body 720, such as spring-loaded buttons) to separate the watch body 720 from the wearable band 710. The release mechanism 729 may include, but is not limited to, a button, a knob, a plug, a handle, a lever, a fastener, a buckle, a dial, a latch, or a combination thereof.

[0160] The user can actuate the release mechanism 729 by pushing, turning, lifting, pressing, moving the release mechanism 729, or performing other actions on the release mechanism 729. Actuation of the release mechanism 729 can release (e.g., separate) the watch body 720 from the coupling mechanism 716 of the wearable band 710, thereby allowing the user to use the watch body 720 independently of the wearable band 710, and vice versa. For example, separating the watch body 720 from the wearable band 710 can allow the user to use the rear camera 725b to capture images. Although the coupling mechanism 716 is shown as being positioned at a corner of the watch body 720, the release mechanism 729 can be positioned at any position on the watch body 720 that is convenient for user actuation. In addition, in some embodiments, the wearable band 710 may also include a corresponding release mechanism for separating the watch body 720 from the coupling mechanism 716. In some embodiments, the release mechanism 729 is optional, and as described above, the watch body 720 can be separated from the coupling mechanism 716 (e.g., via twisting or rotation).

[0161] The watch body 720 may include one or more peripheral buttons 723 and 727 for performing various operations at the watch body 720. For example, the peripheral buttons 723 and 727 may be used to turn on or wake up the display 705 (e.g., to convert the display 705 from a dormant state to an active state), unlock the watch body 720, increase or decrease the volume, increase or decrease the brightness, interact with one or more applications, interact with one or more user interfaces. Additionally or alternatively, in some embodiments, the display 705 acts as a touch screen and allows a user to provide one or more inputs for interacting with the watch body 720.

[0162] In some embodiments, the watch body 720 includes one or more sensors 721. The sensors 721 of the watch body 720 may be the same as or different from the sensors 713 of the wearable band 710. The sensors 721 of the watch body 720 may be distributed on the inner surface and / or the outer surface of the watch body 720. In some embodiments, the sensors 721 are configured to contact the user's skin when the user wears the watch body 720. For example, the sensor 721 may be placed on the bottom side of the watch body 720, and the coupling mechanism 716 may be a bracket with an opening that allows the bottom side of the watch body 720 to directly contact the user's skin. Alternatively, in some embodiments, the watch body 720 does not include a sensor configured to contact the user's skin (e.g., includes sensors inside and / or outside the watch body 720 that are configured to sense data of the watch body 720 and data of the surrounding environment of the watch body 720). In some embodiments, the sensors 721 are configured to track the position and / or movement of the watch body 720.

[0163] The watch body 720 and the wearable band 710 can share data using a wired communication method (e.g., a Universal Asynchronous Receiver / Transmitter (UART) or a USB transceiver) and / or a wireless communication method (e.g., near field communication or Bluetooth). For example, the watch body 720 and the wearable band 710 can share data sensed by sensors 713 and 721, as well as application-specific and device-specific information (e.g., active and / or available applications), output devices (e.g., a display or a speaker), and / or input devices (e.g., a touch screen, a microphone, or an imaging sensor).

[0164] In some embodiments, the watch body 720 may include, but is not limited to, a front camera 725a and / or a rear camera 725b, sensors 721 (e.g., biometric sensors, IMU sensors, heart rate sensors, saturated oxygen sensors, neuromuscular signal sensors, altimeter sensors, temperature sensors, bioimpedance sensors, pedometer sensors, optical sensors (e.g., Figure 7B ; imaging sensor 763), touch sensor, sweat sensor). In some embodiments, the watch body 720 may include one or more tactile devices 776 ( Figure 7B ; vibrotactile actuator), the one or more tactile devices 776 are configured to provide tactile feedback (e.g., skin sensation and / or kinesthetic perception) to the user. The sensor 721 and / or the tactile device 776 may also be configured to operate in conjunction with multiple applications, including but not limited to health monitoring applications, social media applications, gaming applications, and AR applications (e.g., applications associated with AR).

[0165] As described above, the watch body 720 and the wearable band 710 can form the wrist wearable device 700 when coupled. The watch body 720 and the wearable band 710 act as a single device when coupled to perform the functions described herein (e.g., operation, detection, or communication). In some embodiments, each device is provided with specific instructions for performing one or more operations of the wrist wearable device 700. For example, based on determining that the watch body 720 does not include a neuromuscular signal sensor, the wearable band 710 may include alternative instructions for executing the associated instructions (e.g., providing the sensed neuromuscular signal data to the watch body 720 via a different electronic device). The various operations of the wrist wearable device 700 can be performed by the watch body 720 alone or by the watch body in combination with the wearable band 710 (e.g., via corresponding processors and / or hardware components), or vice versa. In some embodiments, the various operations of the wrist wearable device 700, the watch body 720, and / or the wearable band 710 can be communicated with another communicatively coupled device (e.g., Fig. 9A and Fig. 9B; One or more processors and / or hardware components of HIPD 900) are executed in combination.

[0166] As the following reference Figure 7B As described in the block diagram of FIG. 7 , the wearable band 710 and / or the body 720 may each include independent resources required to independently perform functions. For example, the wearable band 710 and / or the body 720 may each include a power source (e.g., a battery), a memory, a data storage device, a processor (e.g., a CPU), a communication, a light source, and / or an input / output device.

[0167] Figure 7B A block diagram of a computing system 730 corresponding to a wearable band 710 and a computing system 760 corresponding to a watch body 720 is shown according to some embodiments. According to some embodiments, the computing system of wrist wearable device 700 includes a combination of components of the wearable band computing system 730 and components of the watch body computing system 760.

[0168] The watch body 720 and / or the wearable band 710 may include one or more components shown in the watch body computing system 760. In some embodiments, a single integrated circuit includes all or most of the components of the watch body computing system 760, which are included in a single integrated circuit. Alternatively, in some embodiments, the components of the watch body computing system 760 are included in multiple integrated circuits that are communicatively coupled. In some embodiments, the watch body computing system 760 is configured to be coupled to the wearable band computing system 730 (e.g., via a wired connection or a wireless connection), which allows the two computing systems to share components, distribute tasks, and / or perform other operations described herein (separately or as a single device).

[0169] The meter body computing system 760 may include one or more processors 779, a controller 777, a peripheral interface 761, a power system 795, and a memory (eg, memory 780), each of which is defined above and described in more detail below.

[0170] The power system 795 may include a charger input 796, a power management integrated circuit (PMIC) 797, and a battery 798, each of which is defined above. In some embodiments, the body 720 and the wearable band 710 may have respective charger inputs (e.g., charger inputs 796 and 757), respective batteries (e.g., batteries 798 and 759), and may share power with each other (e.g., the body 720 may power and / or charge the wearable band 710, and vice versa). Although the body 720 and / or the wearable band 710 may include respective charger inputs, a single charger input may charge the two devices when the two devices are coupled. The body 720 and the wearable band 710 may receive a charge using various techniques. In some embodiments, the body 720 and the wearable band 710 may receive a charge using a wired charging component (e.g., a power cord). Alternatively or additionally, the body 720 and / or the wearable band 710 may be configured for wireless charging. For example, the portable charging device can be designed to match a portion of the watch body 720 and / or a portion of the wearable band 710 and wirelessly transmit available power to the battery of the watch body 720 and / or the battery of the wearable band 710. The watch body 720 and the wearable band 710 can have independent power systems (e.g., power systems 795 and 756) to enable each to operate independently. The watch body 720 and the wearable band 710 can also share power (e.g., one can charge the other) via their respective PMICs (e.g., PMICs 797 and 758), which can share power through power conductors and ground conductors and / or through wireless charging antennas.

[0171] In some embodiments, the peripheral interface 761 may include one or more sensors 721, many of which are defined above. The sensor 721 may include one or more coupling sensors 762, which are used to detect when the watch body 720 is coupled to another electronic device (e.g., the wearable band 710). The sensor 721 may include an imaging sensor 763 (one or more of the cameras 725 and / or a separate imaging sensor 763 (e.g., a thermal imaging sensor)). In some embodiments, the sensor 721 includes one or more SpO2 sensors 764. In some embodiments, the sensor 721 includes one or more biopotential signal sensors (e.g., an EMG sensor 765, which may be disposed on a user-facing portion of the watch body 720 and / or the wearable band 710). In some embodiments, the sensor 721 includes one or more capacitive sensors 766. In some embodiments, the sensor 721 includes one or more heart rate sensors 767. In some embodiments, the sensor 721 includes one or more IMU sensors 768. In some embodiments, one or more IMU sensors 768 may be configured to detect movement of a user's hand, or movement of other locations where watch body 720 is placed or held.

[0172] In some embodiments, the peripheral interface 761 includes an NFC component 769, a GPS component 770, a long-term evolution (LTE) component 771, and / or a Wi-Fi and / or Bluetooth communication component 772. In some embodiments, the peripheral interface 761 includes one or more buttons 773 (e.g., Fig. 7A The peripheral interface 761 may include peripheral buttons 723 and 727 in the watch body 720, which, when selected by the user, cause an operation to be performed at the watch body 720. In some embodiments, the peripheral interface 761 includes one or more indicators (e.g., light-emitting diodes (LEDs)) to provide visual indicators to the user (e.g., received messages, low battery, active microphone and / or camera).

[0173] The watch body 720 may include at least one display 705 for displaying a visual representation of information or data to the user, the visual representation including user interface elements and / or three-dimensional (3D) virtual objects. The display may also include a touch screen for inputting user input (e.g., touch gestures and sliding gestures, etc.). The watch body 720 may include at least one speaker 774 and at least one microphone 775 for providing audio signals to the user and receiving audio input from the user. The user may provide user input through the microphone 775, and may also receive audio output from the speaker 774 as part of the tactile event provided by the tactile controller 778. The watch body 720 may include at least one camera 725, the at least one camera 725 including a front camera 725a and a rear camera 725b. The camera 725 may include an ultra-wide-angle camera, a wide-angle camera, a fisheye camera, a spherical camera, a telephoto camera, a depth sensing camera, or other types of cameras.

[0174] The watch body computing system 760 may include one or more haptic controllers 778 and associated components (e.g., haptic devices 776) for providing haptic events at the watch body 720 (e.g., vibration sensations or audio outputs in response to events at the watch body 720). The haptic controller 778 may communicate with one or more haptic devices 776 (e.g., electroacoustic devices) including one or more speakers 774 and / or other audio components and / or electromechanical devices that convert energy into linear motion (e.g., motors, electromagnetic coils, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components that convert electrical signals into haptic outputs on the device)). The haptic controller 778 may provide haptic events that can be felt by a user of the watch body 720 to corresponding haptic actuators. In some embodiments, the one or more haptic controllers 778 may receive input signals from an application in the applications 782.

[0175] In some embodiments, the computing system 760 may include a memory 780, which may be controlled by one or more controllers 777 and / or a memory controller of one or more processors 779. In some embodiments, the software components stored in the memory 780 include one or more applications 782 configured to perform operations at the watch body 720. In some embodiments, the one or more applications 782 include games, word processors, messaging applications, call applications, web browsers, social media applications, media streaming applications, financial applications, calendars, clocks, etc. In some embodiments, the software components stored in the memory 780 include one or more communication interface modules 783 as defined above. In some embodiments, the software components stored in the memory 780 include: one or more graphics modules 784 for rendering, encoding and / or decoding audio data and / or video data; and one or more data management modules 785 for collecting, organizing and / or providing access to data 787 stored in the memory 780. In some embodiments, the software components stored in the memory 780 include a multi-modal gesture (MMG) processing module 786A, which is configured to perform the above referenced Figures 1A to 3C For example, the MMG processing module includes processing as described above Figures 1A to 3C 720 and performed by the user 105. In some embodiments, the gestures processed by the MMG processing module include air gestures, touch gestures performed at another communicatively coupled device (e.g., the user 105 touches the watch body 720 of the wrist wearable device 700), inputs performed using one or more buttons at the head wearable device 120 and / or the wrist wearable device 110 (e.g., at the watch body 720 of the wrist wearable device, etc.). In some embodiments, one or more applications and / or one or more modules in the application 782 can work in conjunction with each other to perform various tasks at the watch body 720.

[0176] In some embodiments, the software components stored in the memory 780 may include one or more operating systems 781 (e.g., a Linux-based operating system, an Android operating system, etc.). The memory 780 may also include data 787. The data 787 may include profile data 788A, sensor data 789A, media content data 790, application data 791, and MMG data 792A, which stores the same information as described above. Figures 1A to 3C Data related to the execution of the various features described. MMG data 792A also represents data collected by the MMG processing module 786A.

[0177] It should be appreciated that watch body computing system 760 is an example of a computing system within watch body 720, and that watch body 720 may have more components or fewer components, may combine two or more components, and / or may have a different configuration and / or arrangement of the components than those shown in watch body computing system 760. The various components shown in watch body computing system 760 are implemented in hardware, software, firmware, or a combination thereof, including one or more signal processing circuits and / or application-specific integrated circuits (ASICs).

[0178] Turning to the wearable band computing system 730, one or more components that may be included in the wearable band 710 are shown. The wearable band computing system 730 may include more components or fewer components than those shown in the body computing system 760, may combine two or more components, and / or may have a different configuration and / or arrangement of some or all of these components. In some embodiments, all or most of the multiple components of the wearable band computing system 730 are included in a single integrated circuit. Alternatively, in some embodiments, the multiple components of the wearable band computing system 730 are included in multiple integrated circuits that are communicatively coupled. As described above, in some embodiments, the wearable band computing system 730 is configured to be coupled to the body computing system 760 (e.g., via a wired connection or a wireless connection), which allows the two computing systems to share components, distribute tasks, and / or perform other operations described herein (separately or as a single device).

[0179] Similar to the body computing system 760, the wearable band computing system 730 may include: one or more processors 749; one or more controllers 747 (including one or more tactile controllers 748); a peripheral interface 731, which may include one or more sensors 713 and other peripheral devices; a power source (e.g., a power system 756); and a memory (e.g., a memory 750), which includes an operating system (e.g., an operating system 751), data (e.g., data 754, which includes profile data 788B, sensor data 789B, MMG data 792B, etc.) and one or more modules (e.g., a communication interface module 752, a data management module 753, an MMG processing module 786B, etc.).

[0180] In view of the above definition, the one or more sensors 713 can be similar to the sensor 721 of the computing system 760. For example, the sensor 713 can include one or more coupling sensors 732, one or more SpO2 sensors 734, one or more EMG sensors 735, one or more capacitive sensors 736, one or more heart rate sensors 737, and one or more IMU sensors 738.

[0181] Peripheral interface 731 may also include other components similar to those included in peripheral interface 761 of computing system 760, as described above with reference to peripheral interface 761, including NFC component 739, GPS component 740, LTE component 741, Wi-Fi and / or Bluetooth communication component 742, and / or one or more haptic devices 746. In some embodiments, peripheral interface 731 includes one or more buttons 743, display 733, speaker 744, microphone 745, and camera 755. In some embodiments, peripheral interface 731 includes one or more indicators, such as LEDs.

[0182] It should be appreciated that the wearable band computing system 730 is an example of a computing system within the wearable band 710, and that the wearable band 710 may have more components or fewer components than those shown in the wearable band computing system 730, may combine two or more components, and / or may have a different configuration and / or arrangement of the components. The various components shown in the wearable band computing system 730 may be implemented in one or a combination of hardware, software, and firmware, including one or more signal processing circuits and / or application specific integrated circuits.

[0183] refer to Fig. 7A The wrist wearable device 700 is an example of a wearable band 710 and a watch body 720 coupled, and thus the wrist wearable device 700 will be understood to include the components shown and described for the wearable band computing system 730 and the watch body computing system 760. In some embodiments, the wrist wearable device 700 has a split architecture (e.g., a split mechanical architecture or a split electronic architecture) between the watch body 720 and the wearable band 710. In other words, all the components shown in the wearable band computing system 730 and the watch body computing system 760 can be housed or otherwise arranged in the combined wrist wearable device 700 (e.g., a watch device), or can be housed or otherwise arranged in a single component in the watch body 720, the wearable band 710, and / or portions thereof (e.g., the coupling mechanism 716 of the wearable band 710).

[0184] The above technology can be used with any device for sensing neuromuscular signals (including Fig. 7A and Figure 7BThe invention is intended for use with arm wearables in the context of neuromuscular control, but may also be used with other types of wearable devices for sensing neuromuscular signals, such as body wearables or head wearables that may have neuromuscular sensors closer to the brain or spine.

[0185] In some embodiments, the wrist wearable device 700 can be used in conjunction with the head wearable devices described below (e.g., AR device 800 and VR device 810) and / or HIPD 900; and the wrist wearable device 700 can also be configured to allow the user to control aspects of the artificial reality (e.g., by using EMG-based gestures to control user interface objects in the artificial reality, and / or by allowing the user to interact with a touch screen on the wrist wearable device to also control aspects of the artificial reality). Having described example wrist wearable devices in this manner, attention is now turned to example head wearable devices, such as AR device 800 and VR device 810.

[0186] Example head wearable device

[0187] Fig. 8A , Figure 8B-1 , Figure 8B-2 and Figure 8C An example head wearable device according to some embodiments is shown. The head wearable device may include, but is not limited to, an AR device 800 (e.g., an AR eyewear device or a smart eyewear device, such as smart glasses, smart monocles, smart contact lenses, etc.), a VR device 810 (e.g., a VR head-mounted device or a head-mounted display (HMD)), or other visually coupled devices. The AR device 800 and the VR device 810 are referenced herein. FIG. 1A to FIG. 1T The example of the head wearable device 120 described above is such that the head wearable device should be understood to have features of the AR device 800 and / or the VR device 810, and vice versa. The AR device 800 and the VR device 810 can perform various functions and / or operations associated with navigating in the user interface and selectively launching applications, as well as the above referenced FIG. 1A to FIG. 1T as well as Figures 4 to 5B Describes the functionality and / or operation.

[0188] In some embodiments, an AR system (e.g., Figure 6A to Figure 6C-2 ; AR system 600a to 600c) includes an AR device 800 (such as Fig. 8A ) and / or VR device 810 (as shown Figure 8B-1 and Figure 8B-2In some embodiments, the AR device 800 and the VR device 810 may include one or more similar components (e.g., components for presenting an interactive AR environment, such as a processor, a memory, and / or a presentation device, the presentation device including one or more displays and / or one or more waveguides), as described in detail in the accompanying drawings. Figure 8C Some of these components are described in more detail. The head wearable device may use a display projector (e.g., display projector components 807A and 807B) and / or a waveguide for projecting a representation of data to a user. Some embodiments of the head wearable device do not include a display.

[0189] Fig. 8A An example visual depiction of an AR device 800 (e.g., the AR device 800 may also be described herein as augmented reality glasses and / or smart glasses) is shown. The AR device 800 may be used with Fig. 8A 800, which are in electronic communication with or otherwise configured to be used in conjunction with the AR device 800. In some embodiments, the wearable accessory device and / or the intermediate processing device may be configured to communicate with the AR device 800 via a coupled sensor 824 (e.g., Figure 8C The AR device 800 may be coupled to the AR device 800 by a coupling mechanism for electronic communication with the AR device 800 (as shown), wherein the coupling sensor 824 may detect when the electronic device is physically or electronically coupled to the AR device 800. In some embodiments, the AR device 800 may be configured to couple to a housing (e.g., a portion of the frame 804 or a portion of the temple 805), which may include one or more additional coupling mechanisms configured to couple with additional accessory devices. Fig. 8A The components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, firmware, or a combination thereof, including one or more signal processing components and / or application specific integrated circuits (ASICs).

[0190] The AR device 800 includes mechanical eyeglass components, which include a frame 804 configured to hold one or more lenses (e.g., one or both of lenses 806-1 and 806-2). It will be appreciated by those of ordinary skill in the art that the AR device 800 may include additional mechanical components, such as a hinge configured to allow a portion of the frame 804 of the AR device 800 to fold and unfold, a bridge configured to span the gap between the lens 806-1 and the lens 806-2 and rest on the user's nose, a nose pad configured to rest on the bridge of the nose and provide support for the AR device 800, an earpiece configured to rest on the user's ear and provide additional support for the AR device 800, and a temple 805 configured to extend from the hinge of the AR device 800 to the earpiece, etc. It will also be appreciated by those of ordinary skill in the art that some examples of the AR device 800 may not include the mechanical components described herein. For example, a smart contact lens configured to present AR to a user may not include any components of the AR device 800.

[0191] Lenses 806-1 and 806-2 can be separate displays or display devices (e.g., waveguides for projected representations). Lenses 806-1 and 806-2 can work together or independently to present an image or a series of images to a user. In some embodiments, lenses 806-1 and 806-2 can operate in conjunction with one or more display projector components 807A and 807B to present image data to a user. Although AR device 800 includes two displays, embodiments of the present disclosure can be implemented in an AR device with a single near-eye display (NED) or more than two NEDs.

[0192] The AR device 800 includes a number of electronic components, many of which will be described below with reference to FIG. Figure 8C Describe in more detail. Fig. 8A Some example electronic components are shown in FIG. 8 , which include sensors 823 - 1 , 823 - 2 , 823 - 3 , 823 - 4 , 823 - 5 , and 823 - 6 , which may be distributed along a majority of the frame 804 of the AR device 800 . Figure 8C Different types of sensors are described. The AR device 800 also includes a left camera 839A and a right camera 839B located on different sides of the frame 804. And the eyewear device includes one or more processors 848A-1 and 848A-2 (e.g., an integrated microprocessor such as an ASIC) embedded in a portion of the frame 804.

[0193] Figure 8B-1 and Figure 8B-2An example visual depiction of a VR device 810 (e.g., a head mounted display (HMD) 812, also referred to herein as an AR headset, a head wearable device, or a VR headset) is shown. HMD 812 includes a front body 814 and a frame 816 (e.g., a strip or band) shaped to fit the user's head. In some embodiments, front body 814 and / or frame 816 include one or more electronic components (e.g., a display, a processor (e.g., processor 848A-1), an IMU, a tracking emitter or detector, or a sensor) that facilitates presentation of, and / or interaction with, an AR system and / or a VR system. In some embodiments, as Figure 8B-2 As shown, HMD 812 includes an output audio transducer (e.g., audio transducer 818). In some embodiments, as Figure 8B-2 As shown, one or more components (e.g., one or more output audio transducers 818 and frame 816) (e.g., part or all of frame 816 and / or output audio transducer 818) can be configured to be attached to and detached from HMD 812 (e.g., it is detachably attached to HMD 812). In some embodiments, coupling the detachable component to HMD 812 causes the detachable component to enter into electronic communication with HMD 812. VR device 810 includes electronic components, many of which will be described below with reference to Figure 8C Describe in more detail.

[0194] Figure 8B-1 and Figure 8B-2 The VR device 810 is also shown to have one or more cameras, such as a left camera 839A and a right camera 839B, which can be similar to the left camera and the right camera on the frame 804 of the AR device 800. In some embodiments, the VR device 810 includes one or more additional cameras (e.g., cameras 839C and 839D), which can be configured to enhance the image data acquired by the cameras 839A and 839B by providing more information. For example, camera 839C can be used to provide color information that is not recognized by cameras 839A and 839B. In some embodiments, one or more of the cameras 839A to 839D can include an optional infrared (IR) cut filter that is configured to remove IR light received at the corresponding camera sensor.

[0195] The VR device 810 may include a housing 890 that stores one or more components of the VR device 810 and / or additional components of the VR device 810. The housing 890 may be a modular electronic device that is configured to couple with the VR device 810 (or AR device 800) and to supplement and / or expand the capabilities of the VR device 810 (or AR device 800). For example, the housing 890 may include additional sensors, cameras, power supplies, and processors (e.g., processor 848A-2) to improve and / or increase the functionality of the VR device 810. The following references Figure 8C Examples of different components included in housing 890 are described.

[0196] Alternatively or additionally, in some embodiments, the head wearable device (e.g., VR device 810 and / or AR device 800) includes another external device (e.g., a pairing device) or is communicatively coupled to another external device, such as (hereinafter referred to as Fig.9A and Fig. 9B The HIPD 900 and / or the optional neckband discussed herein may be coupled to the head wearable device via one or more connectors (e.g., a wired connector or a wireless connector). The head wearable device and the neckband may operate independently without any wired or wireless connection between them. In some embodiments, the components of the head wearable device and the components of the neckband are located on one or more additional peripheral devices that are paired with the head wearable device, the neckband, or some combination thereof. In addition, the neckband is intended to represent any suitable type or form of paired device. Therefore, the following discussion of the neckband may also apply to various other paired devices, such as smart watches, smart phones, wristbands, other wearable devices, handheld controllers, tablet computers, or laptop computers.

[0197] In some cases, pairing an external device, such as an intermediate processing device (e.g., HIPD 900, an optional neckband, and / or a wearable accessory device) with a head wearable device (e.g., AR device 800 and / or VR device 810) enables the head wearable device to achieve a form factor similar to a pair of glasses while still providing sufficient battery power and computing power for the expanded capabilities. Some or all of the battery power, computing resources, and / or additional features of the head wearable device can be provided by the paired device, or shared between the paired device and the head wearable device, thereby reducing the weight, heat profile, and form factor of the head wearable device overall while allowing the head wearable device to maintain its desired functionality. For example, an intermediate processing device (e.g., HIPD 900) can allow components that are otherwise included in the head wearable device to be included in the intermediate processing device (and / or wearable device or accessory device), thereby shifting the weight load from the user's head and neck to one or more other parts of the user's body. In some embodiments, the intermediate processing device has a large surface area to diffuse and disperse heat to the surrounding environment through the large surface area. Thus, the intermediate processing device may allow for greater battery capacity and computing power than would otherwise be feasible on a separate head wearable device. Because the weight carried in the intermediate processing device may be less intrusive to the user than the weight carried in the head wearable device, the user may be able to tolerate wearing a lighter eyewear device and carrying or wearing a paired device for longer periods of time than if the user were wearing a heavier separate eyewear device, thereby enabling the AR environment to be more fully integrated into the user's daily activities.

[0198] In some embodiments, the intermediate processing device is communicatively coupled to the head wearable device and / or communicatively coupled to other devices. These other devices can provide certain functions (e.g., tracking, positioning, depth map construction, processing, and / or storage) to the head wearable device. In some embodiments, the intermediate processing device includes a controller and a power supply. In some embodiments, the sensor of the intermediate processing device is configured to sense additional data, which can be shared with the head wearable device in an electronic format (analog or digital).

[0199] The controller of the intermediate processing device processes information generated by sensors on the intermediate processing device and / or on the head wearable device. The intermediate processing device (e.g., HIPD 900) can process information generated by one or more of its multiple sensors and / or information provided by other communicatively coupled devices. For example, the head wearable device may include an IMU, and the intermediate processing device (neckband and / or HIPD 900) can calculate all inertial calculations and spatial calculations based on the IMU located on the head wearable device. The following reference Fig.9A and Fig. 9B Additional examples of processing performed by a communicatively coupled device (eg, HIPD 900 ) are provided.

[0200] The AR system may include various types of visual feedback mechanisms. For example, the display device in the AR device 800 and / or the VR device 810 may include one or more liquid crystal displays (LCD), one or more light emitting diode (LED) displays, one or more organic LED (OLED) displays, and / or any other suitable type of display screen. The AR system may include a single display screen for both eyes or may provide a display screen for each eye, which may allow additional flexibility for zoom adjustment or for correcting refractive errors associated with the user's vision. Some AR systems also include an optical subsystem having one or more lenses (e.g., conventional concave or convex lenses, Fresnel lenses, or adjustable liquid lenses) through which the user can view the display screen. In addition to or instead of using a display screen, some AR systems include one or more projection systems. For example, the display device in the AR device 800 and / or the VR device 810 may include a micro LED projector (e.g., using a waveguide) that projects light to the display device, such as a transparent combination lens that allows ambient light to penetrate. The display device may refract the projected light to the user's pupil, and may enable the user to view both the AR content and the real world simultaneously. The AR system may also be configured with any other suitable type or form of image projection system. As noted, some AR systems may not mix artificial reality with real reality, but rather substantially replace one or more of the user's multiple sensory perceptions of the real world with a virtual experience.

[0201] Although multiple example head wearable devices are described herein as AR device 800 and VR device 810, respectively, any or both of the multiple example head wearable devices described herein can be configured to present a fully immersive VR scene presented in substantially all of the user's field of view, in addition to or in lieu of a more subtle augmented reality scene presented within a portion of the user's field of view (less than the full field of view).

[0202] In some embodiments, the AR device 800 and / or the VR device 810 may include a tactile feedback system. The tactile feedback system can provide various types of skin feedback, including vibration, force, traction, shear stress, texture and / or temperature. The tactile feedback system can also provide various types of kinesthetic feedback, such as motion and compliance. Tactile feedback can be implemented using motors, piezoelectric actuators, fluid systems and / or various other types of feedback mechanisms. The tactile feedback system can be implemented independently of other AR devices, implemented in other AR devices, and / or in combination with other AR devices (e.g., wrist wearable devices that can be incorporated into headwear, gloves, jumpsuits; handheld controllers; environmental devices (e.g., chairs or carpets); and / or any other type of device or system, such as wrist wearable devices 700, HIPD 900, smart textile-based clothing) and / or other devices described herein.

[0203] Figure 8C A computing system 820 and an optional housing 890 are shown, each of which shows components that may be included in a head wearable device (e.g., AR device 800 and / or VR device 810). In some embodiments, more components or fewer components may be included in the optional housing 890, depending on the practical constraints of the respective head wearable device being described. Additionally or alternatively, the optional housing 890 may include additional components to expand and / or enhance the functionality of the head wearable device.

[0204] In some embodiments, computing system 820 and / or optional housing 890 may include one or more peripheral interfaces 822A and 822B, one or more power systems 842A and 842B (including charger input 843, PMIC 844, and battery 845), one or more controllers 846A and 846B (including one or more haptic controllers 847), one or more processors 848A and 848B (as defined above, including any of the multiple examples provided), and memory 850A and 850B, all of which may be in electronic communication with each other. For example, one or more processors 848A and / or 848B may be configured to execute instructions stored in memory 850A and / or 850B, which may cause one of the one or more controllers 846A and / or 846B to cause multiple operations to be performed at one or more peripheral devices of peripheral interfaces 822A and / or 822B. In some embodiments, each of the operations described may be performed based on power provided by power systems 842A and / or 842B.

[0205] In some embodiments, the peripheral interface 822A may include one or more devices configured as part of the computing system 820, many of which have been defined and / or referenced above. Fig. 7A and Figure 7B 823A. For example, the peripheral interface may include one or more sensors 823A. Some example sensors include one or more coupling sensors 824, one or more acoustic sensors 825, one or more imaging sensors 826, one or more EMG sensors 827, one or more capacitive sensors 828, and / or one or more IMU sensors 829. In some embodiments, sensor 823A also includes: a depth sensor 867; an optical sensor 868; and / or any other type of sensor defined above or described with reference to any other embodiment discussed herein.

[0206] In some embodiments, the peripheral interface may include one or more additional peripheral devices, including: one or more NFC devices 830; one or more GPS devices 831; one or more LTE devices 832; one or more Wi-Fi and / or Bluetooth devices 833; one or more buttons 834 (e.g., including slidable or otherwise adjustable buttons); one or more displays 835A; one or more speakers 836A; one or more microphones 837A; one or more cameras 838A (e.g., including a first camera 839-1 to an nth camera 839-n, which are similar to the left camera 839A and / or the right camera 839B); one or more tactile devices 840; and / or any other type of peripheral device defined above or described with reference to any other embodiments discussed herein.

[0207] The head wearable device may include various types of visual feedback mechanisms (e.g., presentation devices). For example, the display device in the AR device 800 and / or the VR device 810 may include one or more liquid crystal displays (LCDs), one or more light emitting diode (LED) displays, one or more organic LED (OLED) displays, one or more micro-LEDs, and / or any other suitable type of display screen. The head wearable device may include a single display screen (e.g., configured to be viewed by both eyes), and / or a separate display screen may be provided for each eye, which may allow for additional flexibility for zoom adjustment and / or for correcting refractive errors associated with the user's vision. Some embodiments of the head wearable device also include an optical subsystem having one or more lenses (e.g., conventional concave or convex lenses, Fresnel lenses, or adjustable liquid lenses) through which the user can view the display screen. For example, a corresponding display 835A may be coupled to each of the lenses 806-1 and 806-2 of the AR device 800. The display 835A coupled to each of the lenses 806-1 and 806-2 may work in conjunction or independently to present an image or a series of images to the user. In some embodiments, the AR device 800 and / or VR device 810 includes a single display 835A (e.g., a near-eye display) or more than two displays 835A.

[0208] In some embodiments, one or more displays 835A of the first group may be used to present an augmented reality environment, and one or more display devices 835A of the second group may be used to present a VR environment. In some embodiments, one or more waveguides are used in conjunction with presenting AR content to a user of an AR device 800 and / or a VR device 810 (e.g., as a means of transmitting light from a display projector assembly and / or one or more displays 835A to the user's eyes). In some embodiments, one or more waveguides are fully or partially integrated into an AR device 800 and / or a VR device 810. In addition to or in lieu of a display screen, some AR systems include one or more projection systems. For example, a display device in an AR device 800 and / or a VR device 810 may include a micro LED projector that projects light to a display device (e.g., using a waveguide), such as a transparent combination lens that allows ambient light to penetrate. The display device may refract the projected light to the user's pupil, and may enable the user to simultaneously view both AR content and the real world. The head wearable device may also be configured with any other suitable type or form of image projection system. In some embodiments, in addition to or in lieu of one or more displays 835A, one or more waveguides are provided.

[0209] In some embodiments of the head wearable device, ambient light and / or a real-world real-time view (e.g., a real-time feed of the surrounding environment that a user would normally see) can be made to penetrate through display elements of the corresponding head wearable device that is presenting aspects of the AR system. In some embodiments, ambient light and / or a real-world real-time view can be made to penetrate a portion (less than the entire AR environment) of the AR environment presented within the user's field of view (e.g., a portion of the AR environment that is co-located with physical objects in the user's real-world environment that are within a specified boundary (e.g., a guard boundary) that is configured to be used by the user when they are interacting with the AR environment). For example, a visual user interface element (e.g., a notification user interface element) can be presented at the head wearable device, and a certain amount of ambient light and / or a real-world real-time view (e.g., 15% to 50% of the ambient light and / or real-world real-time view) can be made to penetrate the user interface element so that the user can distinguish at least a portion of the physical environment on which the user interface element is displayed.

[0210] The head wearable device may include one or more external displays 835A for presenting information to the user. For example, the external display 835A may be used to display the current battery level, network activity (e.g., connected, disconnected), current activity (e.g., playing a game, talking, in a meeting, or watching a movie), and / or other relevant information. In some embodiments, the external display 835A may be used to communicate with other people. For example, a user of the head wearable device may cause the external display 835A to present a "Do Not Disturb" notification. The user may also use the external display 835A to share any information collected by one or more components of the peripheral interface 822A and / or generated by the head wearable device (e.g., during the operation and / or execution of one or more applications).

[0211] The memory 850A may include instructions and / or data executable by one or more processors 848A (and / or processor 848B of housing 890) and / or a storage controller of one or more controllers 846A (and / or controller 846B of housing 890). The memory 850A may include: one or more operating systems 851; one or more applications 852; one or more communication interface modules 853A; one or more graphics modules 854A; one or more AR processing modules 855A; an MMG processing module 856A configured to process one or more input gestures performed by the user 105 detected by one or more sensors 823A at the head wearable device 120; and / or any other type of module or component as defined above or described with reference to any other embodiment discussed herein.

[0212] The data 860 stored in the memory 850A may be used in conjunction with one or more of the applications and / or programs discussed above. The data 860 may include: profile data 861; sensor data 862; media content data 863; AR application data 864; MMG data 865 representing one or more input gestures performed by the user 105 detected at the head wearable device 120 and / or another communicatively coupled device; and / or any other type of data defined above or described with reference to any other embodiment discussed herein.

[0213] In some embodiments, the controller 846A of the head wearable device processes information generated by the sensor 823A on the head wearable device, and / or another component of the head wearable device, and / or another component communicatively coupled to the head wearable device (e.g., a component of the housing 890, such as a component of the peripheral interface 822B). For example, the controller 846A can process information from the acoustic sensor 825 and / or the image sensor 826. For each detected sound, the controller 846A can perform a direction of arrival (DOA) estimation to estimate the direction from which the detected sound arrived at the head wearable device. When one or more of the plurality of acoustic sensors 825 detects a sound, the controller 846A can populate an audio data set with information (e.g., represented by the sensor data 862).

[0214] In some embodiments, a physical electronic connector can transmit information between a head wearable device and another electronic device, and / or transmit information between one or more processors 848A and a controller 846A of a head wearable device. The information can be in the form of: optical data form; electrical data form; wireless data form; or any other transmittable data form. Moving the processing of information generated by the head wearable device to an intermediate processing device can reduce the weight and heat of the eyewear device, making it more comfortable and safer for the user. In some embodiments, an optional accessory device (e.g., an electronic neckband or HIPD 900) is coupled to the head wearable device via one or more connectors. Each connector can be a wired connector or a wireless connector, and can include electronic components and / or non-electronic (e.g., structural) components. In some embodiments, the head wearable device and the accessory device can operate independently without any wired or wireless connection between them.

[0215] The head wearable device may include various types of computer vision components and subsystems. For example, the AR device 800 and / or the VR device 810 may include one or more optical sensors, such as a two-dimensional (2D) camera or a three-dimensional (3D) camera, a ToF depth sensor, a single-beam rangefinder or a scanning laser rangefinder, a 3D laser radar (LiDAR) sensor, and / or any other suitable type or form of optical sensor. The head wearable device may process data from one or more of these sensors to identify the user's position and / or multiple aspects of the user's real-world physical environment (including the position of real-world objects in the real-world physical environment). In some embodiments, the methods described herein are used to map the real world to provide the user with context about the real-world environment, and / or to generate interactive virtual objects (which may be copies or digital twins of real-world objects that can interact with the AR environment) and other various functions. For example, Figure 8B-1 and Figure 8B-2 A VR device 810 is shown having cameras 839A to 839D that can be used to provide depth information for creating a voxel field and a 2D mesh to provide object information to the user for collision avoidance.

[0216] The optional housing 890 may include components similar to those described above with reference to the computing system 820. For example, the optional housing 890 may include a corresponding peripheral interface 822B, which includes more components or fewer components than those described above with reference to the peripheral interface 822A. As described above, the components of the optional housing 890 can be used to enhance and / or expand the functionality of the head wearable device. For example, the optional housing 890 may include corresponding sensors 823B, speakers 836B, displays 835B, microphones 837B, cameras 838B, and / or other components for collecting and / or presenting data. Similarly, the optional housing 890 may include one or more processors 848B, controllers 846B, and / or memory 850B (including corresponding communication interface modules 853B; one or more graphics modules 854B; one or more AR processing modules 855B; MMG processing modules 856A), which may be used alone and / or in combination with components of the computing system 820.

[0217] Above FIG. 8A to FIG. 8CThe techniques described in the can be used with different head wearable devices. In some embodiments, the head wearable device (e.g., AR device 800 and / or VR device 810) can be used in conjunction with one or more wearable devices, such as wrist wearable device 700 (or its components). Having described the example head wearable device in this manner, we now turn our attention to an example handheld intermediate processing device, such as HIPD 900.

[0218] Sample Handheld Intermediate Processing Device

[0219] Fig.9A and Fig. 9B An example handheld intermediate processing device (HIPD) 900 is shown in accordance with some embodiments. FIG. 1A to FIG. 1T An instance of HIPD 900 may be included that receives and processes one or more input gestures performed by user 105, such that HIPD 900 should be understood to have the features described with reference to any intermediate device defined above or otherwise described herein, and vice versa. HIPD 900 may perform various functions and / or operations associated with navigating in a user interface and selectively launching applications, as well as the features described with reference to any intermediate device defined above. Figures 1A to 3C The functions and / or operations described. For example, Figure 1E , user 105 performs a directional key sliding gesture detected by one or more sensors coupled to wrist wearable device 110. HIPD 900 can receive input gesture data, process the input gesture data using MMG processing module 987, and cause an operation to be performed at head wearable device 120 or wrist wearable device 110 based on determining that the input gesture data corresponds to an operation.

[0220] Fig.9A A top view 905 and a side view 925 of the HIPD 900 are shown. The HIPD 900 is configured to be communicatively coupled to one or more wearable devices (or other electronic devices) associated with the user. For example, the HIPD 900 is configured to be communicatively coupled to the user's wrist wearable device 700 (or its components, such as the watch body 720 and the wearable band 710), the AR device 800, and / or the VR device 810. The HIPD 900 can be configured to be held by the user (e.g., as a handheld controller), carried by the user (e.g., in his pocket or in his bag), placed near the user (e.g., on his desk when the user is sitting at a desk, or placed on a charging base), and / or placed at or within a predetermined distance from the wearable device or other electronic device (e.g., where, in some embodiments, the predetermined distance is the maximum distance (e.g., 10 meters) at which the HIPD 900 can successfully communicate with the electronic device (e.g., wearable device)).

[0221] HIPD 900 can perform various functions independently and / or in combination with one or more wearable devices (e.g., wrist wearable device 700, AR device 800, and / or VR device 810). HIPD 900 is configured to increase and / or improve the functionality of a communicatively coupled device (e.g., a wearable device). HIPD 900 is configured to perform one or more functions or operations associated with: interacting with a user interface and application of a communicatively coupled device, interacting with an AR environment, interacting with a VR environment, and / or acting as a human-machine interface controller, as well as the above references. Figures 1A to 3C The functions and / or operations described. In addition, as will be described in more detail below, the functions and / or operations of the HIPD 900 may include, but are not limited to: task transfer and / or transmission; heat transfer and / or transmission; 6 degrees of freedom (6DoF) ray casting and / or gaming (e.g., using imaging devices or cameras 914A and 914B, which can be used for simultaneous localization and mapping (SLAM) and / or used with other image processing techniques); portable charging; messaging; image acquisition via one or more imaging devices or cameras (e.g., cameras 922A and 922B); sensing user input (e.g., sensing touch on the multi-touch input surface 902); wireless communication and / or interconnection (e.g., cellular, near field, Wi-Fi, or personal area network); location determination; financial transactions; providing tactile feedback; alarms; notifications; biometric authentication; health monitoring; sleep monitoring. The above example functions may be performed independently in the HIPD 900 and / or in communication between the HIPD 900 and another wearable device described herein. In some embodiments, the functions may be performed in conjunction with an AR environment on the HIPD 900. As will be appreciated by those skilled in the art upon reading the various descriptions provided herein, the novel HIPD 900 described herein may be used with any type of suitable AR environment.

[0222] When the HIPD 900 is communicatively coupled with a wearable device and / or other electronic device, the HIPD 900 is configured to perform one or more operations initiated at the wearable device and / or the other electronic device. In particular, one or more operations of the wearable device and / or other electronic device can be transferred to the HIPD 900 for execution. The HIPD 900 performs one or more operations of the wearable device and / or other electronic device, and provides data corresponding to the completed operations to the wearable device and / or the other electronic device. For example, a user can initiate a video stream using an AR device 800, and a back-end task (e.g., video rendering) associated with executing the video stream can be transferred to the HIPD 900, which performs the back-end task and provides corresponding data to the AR device 800 to perform the remaining front-end tasks associated with the video stream (e.g., presenting the rendered video data via the display of the AR device 800). In this way, the HIPD 900, which has more computing resources and greater thermal margin than a wearable device, can perform computationally intensive tasks for the wearable device, thereby improving the performance of the operations performed by the wearable device.

[0223] HIPD 900 includes a multi-touch input surface 902 on a first side (e.g., a front surface), which is configured to detect one or more user inputs. In particular, the multi-touch input surface 902 can detect single-click input, multi-tap input, sliding gestures and / or inputs, force-based and / or pressure-based touch inputs and press-and-hold clicks, etc. The multi-touch input surface 902 is configured to detect capacitive touch inputs and / or force (and / or pressure) touch inputs. The multi-touch input surface 902 includes a first touch input surface 904 defined by a surface depression and a second touch input surface 906 defined by a substantially flat portion. The first touch input surface 904 can be arranged adjacent to the second touch input surface 906. In some embodiments, the first touch input surface 904 and the second touch input surface 906 can be different sizes, shapes, and / or can cover different parts of the multi-touch input surface 902. For example, the first touch input surface 904 can be substantially circular, and the second touch input surface 906 is substantially rectangular. In some embodiments, the surface depression of the multi-touch input surface 902 is configured to guide the user's manipulation of the HIPD 900. In particular, the surface depression is configured to enable the user to hold the HIPD 900 upright when holding the HIPD 900 in one hand (e.g., such that the imaging device or cameras 914A and 914B used are pointed toward the ceiling or the sky). In addition, the surface depression is configured to enable the user's thumb to be located within the first touch input surface 904.

[0224] In some embodiments, different touch input surfaces include multiple touch input areas. For example, the second touch input surface 906 (also referred to as the second touch input area 906 in this embodiment) includes at least a first touch input area 908 in the second touch input area 906, and a third touch input area 910 in the first touch input area 908. In some embodiments, one or more touch input areas in the multiple touch input areas are optional and / or user-defined (for example, the user can specify the touch input area based on his or her preferences). In some embodiments, each touch input surface and / or touch input area is associated with a predetermined set of commands. For example, the user input detected in the first touch input area 908 causes the HIPD 900 to execute a first command, and the user input detected in the second touch input area 906 causes the HIPD 900 to execute a second command different from the first command. In some embodiments, different touch input surfaces and / or touch input areas are configured to detect one or more types of user input. Different touch input surfaces and / or touch input areas can be configured to detect the same type or different types of user input. For example, the first touch input region 908 may be configured to detect force touch input (eg, the magnitude of a user's press) and capacitive touch input, and the second touch input region 906 may be configured to detect capacitive touch input.

[0225] HIPD 900 includes one or more sensors 951 (e.g. Fig. 9B As shown in FIG. 1 ), the one or more sensors 951 are used to sense data used when performing one or more operations and / or functions. For example, the HIPD 900 may include an IMU that is used in conjunction with the cameras 914A and 914B to perform three-dimensional object manipulation in an AR environment or a VR environment (e.g., enlarging, moving, destroying, etc. an object). Non-limiting examples of the sensors 951 included in the HIPD 900 include light sensors, magnetometers, depth sensors, pressure sensors, and force sensors. Fig. 9B Additional examples of sensor 951 are provided.

[0226] HIPD 900 may include one or more light indicators 912 to provide one or more notifications to the user. In some embodiments, the light indicator is an LED or other type of lighting device. The light indicator 912 can act as a privacy light to notify the user and / or other people near the user that the imaging device and / or microphone are active. In some embodiments, the light indicator is positioned near one or more touch input surfaces. For example, the light indicator can be positioned around the first touch input surface 904. The light indicator can be illuminated in different colors and / or patterns to provide the user with one or more notifications and / or information about the device. For example, the light indicator positioned around the first touch input surface 904 can flash when the user receives a notification (e.g., a message), turn red when the HIPD 900 loses power, act as a progress bar (e.g., a halo that closes when the task is completed (e.g., 0% to 100%)), act as a volume indicator, etc.

[0227] In some embodiments, HIPD 900 includes one or more additional sensors on another surface. Fig.9A As shown, the HIPD 900 includes a set of one or more sensors (e.g., sensor group 920) on the edge of the HIPD 900. When the sensor group 920 is positioned on the edge of the HIPD 900, it can be positioned at a predetermined tilt angle (e.g., 26 degrees), which allows the sensor group 920 to be tilted toward the user when placed on a table or other flat surface. Alternatively, in some embodiments, the sensor group 920 is positioned on a surface (e.g., the back) opposite to the multi-touch input surface 902. One or more sensors in the sensor group 920 are discussed in detail below.

[0228] The side view 925 of HIPD 900 shows sensor group 920 and camera 914B. Sensor group 920 includes one or more cameras 922A and 922B, a depth projector 924, an ambient light sensor 928, and a depth receiver 930. In some embodiments, sensor group 920 includes a light indicator 926. Light indicator 926 can act as a privacy indicator to let the user and / or people around the user know that the camera and / or microphone are active. Sensor group 920 is configured to collect the user's facial expressions so that the user can manipulate a customized avatar (e.g., display emotions (e.g., smile, laugh, etc.) on the user's avatar or digital representation). Sensor group 920 can be configured as a stereo red-green-blue (RGB) system on the side, an indirect time-of-flight (iToF) system at the rear, or a stereo RGB system at the rear. As will be appreciated by those skilled in the art upon reading the description provided herein, the novel HIPD 900 described herein may use different sensor group 920 configurations and / or sensor group 920 arrangements.

[0229] In some embodiments, the HIPD 900 includes one or more haptic devices 971 ( Fig. 9B ; for example, a vibrotactile actuator), the one or more tactile devices being configured to provide tactile feedback (e.g., kinesthetic perception). Sensor 951 and / or tactile device 971 may be configured to operate in conjunction with a plurality of applications and / or communicatively coupled devices, including but not limited to wearable devices, health monitoring applications, social media applications, gaming applications, and AR applications (e.g., applications associated with AR).

[0230] HIPD 900 is configured to operate without a display. However, in an alternative embodiment, HIPD 900 may include a display 968 ( Fig. 9B ). The HIPD 900 may also be provided with one or more optional peripheral buttons 967 ( Fig. 9B). For example, the peripheral button 967 can be used to turn the HIPD 900 on or off. In addition, the housing of the HIPD 900 can be formed of a polymer and / or an elastomer. The HIPD 900 can be configured to have an anti-slip surface to allow the HIPD 900 to be placed on a surface without the user having to look after the HIPD 900. In other words, the HIPD 900 is designed so that it does not easily slide off a surface. In some embodiments, the HIPD 900 includes one or more magnets that are used to couple the HIPD 900 to another surface. This allows the user to place the HIPD 900 on different surfaces and provides the user with greater flexibility in using the HIPD 900.

[0231] As described above, the HIPD 900 may distribute and / or provide instructions for performing one or more tasks at the HIPD 900 and / or the communicatively coupled devices. For example, the HIPD 900 may identify one or more back-end tasks to be performed by the HIPD 900 and one or more front-end tasks to be performed by the communicatively coupled devices. Although the HIPD 900 is configured to transfer and / or transmit tasks of the communicatively coupled devices, the HIPD 900 may (e.g., via one or more processors, such as the CPU 977 ( Fig. 9B )) performs both back-end tasks and front-end tasks. The HIPD 900 may be used, but is not limited to, to perform: enhanced calls (e.g., receiving and / or sending 3D or 2.5D real-time volumetric calls, real-time digital human representation calls and / or avatar calls), unobtrusive messaging, 6DoF portrait / landscape games, AR / VR object manipulation, AR / VR content display (e.g., presenting content via a virtual display), and / or other AR / VR interactions. The HIPD 900 may perform the above operations alone or in combination with a wearable device (or other communicatively coupled electronic device).

[0232] Fig. 9B A block diagram of a computing system 940 of a HIPD 900 according to some embodiments is shown. The HIPD 900 described in detail above may include one or more components shown in the HIPD computing system 940. The HIPD 900 will be understood to include the components shown and described below for the HIPD computing system 940. In some embodiments, all or most of the multiple components of the HIPD computing system 940 are included in a single integrated circuit. Alternatively, in some embodiments, the multiple components of the HIPD computing system 940 are included in multiple integrated circuits that are communicatively coupled.

[0233] The HIPD computing system 940 may include: a processor (e.g., a CPU 977, a GPU, and / or a CPU with integrated graphics); a controller 975; a peripheral interface 950, which includes one or more sensors 951 and other peripheral devices; a power source (e.g., a power system 995); and a memory (e.g., a memory 978), which includes an operating system (e.g., an operating system 979), data (e.g., data 988), one or more applications (e.g., application 980), and one or more modules (e.g., a communication interface module 981, a graphics module 982, a task and processing management module 983, an interoperability module 984, an AR processing module 985, a data management module 986, an MMG processing module 987, etc.). The MMG processing module 987 is configured to process one or more input gestures detected by a communication-coupled device (e.g., a head wearable device 120 and / or a wrist wearable device 110). After the input gesture is processed at the MMG processing module 987, and based on determining that the input gesture corresponds to one or more operations, one or more operations are performed at the head wearable device 120 or the wrist wearable device 110. The HIPD computing system 940 also includes a power system 995, which includes a charger input and output 996, a PMIC 997, and a battery 998, all of which are defined above.

[0234] In some embodiments, the peripheral interface 950 may include one or more sensors 951. The sensor 951 may include the same Figure 7B Sensors similar to those described above. For example, sensors 951 may include imaging sensors 954, (optional) EMG sensors 956, IMU sensors 958, and capacitive sensors 960. In some embodiments, sensors 951 may include: one or more pressure sensors 952 for sensing pressure data; an altimeter 953 for sensing the height of the HIPD 900; a magnetometer 955 for sensing magnetic fields; a depth sensor 957 (or time of flight sensor) for determining the difference between the camera and an object in the image; a position sensor 959 (e.g., a flexible position sensor) for sensing relative displacement or position change of a portion of the HIPD 900; a force sensor 961 for sensing a force applied to a portion of the HIPD 900; and a light sensor 962 (e.g., an ambient light sensor) for detecting the amount of light emitted. Sensors 951 may include Fig. 9B One or more sensors not shown.

[0235] With the above reference Figure 7BSimilar to the peripheral devices described above, the peripheral interface 950 may also include an NFC component 963, a GPS component 964, an LTE component 965, a Wi-Fi and / or Bluetooth communication component 966, a speaker 969, a haptic device 971, and a microphone 973. Fig.9A As described, the HIPD 900 may optionally include a display 968 and / or one or more buttons 967. The peripheral interface 950 may also include one or more cameras 970, a touch surface 972, and / or one or more light emitters 974. Fig.9A The multi-touch input surface 902 depicted is an example of a touch surface 972. The light emitter 974 may be one or more LEDs, lasers, etc., and may be used to project or present information to a user. For example, the light emitter 974 may include the above referenced Fig.9A The light indicators 912 and 926 described above. The camera 970 (e.g., Fig.9A The cameras 914A, 914B, 922A and 922B described in the figure may include one or more wide-angle cameras, fisheye cameras, spherical cameras, compound eye cameras (e.g., stereo and multi-cameras), depth cameras, RGB cameras, ToF cameras, RGB-D cameras (depth and ToF cameras) and / or other available cameras. Camera 970 can be used for: SLAM; 6DoF ray casting, games, object manipulation and / or other rendering; facial recognition and facial expression recognition, etc.

[0236] With the above reference Figure 7B Similar to the described body computing system 760 and wearable band computing system 730 , the HIPD computing system 940 may include one or more haptic controllers 976 and associated components (e.g., haptic device 971 ) for providing haptic events at the HIPD 900 .

[0237] The memory 978 may include high-speed random access memory and / or non-volatile memory, such as one or more disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The access of other components of the HIPD 900 (such as one or more processors and the peripheral interface 950) to the memory 978 may be controlled by a storage controller in the controller 975.

[0238] In some embodiments, the software components stored in the memory 978 include one or more operating systems 979, one or more applications 980, one or more communication interface modules 981, one or more graphics modules 982, and one or more data management modules 986. Figure 7BThe software components stored in the memory 978 may also include an MMG processing module 987, which is configured to execute the above reference Figures 1A to 3C The MMG processing module 987 includes the functions described above. Figure 7B and Figure 8C The features described in the MMG processing module.

[0239] In some embodiments, the software components stored in the memory 978 include a task and process management module 983, which is used to: identify one or more front-end tasks and back-end tasks associated with the operation performed by the user; execute one or more front-end tasks and / or back-end tasks; and / or provide instructions to one or more communication-coupled devices to perform the one or more front-end tasks and / or back-end tasks. In some embodiments, the task and process management module 983 uses data 988 (e.g., device data 990) to distribute one or more front-end tasks and / or back-end tasks based on the following items: computing resources, available power, thermal headroom, ongoing operations, and / or other factors of the communication-coupled devices. For example, the task and process management module 983 can cause one or more back-end tasks (of the operation performed at the communication-coupled AR device 800) to be performed at the HIPD 900 based on determining that the operation is utilizing a predetermined amount (e.g., at least 70%) of computing resources available at the AR device 800.

[0240] In some embodiments, the software components stored in the memory 978 include an interoperability module 984 for exchanging and utilizing information received and / or provided to different communicatively coupled devices. The interoperability module 984 allows different systems, devices, and / or applications to connect and communicate in a coordinated manner without user input. In some embodiments, the software components stored in the memory 978 include an AR processing module 985, which is configured to process signals based at least on sensor data for use in an AR and / or VR environment. For example, the AR processing module 985 can be used for 3D object manipulation, gesture recognition, facial recognition, and facial expression recognition, etc.

[0241] The memory 978 may also include data 988, which includes structured data. In some embodiments, the data 988 may include: profile data 989; device data 990 (including device data of one or more devices coupled to the HIPD 900, such as device type, hardware, software, configuration, etc.); sensor data 991; media content data 992; application data 993; and MMG data 994, which stores the same information as the above referenced data. Figures 1A to 3CData related to the execution of the functions described.

[0242] It should be appreciated that HIPD computing system 940 is an example of a computing system within HIPD 900, and that HIPD 900 may have more components or fewer components, may combine two or more components, and / or may have a different configuration and / or arrangement of components than those shown in HIPD computing system 940. The various components shown in HIPD computing system 940 are implemented in hardware, software, firmware, or a combination thereof, including one or more signal processing circuits and / or application specific integrated circuits.

[0243] Above Fig.9A and Fig. 9B The various techniques described in can be used with any device used as a human-machine interface controller. In some embodiments, the HIPD 900 can be used in conjunction with one or more wearable devices, such as head wearable devices (e.g., AR devices 800 and VR devices 810) and / or wrist wearable devices 700 (or parts thereof). Any data collection performed by the various devices described herein and / or any device configured to perform or enable the execution of the different embodiments described above with reference to any of the accompanying drawings (hereinafter referred to as "devices") is performed with the user's consent and in a manner that complies with all applicable privacy laws. The user is provided with options for allowing the device to collect data and options for limiting or denying the device from collecting data. The user is able to choose to enable or disable any data collection at any time. In addition, the user is provided with an option to request deletion of any collected data.

[0244] It will be understood that, although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another.

[0245] The terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the claims. As used in the description of each embodiment and the appended claims, unless the context clearly indicates otherwise, the singular forms "one", "an" and "the" are intended to also include plural forms. It will also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "include" and / or "comprise" when used in this specification specify the presence of stated features, unity, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, unity, steps, operations, elements, parts and / or their combinations.

[0246] As used herein, the term “if” may be interpreted to mean, depending on the context: “when the stated conditional precedent is true”; or “upon” the stated conditional precedent is true; or “in response to determining” the stated conditional precedent is true; or “upon determining” the stated conditional precedent is true; or “in response to detecting” the stated conditional precedent is true. Similarly, depending on the context, the phrase “if it is determined that [the stated conditional precedent is true]” or “if [the stated conditional precedent is true]” or “when [the stated conditional precedent is true]” may be interpreted to mean: “upon determining” the stated conditional precedent is true; or “upon determining” the stated conditional precedent is true; or “upon detecting” the stated conditional precedent is true; or “in response to detecting” the stated conditional precedent is true.

[0247] For purposes of explanation, the foregoing description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the claims to the precise forms disclosed. In view of the above teachings, many modifications and variations are possible. The various embodiments are selected and described in order to best explain the principles of operation and practical application, thereby enabling others skilled in the art to realize.

Claims

1. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors, cause the one or more processors to: While the user is wearing an artificial reality headset: At a first point in time, and while a multimodal gesture is available via at least two input devices associated with the artificial reality head mounted device: In response to detecting a first input via a first input device of the at least two input devices, performing a first action at the artificial reality head mounted device; In response to detecting a second input via a second input device of the at least two input devices, performing a second action within a user interface presented via the artificial reality head mounted device; as well as At a second point in time after the first point in time, and when a multimodal gesture is no longer available via the at least two input devices associated with the artificial reality head mounted device: In response to detecting the first input via the first input device of the at least two input devices, performing the second action within the user interface presented via the artificial reality head mounted device.

2. The non-transitory computer-readable storage medium of claim 1, wherein: The first input is one of a first input level group, the first input level group comprising a first input level and a second input level, The first input level of the first input level group includes one or more inputs associated with navigation control commands, and The second input level of the first input level group includes one or more inputs associated with system control commands for the artificial reality head mounted device.

3. The non-transitory computer-readable storage medium of claim 2, wherein: The first input level of the first input level group is associated with input performed with a single finger of the user's hand, and The second input level of the first input level group is associated with input performed with at least two fingers of the user's hand.

4. The non-transitory computer-readable storage medium of claim 2, wherein: The first input level group includes a third input level, the third input level of the first input level group including one or more inputs associated with media control commands.

5. The non-transitory computer-readable storage medium of claim 4, wherein: The third input level of the first input level group is associated with a swipe gesture performed with at least two fingers of the user's hand.

6. The non-transitory computer-readable storage medium of claim 1, wherein: The second action includes a navigation control command and / or a system control command for the artificial reality head mounted device.

7. The non-transitory computer-readable storage medium of claim 1, wherein: The second input is one of a second set of input levels, the second set of input levels comprising a first input level and a second input level, The first input level of the second input level group includes one or more inputs associated with navigation control commands, and The second input level of the second input level group includes one or more inputs associated with system control commands for the artificial reality head mounted device.

8. The non-transitory computer-readable storage medium of claim 7, wherein: The first input level of the second input level group is associated with a gesture performed using a first finger and a second finger of the user's hand; and The second input level of the second input level group is associated with a gesture performed using the first and third fingers of the user.

9. The non-transitory computer-readable storage medium of claim 8, wherein: The second input level group includes a third input level, the third input level of the second input level group including one or more inputs associated with media control commands.

10. The non-transitory computer-readable storage medium of claim 9, wherein: The third input level of the second input level group is associated with a gesture performed using a first phalanx and a fourth phalanx of the user.

11. The non-transitory computer-readable storage medium of claim 1, wherein: causing the artificial reality head mounted device to execute commands specific to the artificial reality head mounted device includes launching a virtual assistant, and the non-transitory computer readable storage medium further includes instructions for causing the one or more processors to: activating a microphone of the artificial reality head mounted device and / or the second input device; obtaining audio data via the microphone; as well as In response to detecting one or more user input commands from the audio data, causing the second input device and / or the artificial reality head mounted device to execute the one or more user input commands.

12. A method comprising: While the user is wearing an artificial reality headset: At a first point in time, and while a multimodal gesture is available via at least two input devices associated with the artificial reality head mounted device: In response to detecting a first input via a first input device of the at least two input devices, performing a first action at the artificial reality head mounted device; In response to detecting a second input via a second input device of the at least two input devices, performing a second action within a user interface presented via the artificial reality head mounted device; as well as At a second point in time after the first point in time, and when a multimodal gesture is no longer available via the at least two input devices associated with the artificial reality head mounted device: In response to detecting the first input via the first input device of the at least two input devices, performing the second action within the user interface presented via the artificial reality head mounted device.

13. The method according to claim 12, wherein: The first input is one of a first input level group, the first input level group comprising a first input level and a second input level, The first input level of the first input level group includes one or more inputs associated with navigation control commands, and The second input level of the first input level group includes one or more inputs associated with system control commands for the artificial reality head mounted device.

14. The method according to claim 13, wherein: The first input level of the first input level group is associated with input performed with a single finger of the user's hand, and The second input level of the first input level group is associated with input performed with at least two fingers of the user's hand.

15. The method according to claim 13, wherein: The first input level group includes a third input level, the third input level of the first input level group including one or more inputs associated with media control commands.

16. The method according to claim 15, wherein: The third input level of the first input level group is associated with a swipe gesture performed with at least two fingers of the user's hand.

17. An artificial reality head mounted device comprising: one or more processors; as well as a memory comprising instructions that, when executed by the one or more processors, cause the one or more processors to: When the user wears the artificial reality head mounted device: At a first point in time, and while a multimodal gesture is available via at least two input devices associated with the artificial reality head mounted device: In response to detecting a first input via a first input device of the at least two input devices, performing a first action at the artificial reality head mounted device; In response to detecting a second input via a second input device of the at least two input devices, performing a second action within a user interface presented via the artificial reality head mounted device; as well as At a second point in time after the first point in time, and when a multimodal gesture is no longer available via the at least two input devices associated with the artificial reality head mounted device: In response to detecting the first input via the first input device of the at least two input devices, performing the second action within the user interface presented via the artificial reality head mounted device.

18. The artificial reality head mounted device of claim 17, wherein: The first input is one of a first input level group, the first input level group comprising a first input level and a second input level, The first input level of the first input level group includes one or more inputs associated with navigation control commands, and The second input level of the first input level group includes one or more inputs associated with system control commands for the artificial reality head mounted device.

19. The artificial reality head mounted device of claim 18, wherein: The first input level of the first input level group is associated with input performed with a single finger of the user's hand, and The second input level of the first input level group is associated with input performed with at least two fingers of the user's hand.

20. The artificial reality head mounted device of claim 18, wherein: The first input level group includes a third input level, the third input level of the first input level group including one or more inputs associated with media control commands.

Images