Wrist wearable controller for head-mounted device (HMD) or other user device with camera input

Abstract

New tools and techniques are provided for implementing a wrist wearable controller for a head-mounted device (HMD) or other user device with camera input (collectively referred to as a "user device"). In various embodiments, a computing system may analyze first sensor data to identify a gesture corresponding to movement of a user's finger and analyze second sensor data to detect movement of the wrist wearable control device relative to one or more axes of rotation, the first sensor data and the second sensor data are received from a first sensor and a second sensor disposed on the wrist wearable control device, respectively. If a combination of the detected motion and the gesture corresponds to a gesture-based command to control a user interface (UI) of the user device, the computing system may generate an instruction based on the gesture-based command and send the instruction to the user device, the instruction serving as an input to control the user device UI.

Description

Copyright Notice

[0001] Portions of the disclosure of this patent document contain material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. Technical Field

[0002] The present disclosure generally relates to methods, systems, and apparatus for implementing user interface (UI) controls, and more specifically, to methods, systems, and apparatus for implementing a wrist-based wearable controller for a head-mounted device (HMD) or other user device with camera input. Background Art

[0003] Most traditional controllers for HMDs or other user devices are designed as handheld control devices. However, such traditional handheld control devices always occupy one or both hands of the user for UI interaction. Although such devices may be suitable for virtual reality (VR) games, for other use cases, such as daily use of augmented reality (AR) or mixed reality (MR) devices or outdoor use where it is inconvenient to carry additional handheld controllers, these devices are not suitable and inconvenient for users. In addition, these traditional controllers will not be able to accurately provide gesture recognition when they are out of the camera's field of view or in the presence of visual occlusion.

[0004] Therefore, a more robust and scalable solution is needed to achieve UI control. Summary of the invention

[0005] The technology of the present disclosure generally relates to tools and techniques for implementing user interface (UI) controls, and more specifically, to methods, systems, and apparatus for implementing a wrist-worn wearable controller for a head-mounted device (HMD) or other user device with camera input.

[0006] In one aspect, a wrist-type wearable control device may include one or more first sensors, one or more second sensors, at least one processor, and a non-transitory computer-readable medium. The one or more first sensors are used to detect a gesture of one or more fingers of a user when the wrist-type wearable control device is worn by the user without any fingers of the user contacting the wrist-type wearable control device. The one or more second sensors are used to detect movement of the wrist-type wearable control device relative to at least one of three rotation axes. The non-transitory computer-readable medium is communicatively coupled to the at least one processor. A non-transitory computer-readable medium may store computer software, the computer software including an instruction set, which, when executed by at least one processor, causes the wrist-type wearable control device to: analyze first sensor data to identify a first gesture corresponding to a movement of one or more fingers of a user, wherein the first sensor data is received from one or more first sensors disposed on the wrist-type wearable control device when the wrist-type wearable control device is worn by the user; analyze second sensor data to detect movement of the wrist-type wearable control device relative to at least one of three rotation axes, wherein the second sensor data is received from one or more second sensors disposed on the wrist-type wearable control device when the wrist-type wearable control device is worn by the user; based on determining that a combination of the detected movement of the wrist-type wearable control device and the first gesture corresponds to a command based on the first gesture, generate a first instruction based on the command based on the first gesture, and send the generated first instruction to a head-mounted device (HMD) or other user device with a camera input, wherein the command based on the first gesture is used to control a user interface (UI) of the HMD or other user device, and the first instruction is used as an input for controlling the UI of the HMD or other user device.

[0007] On the other hand, a method may include analyzing first sensor data using a computing system to identify a first gesture corresponding to a movement of one or more fingers of a user, when the wrist wearable control device is worn by the user, the first sensor data is received from one or more first sensors disposed on the wrist wearable control device, the one or more first sensors are used to detect the gesture of the one or more fingers of the user when the wrist wearable control device is worn by the user without any finger of the user contacting the wrist wearable control device. The method may also include: analyzing second sensor data using a computing system to detect the movement of the wrist wearable control device relative to at least one of three rotation axes, when the wrist wearable control device is worn by the user, the second sensor data is received from one or more second sensors disposed on the wrist wearable control device; based on determining that the detected combination of the movement of the wrist wearable control device and the first gesture corresponds to a command based on the first gesture, using the computing system to generate a first instruction based on the command based on the first gesture, and sending the generated first instruction to a head mounted device (HMD) or other user device with a camera input, the command based on the first gesture is used to control the user interface (UI) of the HMD or other user device, and the first instruction is used as an input to control the UI of the HMD or other user device.

[0008] Various modifications and additions may be made to the embodiments discussed without departing from the scope of the present disclosure. For example, although the embodiments described above refer to particular features, the scope of the present disclosure also includes embodiments having different combinations of features and embodiments that do not include all of the above features.

[0009] The details of one or more aspects of the disclosure are set forth in the accompanying drawings and the following description. Other features, objects, and advantages of the techniques described in this disclosure will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The nature and advantages of particular embodiments may be further understood by reference to the remainder of the specification and the drawings, wherein the same reference numerals are used to refer to similar components. In some cases, a sub-label is associated with a reference numeral to denote one of multiple similar components. When a reference numeral is referenced without specifying an existing sub-label, it is intended to refer to all such similar components.

[0011] Figure 1 A schematic diagram of a system for implementing a wrist-type wearable control device according to various embodiments, wherein the wrist-type wearable control device is used to control a user interface (UI) of a head-mounted device (HMD) or other user device with camera input.

[0012] Figures 2A to 2P Schematic diagrams of various non-limiting examples of wrist-worn wearable control devices for controlling a UI of an HMD or other user device with camera input according to various embodiments.

[0013] Figure 3A and Figure 3B Schematic diagram of various non-limiting examples of using a wrist wearable control device to control a UI of an HMD or other user device with camera input according to various embodiments.

[0014] Figure 4 1 is a flow chart of a method for implementing a wrist-type wearable control device for controlling a UI of an HMD or other user device with camera input according to various embodiments.

[0015] Figure 5 is a block diagram of an example of a computer or system hardware architecture according to various embodiments. DETAILED DESCRIPTION

[0016] Overview

[0017] Various embodiments provide tools and techniques for implementing user interface (UI) controls, and more specifically, provide methods, systems, and apparatus for implementing a wrist-worn wearable controller for a head-mounted device (HMD) or other user device with camera input.

[0018] In various embodiments, a wrist-type wearable control device may include one or more first sensors, one or more second sensors, at least one processor, and a non-transitory computer-readable medium. The one or more first sensors are used to detect a gesture of one or more fingers of a user when the wrist-type wearable control device is worn by the user without any fingers of the user contacting the wrist-type wearable control device. The one or more second sensors are used to detect movement of the wrist-type wearable control device relative to at least one of three rotation axes. The non-transitory computer-readable medium is communicatively coupled to the at least one processor. A non-transitory computer-readable medium may store computer software, the computer software including an instruction set, which, when executed by at least one processor, causes the wrist-type wearable control device to: analyze first sensor data to identify a first gesture corresponding to a movement of one or more fingers of a user, wherein the first sensor data is received from one or more first sensors disposed on the wrist-type wearable control device when the wrist-type wearable control device is worn by the user; analyze second sensor data to detect movement of the wrist-type wearable control device relative to at least one of three rotation axes, wherein the second sensor data is received from one or more second sensors disposed on the wrist-type wearable control device when the wrist-type wearable control device is worn by the user; generate a first instruction based on the command based on the first gesture based on determining that the detected combination of the movement of the wrist-type wearable control device and the first gesture corresponds to a command based on the first gesture, and send the generated first instruction to a head-mounted device (HMD) or other user device with a camera input, wherein the command based on the first gesture is used to control a user interface (UI) of the HMD or other user device, and the first instruction is used as an input for controlling the UI of the HMD or other user device.

[0019] In some embodiments, the wrist-type wearable control device may include one of a wristwatch-based wearable control device or a wristband-based wearable control device, each of which includes one or more band portions, and the band portions may be linked to adjacent band portions by a connector in a manner that allows the wrist-type wearable control device to be detachably wrapped around the wrist of the user. In some cases, at least one of the one or more first sensors and at least one of the one or more second sensors may be disposed in each of the one or more band portions. Alternatively, the one or more first sensors and the one or more second sensors may each be disposed in only one of the one or more band portions, wherein the band portion containing the one or more first sensors may be one of the following: the same band portion containing the one or more second sensors, or a band portion different from the band portion containing the one or more second sensors. Alternatively, at least one of the one or more first sensors may be disposed in each of the one or more band portions, and the one or more second sensors may be disposed in only one of the one or more band portions.

[0020] According to some embodiments, the HMD or other user device with camera input may include one of the following: virtual reality (VR) goggles, augmented reality (AR) goggles, mixed reality (MR) goggles, VR-enabled glasses, AR-enabled glasses, MR-enabled glasses, a VR-enabled smartphone installed in a headset, an AR-enabled smartphone installed in a headset, an MR-enabled smartphone installed in a headset, a smart TV (television, TV) with a built-in camera, a smart TV with an external camera, a game console with a built-in camera, a game console with an external camera, a monitor with a built-in camera, a monitor with an external camera, or a projector with an external camera, etc.

[0021] By way of example only, in some cases, the one or more first sensors may each include at least one biosensor, the at least one biosensor may include at least one of one or more photoplethysmography (PPG) sensors, one or more electromyography (EMG) sensors, one or more sound sensors, or one or more motion sensors, each of which is used to detect a biometric feature corresponding to a soft tissue movement in or near the wrist of a user wearing a wrist-type wearable control device. In some cases, the soft tissue may include at least one of one or more muscles, one or more tendons, or one or more blood vessels in or near the wrist of the user. In some cases, the one or more PPG sensors may be used to optically detect a change in blood volume in one or more blood vessels under the skin tissue at or near the wrist of the user corresponding to the movement of one or more fingers of the user. In some cases, the one or more EMG sensors may be used to detect the electrical activity of one or more muscles at or near the wrist of the user corresponding to the movement of one or more fingers of the user. In some cases, the one or more sound sensors may be used to detect sounds in the user's body corresponding to the movement of one or more fingers of the user. In some cases, the one or more motion sensors may be used to detect the movement of one or more muscles, one or more tendons, and one or more blood vessels corresponding to the movement of one or more fingers of the user.

[0022] In some embodiments, the one or more second sensors may each include at least one of: an inertial measurement unit (IMU) sensor or a gyroscope-based sensor.

[0023] According to some embodiments, the wrist-type wearable control device may further include one or more band portions and a plurality of outward-facing lights disposed on the one or more band portions, wherein the plurality of outward-facing lights may be arranged on the one or more band portions in a predetermined pattern and may be used as tracking points for one or more cameras of an HMD or other user device to track the translational movement of the wrist-type wearable control device relative to the HMD or other user device along three axes, wherein the tracking of the translational movement of the wrist-type wearable control device along three axes may be used as an additional input for controlling a UI of the HMD or other user device. In some cases, the plurality of outward-facing lights may include at least one of: a plurality of infrared (IR) light emitting diodes (LEDs) or a plurality of color LEDs.

[0024] In some embodiments, the command based on the first gesture may include at least one of the following: a sliding-based command, a dragging-based command, a tapping command, a double-clicking command, a pointing command, a pinching-based command, a grip-based command, a rotation command, a rolling command, a pitching command, or a yaw command, etc. In some cases, the first instruction may include at least one of the following: a rolling command, a moving command, a selecting command, a confirming command, a highlighting command, a returning command, a cancelling command, a moving cursor command, or a navigating command, etc.

[0025] On the other hand, the computing system may analyze the first sensor data to identify a first gesture corresponding to the movement of one or more fingers of the user, and when the wrist wearable control device is worn by the user, the first sensor data is received from one or more first sensors disposed on the wrist wearable control device. The computing system may analyze the second sensor data to detect the movement of the wrist wearable control device relative to at least one of the three rotation axes, and when the wrist wearable control device is worn by the user, the second sensor data is received from one or more second sensors disposed on the wrist wearable control device. Based on determining that the detected combination of the movement of the wrist wearable control device and the first gesture corresponds to a command based on the first gesture, the computing system may generate a first instruction based on the command based on the first gesture, and may send the generated first instruction to a head mounted device (HMD) or other user device with a camera input, the command based on the first gesture is used to control the user interface (UI) of the HMD or other user device, and the first instruction is used as an input to control the UI of the HMD or other user device.

[0026] In some embodiments, the computing system may include at least one of the following: one or more processors on the wrist-type wearable control device, one or more processors on a mobile device communicatively coupled to the wrist-type wearable control device, a machine learning system, an artificial intelligence (AI) system, a deep learning system, a neural network, a convolutional neural network (CNN), or a fully convolutional network (FCN), etc. In some cases, the wrist-type wearable control device may include one of a wristwatch-based wearable control device or a wristband-based wearable control device, each of which includes one or more band portions, and the band portions may be linked to adjacent band portions by a connector in a manner that allows the wrist-type wearable control device to be detachably wrapped around the user's wrist.

[0027] In some cases, the one or more first sensors may each include at least one biosensor, and the at least one biosensor may include at least one of one or more photoplethysmography (PPG) sensors, one or more electromyography (EMG) sensors, one or more sound sensors, or one or more motion sensors, each of which is used to detect a biometric feature corresponding to a soft tissue movement in or near the wrist of a user wearing a wrist-type wearable control device. In some cases, the soft tissue may include at least one of one or more muscles, one or more tendons, or one or more blood vessels in or near the wrist of the user. In some cases, the one or more PPG sensors may be used to optically detect a change in blood volume in one or more blood vessels under the skin tissue at or near the wrist of the user corresponding to the movement of one or more fingers of the user. In some cases, the one or more EMG sensors may be used to detect the electrical activity of one or more muscles at or near the wrist of the user corresponding to the movement of one or more fingers of the user. In some cases, the one or more sound sensors may be used to detect sounds in the user's body corresponding to the movement of one or more fingers of the user. In some cases, the one or more motion sensors may be used to detect the movement of one or more muscles, one or more tendons, and one or more blood vessels corresponding to the movement of one or more fingers of the user.

[0028] In some embodiments, the one or more second sensors may each include at least one of: an inertial measurement unit (IMU) sensor or a gyroscope-based sensor.

[0029] According to some embodiments, the wrist-type wearable control device may further include one or more band portions and a plurality of outward-facing lights disposed on the one or more band portions, wherein the plurality of outward-facing lights may be arranged on the one or more band portions in a predetermined pattern and may be used as tracking points for one or more cameras of an HMD or other user device to track translational movement of the wrist-type wearable control device relative to the HMD or other user device along three axes, wherein tracking of the translational movement of the wrist-type wearable control device along three axes may be used as additional input for controlling a UI of the HMD or other user device. In some cases, the plurality of outward-facing lights may include at least one of: a plurality of infrared (IR) light emitting diodes (LEDs) or a plurality of color LEDs.

[0030] In various aspects described herein, a wrist-type wearable control device for controlling the UI of an HMD or other user device with camera input (collectively referred to as a user device) is provided. High-precision six-degrees-of-freedom (6DOF) control of the UI of the user device is achieved by combining micro-gesture-based UI control (using one or more first sensors or biosensors), motion (e.g., rotational and / or linear motion, etc.) detection-based UI control (using at least one second sensor or IMU sensor), and translation detection-based UI control (using one or more outward-facing lights as tracking points for the camera of the user device to serve as additional UI control input).

[0031] These and other aspects of systems and methods for implementing a wrist-based wearable control device for controlling a UI of an HMD or other user device with camera input will be described in more detail in conjunction with the accompanying drawings.

[0032] The following detailed description shows several embodiments in more detail to enable those skilled in the art to practice these embodiments. The examples are only for illustrative purposes and are not intended to limit the scope of the present invention.

[0033] In the following description, for the purpose of explanation, many details are set forth in order to provide a thorough understanding of the embodiments. However, it is obvious to those skilled in the art that other embodiments of the present invention may be implemented without these details. In other examples, some structures and devices are shown in block diagram form. Several embodiments are described herein, and although various features are derived from different embodiments, it should be understood that the features described for one embodiment may also be combined with other embodiments. However, in the same way, any single or multiple features of the embodiments should not be considered as essential to each embodiment of the present invention, and other embodiments of the present invention may omit these features.

[0034] Unless otherwise stated, all numbers used herein to express quantities, sizes, etc. are to be understood as being modified in all cases by the term "about". In this application, unless otherwise expressly stated, the use of the singular includes the plural, and the use of the terms "and" and "or" means "and / or" unless otherwise stated. In addition, the use of the term "include" as well as other forms such as "include" and "comprising" are to be considered non-exclusive. In addition, unless otherwise expressly stated, terms such as "element" or "component" include both elements and components consisting of one unit and elements and components consisting of a plurality of units.

[0035] The various embodiments described herein - while (in some cases) embodying software products, computer-implemented methods and / or computer systems - represent tangible, concrete improvements over the prior art fields, including but not limited to wrist-worn wearable device technology, gesture control technology, gesture control technology for wrist-worn wearable devices, biosensor technology for gesture control, motion tracking technology for gesture control, user interface (UI) control technology, UI control technology for wrist-worn wearable devices, 3DOF control technology for UI, 6DOF control technology for UI, and / or the like. In other aspects, some embodiments may improve the functionality of a user device or system (e.g., a wrist wearable device, a gesture control system, a gesture control system for a wrist wearable device, a biosensor system for gesture control, a motion tracking system for gesture control, a UI control system, a UI control system for a wrist wearable device, a 3DOF control system of a UI, etc.) itself, for example, by: using a computing system to analyze first sensor data to identify a first gesture corresponding to movement of one or more fingers of a user, when the wrist wearable control device is worn by the user, the first sensor data is received from one or more first sensors disposed on the wrist wearable control device, the one or more first sensors being used to identify a first gesture corresponding to movement of one or more fingers of the user when the wrist wearable control device is worn by the user, and the one or more first sensors being used to identify a first gesture corresponding to movement of one or more fingers of the user when the wrist wearable control device is worn by the user. The invention relates to a method of detecting a gesture of one or more fingers of a user under a condition of detecting a gesture of one or more fingers of a user; using a computing system to analyze second sensor data to detect movement of a wrist-type wearable control device relative to at least one of three rotation axes, and when the wrist-type wearable control device is worn by the user, the second sensor data is received from one or more second sensors arranged on the wrist-type wearable control device; based on determining that the detected movement of the wrist-type wearable control device and the combination of the first gesture correspond to a command based on the first gesture, using a computing system to generate a first instruction based on the command based on the first gesture, and sending the generated first instruction to a head mounted device (HMD) or other user device with a camera input, the command based on the first gesture is used to control a user interface (UI) of the HMD or other user device, and the first instruction is used as an input to control the UI of the HMD or other user device; and the like.

[0036] Specifically, to the extent that any abstract concepts exist in various embodiments, these concepts can be implemented as described herein by devices, software, systems, and methods involving new functions (e.g., steps or operations), for example, by combining micro-gesture-based UI control (using one or more first sensors or biosensors), motion detection-based UI control (using one or more second sensors or IMU sensors), and translation detection-based UI control (using one or more outward-facing lights as tracking points for the camera of the user device to serve as additional UI control inputs), etc., and these new functions go beyond simple traditional computer processing operations. These functions can produce tangible results outside of the implemented computer system, including (by way of example only) an optimized wrist-worn wearable control device that enables 6DOF UI control of another user device by using biosensors (in some cases in combination with sound / motion sensors (such as microphones, etc.) and / or IMU sensors) to monitor, track, and identify micro-gestures of fingers on the arm wearing the wrist-worn wearable control device, using these detected and identified micro-gestures in combination with motion detection (using IMU sensors) and translation detection (using outward-facing lights that serve as tracking points for the camera of the controlled user device) of the wrist-worn wearable control device, at least some of which can be observed or measured by the user, the wrist-worn wearable device manufacturer, the user device manufacturer, and / or the universal remote control manufacturer.

[0037] Some embodiments

[0038] The embodiments shown in the drawings will now be described. Figures 1 to 5 Some features of methods, systems, and apparatus for implementing user interface (UI) control are shown, and more specifically, methods, systems, and apparatus for implementing a wrist-worn wearable controller for controlling an HMD or other user device with camera input as described above are shown. Figures 1 to 5 The methods, systems, and apparatus shown refer to examples of different embodiments including various components and steps, which components and steps may be considered alternatives to each other or may be used in combination with each other in various embodiments. Figures 1 to 5 The descriptions of the methods, systems, and devices shown in the present invention are for illustration purposes only and should not be considered to limit the scope of the different embodiments.

[0039] With reference to the accompanying drawings, Figure 1 A schematic diagram of a system 100 for implementing a wrist-based wearable control device for controlling a UI for an HMD or other user device with camera input is shown in accordance with various embodiments.

[0040] exist Figure 1In a non-limiting embodiment of the present invention, the system 100 may include a wrist wearable device 105, which is used to be worn on the wrist of the user 110. In some cases, the wrist wearable device 105 may include but is not limited to a smart watch, a wrist wearable display device, a wrist wearable control device, or one of other wrist wearable user devices. According to some embodiments, the wrist wearable control device may include but is not limited to one of a wristwatch-based wearable control device or a wristband-based wearable control device, each of which includes one or more band portions, which can be connected to adjacent band portions by connectors, and the connection method is configured to detachably wrap the wrist wearable control device around the wrist of the user 110 (for example, as Figure 2A-2P , Figure 3A and Figure 3B The wrist wearable device 105 may include, but is not limited to, a computing system 115, one or more first sensors 120, one or more second sensors 125, a display screen 130 (optional), a communication system 135, and one or more outward-facing lights 140a-140n (optional; collectively referred to as "outward-facing lights 140" or "lights 140", etc.). According to some embodiments, the outward-facing lights 140 may include, but is not limited to, at least one of the following: a plurality of infrared (IR) light emitting diodes (LEDs) or a plurality of color LEDs (e.g., LEDs that emit light in the visible spectrum, each LED being monochromatic or multi-color variable), etc.

[0041] In some embodiments, the computing system 115 may include, but is not limited to, at least one of the following: one or more processors on the wrist wearable device (e.g., processor 115a, etc.), one or more processors on a mobile device communicatively coupled to the wrist wearable device (e.g., a processor (not shown) on a mobile device 145 communicatively coupled to the wrist wearable device 105 via the communication system 135) (e.g., a processor (not shown) on a mobile device 145 communicatively coupled to the wrist wearable device 105 via the communication system 135). Figure 1 The mobile device 145 may include, but is not limited to, a mobile device 145 ...

[0042] Merely by way of example, in some cases, display device 130 may include, but is not limited to, at least one of a touch screen display device, a non-touch screen display device, a projection-based display device, a holographic display device, etc. In some cases, communication system 135 may include a wireless communication device capable of communicating using a protocol, including but not limited to at least one of a Bluetooth communication protocol, a WiFi communication protocol, or other 802.11 communication protocol suites, a ZigBee communication protocol, a Z-wave communication protocol, or other 802.15.4 communication protocol suites, a cellular communication protocol (e.g., 3G, 4G, 4G LTE, 5G, etc.), or other suitable communication protocols, etc.

[0043] In some embodiments, the system 100 may also include a user device 150, which may include, but is not limited to, at least one of one or more processors 150a, a transceiver 150b, one or more cameras 150c, or a display device 150d, etc. In some cases, the transceiver 150b and / or the one or more cameras 150c may each be integrated within the user device 150, or external to the user device 150 but communicatively coupled to the user device 150, or partially integrated within the user device 150 and partially external to the user device 150 but communicatively coupled to the user device 150, etc. In some cases, the camera 150c may have a field of view (FOV) 155 that, when directed toward the wrist wearable device 105, may be used to monitor or track the position and / or orientation of the wrist wearable device 105 (in some cases, performed at least in part based on an outward-facing light 140 (if any) disposed on the wrist wearable device 105). In some cases, UI 160 can be displayed or presented on display device 150d. In some cases, wrist wearable device 105 can be communicatively coupled to user device 150 via communication system 135 (e.g., Figure 1In some cases, the user device 150 may include, but is not limited to, a head mounted device (HMD) or other user devices with camera input, etc. In some cases, the HMD may include, but is not limited to, one of the following: virtual reality (VR) goggles, augmented reality (AR) goggles, mixed reality (MR) goggles, VR-enabled glasses, AR-enabled glasses, MR-enabled glasses, VR-enabled smartphones installed in headsets, AR-enabled smartphones installed in headsets, MR-enabled smartphones installed in headsets, etc. In some cases, other user devices with camera input may include, but are not limited to, one of the following: a smart TV (TV) with a built-in camera, a smart TV with an external camera, a game console with a built-in camera, a game console with an external camera, a monitor with a built-in camera, a monitor with an external camera, or a projector with an external camera, etc.

[0044] According to some embodiments, the one or more first sensors 120 may each include at least one biosensor. In some cases, the at least one biosensor may include but is not limited to the following one or more photoplethysmography (PPG) sensors, one or more electromyography (EMG) sensors, one or more sound sensors, or at least one of one or more motion sensors, each of which is used to detect a biometric feature corresponding to a soft tissue movement in or near the wrist of a user wearing a wrist wearable device. In some cases, the soft tissue may include but is not limited to at least one of one or more muscles, one or more tendons, or one or more blood vessels in or near the wrist of the user. In some cases, one or more PPG sensors may be used to optically detect changes in blood volume in one or more blood vessels under the skin tissue at or near the wrist of the user corresponding to the movement of one or more fingers of the user. In some cases, one or more EMG sensors may be used to detect the electrical activity of one or more muscles at or near the wrist of the user corresponding to the movement of one or more fingers of the user. In some cases, one or more sound sensors (including but not limited to microphones, etc. in some cases) may be used to detect sounds in the user's body corresponding to the movement of one or more fingers of the user. In some cases, one or more motion sensors (including but not limited to microphones, etc. in some cases) may be used to detect the movement of one or more muscles, one or more tendons, and one or more blood vessels corresponding to the movement of one or more fingers of the user.

[0045] By way of example only, in some cases, the one or more second sensors 125 may each include, but are not limited to, at least one of an inertial measurement unit (IMU) sensor or a gyroscope-based sensor, or a similar sensor, etc. In some cases, at least one of the one or more first sensors 120 and at least one of the one or more second sensors 125 may be disposed within each of the one or more belt portions. Alternatively, the one or more first sensors 120 and the one or more second sensors 125 may each be disposed within only one of the one or more belt portions, wherein the belt portion containing the one or more first sensors 120 may be the same belt portion containing the one or more second sensors 125, or a different belt portion from the belt portion containing the one or more second sensors 125. Alternatively, at least one of the one or more first sensors 120 may be disposed within each of the one or more belt portions, and the one or more second sensors 125 may be disposed within only one of the one or more belt portions.

[0046] In some cases, outward-facing lights 140 can be disposed on one or more band portions, and in some cases, can be arranged in a predetermined pattern on one or more band portions, and can be used as tracking points for one or more cameras 150 c of user device 150 (as captured within FOV 155) to track translational motion of wrist wearable device 105 along three axes relative to user device 150. In some cases, tracking of translational motion of the wrist wearable device along three axes can serve as an additional input for controlling UI 160 of user device 150.

[0047] During operation, the computing system 115, the processor 115a, the signal processing system 115b, the AI ​​system 115c, the other computing systems 115d, and / or the mobile device 145, etc. (collectively referred to as “computing systems”, etc.) may analyze the first sensor data to identify a first gesture corresponding to the movement of one or more fingers of a user (e.g., the user 110, etc.). When the wrist-type wearable control device is worn by the user, the first sensor data is received from one or more first sensors (e.g., the first sensor or the biosensor 120, etc.) provided on the wrist-type wearable control device (e.g., the wrist-type wearable device 105, etc.). The one or more first sensors are used to detect the gestures of one or more fingers of the user when the wrist-type wearable control device is worn by the user without any fingers of the user contacting the wrist-type wearable control device. The computing system may analyze the second sensor data to detect the movement of the wrist wearable control device relative to at least one of the three rotation axes (e.g., rotation and / or linear movement, etc.), and when the wrist wearable control device is worn by the user, the second sensor data is received from one or more second sensors (e.g., the second sensor or the IMU sensor 125, etc.) provided on the wrist wearable control device. Based on determining that the detected combination of the movement of the wrist wearable control device and the first gesture corresponds to a command based on the first gesture, the computing system may generate a first instruction based on the command based on the first gesture, and may send the generated first instruction to the HMD or other user device with camera input (e.g., the user device 150, etc.), the command based on the first gesture is used to control the user interface (UI) (e.g., UI 160, etc.) of the HMD or other user device with camera input, and the first instruction is used as an input for controlling the UI of the HMD or other user device with camera input.

[0048] In some embodiments, the command based on the first gesture may include but is not limited to at least one of the following: a sliding-based command, a dragging-based command, a tapping command, a double-clicking command, a pointing command, a pinching-based command, a grip-based command, a rotation command, a rolling command, a pitching command, or a yaw command, etc. In some cases, the first instruction may include but is not limited to at least one of the following: a rolling command, a moving command, a selecting command, a confirming command, a highlighting command, a returning command, a cancelling command, a moving cursor command, or a navigating command, etc.

[0049] According to some embodiments, for a slide-based command gesture (or finger slide) command (not shown), a user can start from a relaxed position of the finger (also referred to herein as an "initial position"; e.g., Figure 3A and Figure 3BThe command may be defined as a swipe up command in which the fingers of the user's hand (e.g., five fingers) are naturally stretched and loosened or relaxed (i.e., there is no muscle tension to straighten the fingers, nor is there muscle tension to bend the fingers (e.g., making a fist, etc.), or similar postures). For a swipe up command, a finger (e.g., but not limited to an index finger, etc.) may be lifted relative to other fingers from an initial or relaxed posture or from a lowered position (as described below), for example, at a moderate speed (the moderate speed in this article may refer to a speed greater than the speed of slow, deliberate movement of the finger positioning, etc.), with the other fingers remaining in the initial or relaxed posture. In some cases, the finger lifting may be performed quickly (e.g., according to definitions such as blurred vision of the finger due to movement speed). For a swipe down command, a finger may be lowered relative to other fingers from an initial or relaxed posture or a lifted position (as described above) at a moderate speed, with the other fingers remaining in the initial or relaxed posture. In some cases, the finger lowering may be performed quickly.

[0050] In some embodiments, starting from the initial posture, if the user lifts the index finger, the computing system may recognize this micro-gesture as a swipe up command. Conversely, starting from the initial posture, if the user pulls down the index finger, the computing system may recognize this micro-gesture as a swipe down command. Taking the page navigation function as an example, the user can use an activation gesture (such as a pinch command) to open a page. Here, the pinch command or pinch-based gesture command may include the thumb and index finger touching or pressing together, while the other fingers of the hand (not shown) are almost or completely motionless. The user can then pull down the index finger to trigger the swipe down function to navigate the page downward. If the user lifts the index finger above the initial posture, this gesture will trigger the swipe up function.

[0051] According to some embodiments, for a drag-based gesture (or finger drag) command (not shown), the user may start from an initial or relaxed position of the finger, similar to the slide-based command gesture (or finger slide) command described above. For an upward drag command, a finger (such as but not limited to an index finger, etc.) may be lifted relative to other fingers from an initial or relaxed position or from a lowered position (as described below), for example, at a slow speed (here, slow speed may refer to a speed slower than the above-mentioned medium speed, etc.), and the other fingers remain in the initial or relaxed position. For a downward drag command, a finger may be lowered relative to other fingers from an initial or relaxed position or a lifted position (as described above), for example, at a slow speed, and the other fingers remain in the initial or relaxed position.

[0052] If the UI (e.g., UI 160, etc.) only provides one dimension for the user to scroll or drag within the UI (regardless of whether the wrist wearable device includes a second sensor or an IMU sensor (e.g., second sensor or IMU sensor 125, etc.)), the lifting and lowering of the finger (as described above with respect to the slide-based command and the drag-based command) can be recognized as a micro-gesture that causes the computing system (e.g., computing system 115, etc.) to cause the UI to perform one of scrolling or dragging in a first direction (in response to the finger lifting) or scrolling or dragging in a second direction (in response to the finger lowering). For example, if the UI only provides vertical movement, the first direction can cause the computing system to cause the UI to perform one of a scroll up function or an upward drag function, while the second direction can cause the computing system to cause the UI to perform one of a scroll down function or a downward drag function, and so on. Similarly, if the UI only provides horizontal movement, the first direction may cause the computing system to cause the UI to perform one of a left scroll function or a left drag function (if the left hand is used; or a right scroll function or a right drag function if the right hand is used), while the second direction may cause the computing system to cause the UI to perform one of a right scroll function or a right drag function (if the left hand is used; or a left scroll function or a left drag function if the right hand is used), and so on.

[0053] If a UI (e.g., UI 160, etc.) provides two dimensions for a user to scroll or drag within the UI, and if the wrist wearable device includes a second sensor or IMU sensor (e.g., second sensor or IMU sensor 125, etc.), then the lifting and lowering of a finger (as described above with respect to slide-based commands and drag-based commands) combined with the movement of the user's wrist (detected by the second sensor or IMU sensor 125, etc.) can be recognized as a micro-gesture that causes a computing system (e.g., computing system 115, etc.) to cause the UI to perform one of scrolling or dragging, the direction of which is based on the degree of pronation or supination of the forearm connected to the wrist of the user wearing the wrist wearable device, which is detected by the second sensor or IMU sensor that tracks the movement of the wrist wearable device. Here, forearm supination may refer to when the user's arm is hanging by the side or when the forearm is raised from the arm-hanging position (regardless of whether the fingers are open in a palm or clenched into a fist), the forearm is rotated so that the ventral side of the forearm (or the palm) faces forward and / or upward relative to the user's body in a sitting or standing position, and forearm pronation may refer to when the user's arm is hanging by the side or when the forearm is raised from the arm-hanging position (regardless of whether the fingers are open in a palm or clenched into a fist), the forearm is rotated so that the ventral side of the forearm (or the palm) faces backward and / or downward relative to the user's body in a sitting or standing position, and the neutral position of the forearm may refer to when the user's arm is hanging by the side or when the forearm is raised from the arm-hanging position (regardless of whether the fingers are open in a palm or clenched into a fist), the forearm is rotated so that the ventral side of the forearm (or the palm) faces the sagittal plane or the median plane, which divides the body into left and right parts along the midline of the body and is located in the middle or halfway between the pronated position and the supinated position.

[0054] In a non-limiting example, if the forearm is pronated (or fully pronated), the lifting and lowering of the fingers may be recognized as a micro-gesture that causes the computing system to cause the UI to perform one of scrolling or dragging in a vertical direction (i.e., up or down, similar to the one-dimensional vertical motion embodiment described above), regardless of whether the user has an elbow straight, an elbow bent and the forearm is beside the torso, or an elbow bent and the forearm is in front of the torso. Similarly, if the forearm is in a neutral position such that the ventral side of the forearm (or palm) faces the sagittal or median plane, the lifting and lowering of the fingers may be recognized as a micro-gesture that causes the computing system to cause the UI to perform one of scrolling or dragging in a horizontal direction (i.e., left or right, similar to the one-dimensional horizontal motion embodiment described above), regardless of whether the user has an elbow straight, an elbow bent and the forearm is beside the torso, or an elbow bent and the forearm is in front of the torso. If the forearm is between the pronated and supinated positions (but not in the pronated or neutral position), the lifting and lowering of the fingers can be recognized as a micro-gesture that causes the computing system to cause the UI to perform one of a diagonal scroll or drag based on the degree of rotation between the pronated and supinated positions, regardless of whether the user has their elbows straight, their elbows bent and their forearms next to their torsos, or their elbows bent and their forearms in front of their torsos.

[0055] In some embodiments, a machine learning system (e.g., AI system 115c or other computing system 115d, etc.) can be used to further enhance the operation of the wrist wearable device by learning and fine-tuning the detection of micro-gestures, motion-based gesture detection, translation-based gesture detection, and / or instances of user-intended gesture control (as compared to regular motions that are not gesture-controlled motions), etc., to adapt to user use over time, etc.

[0056] Reference will be made to Figure 2- Figure 4 These and other functions of system 100 (and its components) are described in further detail.

[0057] Figures 2A to 2P (collectively, “ FIG. 2 ”) is a schematic diagram illustrating various non-limiting examples 200 and 200 ′ of wrist-type wearable control devices for controlling a UI of an HMD or other user device with camera input in accordance with various embodiments. FIG. 2A to FIG. 2H Various non-limiting examples 200 of wristband-based wearable control devices 205a - 205h are depicted respectively. Figure 2I to Figure 2P Various non-limiting examples 200 ′ of wristwatch-based wearable control devices 205 i - 205 p are depicted.

[0058] In FIG. 2 , the wristband-based wearable control devices 205a-205h or the wristwatch-based wearable control devices 205i-205p (these two groups of devices are collectively referred to as “wrist-type wearable control devices 205”, etc.) may each include, but are not limited to, a plurality of band portions 210 and a plurality of connectors 215. Each band portion 210 may include, but are not limited to, at least one of the following: one or more first sensors or biosensors 220 (similar to Figure 1 a first sensor or biosensor 120, etc.), at least one second sensor or IMU sensor 225 (similar to Figure 1 a second sensor or IMU sensor 125, etc.), one or more outward-facing lights 230 (similar to Figure 1 one or more outward-facing lights 140a-140n, etc.), etc.), or may include a belt portion that does not have any of the first sensor or biosensor 220, the second sensor or IMU sensor 225, or the outward-facing lights 230, etc.

[0059] For example, the wristband-based wearable control devices 205a-205h may include, but are not limited to: (a) a wristband-based wearable control device 205a having a plurality of band portions 210a, each band portion 210a including one or more first sensors or biosensors 220 and at least one second sensor or IMU sensor 225 (e.g., Figure 2A (b) a wristband-based wearable control device 205b having a plurality of band portions 210b, each band portion 210b including one or more first sensors or biosensors 220, at least one second sensor or IMU sensor 225, and one or more outward-facing lights 230 (such as Figure 2B (c) a wristband-based wearable control device 205c having a plurality of band portions 210c and a single band portion 210a, each band portion 210c including one or more first sensors or biosensors 220, and a single band portion 210a including one or more first sensors or biosensors 220 and at least one second sensor or IMU sensor 225 (as shown in FIG. Figure 2C (d) a wristband-based wearable control device 205d having a plurality of band portions 210d and a single band portion 210b, each band portion 210d including one or more first sensors or biosensors 220 and one or more outward-facing lights 230, and a single band portion 210b including one or more first sensors or biosensors 220, at least one second sensor or IMU sensor 225, and one or more outward-facing lights 230 (as shown in FIG. Figure 2D(e) a wristband-based wearable control device 205e having a plurality of band portions 210e and a single band portion 210a, the band portion 210e not including any first sensor or biosensor 220, second sensor or IMU sensor 225, or outward-facing light 230, the single band portion 210a including one or more first sensors or biosensors 220 and at least one second sensor or IMU sensor 225 (as shown in FIG. 1 ). Figure 2E (f) a wristband-based wearable control device 205f having a plurality of band portions 210f and a single band portion 210b, each band portion 210f including one or more outwardly facing lights 230, and a single band portion 210b including one or more first sensors or biosensors 220, at least one second sensor or IMU sensor 225, and one or more outwardly facing lights 230 (as shown in FIG. Figure 2F (g) a wristband-based wearable control device 205g having a single band portion 210g, a (separate) single band portion 210c, and a plurality of band portions 210e, band portion 210g including at least one second sensor or IMU sensor 225, band portion 210c including one or more first sensors or biosensors 220, and band portion 210e not including any first sensor or biosensor 220, second sensor or IMU sensor 225, or outward-facing light 230 (as shown in FIG. Figure 2G (h) a wristband-based wearable control device 205h having a single band portion 210h, a (separate) single band portion 210d, and a plurality of band portions 210f, the band portion 210h including at least one second sensor or IMU sensor 225 and one or more outward-facing lights 230, the band portion 210d including one or more first sensors or biosensors 220 and one or more outward-facing lights 230, each band portion 210f including one or more outward-facing lights 230 (as shown in FIG. Figure 2H shown); and so on.

[0060] Each band portion 210 can be connected to an adjacent band portion 210 via a connector 215, and the connection method is configured so that when the wrist wearable control device 205 is worn by a user (e.g., user 110, etc.), the wrist wearable control device 205 can be removably wrapped around the user's wrist. In some cases, the connector 215 may include an elastic material-based connector that allows the band portions to be pulled apart from each other to slide over and put on the user's hand and wrist (or slide over and take off the user's hand and wrist). Alternatively, the connector 215 may include a plurality of links and buckles (e.g., made of metal, plastic, or other solid materials), etc. Although FIG. 2A to FIG. 2H 2, but at least one band portion 210 may also include, but is not limited to, a computing system (similar to Figure 1computing system 115, etc.) or a communication system (similar to Figure 1 at least one of the communication systems 135, etc.

[0061] Similarly, the wristwatch-based wearable control devices 205i-205p may include, but are not limited to: (i) a wristwatch-based wearable control device 205i having a watch portion 235 and a plurality of band portions 210a, each band portion 210a including one or more first sensors or biosensors 220 and at least one second sensor or IMU sensor 225 (e.g., Fig.2I (j) a wristwatch-based wearable control device 205j having a watch portion 235 and a plurality of band portions 210b, each band portion 210b including one or more first sensors or biosensors 220, at least one second sensor or IMU sensor 225, and one or more outward-facing lights 230 (e.g. Figure 2J (k) a wristwatch-based wearable control device 205k having a watch portion 235, a plurality of band portions 210c, and a single band portion 210a, each band portion 210c including one or more first sensors or biosensors 220, and a single band portion 210a including one or more first sensors or biosensors 220 and at least one second sensor or IMU sensor 225 (as shown in Figure 2K (l) a wristwatch-based wearable control device 205l having a watch portion 235, a plurality of band portions 210d, and a single band portion 210b, each band portion 210d including one or more first sensors or biosensors 220 and one or more outward-facing lights 230, and a single band portion 210b including one or more first sensors or biosensors 220, at least one second sensor or IMU sensor 225, and one or more outward-facing lights 230 (as shown in FIG. Figure 2L (m) a wristwatch-based wearable control device 205m having a watch portion 235, a plurality of band portions 210e, and a single band portion 210a, the band portion 210e not including any first sensor or biosensor 220, second sensor or IMU sensor 225, or outward-facing light 230, the band portion 210a including one or more first sensors or biosensors 220 and at least one second sensor or IMU sensor 225 (as shown in FIG. 1 ); Figure 2M (n) a wristwatch-based wearable control device 205n having a watch portion 235, a plurality of band portions 210f, and a single band portion 210b, each band portion 210f including one or more outwardly facing lights 230, and band portion 210b including one or more first sensors or biosensors 220, at least one second sensor or IMU sensor 225, and one or more outwardly facing lights 230 (as shown in FIG. Figure 2N (o) a wristwatch-based wearable control device 205o having a watch portion 235, a single band portion 210g, a (separate) single band portion 210c, and a plurality of band portions 210e, band portion 210g including at least one second sensor or IMU sensor 225, band portion 210c including one or more first sensors or biosensors 220, and band portion 210e not including any first sensor or biosensor 220, second sensor or IMU sensor 225, or outward-facing light 230 (e.g. Fig.2O (p) a wristwatch-based wearable control device 205p having a watch portion 235, a single band portion 210h, a (separate) single band portion 210d, and a plurality of band portions 210f, the band portion 210h including at least one second sensor or IMU sensor 225 and one or more outward-facing lights 230, the band portion 210d including one or more first sensors or biosensors 220 and one or more outward-facing lights 230, each band portion 210f including one or more outward-facing lights 230 (as shown in FIG. Figure 2P shown); and so on.

[0062] Each band portion 210 (and the watch portion 235) can be connected to an adjacent band portion 210 via a connector 215, and the connection method is configured to removably wrap the wrist-type wearable control device 205 around the user's wrist when the user (e.g., user 110, etc.) wears the wrist-type wearable control device 205. In some cases, the connector 215 may include an elastic-based connector that allows the band portions to be pulled apart from each other to slide over and put on the user's hand and wrist (or slide over and take off the user's hand and wrist). Alternatively, the connector 215 may include a plurality of links and buckles (e.g., made of metal, plastic, or other solid materials), etc. Although Figure 2I to Figure 2P Not shown, however, at least one band portion 210 may also include, but is not limited to, a computing system (similar to Figure 1 computing system 115, etc.) or a communication system (similar to Figure 1 at least one of the communication systems 135, etc.

[0063] although Figure 2I to Figure 2P 2, but the table portion 235 (in addition to including a table component or a smart table component) may include, but is not limited to, at least one of the following: a computing system (similar to Figure 1 computing system 115, etc.), communication system (similar to Figure 1 communication system 135, etc.), one or more first sensors or biosensors 220, at least one second sensor or IMU sensor 225, or one or more outward-facing lights 230, etc.

[0064] Here we will refer to Figure 1 , Figure 3 and Figure 4 These and other functions of examples 200 and 200' (and components thereof) are described in further detail.

[0065] Figure 3A and Figure 3B (collectively referred to as “FIG. 3”) is a diagram illustrating the use of a wrist-type wearable control device to control an HMD ( Figure 3A ) or other user devices with camera input ( Figure 3B ) are schematic diagrams of various non-limiting examples 300 and 300' of a UI.

[0066] In FIG. 3 , a wrist wearable device 205 (similar to Figure 1 and wrist wearable devices 105 and 205 in FIG. 2 ) are shown as being worn on a user (similar to Figure 1 The wrist wearable device 205 may each include, but is not limited to, a plurality of band portions 210 and a plurality of connectors 215. The wrist wearable device 205 includes a wristwatch-based wearable control device (similar to Figure 2I-2P In the case of the wrist-based wearable control device 205i-205p, etc. in FIG. 3 ), the wrist-type wearable device 205 may further include a watch portion 235 (not shown in FIG. 3 ; similar to Figure 2I-2P Each band portion 210 (and in some cases each band portion 235) may include, but is not limited to, at least one of the following: one or more first sensors or biosensors 220 (not shown in FIG. 3; similar to Figure 1 2 ), at least one second sensor or IMU sensor 225 (not shown in FIG. 3 ; similar to the second sensor or IMU sensor 125 and 225 in FIG. 1 and FIG. 2 , etc.), or one or more outward-facing lights 230 (similar to Figure 1 and one or more outward-facing lights 140a-140n and 230 in Figure 2, etc.), and so on.

[0067] refer to Figure 3A In a non-limiting embodiment 300, HMD 310a (similar to Figure 1 The user equipment 150, etc.) may include but is not limited to at least one processor (not shown; similar to Figure 1 One or more processors 150a, etc.), at least one transceiver (not shown; similar to Figure 1 transceiver 150b, etc.), one or more cameras (not shown; similar to Figure 1 One or more cameras 150c, etc.), display device 315a (similar to Figure 1display device 150d, etc.), and UI 320a (similar to Figure 1 In some cases, the HMD 310a may include, but is not limited to, one of the following: virtual reality (VR) goggles, augmented reality (AR) goggles, mixed reality (MR) goggles, VR-enabled glasses, AR-enabled glasses, MR-enabled glasses, a VR-enabled smartphone installed in a headset, an AR-enabled smartphone installed in a headset, an MR-enabled smartphone installed in a headset, etc.

[0068] Similarly, in Figure 3B In the non-limiting embodiment 300', the user device 310b (similar to Figure 1 The user equipment 150, etc.) may include but is not limited to at least one processor (not shown; similar to Figure 1 One or more processors 150a, etc.), at least one transceiver (not shown; similar to Figure 1 transceiver 150b, etc.), one or more cameras (similar to Figure 1 One or more cameras 150c, etc.), a display screen or display device 315b (similar to Figure 1 display device 150d, etc.), and UI 320b (similar to Figure 1 UI 160, etc.). Figure 3B In the embodiment, the camera 340 is shown as an external but communicatively coupled camera, but various embodiments are not limited thereto, and the camera 340 may be integrated into the user device 310b. In some cases, the user device 310b may include, but is not limited to, a smart TV (TV) with a built-in camera, a smart TV with an externally connected camera, a game console with a built-in camera, a game console with an externally connected camera, a monitor with a built-in camera, a monitor with an externally connected camera, or a projector with an externally connected camera, etc. In some cases, the monitor or projector may also be communicatively coupled with one of a smart phone, a tablet computer, a laptop computer, a desktop computer, or a portable gaming device, etc., to present content to be displayed on the monitor or projector.

[0069] exist Figure 3A and Figure 3BIn the example, UI 320a or 320b (collectively referred to as “UI 320”, etc.) presented within display device 315a or 315b (collectively referred to as “display device 315”, etc.) may display or present content (in this example, a menu screen having a plurality of system icons 325 (in this example, icons of a connected VR headset and mobile device, etc.) and / or a plurality of icons 330 associated with a corresponding plurality of software applications (apps), etc., but not limited thereto). In some cases (although not shown), the content that may be displayed or presented may include icons, text, graphics, objects, thumbnails, or picture-in-picture (PIP) type mini-windows of media content, the media content including but not limited to video content, image content, representations of audio content (e.g., album cover art, artist photos or posters, etc.), game content, demonstration content, etc.

[0070] In some embodiments, as described above with respect to Figure 1 As described in detail, one or more first sensors or biosensors 220 (including but not limited to at least one of a PPG sensor, an EMG sensor, a sound sensor, or a motion sensor, etc.) can detect biometrics corresponding to the movement of soft tissue in or near the wrist of a user wearing the wrist wearable device. In some cases, one or more first sensors or biosensors 220 can be used to track, monitor, and / or recognize gesture-based commands, which include but are not limited to at least one of the following: sliding-based commands (e.g., as described above with respect to Figure 1 As described above, finger 305c is raised from or lowered toward thumb 305d at a moderate speed, and the other fingers of hand 305b are maintained in an initial or relaxed position, etc.), drag-based commands (e.g., as described above with respect to Figure 1 As described above, finger 305c is slowly raised from thumb 305d or lowered toward thumb 305d, and other fingers of hand 305b are kept in initial or relaxed posture, etc.), tap command (for example, finger 305c is quickly lowered toward thumb 305d, or finger 305c is bent or folded, and other fingers of hand 305b are kept in initial or relaxed posture, etc.), double-click command (for example, similar to tap command, but such gesture is performed twice in rapid succession, etc.), pointing command (for example, finger 305c is straightened, and other fingers of hand 305b are kept in initial or relaxed posture, etc.), pinch-based command (for example, as described above with respect to Figure 1In some cases, the wrist wearable device 205 may generate instructions based on the tracked, monitored, and / or recognized gesture-based commands, including but not limited to at least one of a scroll command, a move command, a selection command, a confirmation command, a highlight command, a return command, or a cancel command, etc.

[0071] According to some embodiments, at least one second sensor or IMU sensor 225 (including but not limited to at least one of an inertial measurement unit (IMU) sensor or a gyroscope-based sensor, etc.) can detect the movement of the wrist wearable device 205 relative to at least one of the three rotation axes (for example, as shown in FIG. 3, the rotation R relative to the X axis). x , rotation R relative to the Y axis y , and / or rotation R about the Z axis z (also referred to as pitch, roll, and / or yaw, respectively), etc.). In some cases, the Y-axis can be parallel to at least one of the longitudinal extension of the user's arm or the main axis of the wrist wearable device 205, etc., while the X-axis can be in a plane parallel to a plane defined by one of the dorsal side of the forearm or the ventral side of the forearm, and the Z-axis can be parallel to a line perpendicular to a plane defined by one of the dorsal side of the forearm or the ventral side of the forearm. Reference Figure 1 and Figure 3, supination or pronation can refer to the rotation R about the Y axis y , the direction of rotation depends on which arm the wrist wearable device is worn on. In some cases, the wrist wearable device 205 can generate instructions based on the detected movement of the wrist wearable device 205 relative to at least one of the three rotation axes, including but not limited to at least one of a cursor movement command or a navigation command.

[0072] In some cases, one or more outward-facing lights 230 can be disposed on one or more band portions 210, and in some cases, can be arranged in a predetermined pattern on one or more band portions, and can be used as tracking points for one or more cameras 340 of a user device 310a or 310b (collectively referred to as "user device 310", etc.) to track the translational movement of the wrist wearable device 205 along three axes (e.g., X-axis, Y-axis, Z-axis, etc.) relative to the user device 310. In some cases, the tracking of the translational movement of the wrist wearable device along three axes can be used as an additional input for controlling the UI 320 displayed within the display device 315 of the user device 310.

[0073] The combination of one or more first sensors or biosensors 220, at least one second sensor or IMU sensor 225, and one or more outward-facing lights 230 can achieve high-precision six-degree-of-freedom (6DOF) control of UI 320 displayed within display device 315 of user device 310 by combining micro-gesture-based UI control (using one or more first sensors or biosensors 220), motion detection-based UI control (using at least one second sensor or IMU sensor 225), and translation detection-based UI control (using one or more outward-facing lights 230 as tracking points for the user's camera 340, used as additional UI control input). In some cases, there is some overlap between two or more of the micro-gesture-based UI control, motion detection-based UI control, and translation detection-based UI control. For example, Figure 3A and Figure 3B As shown in the non-limiting embodiments 300 and 300' of FIG. 3, 6DOF controls can be used to navigate within UI 320. According to some embodiments, navigating and interacting within UI 320 can include selecting or highlighting icons, text, graphics, objects, thumbnails, or picture-in-picture (PIP) type media content mini windows, etc. (collectively referred to as "UI objects 330", etc.) (as shown, for example, by superimposing a bounding box 335 on an app icon in FIG. 3, etc.), moving a selected or highlighted field (e.g., bounding box 335, etc.) from one UI object to another UI object, etc. Alternatively or additionally, navigating and interacting within UI 320, whether or not there is a selected or highlighted field, can include scrolling or flipping through menus, pages, windows, and / or other UI display areas, etc., and responding to prompts (using micro-gesture-based acceptance or rejection input commands, or using micro-gesture-based "yes" or "no" input commands, or using micro-gesture-based "OK" or "cancel" input commands, etc.), etc.

[0074] In some cases, such as when conditions or circumstances for tracking or detecting one or more corresponding gesture-based commands are less than ideal, the combination of one or more first sensors or biosensors 220, at least one second sensor or IMU sensor 225, and one or more outward-facing lights 230 may also supplement and / or assist gesture-based input or commands. For example, in low-light or dark conditions (or extremely bright conditions), the camera 340 may have problems properly tracking the position and / or orientation of the one or more outward-facing lights 230. The above situation may also occur if the wrist wearable device 205 (and one or more outward-facing lights 230) moves out of the field of view of the camera 340 (which is more likely in HMD embodiments). In this case, micro-gesture-based control and / or motion detection-based control (constituting three-degree-of-freedom (“3DOF”) control) can still enable navigation and interaction in the user interface 320. In some cases, multiple wrist wearable devices 205 can be used (one on each arm of the user) to provide additional 6DOF control of UI 320, and can facilitate immersive experiences (e.g., games or AR / VR / MR experiences, etc.), wherein one wrist wearable device can be used to navigate within an environment displayed or presented in UI 320, while another wrist wearable device can be used to navigate UI objects or sub-UIs within the environment within UI320 and interact with UI objects or sub-UIs, etc.

[0075] In some embodiments, a machine learning system (e.g., Figure 1 The machine learning system 115b, AI system 115c, or other computing system 115d, etc.) can be used to further enhance the operation of the wrist wearable device by learning and fine-tuning the detection of micro-gestures, motion-based gesture detection, translation-based gesture detection, and / or instances of user intended gesture control (compared to regular motions that are not gesture control motions) to adapt to user use over time, etc.

[0076] This article will refer to Figure 1 , Figure 2, and Figure 4 These and other functions of examples 300 and 300 ′ (and components thereof) are described in further detail.

[0077] Figure 4 is a flow chart illustrating a method 400 for implementing a wrist-based wearable control device for controlling a UI for an HMD or other user device with camera input, according to various embodiments.

[0078] Although techniques and procedures are depicted and / or described in a particular order for purposes of illustration, it should be understood that certain procedures may be reordered and / or omitted within the scope of various embodiments. Figure 4The method 400 shown may be performed by or using Figure 1 , Figure 2A-2H , Figure 2I-2P , Figure 3A and Figure 3B Although the method 400 is implemented with respect to the systems, examples, or embodiments 100, 200, 200', 300, and 300' (or components thereof) (and in some cases, the method 400 will be described below with respect to these systems, examples, or embodiments), the method may also be implemented using any suitable hardware (or software) implementation. Similarly, although Figure 1 , Figure 2A-2H , Figure 2I-2P , Figure 3A and Figure 3B Each of the systems, examples, or embodiments 100, 200, 200', 300 and 300' can be based on Figure 4 The method 400 shown is operated (eg, by executing instructions embodied on a computer-readable medium), however Figure 1 , Figure 2A-2H , Figure 2I-2P , Figure 3A and Figure 3B The systems, examples or embodiments 100, 200, 200', 300 and 300' may also operate according to other operating modes and / or execute other suitable procedures, respectively.

[0079] exist Figure 4 In a non-limiting embodiment of A, at block 405, method 400 may include, when the wrist wearable control device is worn by the user, using a computing system to receive first sensor data from one or more first sensors disposed on the wrist wearable control device. The one or more first sensors may be used to detect a gesture of one or more fingers of the user when the wrist wearable control device is worn by the user without any fingers of the user contacting the wrist wearable control device. At block 410, method 400 may include analyzing the first sensor data using a computing system to identify a first gesture corresponding to a movement of one or more fingers of the user.

[0080] In some embodiments, the computing system may include, but is not limited to, at least one of the following: one or more processors on a wrist-worn wearable control device, one or more processors on a mobile device communicatively coupled to the wrist-worn wearable control device, a machine learning system, an artificial intelligence (AI) system, a deep learning system, a neural network, a convolutional neural network (CNN), or a fully convolutional network (FCN), etc.

[0081] According to some embodiments, the one or more first sensors may each include at least one biosensor, the at least one biosensor including at least one of one or more photoplethysmography (PPG) sensors, one or more electromyography (EMG) sensors, one or more sound sensors, or one or more motion sensors, each sensor for detecting a biometric feature corresponding to a soft tissue movement in or near the wrist of a user wearing a wrist wearable device. In some cases, the soft tissue may include but is not limited to at least one of one or more muscles, one or more tendons, or one or more blood vessels in or near the wrist of the user. In some cases, the one or more PPG sensors may be used to optically detect changes in blood volume in one or more blood vessels under the skin tissue at or near the wrist of the user corresponding to the movement of one or more fingers of the user. In some cases, the one or more EMG sensors may be used to detect the electrical activity of one or more muscles at or near the wrist of the user corresponding to the movement of one or more fingers of the user. In some cases, the one or more sound sensors may be used to detect sounds in the user's body corresponding to the movement of one or more fingers of the user. In some cases, the one or more motion sensors may be used to detect the movement of one or more muscles, one or more tendons, and one or more blood vessels corresponding to the movement of one or more fingers of the user.

[0082] At block 415, the method 400 may include: when the wrist-type wearable control device is worn by the user, using the computing system to receive second sensor data from one or more second sensors disposed on the wrist-type wearable control device. When the user wears the wrist-type wearable control device, the one or more second sensors may be used to detect movement of the wrist-type wearable control device relative to at least one of the three rotation axes. At block 420, the method 400 may include: using the computing system to analyze the second sensor data to detect movement of the wrist-type wearable control device relative to at least one of the three rotation axes.

[0083] In some embodiments, the one or more second sensors may each include, but are not limited to, at least one of an inertial measurement unit (IMU) sensor or a gyroscope-based sensor, etc. In some cases, the wrist-type wearable control device includes one of a wristwatch-based wearable control device or a wristband-based wearable control device, each of the wristwatch-based wearable control device and the wristband-based wearable control device includes one or more band portions, and the band portions are linked to adjacent band portions by connectors in a manner that allows the wrist-type wearable control device to be detachably wrapped around the wrist of the user. In some cases, at least one of the one or more first sensors and at least one of the one or more second sensors may be disposed in each of the one or more band portions. Alternatively, the one or more first sensors and the one or more second sensors are each disposed in only one of the one or more band portions, wherein the band portion containing the one or more first sensors may be one of the following: the same band portion containing the one or more second sensors, or a band portion different from the band portion containing the one or more second sensors. Alternatively, at least one of the one or more first sensors is disposed in each of the one or more band portions, and the one or more second sensors are disposed in only one of the one or more band portions.

[0084] At block 425, method 400 may also include using a computing system to determine whether the detected combination of the movement of the wrist-type wearable control device and the first gesture corresponds to a command based on the first gesture, the command based on the first gesture being used to control the user interface (UI) of the HMD or other user device with camera input. If so, method 400 may also include using a computing system to generate a first instruction based on the command based on the first gesture (block 430), and using a computing system to send the generated first instruction to the HMD or other user device with camera input, the first instruction being used as input to control the UI of the HMD or other user device (block 435).

[0085] In some cases, the HMD or other user device with camera input may include, but is not limited to, one of the following: virtual reality (VR) goggles, augmented reality (AR) goggles, mixed reality (MR) goggles, VR-enabled glasses, AR-enabled glasses, MR-enabled glasses, a VR-enabled smartphone mounted in a headset, an AR-enabled smartphone mounted in a headset, an MR-enabled smartphone mounted in a headset, a smart television (TV) with a built-in camera, a smart TV with an externally connected camera, a game console with a built-in camera, a game console with an externally connected camera, a monitor with a built-in camera, a monitor with an externally connected camera, or a projector light with an externally connected camera.

[0086] According to some embodiments, the command based on the first gesture may include but is not limited to at least one of the following: a sliding-based command, a dragging-based command, a tapping command, a double-clicking command, a pointing command, a pinching-based command, a grip-based command, a rotation command, a rolling command, a pitching command, or a yaw command. In some cases, the first instruction may include but is not limited to at least one of the following: a rolling command, a moving command, a selecting command, a confirming command, a highlighting command, a returning command, a cancelling command, a moving cursor command, or a navigating command, etc.

[0087] In some cases, at block 440, method 400 may also include activating, using the computing system, a plurality of outward-facing lights disposed on one or more band portions of the wrist-type wearable control device, wherein the plurality of outward-facing lights are arranged in a predetermined pattern on the one or more band portions and are used as tracking points for one or more cameras of an HMD or other user device to track translational movement of the wrist-type wearable control device relative to the HMD or other user device along three axes. In some cases, the plurality of outward-facing lights may include, but are not limited to, at least one of: a plurality of infrared (IR) light emitting diodes (LEDs) or a plurality of color LEDs.

[0088] System and Hardware Implementation Examples

[0089] Figure 5 is a block diagram illustrating an example of a computer or system hardware architecture according to various embodiments. Figure 5 A schematic diagram of one embodiment of a computer system 500 of a service provider system hardware is provided, which can perform the methods provided by various other embodiments described herein, and / or can perform the functions of the computer or hardware system described above (i.e., wrist wearable devices 105, 205a-205p and 205, computing system 115, display screen 130, and user devices 150, 310a, and 310b, etc.). It should be noted that Figure 5 It is intended only to provide a general description of the various components, wherein one or more (or none) of each component may be used as appropriate. Figure 5 It is broadly described how various system elements may be implemented in a relatively discrete or relatively more integrated manner.

[0090] Computer or hardware system 500 may represent the above Figures 1 to 4In the embodiment of the computer or hardware system described (i.e., wrist wearable devices 105, 205a-205p and 205, computing system 115, display screen 130, and user devices 150, 310a and 310b, etc.), the computer or hardware system 500 is shown to include hardware elements that can be electrically coupled via a bus 505 (or can communicate in other suitable ways). The hardware elements may include: one or more processors 510, including but not limited to one or more general-purpose processors and / or one or more special-purpose processors (e.g., microprocessors, digital signal processing chips, graphics acceleration processors, etc.); one or more input devices 515, which may include but are not limited to a mouse, a keyboard, etc.; and one or more output devices 520, which may include but are not limited to a display device, a printer, etc.

[0091] The computer or hardware system 500 may also include (and / or communicate with) one or more storage devices 525, which may include, but are not limited to, local and / or network accessible storage, and / or may include, but are not limited to, disk drives, drive arrays, optical storage devices, solid-state storage devices (e.g., random access memory (RAM) and / or read-only memory (ROM)), which may be programmable, flash-updatable, etc. Such storage devices may be used to implement any appropriate data storage, including, but not limited to, various file systems, database structures, and the like.

[0092] The computer or hardware system 500 may also include a communication subsystem 530, which may include, but is not limited to, a modem, a (wireless or wired) network card, an infrared communication device, a wireless communication device, and / or a chipset (e.g., a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a WWAN device, a cellular communication facility, etc.), etc. The communication subsystem 530 may allow data to be exchanged with a network (e.g., the network described below), with other computers or hardware systems, and / or with any other device described herein. In many embodiments, the computer or hardware system 500 will also include a working memory 535, which may include a RAM or ROM device as described above.

[0093] The computer or hardware system 500 may also include software elements, which are shown in the figure as currently located in the working memory 535, including an operating system 540, a device driver, an executable library, and / or other code, such as one or more application programs 545, which may include computer programs provided by various embodiments (including but not limited to virtual machine management programs, VMs, etc.), and / or may be designed to implement methods and / or configure systems provided by other embodiments as described herein. By way of example only, one or more programs described with respect to the above methods may be implemented as codes and / or instructions executable by a computer (and / or a processor within a computer). Then, in one aspect, such codes and / or instructions may be used to configure and / or adjust a general-purpose computer (or other device) to perform one or more operations according to the described methods.

[0094] The set of these instructions and / or codes may be encoded and / or stored on a non-transitory computer-readable storage medium, such as the storage device 525 described above. In some cases, the storage medium may be incorporated into a computer system, such as the system 500. In other embodiments, the storage medium may be separate from the computer system (i.e., removable media, such as a compact disc, etc.), and / or provided in the form of an installation package, such that the storage medium may be used to program, configure, and / or adjust a general-purpose computer through the instructions / code stored thereon. These instructions may be in the form of executable code, executable by the computer or hardware system 500, and / or may be in the form of source code and / or installable code, which, when compiled and / or installed on the computer or hardware system 500 (e.g., using various commonly available compilers, installers, compression / decompression utilities, etc.), is in the form of executable code.

[0095] Those skilled in the art will appreciate that a number of changes may be made depending on specific requirements. For example, custom hardware (e.g., programmable logic controllers, field programmable gate arrays, application specific integrated circuits, etc.) may also be used and / or specific elements may be implemented in hardware and / or software (including portable software, such as applets, etc.). In addition, connections to other computing devices (e.g., network input / output devices) may be employed.

[0096] As described above, in one aspect, some embodiments may employ a computer or hardware system (e.g., computer or hardware system 500) to perform methods according to various embodiments of the invention. According to one set of embodiments, in response to processor 510 executing one or more sequences of one or more instructions contained in working memory 535 (which may be incorporated into operating system 540 and / or other code, such as application 545), computer or hardware system 500 performs some or all of the procedures of these methods. These instructions may be read into working memory 535 from another computer-readable medium (e.g., one or more storage devices 525). By way of example only, execution of the sequences of instructions contained in working memory 535 may cause processor 510 to perform one or more procedures of the methods described herein.

[0097] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any medium that participates in providing data that causes a machine to operate in some manner. In embodiments implemented using a computer or hardware system 500, various computer-readable media may participate in providing instructions / codes to the processor 510 for execution, and / or may be used to store and / or carry such instructions / codes (e.g., as signals). In many embodiments, a computer-readable medium is a non-temporary, physical, and / or tangible storage medium. In some embodiments, a computer-readable medium may take a variety of forms, including but not limited to non-volatile media, volatile media, and the like. Non-volatile media include, for example, optical and / or magnetic disks, such as storage devices 525. Volatile media include, but are not limited to, dynamic memory, such as working memory 535. In some alternative embodiments, a computer-readable medium may take the form of a transmission medium, including but not limited to coaxial cables, copper wires, and optical fibers, including wires that make up the bus 505, and various components of the communication subsystem 530 (and / or the medium by which the communication subsystem 530 communicates with other devices). In an alternative set of embodiments, transmission media may also take the form of waves (including but not limited to radio waves, acoustic waves, and / or light waves, such as those generated during radio-wave and infrared data communications).

[0098] Common forms of physical and / or tangible computer readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, or any other magnetic medium, CD-ROMs, any other optical medium, punch cards, paper tapes, any other physical medium with a pattern of holes, RAM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cassettes, a carrier wave as described below, or any other medium from which a computer can read instructions and / or code.

[0099] Various forms of computer-readable media may be used to carry one or more sequences of one or more instructions to processor 510 for execution. By way of example only, the instructions may initially be carried on a magnetic disk and / or optical disk of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions as signals over a transmission medium for receipt and / or execution by the computer or hardware system 500. According to various embodiments of the present invention, these signals may be in the form of electromagnetic signals, acoustic signals, optical signals, etc., which are all examples of carrier waves that may encode instructions.

[0100] The communication subsystem 530 (and / or its components) typically receives the signal, and the bus 505 may then transfer the signal (and / or the data, instructions, etc. carried by the signal) to the working memory 535, from which the processor 505 retrieves and executes the instructions. The instructions received by the working memory 535 may optionally be stored on the storage device 525 before or after execution by the processor 510.

[0101] Although specific features and aspects have been described for certain embodiments, it will be appreciated by those skilled in the art that various modifications may be made. For example, the methods and processes described herein may be implemented using any combination of hardware components, software components, and / or hardware and software. In addition, although the various methods and processes described herein may be described for specific structures and / or functional components for ease of description, the methods provided by the various embodiments are not limited to any specific structures and / or functional architectures, but may be implemented on any suitable hardware, firmware, and / or software configurations. Similarly, although specific functions originate from specific system components, unless the context otherwise provides, the functions need not be limited thereto, and may be distributed in various other system components according to several embodiments.

[0102] In addition, although the procedures of the methods and processes described herein are described in a particular order for ease of description, various procedures may be reordered, added, and / or omitted according to various embodiments unless the context dictates otherwise. In addition, the procedures described for a method or process may be incorporated into other described methods or processes; similarly, system components described according to a specific structural architecture and / or for a system may be organized into alternative structural architectures and / or incorporated into other described systems. Therefore, although various embodiments are described in the form of specific features or non-specific features for ease of description and explanation of certain aspects of these embodiments, various components and / or features described herein for specific embodiments may be replaced, added, and / or deleted in other embodiments unless the context dictates otherwise. Therefore, although several embodiments are described above, it should be understood that the present invention is intended to cover all modifications and equivalents within the scope of the claims.

Claims

1. A wrist-type wearable control device, include: One or more first sensors, for detecting gestures of one or more fingers of the user when the wrist-type wearable control device is worn by the user without any fingers of the user contacting the wrist-type wearable control device; one or more second sensors for detecting movement of the wrist-type wearable control device relative to at least one of the three rotation axes; at least one processor; as well as a non-transitory computer-readable medium communicatively coupled to the at least one processor, the non-transitory computer-readable medium having computer software stored thereon, the computer software comprising a set of instructions that, when executed by the at least one processor, causes the wrist-based wearable control device to: analyzing the first sensor data to identify a first gesture corresponding to movement of one or more fingers of the user, When the wrist-type wearable control device is worn by the user, the first sensor data is received from the one or more first sensors disposed on the wrist-type wearable control device; analyzing second sensor data to detect movement of the wrist-type wearable control device relative to the at least one of the three rotation axes, the second sensor data being received from the one or more second sensors disposed on the wrist-type wearable control device when the wrist-type wearable control device is worn by the user; as well as Based on determining that the detected combination of the movement of the wrist-type wearable control device and the first gesture corresponds to a command based on the first gesture, a first instruction is generated based on the command based on the first gesture, and the generated first instruction is sent to a head-mounted device (HMD) or other user device with a camera input, wherein the command based on the first gesture is used to control a user interface (UI) of the HMD or the other user device, and the first instruction is used as an input for controlling the UI of the HMD or the other user device.

2. The wrist-type wearable control device according to claim 1, in, The wrist-type wearable control device includes one of a wristwatch-based wearable control device or a wristband-based wearable control device, each of which includes one or more band portions, and the band portions are linked to adjacent band portions by connectors in a manner that enables the wrist-type wearable control device to be detachably wrapped around the user's wrist.

3. The wrist-type wearable control device according to claim 2, in, At least one first sensor of the one or more first sensors and at least one second sensor of the one or more second sensors are disposed within each of the one or more belt portions.

4. The wrist-type wearable control device according to claim 2, in, The one or more first sensors and the one or more second sensors are each arranged in only one of the one or more belt portions, wherein the belt portion containing the one or more first sensors can be one of the following: the same belt portion containing the one or more second sensors, or a belt portion different from the belt portion containing the one or more second sensors.

5. The wrist-type wearable control device according to claim 2, in, At least one of the one or more first sensors is disposed in each of the one or more belt portions, while the one or more second sensors is disposed in only one of the one or more belt portions.

6. A wrist-type wearable control device according to any one of claims 1 to 5, in, The HMD or other user device with camera input includes one of the following: virtual reality (VR) goggles, augmented reality (AR) goggles, mixed reality (MR) goggles, VR-enabled glasses, AR-enabled glasses, MR-enabled glasses, a VR-enabled smartphone installed in a headset, an AR-enabled smartphone installed in a headset, an MR-enabled smartphone installed in a headset, a smart TV (TV) with a built-in camera, a smart TV with an external camera, a game console with a built-in camera, a game console with an external camera, a monitor with a built-in camera, a monitor with an external camera, or a projector with an external camera.

7. A wrist-type wearable control device according to any one of claims 1 to 6, in, The one or more first sensors each include at least one biosensor, the at least one biosensor including at least one of one or more photoplethysmography (PPG) sensors, one or more electromyography (EMG) sensors, one or more sound sensors, or one or more motion sensors, each sensor being used to detect a biometric feature corresponding to a soft tissue movement in or near a wrist of the user wearing the wrist-type wearable control device, wherein the soft tissue includes at least one of one or more muscles, one or more tendons, or one or more blood vessels in or near the wrist of the user.

8. The wrist-type wearable control device according to claim 7, in, The one or more PPG sensors are used to optically detect blood volume changes in the one or more blood vessels beneath the skin tissue at or near the wrist of the user corresponding to the movement of the one or more fingers of the user, wherein the one or more EMG sensors are used to detect electrical activities of the one or more muscles at or near the wrist of the user corresponding to the movement of the one or more fingers of the user, wherein the one or more sound sensors are used to detect sounds within the user's body corresponding to the movement of the one or more fingers of the user, wherein the one or more motion sensors are used to detect movement of the one or more muscles, the one or more tendons, and the one or more blood vessels corresponding to the movement of the one or more fingers of the user.

9. The wrist-type wearable control device according to any one of claims 1 to 8, in, The one or more second sensors each include at least one of: an inertial measurement unit (IMU) sensor or a gyroscope-based sensor.

10. The wrist-type wearable control device according to any one of claims 1 to 9, further comprising: include: one or more belt sections; as well as a plurality of outward-facing lights disposed on the one or more band portions, wherein the plurality of outward-facing lights are arranged in a predetermined pattern on the one or more band portions and serve as tracking points for one or more cameras of the HMD or the other user device to track translational movement of the wrist-type wearable control device along three axes relative to the HMD or the other user device, wherein tracking of the translational movement of the wrist-type wearable control device along the three axes is used as additional input for controlling the UI of the HMD or the other user device.

11. The wrist-type wearable control device according to claim 10, in, The plurality of outwardly facing lights include at least one of: a plurality of infrared (IR) light emitting diodes (LEDs) or a plurality of colored LEDs.

12. A wrist-type wearable control device according to any one of claims 1 to 11, in, The command based on the first gesture includes at least one of the following: a sliding-based command, a dragging-based command, a tap command, a double-click command, a pointing command, a pinch-based command, a grip-based command, a rotation command, a scrolling command, a pitch command, or a yaw command, wherein the first instruction includes at least one of the following: a scrolling command, a moving command, a selection command, a confirmation command, a highlighting command, a return command, a cancel command, a cursor moving command, or a navigation command.

13. A method, include: analyzing first sensor data using a computing system to identify a first gesture corresponding to movement of one or more fingers of a user, the first sensor data being received from one or more first sensors disposed on the wrist-type wearable control device when the wrist-type wearable control device is worn by the user, the one or more first sensors being used to detect the gesture of the one or more fingers of the user when the wrist-type wearable control device is worn by the user without any fingers of the user contacting the wrist-type wearable control device; analyzing second sensor data using a computing system to detect movement of the wrist-type wearable control device relative to at least one of three rotational axes, the second sensor data being received from one or more second sensors disposed on the wrist-type wearable control device when the wrist-type wearable control device is worn by the user; as well as Based on determining that the detected combination of the movement of the wrist-type wearable control device and the first gesture corresponds to a command based on the first gesture, a first instruction is generated based on the command based on the first gesture using a computing system, and the generated first instruction is sent to a head-mounted device (HMD) or other user device with a camera input, wherein the command based on the first gesture is used to control a user interface (UI) of the HMD or the other user device, and the first instruction is used as an input to control the UI of the HMD or the other user device.

14. The method according to claim 13, in, The computing system includes at least one of the following: one or more processors on the wrist-mounted wearable control device, one or more processors on a mobile device communicatively coupled to the wrist-mounted wearable control device, a machine learning system, an artificial intelligence (AI) system, a deep learning system, a neural network, a convolutional neural network (CNN), or a fully convolutional network (FCN).

15. The method according to claim 13 or 14, in, The wrist-type wearable control device includes one of a wristwatch-based wearable control device or a wristband-based wearable control device, each of which includes one or more band portions, and the band portions are linked to adjacent band portions by connectors in a manner that enables the wrist-type wearable control device to be detachably wrapped around the user's wrist.

16. The method according to any one of claims 13 to 15, in, The one or more first sensors each include at least one biosensor, the at least one biosensor including at least one of one or more photoplethysmography (PPG) sensors, one or more electromyography (EMG) sensors, one or more sound sensors, or one or more motion sensors, each sensor being used to detect a biometric feature corresponding to a soft tissue movement in or near a wrist of the user wearing the wrist-type wearable control device, wherein the soft tissue includes at least one of one or more muscles, one or more tendons, or one or more blood vessels in or near the wrist of the user.

17. The method according to claim 16, in, The one or more PPG sensors are used to optically detect blood volume changes in the one or more blood vessels beneath the skin tissue at or near the wrist of the user corresponding to the movement of the one or more fingers of the user, wherein the one or more EMG sensors are used to detect electrical activities of the one or more muscles at or near the wrist of the user corresponding to the movement of the one or more fingers of the user, wherein the one or more sound sensors are used to detect sounds within the user's body corresponding to the movement of the one or more fingers of the user, wherein the one or more motion sensors are used to detect movement of the one or more muscles, the one or more tendons, and the one or more blood vessels corresponding to the movement of the one or more fingers of the user.

18. The method according to any one of claims 13 to 17, in, The one or more second sensors each include at least one of: an inertial measurement unit (IMU) sensor or a gyroscope-based sensor.

19. The method according to any one of claims 13 to 18, in, The wrist-type wearable control device also includes one or more band portions and a plurality of outward-facing lights disposed on the one or more band portions, wherein the plurality of outward-facing lights are arranged on the one or more band portions in a predetermined pattern and serve as tracking points for one or more cameras of the HMD or the other user device to track the translational movement of the wrist-type wearable control device along three axes relative to the HMD or the other user device, wherein the tracking of the translational movement of the wrist-type wearable control device along the three axes is used as an additional input for controlling the UI of the HMD or the other user device.

20. The method according to claim 19, in, The plurality of outwardly facing lights include at least one of: a plurality of infrared (IR) light emitting diodes (LEDs) or a plurality of colored LEDs.

Images