Control of user interface (UI) for other devices based on smartwatch

Abstract

New tools and techniques are provided for enabling control of UIs of other devices based on a wrist wearable device (e.g., a smartwatch). In various embodiments, a computing system may analyze sensor data received from one or more first sensors disposed on the wrist wearable device when the wrist wearable device is worn by a user to detect motion of the wrist wearable device relative to at least one of the three axes of rotation. Based on determining that the detected motion of the wrist wearable device corresponds to a gesture-based command, the computing system may generate an instruction based on the gesture-based command and may send the generated first instruction to a user device separate from the wrist wearable device, the gesture-based command is used for controlling a user interface (UI) of the user equipment, and the instruction is used as an input for controlling the UI of the user equipment.

Description

Copyright Notice

[0001] Portions of the disclosure of this patent document contain material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. Technical Field

[0002] The present disclosure generally relates to methods, systems, and apparatus for implementing user interface (UI) control, and more specifically, to methods, systems, and apparatus for implementing a UI based on a wrist wearable device (e.g., a smart watch) to control other devices. Background Art

[0003] Gesture recognition on wrist-worn devices is becoming a popular perceptual computing UI, which allows wrist-worn devices to capture human gestures and interpret them as commands. There are two methods for UI input for traditional devices. One is to use another physical controller. The other is to use hand tracking. For the first method, a specific or dedicated controller is required to control the traditional device (e.g., head-mounted device (HMD) or smart TV, etc.). Such a controller can have a touchpad (such as Apple Remote controls, etc.), or can be equipped with an inertial measurement unit (IMU) module (such as a virtual reality (VR) controller, etc.), where the user can rotate the controller to move the cursor on the UI. The user can also press a physical button to trigger some specific control functions. For the second method, one or more cameras are installed on the HMD or other device, such as a fisheye camera on the HMD or an HD camera on a smart TV, which can track the user's three-degree-of-freedom ("3DOF") or six-degree-of-freedom ("6DOF") posture and recognize certain gestures as commands.

[0004] However, the first method requires a separate controller device, which increases the cost, and the user needs to physically contact the controller device, which means that the user's hand needs to be close to the controller device to operate. In order to use 3DOF tracking, the user needs to hold the controller device to interact. The second method requires the hand to be within the field of view of one or more cameras. Hand tracking not only requires the user's hand to face the camera, but also requires ideal lighting conditions. Although the visual solution does not require the user to hold any other device, it requires the user's hand to be clearly seen in the picture or video. In this case, hand tracking still has typical visual challenges such as motion blur and visual occlusion.

[0005] Therefore, a more robust and scalable solution is needed to achieve UI control. Summary of the invention

[0006] The technology of the present disclosure generally relates to tools and techniques for implementing user interface (UI) control, and more specifically, to methods, systems, and apparatus for implementing UI based on a wrist-based wearable device (e.g., a smart watch) to control other devices.

[0007] In one aspect, a method may include: using a computing system to analyze first sensor data to detect movement of a wrist wearable device relative to at least one of three rotation axes, when the wrist wearable device is worn by a user, the first sensor data is received from one or more first sensors set on the wrist wearable device; based on determining that the detected movement of the wrist wearable device corresponds to a command based on a first gesture, using the computing system to generate a first instruction based on the command based on the first gesture, and using the computing system to send the generated first instruction to a user device separated from the wrist wearable device, the command based on the first gesture is used to control a user interface (UI) of the above-mentioned user device, and the first instruction is used as an input for controlling the UI of the user device.

[0008] On the other hand, a wrist wearable device may include one or more first sensors, at least one processor, and a non-transitory computer-readable medium. The one or more first sensors are used to detect the movement of the wrist wearable control device relative to at least one of the three rotation axes. The non-transitory computer-readable medium is communicatively coupled to the at least one processor. The non-transitory computer-readable medium stores computer software, which includes an instruction set. When executed by at least one processor, the instruction set enables the wrist wearable device to: analyze first sensor data to detect the movement of the wrist wearable device relative to at least one of the three rotation axes, when the wrist wearable device is worn by a user, the first sensor data is received from one or more first sensors set on the wrist wearable device; based on determining that the detected movement of the wrist wearable device corresponds to a command based on a first gesture, a first instruction is generated based on the command based on the first gesture, and the generated first instruction is sent to a user device separated from the wrist wearable device, the command based on the first gesture is used to control the user interface (UI) of the user device, and the first instruction is used as an input to control the UI of the user device.

[0009] Various modifications and additions may be made to the embodiments discussed without departing from the scope of the present disclosure. For example, although the embodiments described above refer to particular features, the scope of the present disclosure also includes embodiments having different combinations of features and embodiments that do not include all of the above features.

[0010] The details of one or more aspects of the disclosure are set forth in the accompanying drawings and the following description. Other features, objects, and advantages of the techniques described in this disclosure will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The nature and advantages of particular embodiments may be further understood by reference to the remainder of the specification and the drawings, wherein the same reference numerals are used to refer to similar components. In some cases, a sub-label is associated with a reference numeral to denote one of multiple similar components. When a reference numeral is referenced without specifying an existing sub-label, it is intended to refer to all such similar components.

[0012] Figure 1 Schematic diagram of a system for implementing a UI for controlling other devices based on a wrist wearable device (eg, a smart watch) according to various embodiments.

[0013] Figures 2A to 2X Schematic diagrams of various non-limiting examples of wrist-type wearable control devices for controlling UIs of other devices according to various embodiments.

[0014] Figure 3A and Figure 3B Schematic diagram of various non-limiting examples of UIs for controlling other user devices using a wrist-type wearable control device according to various embodiments.

[0015] Figure 4 The present invention is a flowchart of a method for implementing a UI of other devices based on a wrist wearable device (eg, a smart watch) according to various embodiments.

[0016] Figure 5 is a block diagram of an example of a computer or system hardware architecture according to various embodiments. DETAILED DESCRIPTION

[0017] Overview

[0018] Various embodiments provide tools and techniques for implementing user interface (UI) control, and more specifically, provide methods, systems, and apparatus for implementing UI based on a wrist wearable device (e.g., a smart watch) to control other devices.

[0019] In various embodiments, the computing system may analyze the first sensor data to detect the movement of the wrist wearable device relative to at least one of the three rotation axes, and when the wrist wearable device is worn by the user, the first sensor data is received from one or more first sensors disposed on the wrist wearable device. Based on determining that the detected movement of the wrist wearable device corresponds to a command based on a first gesture, the computing system may generate a first instruction based on the command based on the first gesture, and may send the generated first instruction to a user device separated from the wrist wearable device, the command based on the first gesture being used to control a user interface (UI) of the above-mentioned user device, and the first instruction being used as an input for controlling the UI of the user device.

[0020] In some embodiments, the computing system may include at least one of the following: one or more processors on the wrist wearable device, one or more processors on a mobile device communicatively coupled to the wrist wearable device, a machine learning system, an artificial intelligence (AI) system, a deep learning system, a neural network, a convolutional neural network (CNN), or a fully convolutional network (FCN), etc. In some cases, the wrist wearable device may include one of the following: a smart watch, a wrist-based wearable control device, a wristband-based wearable control device, or other wrist wearable user devices, etc. In some cases, at least one first sensor may include at least one of an inertial measurement unit (IMU) sensor or a gyroscope-based sensor, etc. In some cases, the user device separate from the wrist wearable device may include at least one of the following: virtual reality (VR) goggles, augmented reality (AR) goggles, mixed reality (MR) goggles, VR-enabled glasses, AR-enabled glasses, MR-enabled glasses, a VR-enabled smartphone installed in a headset, an AR-enabled smartphone installed in a headset, an MR-enabled smartphone installed in a headset, a smart television (television, TV), a game console communicatively coupled to a display device, a monitor communicatively coupled to a display device, a projector communicatively coupled to a display device, a display device, a tablet computer, a laptop computer, a desktop computer, or a smart home appliance with a display screen, etc.

[0021] According to some embodiments, upward or downward movement of the wrist of a user wearing a wrist wearable device can correspond to movement relative to a first rotation axis ("X-axis"), wherein movement of the wrist of the user wearing the wrist wearable device to the right or left can correspond to movement relative to a second rotation axis ("Z-axis"), wherein the command based on the first gesture includes at least one of the following: an upward or downward gesture command based on movement relative to the X-axis, or a right or left gesture command based on movement relative to the Z-axis, wherein the first instruction may include at least one of the following: a scroll command, a move command, a highlight command, a move cursor command, or a navigation command, etc. In some cases, a longitudinal extension of the arm of the wrist of a user wearing the wrist wearable device can define an axis corresponding to a third rotation axis ("Y axis"), wherein the command based on the first gesture can also include the arm of the wrist of the user wearing the wrist wearable device pointing to a display screen displaying content from the user device, wherein the first instruction can include: based on at least one of an up or down gesture command based on movement relative to the X-axis or a right or left gesture command based on movement relative to the Z-axis, navigating a cursor, selecting a field, or highlighting one of the fields on a UI displayed on the display screen relative to the position on the display screen where the user's arm is pointing.

[0022] In some embodiments, movement of the wrist of a user wearing a wrist wearable device relative to an axis defined by the longitudinal extension of the user's arm can correspond to movement relative to a third rotation axis ("Y-axis"), wherein the command based on the first gesture may include: when the user's fingers are in a relaxed and loose state, at least one of a thumb-up gesture command, a thumb-down gesture command, or a thumb-flat gesture command based on movement relative to the Y-axis, wherein the first instruction may include one of a selection command, a confirmation command, a return command, or a cancel command, etc.

[0023] According to some embodiments, the computing system may analyze the second sensor data to identify a second gesture corresponding to the movement of one or more fingers of the user, and when the user wears the wrist wearable device, the second sensor data is received from one or more second sensors disposed on the wrist wearable device, and the one or more second sensors are used to detect the gesture of one or more fingers of the user without any fingers of the user contacting the wrist wearable control device. Based on determining that the second gesture corresponds to a command based on the second gesture, the computing system may generate a second instruction based on the command based on the second gesture, and send the generated second instruction to the user device, the command based on the second gesture is used to control the UI of the user device separated from the wrist wearable device, and the second instruction is used as an additional input for controlling the UI of the user device.

[0024] In some cases, the one or more second sensors may each include at least one biosensor, the at least one biosensor including at least one of one or more photoplethysmography (PPG) sensors, one or more electromyography (EMG) sensors, one or more sound sensors, or one or more motion sensors, each sensor for detecting a biometric feature corresponding to a soft tissue movement in or near a wrist of a user wearing the wrist wearable device, wherein the soft tissue may include at least one of one or more muscles, one or more tendons, or one or more blood vessels in or near the wrist of the user, and the like. In some cases, the one or more PPG sensors may be used to optically detect a change in blood volume in one or more blood vessels under the skin tissue at or near the wrist of the user corresponding to the movement of one or more fingers of the user. In some cases, the one or more EMG sensors may be used to detect the electrical activity of one or more muscles at or near the wrist of the user corresponding to the movement of one or more fingers of the user. In some cases, the one or more sound sensors may be used to detect sounds in the user's body corresponding to the movement of one or more fingers of the user. In some cases, the one or more motion sensors may be used to detect the movement of one or more muscles, one or more tendons, and one or more blood vessels corresponding to the movement of one or more fingers of the user. In some cases, the command based on the second gesture may include at least one of a tap command, a double-click command, a pinch-based command, or a grip-based command, wherein the second instruction includes at least one of a selection command, a confirmation command, a return command, or a cancel command, etc.

[0025] On the other hand, a wrist wearable device may include one or more first sensors, at least one processor, and a non-transitory computer-readable medium. The one or more first sensors are used to detect the movement of the wrist wearable control device relative to at least one of the three rotation axes. The non-transitory computer-readable medium is communicatively coupled to at least one processor. The non-transitory computer-readable medium stores computer software, which includes an instruction set. When executed by at least one processor, the instruction set enables the wrist wearable device to: analyze first sensor data to detect the movement of the wrist wearable device relative to at least one of the three rotation axes, when the wrist wearable device is worn by a user, the first sensor data is received from one or more first sensors set on the wrist wearable device; based on determining that the detected movement of the wrist wearable device corresponds to a command based on a first gesture, a first instruction is generated based on the command based on the first gesture, and the generated first instruction is sent to a user device separated from the wrist wearable device, the command based on the first gesture is used to control the user interface (UI) of the user device, and the first instruction is used as an input to control the UI of the user device.

[0026] In some embodiments, the wrist wearable device includes one of the following: a smart watch, a wrist-based wearable control device, a wristband-based wearable control device, or other wrist wearable user devices. In some cases, at least one first sensor may include at least one of an inertial measurement unit (IMU) sensor or a gyroscope-based sensor, etc. In some cases, the user device separated from the wrist wearable device may include at least one of the following: virtual reality (VR) goggles, augmented reality (AR) goggles, mixed reality (MR) goggles, VR-enabled glasses, AR-enabled glasses, MR-enabled glasses, VR-enabled smartphones installed in headsets, AR-enabled smartphones installed in headsets, MR-enabled smartphones installed in headsets, smart TVs, game consoles communicatively coupled to a display device, monitors communicatively coupled to a display device, projectors communicatively coupled to a display device, display devices, tablet computers, laptop computers, desktop computers, or smart home appliances with display screens, etc.

[0027] According to some embodiments, upward or downward movement of the wrist of a user wearing a wrist wearable device can correspond to movement relative to a first rotation axis ("X-axis"), wherein movement of the wrist of the user wearing the wrist wearable device to the right or left can correspond to movement relative to a second rotation axis ("Z-axis"), wherein the command based on the first gesture includes at least one of the following: an upward or downward gesture command based on movement relative to the X-axis, or a right or left gesture command based on movement relative to the Z-axis, wherein the first instruction may include at least one of the following: a scroll command, a move command, a highlight command, a move cursor command, or a navigation command, etc. In some cases, a longitudinal extension of the arm of the wrist of a user wearing the wrist wearable device can define an axis corresponding to a third rotation axis ("Y axis"), wherein the command based on the first gesture can also include the arm of the wrist of the user wearing the wrist wearable device pointing to a display screen displaying content from the user device, wherein the first instruction can include: based on at least one of an up or down gesture command based on movement relative to the X-axis or a right or left gesture command based on movement relative to the Z-axis, navigating a cursor, selecting a field, or highlighting one of the fields on a UI displayed on the display screen relative to the position on the display screen where the user's arm is pointing.

[0028] In some embodiments, the wrist wearable device may also include one or more second sensors, which are used to detect gestures of one or more fingers of the user without any fingers of the user contacting the wrist wearable control device, wherein, when executed by at least one processor, the instruction set may also enable the wrist wearable device to: analyze second sensor data to identify a second gesture corresponding to the movement of one or more fingers of the user, when the user wears the wrist wearable device, the second sensor data is received from one or more second sensors set on the wrist wearable device; based on determining that the second gesture corresponds to a command based on the second gesture, generate a second instruction based on the above-mentioned command based on the second gesture, and send the generated second instruction to the user device, the command based on the second gesture is used to control the UI of the user device separated from the wrist wearable device, and the second instruction is used as an additional input for controlling the UI of the user device.

[0029] In some cases, the one or more second sensors may each include at least one biosensor, the at least one biosensor including at least one of one or more photoplethysmography (PPG) sensors, one or more electromyography (EMG) sensors, one or more sound sensors, or one or more motion sensors, each sensor for detecting a biometric feature corresponding to a soft tissue movement in or near a wrist of a user wearing the wrist wearable device, wherein the soft tissue may include at least one of one or more muscles, one or more tendons, or one or more blood vessels in or near the wrist of the user, etc. In some cases, the command based on the second gesture may include at least one of a tap command, a double-click command, a pinch-based command, or a grip-based command, wherein the second instruction includes at least one of a selection command, a confirmation command, a return command, or a cancel command, etc.

[0030] In various aspects described herein, a wrist-worn wearable control device for controlling the UI of another device is provided, and three-degrees-of-freedom (3DOF) control of the UI of a user device is achieved by using motion detection-based UI control (using at least one first sensor or IMU sensor), in some cases combined with micro-gesture-based UI control (using one or more second sensors or biosensors) or similar methods.

[0031] These and other aspects of the system and method for implementing a UI based on a wrist wearable device (eg, a smart watch) to control other devices will be described in more detail in conjunction with the accompanying drawings.

[0032] The following detailed description shows several embodiments in more detail to enable those skilled in the art to practice these embodiments. The examples are only for illustrative purposes and are not intended to limit the scope of the present invention.

[0033] In the following description, for the purpose of explanation, many details are set forth in order to provide a thorough understanding of the embodiments. However, it is obvious to those skilled in the art that other embodiments of the present invention may be implemented without these details. In other examples, some structures and devices are shown in block diagram form. Several embodiments are described herein, and although various features are derived from different embodiments, it should be understood that the features described for one embodiment may also be combined with other embodiments. However, in the same way, any single or multiple features of the embodiments should not be considered as essential to each embodiment of the present invention, and other embodiments of the present invention may omit these features.

[0034] Unless otherwise stated, all numbers used herein to express quantities, sizes, etc. are to be understood as being modified in all cases by the term "about". In this application, unless otherwise expressly stated, the use of the singular includes the plural, and the use of the terms "and" and "or" means "and / or" unless otherwise stated. In addition, the use of the term "include" as well as other forms such as "include" and "comprising" are to be considered non-exclusive. In addition, unless otherwise expressly stated, terms such as "element" or "component" include both elements and components consisting of one unit and elements and components consisting of a plurality of units.

[0035] The various embodiments described herein - while (in some cases) embodying software products, computer-implemented methods and / or computer systems - represent tangible, concrete improvements over the prior art fields, including but not limited to wrist-worn wearable device technology, gesture control technology, gesture control technology for wrist-worn wearable devices, biosensor technology for gesture control, motion tracking technology for gesture control, user interface (UI) control technology, UI control technology for wrist-worn wearable devices, 3DOF control technology for UI, and / or the like. In other aspects, some embodiments may improve the functionality of a user device or system (e.g., a wrist wearable device, a gesture control system, a gesture control system for a wrist wearable device, a biosensor system for gesture control, a motion tracking system for gesture control, a UI control system, a UI control system for a wrist wearable device, a 3DOF control system for UI, etc.) itself, for example, by: using a computing system to analyze first sensor data to detect movement of the wrist wearable device relative to at least one of three rotation axes, and when the wrist wearable device is worn by a user, the first sensor data is received from one or more first sensors disposed on the wrist wearable device; based on determining that the detected movement of the wrist wearable device corresponds to a command based on a first gesture, using a computing system to generate a first instruction based on the command based on the first gesture, and using the computing system to send the generated first instruction to a user device separated from the wrist wearable device, the command based on the first gesture is used to control the user interface (UI) of the above-mentioned user device, and the first instruction is used as input to control the UI of the user device; and the like.

[0036] Specifically, in the event that any abstract concepts exist in various embodiments, these concepts can be implemented as described herein by devices, software, systems, and methods involving new functions (e.g., steps or operations), such as using 3DOF tracking of a wrist wearable device (based on the movement of the wrist wearable device detected using an IMU sensor, etc.) to generate and move a cursor or other object on a UI of another user device (e.g., selecting a field, highlighting a field, etc.); using gesture tracking (and 3DOF tracking) to enable a user to interact with objects within the UI (e.g., using biosensors, cameras, etc. to recognize gesture-based commands such as pinching, clenching, etc., which represent control commands); and so on, and these new functions go beyond simple traditional computer processing operations. These functions can produce tangible results outside of the implemented computer system, including (by way of example only) an optimized wrist-worn wearable control device that enables 3DOF UI control of another user device by using motion detection (using an IMU sensor), in some cases in combination with micro-gestures of fingers on the arm on which the wrist-worn wearable control device is worn, which micro-gestures are detected and recognized using biosensors to monitor, track, and recognize gesture-based commands, and generate UI commands for controlling the UI of another user device, where at least some of the UI commands can be observed or measured by the user, the wrist-worn wearable device manufacturer, the user device manufacturer, and / or the universal remote control manufacturer.

[0037] Some embodiments

[0038] The embodiments shown in the drawings will now be described. Figures 1 to 5 Some features of methods, systems, and devices for implementing user interface (UI) control are shown, and more specifically, methods, systems, and devices for implementing the UI of controlling other devices based on a wrist wearable device (e.g., a smart watch) as described above are shown. Figures 1 to 5 The methods, systems, and apparatus shown refer to examples of different embodiments including various components and steps, which components and steps may be considered alternatives to each other or may be used in combination with each other in various embodiments. Figures 1 to 5 The descriptions of the methods, systems, and devices shown in the present invention are for illustration purposes only and should not be considered to limit the scope of the different embodiments.

[0039] With reference to the accompanying drawings, Figure 1 A schematic diagram of a system 100 for implementing a UI based on a wrist wearable device (eg, a smart watch) to control other devices according to various embodiments is shown.

[0040] exist Figure 1In a non-limiting embodiment of the present invention, the system 100 may include a wrist wearable device 105, which is used to be worn on the wrist of the user 110. In some cases, the wrist wearable device 105 may include but is not limited to a smart watch, a wrist wearable display device, a wrist wearable control device, or one of other wrist wearable user devices. According to some embodiments, the wrist wearable control device may include but is not limited to one of a wristwatch-based wearable control device or a wristband-based wearable control device, each of which includes one or more band portions, which can be connected to adjacent band portions by connectors, and the connection method is configured to detachably wrap the wrist wearable control device around the wrist of the user 110 (for example, as Figure 2A-Figure 2X , Figure 3A and Figure 3B The wrist wearable device 105 may include, but is not limited to, a computing system 115, one or more second sensors 120, one or more first sensors 125, a display screen 130 (optional), a communication system 135, and one or more outward-facing lights 140a-140n (optional; collectively referred to as "outward-facing lights 140" or "lights 140", etc.). According to some embodiments, the outward-facing lights 140 may include, but is not limited to, at least one of the following: a plurality of infrared (IR) light emitting diodes (LEDs) or a plurality of color LEDs (e.g., LEDs that emit light in the visible spectrum, each LED being monochromatic or multi-color variable), etc.

[0041] In some embodiments, the computing system 115 may include, but is not limited to, at least one of the following: one or more processors on the wrist wearable device (e.g., processor 115a, etc.), one or more processors on a mobile device communicatively coupled to the wrist wearable device (e.g., a processor (not shown) on a mobile device 145 communicatively coupled to the wrist wearable device 105 via the communication system 135) (e.g., a processor (not shown) on a mobile device 145 communicatively coupled to the wrist wearable device 105 via the communication system 135). Figure 1 The mobile device 145 may include, but is not limited to, a mobile device 145 ...

[0042] Merely by way of example, in some cases, display device 130 may include, but is not limited to, at least one of a touch screen display device, a non-touch screen display device, a projection-based display device, a holographic display device, etc. In some cases, communication system 135 may include a wireless communication device capable of communicating using a protocol, including but not limited to at least one of a Bluetooth communication protocol, a WiFi communication protocol, or other 802.11 communication protocol suites, a ZigBee communication protocol, a Z-wave communication protocol, or other 802.15.4 communication protocol suites, a cellular communication protocol (e.g., 3G, 4G, 4G LTE, 5G, etc.), or other suitable communication protocols, etc.

[0043] In some embodiments, the system 100 may also include a user device 150, which may include, but is not limited to, at least one of one or more processors 150a, a transceiver 150b, one or more cameras 150c (optional), or a display device 150d, etc. In some cases, the transceiver 150b and / or the one or more cameras 150c may each be integrated within the user device 150, or external to the user device 150 but communicatively coupled to the user device 150, or partially integrated within the user device 150 and partially external to the user device 150 but communicatively coupled to the user device 150, etc. In some cases, the camera 150c may have a field of view (FOV) 155 that, when directed toward the wrist wearable device 105, may be used to monitor or track the position and / or orientation of the wrist wearable device 105 (in some cases, performed at least in part based on an outward-facing light 140 (if any) disposed on the wrist wearable device 105). In some cases, UI 160 can be displayed or presented on display device 150d. In some cases, wrist wearable device 105 can be communicatively coupled to user device 150 via communication system 135 (e.g., Figure 1In some cases, the user device 150 may include, but is not limited to, a head mounted device (HMD) or other user devices. In some cases, the HMD may include, but is not limited to, one of the following: virtual reality (VR) goggles, augmented reality (AR) goggles, mixed reality (MR) goggles, VR-enabled glasses, AR-enabled glasses, MR-enabled glasses, VR-enabled smartphones installed in headsets, AR-enabled smartphones installed in headsets, MR-enabled smartphones installed in headsets, etc. In some cases, other user devices may include, but are not limited to, a smart TV (TV), a game console communicatively coupled to a display device, a monitor communicatively coupled to a display device, a projector communicatively coupled to a display device, a display device, a tablet computer, a laptop computer, a desktop computer, or a smart home appliance with a display screen, etc.

[0044] According to some embodiments, the one or more second sensors 120 may each include at least one biosensor. In some cases, the at least one biosensor may include but is not limited to the following one or more photoplethysmography (PPG) sensors, one or more electromyography (EMG) sensors, one or more sound sensors, or at least one of one or more motion sensors, each of which is used to detect a biometric feature corresponding to a soft tissue movement in or near the wrist of a user wearing a wrist wearable device. In some cases, the soft tissue may include but is not limited to at least one of one or more muscles, one or more tendons, or one or more blood vessels in or near the wrist of the user. In some cases, one or more PPG sensors may be used to optically detect changes in blood volume in one or more blood vessels under the skin tissue at or near the wrist of the user corresponding to the movement of one or more fingers of the user. In some cases, one or more EMG sensors may be used to detect the electrical activity of one or more muscles at or near the wrist of the user corresponding to the movement of one or more fingers of the user. In some cases, one or more sound sensors (including but not limited to microphones, etc. in some cases) may be used to detect sounds in the user's body corresponding to the movement of one or more fingers of the user. In some cases, one or more motion sensors (including but not limited to microphones, etc. in some cases) may be used to detect the movement of one or more muscles, one or more tendons, and one or more blood vessels corresponding to the movement of one or more fingers of the user.

[0045] By way of example only, in some cases, the one or more first sensors 125 may each include, but are not limited to, at least one of an inertial measurement unit (IMU) sensor or a gyroscope-based sensor, or a similar sensor, etc. In some cases, at least one of the one or more first sensors 125 may be disposed in only one belt portion, or in each of the one or more belt portions, without the second sensor 120 in any belt portion. Alternatively, at least one of the one or more second sensors 120 and at least one of the one or more first sensors 125 may be disposed in each of the one or more belt portions. Alternatively, the one or more second sensors 120 and the one or more first sensors 125 may each be disposed in only one of the one or more belt portions, wherein the belt portion containing the one or more second sensors 120 may be the same belt portion containing the one or more first sensors 125, or a different belt portion from the belt portion containing the one or more first sensors 125. Alternatively, at least one of the one or more second sensors 120 may be disposed within each of the one or more belt portions, while the one or more first sensors 125 may be disposed within only one of the one or more belt portions.

[0046] In some cases, outward-facing lights 140 can be disposed on one or more band portions, and in some cases, can be arranged in a predetermined pattern on one or more band portions, and can be used as tracking points for one or more cameras 150 c of user device 150 (as captured within FOV 155) to track translational motion of wrist wearable device 105 along three axes relative to user device 150. In some cases, tracking of translational motion of the wrist wearable device along three axes can serve as an additional input for controlling UI 160 of user device 150.

[0047] In operation, the computing system 115, the processor 115a, the signal processing system 115b, the AI ​​system 115c, the other computing systems 115d, and / or the mobile device 145, etc. (collectively referred to as "computing systems", etc.) can analyze the first sensor data to detect the movement of the wrist wearable device relative to at least one of the three rotation axes (e.g., the rotation and / or linear movement of the wrist wearable device 105, etc.), when the wrist wearable device is worn by a user (e.g., user 110, etc.), the first sensor data is received from one or more first sensors (e.g., the first sensor or the IMU sensor 125, etc.) disposed on the wrist wearable device. Based on determining that the detected movement of the wrist wearable device corresponds to a command based on the first gesture, the computing system can generate a first instruction based on the command based on the first gesture, and can send the generated first instruction to a user device (e.g., user device 150, etc.) separated from the wrist wearable device, the command based on the first gesture is used to control the user interface (UI) of the user device (e.g., UI 160, etc.), and the first instruction is used as an input for controlling the UI of the user device.

[0048] According to some embodiments, the upward or downward movement of the wrist of the user wearing the wrist wearable device may correspond to a movement relative to a first rotation axis ("X axis") (e.g., a rotation R relative to the X axis as shown in FIG. 3 ). x (also called pitch, etc.). Movement of the wrist of the user wearing the wrist wearable device to the right or left may correspond to movement relative to a second rotation axis ("Z axis") (e.g., as shown in FIG. 3 , a rotation R relative to the Z axis) z (also known as yaw), etc.). In this case, the command based on the first gesture may include, but is not limited to, at least one of the following: an up or down gesture command based on movement relative to the X-axis, or a right or left gesture command based on movement relative to the Z-axis. In some cases, the first instruction may include, but is not limited to, at least one of the following: a scroll command, a move command, a highlight command, a move cursor command, or a navigation command, etc. In some cases, the longitudinal extension of the arm of the user wearing the wrist wearable device may define an axis corresponding to the third rotation axis ("Y axis"). In this case, the command based on the first gesture may also include the arm of the user wearing the wrist wearable device pointing to a display screen (e.g., display device 150d, etc.) displaying content from a user device (e.g., user device 150, etc.). In some cases, the first instruction may include, but is not limited to, at least one of an up or down gesture command based on movement relative to the X-axis or a right or left gesture command based on movement relative to the Z-axis, relative to the position on the display screen to which the user's arm is pointing, navigating a cursor, a selection field, or a highlight field on a UI displayed on the display screen.

[0049] In some embodiments, movement of the wrist of a user wearing the wrist wearable device relative to an axis defined by the longitudinal extension of the user's arm can correspond to movement relative to a third rotational axis ("Y axis") (e.g., as shown in FIG. 3 , a rotation R relative to the Y axis). y (also called scrolling, etc.). In this case, when the user's finger is in a relaxed and unfastened state, based on the movement relative to the Y axis, the command based on the first gesture may include but is not limited to at least one of a thumb up gesture command, a thumb down gesture command, or a thumb flat gesture command. In some cases, the first instruction may include but is not limited to one of a selection command, a confirmation command, a return command, or a cancel command, etc.

[0050] As an alternative or additional to the thumbs up, thumbs down, and thumbs flat gesture commands, commands based on the first gesture can be based on the degree of supination or pronation of the forearm connected to the wrist of the user wearing the wrist wearable device, the supination or pronation degree being detected by a first sensor or IMU sensor (e.g., first sensor or IMU sensor 125, etc.) that tracks the movement of the wrist wearable device. Here, forearm supination may refer to when the user's arm is hanging by the side or when the forearm is raised from the arm-hanging position (regardless of whether the fingers are open in a palm or clenched into a fist), the forearm is rotated so that the ventral side of the forearm (or the palm) faces forward and / or upward relative to the user's body in a sitting or standing position, and forearm pronation may refer to when the user's arm is hanging by the side or when the forearm is raised from the arm-hanging position (regardless of whether the fingers are open in a palm or clenched into a fist), the forearm is rotated so that the ventral side of the forearm (or the palm) faces backward and / or downward relative to the user's body in a sitting or standing position, and the neutral position of the forearm may refer to when the user's arm is hanging by the side or when the forearm is raised from the arm-hanging position (regardless of whether the fingers are open in a palm or clenched into a fist), the forearm is rotated so that the ventral side of the forearm (or the palm) faces the sagittal plane or the median plane, which divides the body into left and right parts along the midline of the body and is located in the middle or halfway between the pronated position and the supinated position.

[0051] According to some embodiments, the computing system may analyze the second sensor data to identify a second gesture corresponding to the movement of one or more fingers of the user, and when the user wears the wrist wearable device, the second sensor data is received from one or more second sensors disposed on the wrist wearable device, and the one or more second sensors are used to detect the gesture of one or more fingers of the user without any fingers of the user contacting the wrist wearable control device. Based on determining that the second gesture corresponds to a command based on the second gesture, the computing system may generate a second instruction based on the command based on the second gesture, and may send the generated second instruction to a user device separated from the wrist wearable device, the command based on the second gesture being used to control the UI of the user device, and the second instruction being used as an additional input for controlling the UI of the user device.

[0052] In some cases, the command based on the second gesture may include, but is not limited to, at least one of the following: a tap command, a double-click command, a pinch-based command, or a grip-based command. In some cases, the second instruction may include, but is not limited to, at least one of the following: a selection command, a confirmation command, a return command, or a cancel command, etc. Alternatively or additionally, the command based on the second gesture may further include, but is not limited to, at least one of the following: a slide-based command, a drag-based command, or a pointing command, etc. In some cases, the second instruction may further include, but is not limited to, at least one of the following: a scroll command, a move command, or a highlight command, etc.

[0053] According to some embodiments, for a slide-based command gesture (or finger slide) command (not shown), a user can start from a relaxed position of the finger (also referred to herein as an "initial position"; e.g., Figure 3A and Figure 3B The command may be defined as a swipe up command in which the fingers of the user's hand (e.g., five fingers) are naturally stretched and loosened or relaxed (i.e., there is no muscle tension to straighten the fingers, nor is there muscle tension to bend the fingers (e.g., making a fist, etc.), or similar postures). For a swipe up command, a finger (e.g., but not limited to an index finger, etc.) may be lifted relative to other fingers from an initial or relaxed posture or from a lowered position (as described below), for example, at a moderate speed (the moderate speed in this article may refer to a speed greater than the speed of slow, deliberate movement of the finger positioning, etc.), with the other fingers remaining in the initial or relaxed posture. In some cases, the finger lifting may be performed quickly (e.g., according to definitions such as blurred vision of the finger due to movement speed). For a swipe down command, a finger may be lowered relative to other fingers from an initial or relaxed posture or a lifted position (as described above) at a moderate speed, with the other fingers remaining in the initial or relaxed posture. In some cases, the finger lowering may be performed quickly.

[0054] In some embodiments, starting from the initial posture, if the user lifts the index finger, the computing system may recognize this micro-gesture as a swipe up command. Conversely, starting from the initial posture, if the user pulls down the index finger, the computing system may recognize this micro-gesture as a swipe down command. Taking the page navigation function as an example, the user can use an activation gesture (such as a pinch command) to open a page. Here, the pinch command or pinch-based gesture command may include the thumb and index finger touching or pressing together, while the other fingers of the hand (not shown) are almost or completely motionless. The user can then pull down the index finger to trigger the swipe down function to navigate the page downward. If the user lifts the index finger above the initial posture, this gesture will trigger the swipe up function.

[0055] According to some embodiments, for a drag-based gesture (or finger drag) command (not shown), the user may start from an initial or relaxed position of the finger, similar to the slide-based command gesture (or finger slide) command described above. For an upward drag command, a finger (such as but not limited to an index finger, etc.) may be lifted relative to other fingers from an initial or relaxed position or from a lowered position (as described below), for example, at a slow speed (here, slow speed may refer to a speed slower than the above-mentioned medium speed, etc.), and the other fingers remain in the initial or relaxed position. For a downward drag command, a finger may be lowered relative to other fingers from an initial or relaxed position or a lifted position (as described above), for example, at a slow speed, and the other fingers remain in the initial or relaxed position.

[0056] If the UI (e.g., UI 160, etc.) only provides one dimension for the user to scroll or drag within the UI (regardless of whether the wrist wearable device includes the first sensor or the IMU sensor (e.g., the first sensor or the IMU sensor 125, etc.)), the lifting and lowering of the finger (as described above with respect to the slide-based command and the drag-based command) can be recognized as a micro-gesture that causes the computing system (e.g., computing system 115, etc.) to cause the UI to perform one of scrolling or dragging in a first direction (in response to the finger lifting) or scrolling or dragging in a second direction (in response to the finger lowering). For example, if the UI only provides vertical movement, the first direction can cause the computing system to cause the UI to perform one of a scroll up function or an upward drag function, while the second direction can cause the computing system to cause the UI to perform one of a scroll down function or a downward drag function, and so on. Similarly, if the UI only provides horizontal movement, the first direction may cause the computing system to cause the UI to perform one of a left scroll function or a left drag function (if the left hand is used; or a right scroll function or a right drag function if the right hand is used), while the second direction may cause the computing system to cause the UI to perform one of a right scroll function or a right drag function (if the left hand is used; or a left scroll function or a left drag function if the right hand is used), and so on.

[0057] If a UI (e.g., UI 160, etc.) provides two dimensions for a user to scroll or drag within the UI, and if the wrist wearable device includes a first sensor or an IMU sensor (e.g., first sensor or IMU sensor 125, etc.), then the lifting and lowering of a finger (as described above with respect to slide-based commands and drag-based commands) combined with the movement of the user's wrist (detected by the first sensor or IMU sensor 125, etc.) can be recognized as a micro-gesture that causes a computing system (e.g., computing system 115, etc.) to cause the UI to perform one of scrolling or dragging, the direction of which is based on the degree of pronation or supination of the forearm connected to the wrist of the user wearing the wrist wearable device, which is detected by the first sensor or IMU sensor that tracks the movement of the wrist wearable device.

[0058] In a non-limiting example, if the forearm is pronated (or fully pronated), the lifting and lowering of the fingers may be recognized as a micro-gesture that causes the computing system to cause the UI to perform one of scrolling or dragging in a vertical direction (i.e., up or down, similar to the one-dimensional vertical motion embodiment described above), regardless of whether the user has an elbow straight, an elbow bent and the forearm is beside the torso, or an elbow bent and the forearm is in front of the torso. Similarly, if the forearm is in a neutral position such that the ventral side of the forearm (or palm) faces the sagittal or median plane, the lifting and lowering of the fingers may be recognized as a micro-gesture that causes the computing system to cause the UI to perform one of scrolling or dragging in a horizontal direction (i.e., left or right, similar to the one-dimensional horizontal motion embodiment described above), regardless of whether the user has an elbow straight, an elbow bent and the forearm is beside the torso, or an elbow bent and the forearm is in front of the torso. If the forearm is between the pronated and supinated positions (but not in the pronated or neutral position), the lifting and lowering of the fingers can be recognized as a micro-gesture that causes the computing system to cause the UI to perform one of a diagonal scroll or drag based on the degree of rotation between the pronated and supinated positions, regardless of whether the user has their elbows straight, their elbows bent and their forearms next to their torsos, or their elbows bent and their forearms in front of their torsos.

[0059] In some embodiments, a machine learning system (e.g., AI system 115c or other computing system 115d, etc.) can be used to further enhance the operation of the wrist wearable device by learning and fine-tuning the detection of micro-gestures, motion-based gesture detection, translation-based gesture detection, and / or instances of user-intended gesture control (as compared to regular motions that are not gesture-controlled motions), etc., to adapt to user use over time, etc.

[0060] 2 to 3. Figure 4 These and other functions of system 100 (and its components) are described in further detail.

[0061] Figures 2A to 2X (collectively referred to as "FIG. 2") are schematic diagrams illustrating various non-limiting examples 200 and 200' of wrist-type wearable control devices for controlling UIs of other devices according to various embodiments. Figures 2A to 2L Various non-limiting examples 200 of wristband-based wearable control devices 205a - 2051 are depicted respectively. Figure 2M to Figure 2X Various non-limiting examples 200 ′ of wristwatch-based wearable control devices 205 m - 205 x are depicted.

[0062] In FIG. 2 , the wristband-based wearable control devices 205a-205l or the wristwatch-based wearable control devices 205m-205x (these two groups of devices are collectively referred to as “wrist-type wearable control devices 205”, etc.) may each include, but are not limited to, a plurality of band portions 210 and a plurality of connectors 215. Each band portion 210 may include, but are not limited to, at least one of the following: one or more second sensors or biosensors 220 (similar to Figure 1 a second sensor or biosensor 120, etc.), at least one first sensor or IMU sensor 225 (similar to Figure 1 a first sensor or IMU sensor 125, etc.), one or more outward-facing lights 230 (similar to Figure 1 one or more outward-facing lights 140a-140n, etc.), etc.), or may include a belt portion that does not have any of the second sensor or biosensor 220, the first sensor or IMU sensor 225, or the outward-facing lights 230, etc.

[0063] For example, the wristband-based wearable control devices 205a-205l may include, but are not limited to: (a) a wristband-based wearable control device 205a having a plurality of band portions 210a, each band portion 210a including one or more second sensors or biosensors 220 and at least one first sensor or IMU sensor 225 (e.g., Figure 2A (b) a wristband-based wearable control device 205b having a plurality of band portions 210b, each band portion 210b including one or more second sensors or biosensors 220, at least one first sensor or IMU sensor 225, and one or more outward-facing lights 230 (such as Figure 2B (c) a wristband-based wearable control device 205c having a plurality of band portions 210c and a single band portion 210a, each band portion 210c including one or more second sensors or biosensors 220, and a single band portion 210a including one or more second sensors or biosensors 220 and at least one first sensor or IMU sensor 225 (as shown in FIG. Figure 2C (d) a wristband-based wearable control device 205d having a plurality of band portions 210d and a single band portion 210b, each band portion 210d including one or more second sensors or biosensors 220 and one or more outward-facing lights 230, and a single band portion 210b including one or more second sensors or biosensors 220, at least one first sensor or IMU sensor 225, and one or more outward-facing lights 230 (as shown in FIG. Figure 2D(e) a wristband-based wearable control device 205e having a plurality of band portions 210e and a single band portion 210a, the band portion 210e not including any second sensor or biosensor 220, first sensor or IMU sensor 225, or outward-facing light 230, the single band portion 210a including one or more second sensors or biosensors 220 and at least one first sensor or IMU sensor 225 (as shown in FIG. Figure 2E (f) a wristband-based wearable control device 205f having a plurality of band portions 210f and a single band portion 210b, each band portion 210f including one or more outwardly facing lights 230, and a single band portion 210b including one or more second sensors or biosensors 220, at least one first sensor or IMU sensor 225, and one or more outwardly facing lights 230 (as shown in FIG. Figure 2F (g) a wristband-based wearable control device 205g having a single band portion 210g, a (separate) single band portion 210c, and a plurality of band portions 210e, band portion 210g including at least one first sensor or IMU sensor 225, band portion 210c including one or more second sensors or biosensors 220, and band portion 210e not including any second sensor or biosensor 220, first sensor or IMU sensor 225, or outward-facing light 230 (as shown in FIG. Figure 2G (h) a wristband-based wearable control device 205h having a single band portion 210h, a (separate) single band portion 210d, and a plurality of band portions 210f, the band portion 210h including at least one first sensor or IMU sensor 225 and one or more outward-facing lights 230, the band portion 210d including one or more second sensors or biosensors 220 and one or more outward-facing lights 230, each band portion 210f including one or more outward-facing lights 230 (as shown in FIG. Figure 2H (i) a wristband-based wearable control device 205i having a plurality of strap portions 210g, wherein the strap portions 210g include at least one first sensor or IMU sensor 225 (eg, Fig.2I (j) a wristband-based wearable control device 205j having a plurality of strap portions 210h, the strap portions 210h including at least one first sensor or IMU sensor 225 and one or more outward-facing lights 230 (such as Figure 2J (k) a wristband-based wearable control device 205k having a single band portion 210g and a plurality of band portions 210e, band portion 210g including at least one first sensor or IMU sensor 225, band portion 210e not including any second sensor or biosensor 220, first sensor or IMU sensor 225, or outward-facing light 230 (such as Figure 2K (l) a wristband-based wearable control device 205l having a single band portion 210h and a plurality of band portions 210f, the band portion 210h including at least one first sensor or IMU sensor 225 and one or more outward-facing lights 230, each band portion 210f including one or more outward-facing lights 230 (such as Figure 2L shown); and so on.

[0064] Each band portion 210 can be connected to an adjacent band portion 210 via a connector 215, and the connection method is configured to removably wrap the wrist-type wearable control device 205 around the user's wrist when the wrist-type wearable control device 205 is worn by the user (e.g., user 110, etc.). In some cases, the connector 215 may include an elastic-based connector that allows the band portions to be pulled apart from each other to slide over and put on the user's hand and wrist (or slide over and take off the user's hand and wrist). Alternatively, the connector 215 may include a plurality of links and buckles (e.g., made of metal, plastic, or other solid materials), etc. Although Figures 2A to 2L 2, but at least one band portion 210 may also include, but is not limited to, a computing system (similar to Figure 1 computing system 115, etc.) or a communication system (similar to Figure 1 at least one of the communication systems 135, etc.

[0065] Similarly, the wristwatch-based wearable control devices 205m-205x may include, but are not limited to: (m) a wristwatch-based wearable control device 205m having a watch portion 235 and a plurality of band portions 210a, each band portion 210a including one or more second sensors or biosensors 220 and at least one first sensor or IMU sensor 225 (e.g., Figure 2M (n) a wristwatch-based wearable control device 205n having a watch portion 235 and a plurality of band portions 210b, each band portion 210b including one or more second sensors or biosensors 220, at least one first sensor or IMU sensor 225, and one or more outward-facing lights 230 (such as Figure 2N (o) a wristwatch-based wearable control device 205o having a watch portion 235, a plurality of band portions 210c, and a single band portion 210a, each band portion 210c including one or more second sensors or biosensors 220, and a single band portion 210a including one or more second sensors or biosensors 220 and at least one first sensor or IMU sensor 225 (as shown in Fig.2O(p) a wristwatch-based wearable control device 205p having a watch portion 235, a plurality of band portions 210d, and a single band portion 210b, each band portion 210d including one or more second sensors or biosensors 220 and one or more outward-facing lights 230, and a single band portion 210b including one or more second sensors or biosensors 220, at least one first sensor or IMU sensor 225, and one or more outward-facing lights 230 (as shown in FIG. Figure 2P (q) a wristwatch-based wearable control device 205q having a watch portion 235, a plurality of band portions 210e, and a single band portion 210a, the band portion 210e not including any second sensor or biosensor 220, first sensor or IMU sensor 225, or outward-facing light 230, the band portion 210a including one or more second sensors or biosensors 220 and at least one first sensor or IMU sensor 225 (as shown in FIG. 1 ); Figure 2Q (r) a wristwatch-based wearable control device 205r having a watch portion 235, a plurality of band portions 210f, and a single band portion 210b, each band portion 210f including one or more outwardly facing lights 230, and band portion 210b including one or more second sensors or biosensors 220, at least one first sensor or IMU sensor 225, and one or more outwardly facing lights 230 (as shown in FIG. Figure 2R (s) a wristwatch-based wearable control device 205s having a watch portion 235, a single band portion 210g, a (separate) single band portion 210c, and a plurality of band portions 210e, band portion 210g including at least one first sensor or IMU sensor 225, band portion 210c including one or more second sensors or biosensors 220, and band portion 210e not including any second sensor or biosensor 220, first sensor or IMU sensor 225, or outward-facing light 230 (as shown in FIG. Figure 2S (t) a wristwatch-based wearable control device 205t having a watch portion 235, a single band portion 210h, a (separate) single band portion 210d, and a plurality of band portions 210f, the band portion 210h including at least one first sensor or IMU sensor 225 and one or more outward-facing lights 230, the band portion 210d including one or more second sensors or biosensors 220 and one or more outward-facing lights 230, each band portion 210f including one or more outward-facing lights 230 (as shown in FIG. Figure 2T (u) a wristwatch-based wearable control device 205u having a watch portion 235 and a plurality of strap portions 210g, wherein the strap portion 210g includes at least one first sensor or IMU sensor 225 (such as Figure 2U(v) a wristwatch-based wearable control device 205v having a watch portion 235 and a plurality of strap portions 210h, the strap portions 210h including at least one first sensor or IMU sensor 225 and one or more outward-facing lights 230 (such as Figure 2V (W) a wristwatch-based wearable control device 205w having a watch portion 235, a single band portion 210g, and a plurality of band portions 210e, the band portion 210g including at least one first sensor or IMU sensor 225, the band portion 210e not including any second sensor or biosensor 220, first sensor or IMU sensor 225, or outward-facing light 230 (such as Figure 2W (x) a wristwatch-based wearable control device 205x having a watch portion 235, a single band portion 210h, and a plurality of band portions 210f, the band portion 210h including at least one first sensor or IMU sensor 225 and one or more outward-facing lights 230, each band portion 210f including one or more outward-facing lights 230 (such as Figure 2X shown); and so on.

[0066] Each band portion 210 (and the watch portion 235) can be connected to an adjacent band portion 210 via a connector 215, and the connection method is configured to removably wrap the wrist-type wearable control device 205 around the user's wrist when the user (e.g., user 110, etc.) wears the wrist-type wearable control device 205. In some cases, the connector 215 may include an elastic-based connector that allows the band portions to be pulled apart from each other to slide over and put on the user's hand and wrist (or slide over and take off the user's hand and wrist). Alternatively, the connector 215 may include a plurality of links and buckles (e.g., made of metal, plastic, or other solid materials), etc. Although Figure 2M to Figure 2X Not shown, however, at least one band portion 210 may also include, but is not limited to, a computing system (similar to Figure 1 computing system 115, etc.) or a communication system (similar to Figure 1 at least one of the communication systems 135, etc.

[0067] although Figure 2M to Figure 2X 2, but the table portion 235 (in addition to including a table component or a smart table component) may include, but is not limited to, at least one of the following: a computing system (similar to Figure 1 computing system 115, etc.), communication system (similar to Figure 1 communication system 135, etc.), one or more second sensors or biosensors 220, at least one first sensor or IMU sensor 225, or one or more outward-facing lights 230, etc.

[0068] Here we will refer to Figure 1, Figure 3 and Figure 4 These and other functions of examples 200 and 200' (and components thereof) are described in further detail.

[0069] Figure 3A and Figure 3B (collectively referred to as "FIG. 3") are schematic diagrams illustrating various non-limiting examples 300 and 300' of UIs for controlling other user devices using a wrist wearable control device according to various embodiments.

[0070] In FIG. 3 , a wrist wearable device 205 (similar to Figure 1 2 and the wrist wearable device 105 and 205, etc.) are shown as being worn on a user (similar to Figure 1 The wrist wearable device 205 may each include, but is not limited to, a plurality of band portions 210 and a plurality of connectors 215. The wrist wearable device 205 includes a wristwatch-based wearable control device (similar to Figure 2M-Figure 2X In the case of the wrist-based wearable control device 205m-205x, etc. in FIG. 3 , the wrist-type wearable device 205 may further include a watch portion 235 (not shown in FIG. 3 ; similar to Figure 2M-Figure 2X Each strap portion 210 (and in some cases each strap portion 235) may include, but is not limited to, at least one first sensor or IMU sensor 225 (not shown in FIG. 3; similar to FIG. 3 ). Figure 1 2 ), or at least one of the one or more second sensors or biosensors 220 (not shown in FIG. 3 ; similar to Figure 1 and the second sensor or biosensor 120 and 220 in Figure 2, etc.), and so on.

[0071] refer to Figure 3A In a non-limiting embodiment 300, HMD 310a (similar to Figure 1 The user equipment 150, etc.) may include but is not limited to at least one processor (not shown; similar to Figure 1 One or more processors 150a, etc.), at least one transceiver (not shown; similar to Figure 1 transceiver 150b, etc.), one or more cameras (not shown; similar to Figure 1 One or more cameras 150c, etc.), display device 315a (similar to Figure 1 display device 150d, etc.), and UI 320a (similar to Figure 1In some cases, the HMD 310a may include, but is not limited to, one of the following: virtual reality (VR) goggles, augmented reality (AR) goggles, mixed reality (MR) goggles, VR-enabled glasses, AR-enabled glasses, MR-enabled glasses, a VR-enabled smartphone installed in a headset, an AR-enabled smartphone installed in a headset, or an MR-enabled smartphone installed in a headset, etc.

[0072] Similarly, in Figure 3B In the non-limiting embodiment 300', the user device 310b (similar to Figure 1 The user equipment 150, etc.) may include but is not limited to at least one processor (not shown; similar to Figure 1 One or more processors 150a, etc.), at least one transceiver (not shown; similar to Figure 1 transceiver 150b, etc.), a display screen or display device 315b (similar to Figure 1 display device 150d, etc.), and UI 320b (similar to Figure 1 UI 160 of the present invention, etc.). In some cases, the user device 310b may include, but is not limited to, a smart TV (TV), a game console communicatively coupled to a display device, a monitor communicatively coupled to a display device, a projector communicatively coupled to a display device, a display device, a tablet computer, a laptop computer, a desktop computer, or a smart home appliance with a display screen, etc. In some cases, the monitor or projector may also be communicatively coupled to one of a smart phone, a tablet computer, a laptop computer, a desktop computer, or a portable gaming device, etc., to present the content to be displayed on the monitor or projector.

[0073] exist Figure 3A and Figure 3B In the example, UI 320a or 320b (collectively referred to as “UI 320”, etc.) presented within display device 315a or 315b (collectively referred to as “display device 315”, etc.) may display or present content (in this example, a menu screen having a plurality of system icons 325 (in this example, icons of a connected VR headset and mobile device, etc.) and / or a plurality of icons 330 associated with a corresponding plurality of software applications (apps), etc., but not limited thereto). In some cases (although not shown), the content that may be displayed or presented may include icons, text, graphics, objects, thumbnails, or picture-in-picture (PIP) type mini-windows of media content, the media content including but not limited to video content, image content, representations of audio content (e.g., album cover art, artist photos or posters, etc.), game content, demonstration content, etc.

[0074] According to some embodiments, at least one first sensor or IMU sensor 225 (including but not limited to at least one of an inertial measurement unit (IMU) sensor or a gyroscope-based sensor, etc.) can detect the movement of the wrist wearable device 205 relative to at least one of the three rotation axes (for example, as shown in FIG. 3, the rotation R relative to the X axis). x , rotation R relative to the Y axis y , and / or rotation R about the Z axis z (also referred to as pitch, roll, and / or yaw, respectively), etc.). In some cases, the Y-axis can be parallel to at least one of the longitudinal extension of the user's arm or the main axis of the wrist wearable device 205, etc., while the X-axis can be in a plane parallel to a plane defined by one of the dorsal side of the forearm or the ventral side of the forearm, and the Z-axis can be parallel to a line perpendicular to a plane defined by one of the dorsal side of the forearm or the ventral side of the forearm. Reference Figure 1 and Figure 3, supination or pronation can refer to the rotation R about the Y axis y , the direction of rotation depends on which arm the wrist wearable device is worn on. In some cases, the wrist wearable device 205 can generate instructions based on the detected movement of the wrist wearable device 205 relative to at least one of the three rotation axes, including but not limited to at least one of a cursor movement command or a navigation command.

[0075] In some embodiments, upward or downward movement of the wrist of a user wearing a wrist wearable device may correspond to movement relative to a first rotation axis (“X axis”) (e.g., as shown in FIG. 3 , a rotation R relative to the X axis). x (also called pitch, etc.). Movement of the wrist of the user wearing the wrist wearable device to the right or left may correspond to movement relative to a second rotation axis ("Z axis") (e.g., as shown in FIG. 3 , a rotation R relative to the Z axis) z(also referred to as yaw), etc.). In this case, the command based on the first gesture may include, but is not limited to, at least one of: an upward or downward gesture command based on movement relative to the X-axis, or a rightward or leftward gesture command based on movement relative to the Z-axis. In some cases, the first instruction (which may be generated based on the command based on the first gesture) may include, but is not limited to, at least one of a scroll command, a move command, a highlight command, a move cursor command, or a navigation command, etc. In some cases, the longitudinal extension of the arm of the user wearing the wrist wearable device may define an axis corresponding to the third rotation axis ("Y axis"). In this case, the command based on the first gesture may also include that the arm of the user wearing the wrist wearable device points to a display screen (e.g., display device 150d, etc.) displaying content from a user device (e.g., user device 150, etc.). In some cases, the first instruction may include, but is not limited to, at least one of an upward or downward gesture command based on movement relative to the X-axis or a rightward or leftward gesture command based on movement relative to the Z-axis, navigating a cursor, a selection field, or a highlight field on a UI displayed on the display screen relative to the position on the display screen to which the user's arm points.

[0076] In some embodiments, movement of the wrist of a user wearing the wrist wearable device relative to an axis defined by the longitudinal extension of the user's arm can correspond to movement relative to a third rotational axis ("Y axis") (e.g., as shown in FIG. 3 , a rotation R relative to the Y axis). y (also called scrolling), etc.). In this case, the command based on the first gesture may include, but is not limited to, at least one of a thumb up gesture command, a thumb down gesture command, or a thumb flat gesture command based on movement relative to the Y axis when the user's finger is in a relaxed and unfastened state. In some cases, the first instruction may include, but is not limited to, one of a selection command, a confirmation command, a return command, or a cancel command, etc.

[0077] As an alternative or addition to the thumbs up, thumbs down, and thumbs flat gesture commands, commands based on the first gesture can be based on the degree of supination or pronation of the forearm connected to the wrist of the user wearing the wrist wearable device, which is detected by a first sensor or IMU sensor (e.g., first sensor or IMU sensor 125, etc.) that tracks the movement of the wrist wearable device.

[0078] In some embodiments, as described above with respect to Figure 1As described in detail, one or more second sensors or biosensors 220 (including but not limited to at least one of a PPG sensor, an EMG sensor, a sound sensor, or a motion sensor, etc.) can detect biometrics corresponding to the movement of soft tissue in or near the wrist of a user wearing the wrist wearable device. In some cases, one or more second sensors or biosensors 220 can be used to track, monitor, and / or recognize gesture-based commands, which include but are not limited to at least one of the following: sliding-based commands (e.g., as described above with respect to Figure 1 As described above, finger 305c is raised from or lowered toward thumb 305d at a moderate speed, and the other fingers of hand 305b are maintained in an initial or relaxed position, etc.), drag-based commands (e.g., as described above with respect to Figure 1 As described above, finger 305c is slowly raised from thumb 305d or lowered toward thumb 305d, and other fingers of hand 305b are kept in initial or relaxed posture, etc.), tap command (for example, finger 305c is quickly lowered toward thumb 305d, or finger 305c is bent or folded, and other fingers of hand 305b are kept in initial or relaxed posture, etc.), double-click command (for example, similar to tap command, but such gesture is performed twice in rapid succession, etc.), pointing command (for example, finger 305c is straightened, and other fingers of hand 305b are kept in initial or relaxed posture, etc.), pinch-based command (for example, as described above with respect to Figure 1 In some cases, the wrist wearable device 205 may generate instructions based on the tracked, monitored, and / or recognized gesture-based commands, including but not limited to at least one of a scroll command, a move command, a selection command, a confirmation command, a highlight command, a return command, or a cancel command, etc.

[0079] At least one first sensor or IMU sensor 225 (in some cases in combination with one or more second sensors or biosensors 220, etc.) enables high-precision three-degrees-of-freedom (3DOF) control of UI 320 displayed within display device 315 of user device 310. In some embodiments, wrist wearable device 205 can cause UI 315 of user device 310 to generate a cursor using an initial 3DOF pose (e.g., based on the position and orientation of wrist wearable device 205 detected by at least one first sensor or IMU sensor 225, etc.). In some cases, generating a cursor can include generating a cursor by projecting a laser ray into UI 315. With 3DOF tracking, the user can move the cursor by rotating wrist wearable device 205 (e.g., rotation detected by at least one first sensor or IMU sensor 225, etc.). The wrist wearable device 205 can also use one or more second sensors or biosensors to recognize hand and finger gestures to generate instructions (including but not limited to at least one of a scroll command, a move command, a selection command, a confirmation command, a highlight command, a return command, or a cancel command, etc.) based on detected gesture-based commands (including but not limited to at least one of a slide-based command, a drag-based command, a tap command, a double-click command, a pointing command, a pinch-based command, or a grip-based command, etc.).

[0080] like Figure 3A and Figure 3B As shown in non-limiting embodiments 300 and 300' of FIG. 3 , for example, 3DOF controls may be used to navigate within UI 320. According to some embodiments, navigating and interacting within UI 320 may include selecting or highlighting icons, text, graphics, objects, thumbnails, or picture-in-picture (PIP) type media content mini-windows, etc. (collectively, "UI objects 330", etc.) (as shown, for example, by superimposing bounding boxes 335 on app icons in FIG. 3, etc.), moving selected or highlighted fields (e.g., bounding boxes 335, etc.) from one UI object to another UI object, etc. Alternatively or additionally, navigating and interacting within UI 320, whether or not there is a selection or highlighting of a field, may include scrolling or flipping through menus, pages, windows, and / or other UI display areas, etc., and responding to prompts (using micro-gesture-based acceptance or rejection input commands, or using micro-gesture-based "yes" or "no" input commands, or using micro-gesture-based "OK" or "cancel" input commands, etc.), etc.

[0081] In some cases, a combination of one or more second sensors or biosensors 220, at least one first sensor or IMU sensor 225, and one or more outward-facing lights 230 (e.g., Figure 2B , Figure 2D , Figure 2F , Figure 2H , Figure 2J , Figure 2L , Figure 2N , Figure 2P , Figure 2R , Figure 2T , Figure 2V and Figure 2X As shown in the example of FIG. 2 , micro-gesture based UI control (using one or more first sensors or biometric sensors 220), motion detection based UI control (using at least one second sensor or IMU sensor 225), and translation detection based UI control (using one or more outward-facing lights 230 as tracking points for the user's camera 340, used as additional UI control input) can be combined to achieve high-precision six-degree-of-freedom (6DOF) control of UI 320 displayed in display device 315 of user device 310. In some cases, there is some overlap between two or more of micro-gesture based UI control, motion detection based UI control, and translation detection based UI control. For example, as shown in FIG. 2 , micro-gesture based UI control, motion detection based UI control, and translation detection based UI control can be combined to achieve high-precision six-degree-of-freedom (6DOF) control of UI 320 displayed in display device 315 of user device 310. Figure 3A and Figure 3B As shown in the non-limiting embodiments 300 and 300 ′, 6DOF control may be used to navigate within the UI 320 .

[0082] In some cases, multiple wrist wearable devices 205 (one on each arm of the user) can be used to provide additional 3DOF or 6DOF control of UI 320, and can facilitate immersive experiences (e.g., games or AR / VR / MR experiences, etc.), where one wrist wearable device can be used to navigate within an environment displayed or presented in UI 320, while another wrist wearable device can be used to navigate and interact with UI objects or sub-UIs within the environment within UI 320, etc. In some cases, for wrist wearable devices 205 with touch screen input or physical button input, as an alternative to or in addition to gesture-based commands (as described above), at least one of a selection command, a confirmation command, a return command, a cancel command, etc. can be generated in response to the user's interaction with the touch screen input or physical button input.

[0083] In some embodiments, a machine learning system (e.g., Figure 1 The AI ​​system 115c or other computing system 115d, etc.) further enhances the operation of the wrist wearable device by learning and fine-tuning the detection of micro-gestures, motion-based gesture detection, translation-based gesture detection, and / or instances of user intended gesture control (compared to regular motions that are not gesture control motions) to adapt to user use over time, etc.

[0084] This article will refer to Figure 1 , Figure 2, and Figure 4 These and other functions of examples 300 and 300 ′ (and components thereof) are described in further detail.

[0085] Figure 4 is a flowchart illustrating a method of controlling UIs of other devices based on a wrist wearable device (eg, a smart watch) according to various embodiments.

[0086] Although techniques and procedures are depicted and / or described in a particular order for purposes of illustration, it should be understood that certain procedures may be reordered and / or omitted within the scope of various embodiments. Figure 4 The method 400 shown may be performed by or using Figure 1 , Figure 2A-2L , Figure 2M-Figure 2X , Figure 3A and Figure 3B Although the system, example, or embodiment 100, 200, 200', 300 and 300' (or its components) is implemented (and in some cases, the method 400 will be described below with respect to these systems, examples, or embodiments), the method can also be implemented using any suitable hardware (or software) implementation. Similarly, although Figure 1 , Figure 2A-2L , Figure 2M-Figure 2X , Figure 3A and Figure 3B Each of the systems, examples, or embodiments 100, 200, 200', 300 and 300' can be based on Figure 4 The method 400 shown is operated (eg, by executing instructions embodied on a computer-readable medium), however Figure 1 , Figure 2A-2L , Figure 2M-Figure 2X , Figure 3A and Figure 3B The systems, examples or embodiments 100, 200, 200', 300 and 300' may also operate according to other operating modes and / or execute other suitable procedures, respectively.

[0087] exist Figure 4 In a non-limiting embodiment of A, at block 405, method 400 may include, when the wrist wearable device is worn by a user, receiving first sensor data from one or more first sensors disposed on the wrist wearable device using a computing system. One or more second sensors may be used to detect movement of the wrist wearable control device relative to at least one of the three rotational axes. At block 410, method 400 may include analyzing the first sensor data using a computing system to detect movement of the wrist wearable control device relative to at least one of the three rotational axes.

[0088] In some embodiments, the computing system may include, but is not limited to, at least one of the following: one or more processors on a wrist-type wearable control device, one or more processors on a mobile device communicatively coupled to the wrist-type wearable control device, a machine learning system, an artificial intelligence (AI) system, a deep learning system, a neural network, a convolutional neural network (CNN), or a fully convolutional network (CN), etc. In some cases, the wrist-type wearable device may include, but is not limited to, one of the following: a smart watch, a wrist-based wearable control device, a wristband-based wearable control device, or other wrist-type wearable user devices, etc. In some cases, at least one first sensor may include, but is not limited to, at least one of an inertial measurement unit (IMU) sensor or a gyroscope-based sensor, etc.

[0089] At block 415, method 400 may also include using a computing system to determine whether the detected movement of the wrist wearable device corresponds to a command based on a first gesture, the command based on the first gesture being used to control a user interface (UI) of a user device separate from the wrist wearable device. If so, method 400 may also include using a computing system to generate a first instruction based on the command based on the first gesture (block 420), and using the computing system to send the generated first instruction to the user device, the first instruction being used as an input to control the UI of the user device (block 425).

[0090] In some cases, the user device separate from the wrist wearable device may include, but is not limited to, at least one of the following: virtual reality (VR) goggles, augmented reality (AR) goggles, mixed reality (MR) goggles, VR-enabled glasses, AR-enabled glasses, MR-enabled glasses, a VR-enabled smartphone mounted in a headset, an AR-enabled smartphone mounted in a headset, an MR-enabled smartphone mounted in a headset, a smart television (TV), a game console communicatively coupled to a display device, a monitor communicatively coupled to a display device, a projector communicatively coupled to a display device, a display device, a tablet computer, a laptop computer, a desktop computer, or a smart home appliance with a display screen, etc.

[0091] According to some embodiments, the upward or downward movement of the wrist of the user wearing the wrist wearable device may correspond to the movement relative to the first rotation axis ("X axis"), and the rightward or leftward movement of the wrist of the user wearing the wrist wearable device may correspond to the movement relative to the second rotation axis ("Z axis"), and the command based on the first gesture may include but is not limited to at least one of the following: an upward or downward gesture command based on the movement relative to the X axis, or a rightward or leftward gesture command based on the movement relative to the Z axis. In this case, the first instruction may include at least one of the following: a scroll command, a move command, a highlight command, a cursor move command, or a navigation command, etc.

[0092] In some cases, the longitudinal extension of the arm of the wrist of the user wearing the wrist wearable device can define an axis corresponding to the third rotation axis ("Y axis"). In this case, the command based on the first gesture can also include the arm of the wrist of the user wearing the wrist wearable device pointing to a display screen displaying content from the user device, and the first instruction can include, but is not limited to: based on at least one of an up or down gesture command based on movement relative to the X axis or a right or left gesture command based on movement relative to the Z axis, navigate a cursor on a UI displayed on the display screen, select a field, or highlight one of the fields relative to the position on the display screen pointed to by the user's arm.

[0093] In some embodiments, the movement of the wrist of the user wearing the wrist wearable device relative to the axis defined by the longitudinal extension of the user's arm can correspond to the movement relative to the third rotation axis ("Y axis"). In this case, the command based on the first gesture may include, but is not limited to: when the user's fingers are in a relaxed and loose state, at least one of a thumb up gesture command, a thumb down gesture command, or a thumb flat gesture command based on the movement relative to the Y axis, wherein the first instruction may include one of a selection command, a confirmation command, a return command, or a cancel command, etc.

[0094] According to some embodiments, at block 430, method 400 may further include: when the wrist wearable control device is worn by the user, using the computing system to receive second sensor data from one or more second sensors disposed on the wrist wearable control device. When the user wears the wrist wearable control device, the one or more second sensors may be used to detect gestures of one or more fingers of the user without any fingers of the user contacting the wrist wearable control device. At block 435, method 400 may include: using the computing system to analyze the second sensor data to identify a second gesture corresponding to the movement of the one or more fingers of the user.

[0095] In some embodiments, the one or more second sensors may each include at least one biosensor, the at least one biosensor including at least one of one or more photoplethysmography (PPG) sensors, one or more electromyography (EMG) sensors, one or more sound sensors, or one or more motion sensors, each sensor for detecting a biometric feature corresponding to a soft tissue movement in or near the wrist of a user wearing a wrist wearable device. In some cases, the soft tissue may include, but is not limited to, at least one of one or more muscles, one or more tendons, or one or more blood vessels in or near the wrist of the user. In some cases, the one or more PPG sensors may be used to optically detect changes in blood volume in one or more blood vessels under the skin tissue at or near the wrist of the user corresponding to the movement of one or more fingers of the user. In some cases, the one or more EMG sensors may be used to detect the electrical activity of one or more muscles at or near the wrist of the user corresponding to the movement of one or more fingers of the user. In some cases, the one or more sound sensors may be used to detect sounds in the user's body corresponding to the movement of one or more fingers of the user. In some cases, the one or more motion sensors may be used to detect the movement of one or more muscles, one or more tendons, and one or more blood vessels corresponding to the movement of one or more fingers of the user.

[0096] At block 440, method 400 may include determining whether the second gesture corresponds to a command based on the second gesture, the command based on the second gesture being used to control a UI of a user device separate from the wrist wearable device. If so, method 400 may also include using a computing system to generate a second instruction based on the command based on the second gesture (block 445), and using the computing system to send the generated second instruction to the user device, the second instruction being used as an additional input for controlling the UI of the user device (block 450).

[0097] In some cases, the command based on the second gesture may include at least one of a tap command, a double-click command, a pinch-based command, or a grip-based command, wherein the second instruction includes at least one of a selection command, a confirmation command, a return command, or a cancel command, etc.

[0098] System and Hardware Implementation Examples

[0099] Figure 5 is a block diagram illustrating an example of a computer or system hardware architecture according to various embodiments. Figure 5A schematic diagram of one embodiment of a computer system 500 of a service provider system hardware is provided, which can perform the methods provided by various other embodiments described herein, and / or can perform the functions of the computer or hardware system described above (i.e., wrist wearable devices 105, 205a-205x and 205, computing system 115, display screen 130, and user devices 150, 310a, and 310b, etc.). It should be noted that Figure 5 It is intended only to provide a general description of the various components, wherein one or more (or none) of each component may be used as appropriate. Figure 5 It is broadly described how various system elements may be implemented in a relatively discrete or relatively more integrated manner.

[0100] Computer or hardware system 500 may represent the above Figures 1 to 4 In the embodiment of the computer or hardware system described (i.e., wrist wearable devices 105, 205a-205x and 205, computing system 115, display screen 130, and user devices 150, 310a and 310b, etc.), the computer or hardware system 500 is shown to include hardware elements that can be electrically coupled via a bus 505 (or can communicate in other suitable ways). The hardware elements may include: one or more processors 510, including but not limited to one or more general-purpose processors and / or one or more special-purpose processors (e.g., microprocessors, digital signal processing chips, graphics acceleration processors, etc.); one or more input devices 515, which may include but are not limited to a mouse, a keyboard, etc.; and one or more output devices 520, which may include but are not limited to a display device, a printer, etc.

[0101] The computer or hardware system 500 may also include (and / or communicate with) one or more storage devices 525, which may include, but are not limited to, local and / or network accessible storage, and / or may include, but are not limited to, disk drives, drive arrays, optical storage devices, solid-state storage devices (e.g., random access memory (RAM) and / or read-only memory (ROM)), which may be programmable, flash-updatable, etc. Such storage devices may be used to implement any appropriate data storage, including, but not limited to, various file systems, database structures, and the like.

[0102] The computer or hardware system 500 may also include a communication subsystem 530, which may include, but is not limited to, a modem, a (wireless or wired) network card, an infrared communication device, a wireless communication device, and / or a chipset (e.g., a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a WWAN device, a cellular communication facility, etc.), etc. The communication subsystem 530 may allow data to be exchanged with a network (e.g., the network described below), with other computers or hardware systems, and / or with any other device described herein. In many embodiments, the computer or hardware system 500 will also include a working memory 535, which may include a RAM or ROM device as described above.

[0103] The computer or hardware system 500 may also include software elements, which are shown in the figure as currently located in the working memory 535, including an operating system 540, a device driver, an executable library, and / or other code, such as one or more application programs 545, which may include computer programs provided by various embodiments (including but not limited to virtual machine management programs, VMs, etc.), and / or may be designed to implement methods and / or configure systems provided by other embodiments as described herein. By way of example only, one or more programs described with respect to the above methods may be implemented as codes and / or instructions executable by a computer (and / or a processor within a computer). Then, in one aspect, such codes and / or instructions may be used to configure and / or adjust a general-purpose computer (or other device) to perform one or more operations according to the described methods.

[0104] The set of these instructions and / or codes may be encoded and / or stored on a non-transitory computer-readable storage medium, such as the storage device 525 described above. In some cases, the storage medium may be incorporated into a computer system, such as the system 500. In other embodiments, the storage medium may be separate from the computer system (i.e., removable media, such as a compact disc, etc.), and / or provided in the form of an installation package, such that the storage medium may be used to program, configure, and / or adjust a general-purpose computer through the instructions / code stored thereon. These instructions may be in the form of executable code, executable by the computer or hardware system 500, and / or may be in the form of source code and / or installable code, which, when compiled and / or installed on the computer or hardware system 500 (e.g., using various commonly available compilers, installers, compression / decompression utilities, etc.), is in the form of executable code.

[0105] Those skilled in the art will appreciate that a number of changes may be made depending on specific requirements. For example, custom hardware (e.g., programmable logic controllers, field programmable gate arrays, application specific integrated circuits, etc.) may also be used and / or specific elements may be implemented in hardware and / or software (including portable software, such as applets, etc.). In addition, connections to other computing devices (e.g., network input / output devices) may be employed.

[0106] As described above, in one aspect, some embodiments may employ a computer or hardware system (e.g., computer or hardware system 500) to perform methods according to various embodiments of the invention. According to one set of embodiments, in response to processor 510 executing one or more sequences of one or more instructions contained in working memory 535 (which may be incorporated into operating system 540 and / or other code, such as application 545), computer or hardware system 500 performs some or all of the procedures of these methods. These instructions may be read into working memory 535 from another computer-readable medium (e.g., one or more storage devices 525). By way of example only, execution of the sequences of instructions contained in working memory 535 may cause processor 510 to perform one or more procedures of the methods described herein.

[0107] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any medium that participates in providing data that causes a machine to operate in some manner. In embodiments implemented using a computer or hardware system 500, various computer-readable media may participate in providing instructions / codes to the processor 510 for execution, and / or may be used to store and / or carry such instructions / codes (e.g., as signals). In many embodiments, a computer-readable medium is a non-temporary, physical, and / or tangible storage medium. In some embodiments, a computer-readable medium may take a variety of forms, including but not limited to non-volatile media, volatile media, and the like. Non-volatile media include, for example, optical and / or magnetic disks, such as storage devices 525. Volatile media include, but are not limited to, dynamic memory, such as working memory 535. In some alternative embodiments, a computer-readable medium may take the form of a transmission medium, including but not limited to coaxial cables, copper wires, and optical fibers, including wires that make up the bus 505, and various components of the communication subsystem 530 (and / or the medium by which the communication subsystem 530 communicates with other devices). In an alternative set of embodiments, transmission media may also take the form of waves (including but not limited to radio waves, acoustic waves, and / or light waves, such as those generated during radio-wave and infrared data communications).

[0108] Common forms of physical and / or tangible computer readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, or any other magnetic medium, CD-ROMs, any other optical medium, punch cards, paper tapes, any other physical medium with a pattern of holes, RAM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cassettes, a carrier wave as described below, or any other medium from which a computer can read instructions and / or code.

[0109] Various forms of computer-readable media may be used to carry one or more sequences of one or more instructions to processor 510 for execution. By way of example only, the instructions may initially be carried on a magnetic disk and / or optical disk of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions as signals over a transmission medium for receipt and / or execution by the computer or hardware system 500. According to various embodiments of the present invention, these signals may be in the form of electromagnetic signals, acoustic signals, optical signals, etc., which are all examples of carrier waves that may encode instructions.

[0110] The communication subsystem 530 (and / or its components) typically receives the signal, and the bus 505 may then transfer the signal (and / or the data, instructions, etc. carried by the signal) to the working memory 535, from which the processor 505 retrieves and executes the instructions. The instructions received by the working memory 535 may optionally be stored on the storage device 525 before or after execution by the processor 510.

[0111] Although specific features and aspects have been described for certain embodiments, it will be appreciated by those skilled in the art that various modifications may be made. For example, the methods and processes described herein may be implemented using any combination of hardware components, software components, and / or hardware and software. In addition, although the various methods and processes described herein may be described for specific structures and / or functional components for ease of description, the methods provided by the various embodiments are not limited to any specific structures and / or functional architectures, but may be implemented on any suitable hardware, firmware, and / or software configurations. Similarly, although specific functions originate from specific system components, unless the context otherwise provides, the functions need not be limited thereto, and may be distributed in various other system components according to several embodiments.

[0112] In addition, although the procedures of the methods and processes described herein are described in a particular order for ease of description, various procedures may be reordered, added, and / or omitted according to various embodiments unless the context dictates otherwise. In addition, the procedures described for a method or process may be incorporated into other described methods or processes; similarly, system components described according to a specific structural architecture and / or for a system may be organized into alternative structural architectures and / or incorporated into other described systems. Therefore, although various embodiments are described in the form of specific features or non-specific features for ease of description and explanation of certain aspects of these embodiments, various components and / or features described herein for specific embodiments may be replaced, added, and / or deleted in other embodiments unless the context dictates otherwise. Therefore, although several embodiments are described above, it should be understood that the present invention is intended to cover all modifications and equivalents within the scope of the claims.

Claims

1. A method comprising: analyzing first sensor data using a computing system to detect movement of a wrist wearable device relative to at least one of three rotational axes, the first sensor data being received from one or more first sensors disposed on the wrist wearable device when the wrist wearable device is worn by a user; as well as Based on determining that the detected movement of the wrist wearable device corresponds to a command based on a first gesture, a first instruction is generated based on the command based on the first gesture using the computing system, and the generated first instruction is sent to a user device separated from the wrist wearable device using the computing system, wherein the command based on the first gesture is used to control a user interface (UI) of the user device, and the first instruction is used as an input for controlling the UI of the user device.

2. The method according to claim 1, wherein: The computing system includes at least one of: one or more processors on the wrist wearable device, one or more processors on a mobile device communicatively coupled to the wrist wearable device, a machine learning system, an artificial intelligence (AI) system, a deep learning system, a neural network, a convolutional neural network (CNN), or a fully convolutional network (FCN).

3. The method according to claim 1 or 2, wherein: The wrist wearable device includes one of the following: a smart watch, a wristwatch-based wearable control device, a wristband-based wearable control device, or other wrist wearable user devices.

4. The method according to any one of claims 1 to 3, wherein: The at least one first sensor includes at least one of: an inertial measurement unit (IMU) sensor or a gyroscope-based sensor.

5. The method according to any one of claims 1 to 4, wherein: The user device separated from the wrist wearable device includes at least one of the following: virtual reality (VR) goggles, augmented reality (AR) goggles, mixed reality (MR) goggles, VR-enabled glasses, AR-enabled glasses, MR-enabled glasses, a VR-enabled smartphone installed in a headset, an AR-enabled smartphone installed in a headset, an MR-enabled smartphone installed in a headset, a smart television (TV), a game console communicatively coupled to a display device, a monitor communicatively coupled to a display device, a projector communicatively coupled to a display device, a display device, a tablet computer, a laptop computer, a desktop computer, or a smart home appliance with a display screen.

6. The method according to any one of claims 1 to 5, wherein: The upward or downward movement of the wrist of the user wearing the wrist wearable device corresponds to a movement relative to a first rotation axis ("X axis"), wherein movement of the wrist of the user wearing the wrist wearable device to the right or to the left corresponds to movement relative to the second rotation axis ("Z axis"), The command based on the first gesture includes at least one of the following: an upward or downward gesture command based on movement relative to the X-axis, or a rightward or leftward gesture command based on movement relative to the Z-axis, The first instruction includes at least one of the following: a scroll command, a move command, a highlight command, a cursor move command, or a navigation command.

7. The method according to claim 6, wherein: a longitudinal extension of the arm of the user's wrist wearing the wrist wearable device defines an axis corresponding to a third rotation axis ("Y axis"), The command based on the first gesture further includes: the arm of the wrist of the user wearing the wrist wearable device points to a display screen displaying content from the user device, Wherein, the first instruction includes: navigating a cursor, a selection field, or a highlighted field on the UI displayed on the display screen relative to the position on the display screen pointed to by the user's arm based on at least one of the upward or downward gesture command based on movement relative to the X-axis or the right or left gesture command based on movement relative to the Z-axis.

8. The method according to any one of claims 1 to 7, wherein: movement of the wrist of the user wearing the wrist wearable device relative to an axis defined by a longitudinal extension of the user's arm corresponds to movement relative to a third rotation axis ("Y axis"), The command based on the first gesture includes: when the user's finger is in a relaxed and unfastened state, at least one of a thumb up gesture command, a thumb down gesture command, or a thumb flat gesture command based on movement relative to the Y axis, The first instruction includes at least one of a selection command, a confirmation command, a return command, or a cancel command.

9. The method according to any one of claims 1 to 8, further comprising: analyzing, using the computing system, second sensor data to identify a second gesture corresponding to movement of one or more fingers of the user, the second sensor data being received from one or more second sensors disposed on the wrist wearable device when the user is wearing the wrist wearable device, the one or more second sensors being used to detect the gesture of the one or more fingers of the user when no fingers of the user are in contact with the wrist wearable control device; Based on determining that the second gesture corresponds to a command based on the second gesture, a second instruction is generated based on the command based on the second gesture using the computing system, and the generated second instruction is sent to the user device separated from the wrist wearable device using the computing system, wherein the command based on the second gesture is used to control the UI of the user device, and the second instruction is used as an additional input for controlling the UI of the user device.

10. The method according to claim 9, wherein: The one or more second sensors each include at least one biosensor, and the at least one biosensor includes at least one of the following: one or more photoplethysmography (PPG) sensors, one or more electromyography (EMG) sensors, one or more sound sensors, or one or more motion sensors, each sensor being used to detect a biometric feature corresponding to a soft tissue movement in or near a wrist of the user wearing the wrist-based wearable device, wherein the soft tissue includes at least one of one or more muscles, one or more tendons, or one or more blood vessels in or near the wrist of the user.

11. The method according to claim 9, wherein: The one or more PPG sensors are used to optically detect changes in blood volume in one or more blood vessels beneath the skin tissue at or near the wrist of the user corresponding to the movement of the one or more fingers of the user, wherein the one or more EMG sensors are used to detect electrical activities of one or more muscles at or near the wrist of the user corresponding to the movement of the one or more fingers of the user, wherein the one or more sound sensors are used to detect sounds within the user's body corresponding to the movement of the one or more fingers of the user, wherein the one or more motion sensors are used to detect movements of one or more muscles, one or more tendons, and one or more blood vessels corresponding to the movement of the one or more fingers of the user.

12. The method according to claim 9, wherein: The command based on the second gesture includes at least one of the following: a tap command, a double-click command, a pinch-based command, or a grip-based command, wherein the second instruction includes at least one of the following: a selection command, a confirmation command, a return command, or a cancel command.

13. A wrist wearable device, comprising: One or more first sensors for detecting movement of the wrist-type wearable control device relative to at least one of the three rotation axes; at least one processor; as well as a non-transitory computer-readable medium communicatively coupled to the at least one processor, the non-transitory computer-readable medium storing computer software, the computer software comprising a set of instructions that, when executed by the at least one processor, cause the wrist-wearable device to: analyzing first sensor data to detect movement of the wrist wearable device relative to the at least one of the three rotation axes, the first sensor data being received from the one or more first sensors disposed on the wrist wearable device when the wrist wearable device is worn by a user; Based on determining that the detected movement of the wrist wearable device corresponds to a command based on a first gesture, a first instruction is generated based on the command based on the first gesture, and the generated first instruction is sent to a user device separated from the wrist wearable device, the command based on the first gesture is used to control a user interface (UI) of the user device, and the first instruction is used as an input for controlling the UI of the user device.

14. The wrist wearable device according to claim 13, wherein: The wrist wearable device includes one of the following: a smart watch, a wristwatch-based wearable control device, a wristband-based wearable control device, or other wrist wearable user devices.

15. The wrist wearable device according to claim 13 or 14, wherein: The at least one first sensor includes at least one of: an inertial measurement unit (IMU) sensor or a gyroscope-based sensor.

16. The wrist wearable device according to any one of claims 13 to 15, wherein: The user device separated from the wrist wearable device includes at least one of the following: virtual reality (VR) goggles, augmented reality (AR) goggles, mixed reality (MR) goggles, VR-enabled glasses, AR-enabled glasses, MR-enabled glasses, a VR-enabled smartphone installed in a headset, an AR-enabled smartphone installed in a headset, an MR-enabled smartphone installed in a headset, a smart television (TV), a game console communicatively coupled to a display device, a monitor communicatively coupled to a display device, a projector communicatively coupled to a display device, a display device, a tablet computer, a laptop computer, a desktop computer, or a smart home appliance with a display screen.

17. The wrist wearable device according to any one of claims 13 to 16, wherein: The upward or downward movement of the wrist of the user wearing the wrist wearable device corresponds to a movement relative to a first rotation axis ("X axis"), wherein movement of the wrist of the user wearing the wrist wearable device to the right or to the left corresponds to movement relative to the second rotation axis ("Z axis"), The command based on the first gesture includes at least one of the following: an upward or downward gesture command based on movement relative to the X-axis, or a rightward or leftward gesture command based on movement relative to the Z-axis, The first instruction includes at least one of the following: a scroll command, a move command, a highlight command, a cursor move command, or a navigation command.

18. The wrist wearable device according to any one of claims 13-17, further comprising one or more second sensors, wherein the one or more second sensors are used to detect gestures of the one or more fingers of the user when no fingers of the user are in contact with the wrist wearable control device, wherein: When executed by the at least one processor, the set of instructions further causes the wrist wearable device to: analyzing second sensor data to identify a second gesture corresponding to movement of one or more fingers of the user, the second sensor data being received from one or more second sensors disposed on the wrist wearable device when the user wears the wrist wearable device; as well as Based on determining that the second gesture corresponds to a command based on the second gesture, a second instruction is generated based on the command based on the second gesture, and the generated second instruction is sent to the user device separated from the wrist wearable device using the computing system, wherein the command based on the second gesture is used to control the UI of the user device, and the second instruction is used as an additional input for controlling the UI of the user device.

19. The wrist wearable device according to claim 18, wherein: The one or more second sensors each include at least one biosensor, and the at least one biosensor includes at least one of the following: one or more photoplethysmography (PPG) sensors, one or more electromyography (EMG) sensors, one or more sound sensors, or one or more motion sensors, each sensor being used to detect a biometric feature corresponding to a soft tissue movement in or near a wrist of the user wearing the wrist-based wearable device, wherein the soft tissue includes at least one of one or more muscles, one or more tendons, or one or more blood vessels in or near the wrist of the user.

20. The wrist wearable device according to claim 18, wherein: The command based on the second gesture includes at least one of the following: a tap command, a double-click command, a pinch-based command, or a grip-based command, wherein the second instruction includes at least one of the following: a selection command, a confirmation command, a return command, or a cancel command.

Images