Novel method for controlling user interface (UI) of smart watch based on finger sliding

By analyzing sensor data to identify micro gestures, the problem of inconvenience in using traditional sliding gestures on smart watches is solved, and touchless finger sliding control is achieved, improving the naturalness and convenience of operation.

CN119072678APending Publication Date: 2024-12-03INNOPEAK TECHNOLOGY INC
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202280095299.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2022-05-02
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Traditional swipe gestures require the user's finger to physically contact the screen on a smartwatch, resulting in inconvenience in some cases, such as the inability to effectively control the user interface when the user's other hand is not idle.

Method used

Through the computing system, the first sensor data is analyzed and the micro gesture of the user's finger is identified, that is, the micro gesture is detected without the user's finger touching the device, and its corresponding sliding or drag command is determined, thereby controlling the user interface of the smart watch to execute the corresponding Function.

Benefits of technology

It enables the control of the smartwatch user interface through finger swiping and dragging commands without screen touch, providing a more natural and concealed interaction method, enhancing the operation convenience and nature of the device.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119072678A_ABST
    Figure CN119072678A_ABST
Patent Text Reader

Abstract

Tools and techniques are provided for implementing a novel finger slide based smart watch user interface (UI) control or a novel micro gesture based UI control for a wrist worn device. In various embodiments, a computing system may analyze biosensor data to identify a micro-gesture of a user's finger, the biosensor data being received from a biosensor disposed on a wrist-worn device when the user wears the wrist-worn device. In response to identifying a first micro-gesture of the user's finger in the biosensor data, the computing system may determine whether the first micro-gesture corresponds to one of a finger slide command and a finger drag command for controlling the UI of the wrist-worn device. If it is determined, in response to one of the finger slide command and the finger drag command, the computing system may cause the UI of the wrist-worn device to perform one or more first functions.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Copyright Notice

[0002] Portions of the disclosure of this patent document contain material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. Technical Field

[0003] The present application generally relates to methods, systems and devices for implementing user interface (UI) control, and more specifically, to methods, systems and devices for implementing novel finger-slide-based smart watch UI control or novel micro-gesture-based UI control for wrist-worn devices. Background Art

[0004] Gestures have become a popular approach to perceptual computing user interfaces or UIs, allowing wearable devices such as smartwatches to capture and interpret human gestures as commands. For example, traditional swipe gestures on commercial devices (e.g., touchscreen display devices such as smartphones, smartwatches, tablets, laptops, etc.) provide users with a very natural way to interact with the device in order to navigate through UI elements such as menus or pages. However, traditional swipe gestures require the user's finger to be in physical contact (i.e., touch) with the screen, and sensors are provided below the screen to detect gestures. In the case of smartwatches, users may be inconvenient or unable to use such touch-based swipe gestures, for example, if the user's other hand is not free.

[0005] Therefore, a more robust and scalable solution is needed for implementing UI control. Summary of the invention

[0006] The technology of the present application generally relates to tools and techniques for implementing user interface (UI) control, and more specifically, to methods, systems and devices for implementing novel finger-slide-based smart watch UI control or novel micro-gesture-based UI control for wrist-worn devices.

[0007] In one aspect, a method includes analyzing first sensor data using a computing system to compare with a plurality of preset micro-gestures to identify micro-gestures of a user's finger when the user wears the wrist-worn device from at least one first sensor disposed on the wrist-worn device, the at least one first sensor being configured to detect the micro-gestures of the user's finger when the user does not contact the wrist-worn device; in response to identifying a first micro-gesture of the user's finger in the first sensor data, determining using the computing system whether the first micro-gesture corresponds to one of a finger slide command and a finger drag command for controlling a user interface of the wrist-worn device; and based on determining that the first micro-gesture corresponds to one of a finger slide command and a finger drag command for controlling a user interface of the wrist-worn device, in response to one of the finger slide command and the finger drag command, causing the user interface of the wrist-worn device to perform one or more first functions using the computing system.

[0008] In another aspect, a wrist-worn device includes: at least one first sensor configured to detect micro-gestures of one or more fingers of a user without the user's fingers contacting the wrist-worn device; at least one processor; and a non-transitory computer-readable medium communicatively coupled to the at least one processor, the non-transitory computer-readable medium having computer software stored thereon, the computer software including an instruction set that, when executed by the at least one processor, causes the wrist-worn device to: analyze first sensor data to compare with a plurality of preset micro-gestures to identify micro-gestures of the user's fingers, the first sensor data being received from at least one first sensor disposed on the wrist-worn device when the user wears the wrist-worn device; in response to identifying a first micro-gesture of the user's finger in the first sensor data, determine whether the first micro-gesture corresponds to one of a finger slide command and a finger drag command for controlling a user interface of the wrist-worn device; and based on determining that the first micro-gesture corresponds to one of a finger slide command and a finger drag command for controlling the user interface of the wrist-worn device, in response to one of the finger slide command and the finger drag command, cause the user interface of the wrist-worn device to perform one or more first functions.

[0009] Various modifications and additions may be made to the embodiments discussed without departing from the scope of the invention. For example, while the embodiments described above refer to particular features, the scope of the invention also includes embodiments having different combinations of features and embodiments that do not include all of the features described above.

[0010] The details of one or more aspects of the present application are set forth in the accompanying drawings and the following description. Other features, objects, and advantages of the technology described in this application are apparent in the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The nature and advantages of certain embodiments may be further understood by reference to the remainder of the specification and drawings, wherein the same reference numerals are used to refer to similar components. In some cases, a sub-label is associated with a reference numeral to indicate one of multiple similar components. When a reference numeral is mentioned without specifying an existing sub-label, all such multiple similar components are intended to be referred to.

[0012] Figure 1 is a schematic diagram illustrating a system for implementing a novel micro-gesture based user interface control of a wrist-worn device according to various embodiments.

[0013] Figures 2A-2F is a schematic diagram illustrating various non-limiting examples of slide-based gesture (or finger slide) commands and drag-based gesture (or finger drag) commands according to various embodiments.

[0014] Figures 3A-3F is a schematic diagram showing various non-limiting examples of using swipe-based gesture (or finger swipe) commands and drag-based gesture (or finger drag) commands to control a wrist-worn device user interface during implementation of a novel micro-gesture user interface control for a wrist-worn device according to various embodiments.

[0015] Figure 4 is a flow chart of a method for implementing a novel micro-gesture-based UI control for a wrist-worn device according to various embodiments.

[0016] Figure 5 is a block diagram illustrating an example of a computer or system hardware architecture according to various embodiments. DETAILED DESCRIPTION

[0017] Overview

[0018] Various embodiments of the present application provide tools and techniques for implementing user interface control, and more specifically, provide methods, systems and devices for implementing new finger-slide-based smart watch UI control or new micro-gesture-based UI control for wrist-worn devices.

[0019] In various embodiments, a computing system analyzes first sensor data to compare with a plurality of preset micro-gestures to identify a micro-gesture of a user's finger, the first sensor data being received from at least one first sensor disposed on a wrist-worn device when the user wears the wrist-worn device, the at least one first sensor being configured to detect a micro-gesture of the user's finger without the user's finger contacting the wrist-worn device. In response to identifying a first micro-gesture of the user's finger in the first sensor data, the computing system determines whether the first micro-gesture corresponds to one of a finger slide command and a finger drag command for controlling a user interface of the wrist-worn device. Based on determining that the first micro-gesture corresponds to one of a finger slide command and a finger drag command for controlling a user interface of the wrist-worn device, in response to one of the finger slide command and the finger drag command, the computing system may cause the user interface of the wrist-worn device to perform one or more first functions.

[0020] In some embodiments, the computing system may include at least one of one or more processors on the wrist-worn device, one or more processors on a mobile device communicatively coupled to the wrist-worn device, a machine learning system, an artificial intelligence (AI) system, a deep learning system, a neural network, a convolutional neural network (CNN), or a fully convolutional network (FCN). In some cases, the wrist-worn device may include one of a smart watch, a wrist-worn display device, or other wrist-worn user device.

[0021] According to some embodiments, at least one first sensor includes at least one biosensor, the at least one biosensor includes at least one of one or more photoplethysmograph (PPG) sensors, one or more electromyography (EMG) sensors, one or more sound transducers, and one or more motion transducers, each of the at least one biosensor is configured to detect a biometric feature corresponding to a soft tissue movement in or near a wrist of a user wearing the wrist-worn device. In some cases, the soft tissue may include at least one of one or more muscles, one or more tendons, one or more blood vessels, etc. in or near the user's wrist. In some cases, the one or more photoplethysmographs are configured to optically detect changes in blood volume in one or more blood vessels under the skin tissue at or near the user's wrist corresponding to the user's finger movements. In some cases, the one or more electromyography sensors are configured to detect electrical activity of one or more muscles at or near the user's wrist corresponding to the user's finger movements. In some cases, the one or more sound transducers are configured to detect sounds in the user's body corresponding to the user's finger movements. In some cases, one or more motion transducers are configured to detect movement of one or more muscles, one or more tendons, and one or more blood vessels corresponding to the user's finger movements.

[0022] In some embodiments, the finger slide command may include one of an upward slide command, a downward slide command, a left slide command, or a right slide command, etc. In some cases, the one or more first functions may include one of an upward scroll function, a downward scroll function, a left scroll function, or a right scroll function, etc., for browsing display content on a user interface that may be displayed on a display screen of the wrist-worn device. In some cases, the finger slide command may include one of lifting a finger or lowering a finger, each finger slide command being relative to a relaxed posture of fingers of a user's hand and executed at a medium speed.

[0023] According to some embodiments, the finger drag command may include one of an upward drag command, a downward drag command, a left drag command, or a right drag command, etc. In some cases, the one or more first functions may include one of an upward move function, a downward move function, a left move function, or a right move function, etc., for browsing display content on a user interface that may be displayed on a display screen of the wrist-worn device. In some cases, the finger drag command may include one of lifting a finger or lowering a finger, each finger drag command being relative to a relaxed posture of a finger of a user's hand and executed at a low speed.

[0024] In some embodiments, the computing system may analyze second sensor data to detect movement of the wrist-worn device relative to at least one of the three rotation axes, the second sensor data being received from at least one second sensor disposed on the wrist-worn device when the user wears the wrist-worn device. In response to detecting movement of the wrist-worn device relative to at least one rotation axis in the second sensor data, the computing system determines whether the detected movement of the wrist-worn device is consistent with and operates together with a first micro-gesture corresponding to one of a finger slide command and a finger drag command for controlling a user interface of the wrist-worn device. Based on determining that the detected movement of the wrist-worn device is consistent with and operates together with a first micro-gesture corresponding to one of a finger slide command and a finger drag command for controlling a user interface of the wrist-worn device, in response to one of the finger slide command and the finger drag command, the computing system causes the user interface of the wrist-worn device to perform one or more second functions.

[0025] By way of example only, in some cases, the at least one second sensor may include at least one of an inertial measurement unit (IMU) sensor or a gyroscope-based sensor, etc. In some cases, the one or more second functions may include at least one of a scrolling function or a dragging function, etc., each of the one or more second functions being in a direction based on a degree of supination or pronation of a forearm connected to a wrist of a user on which the user wears the wrist-worn device, the direction being detected by movement of the wrist-worn device.

[0026] In another aspect, a wrist-worn device may include at least one first sensor configured to detect micro-gestures of one or more fingers of a user without the user's fingers contacting the wrist-worn device; at least one processor; and a non-transitory computer-readable medium communicatively coupled to the at least one processor, the non-transitory computer-readable medium having computer software stored thereon, the computer software including an instruction set that, when executed by the at least one processor, causes the wrist-worn device to: analyze first sensor data to compare with a plurality of preset micro-gestures to identify micro-gestures of the user's fingers, the first sensor data being received from at least one first sensor disposed on the wrist-worn device when the user wears the wrist-worn device; in response to identifying a first micro-gesture of the user's fingers in the first sensor data, determine whether the first micro-gesture corresponds to one of a finger slide command and a finger drag command for controlling a user interface of the wrist-worn device; and based on determining that the first micro-gesture corresponds to one of a finger slide command and a finger drag command for controlling the user interface of the wrist-worn device, in response to one of the finger slide command and the finger drag command, cause the user interface of the wrist-worn device to perform one or more first functions.

[0027] In some embodiments, the wrist-worn device may include one of a smart watch, a wrist-worn display device, or other wrist-worn user device. In some cases, the at least one first sensor may include at least one biosensor. In some cases, the at least one biosensor may include at least one of one or more photosensitive blood pressure sensors, one or more electromyography sensors, one or more sound transducers, or one or more motion transducers, etc., each of the at least one biosensor is configured to detect a biometric feature corresponding to soft tissue movement in or near the wrist of a user wearing the wrist-worn device.

[0028] According to some embodiments, the wrist-worn device may further include at least one second sensor configured to detect movement of the wrist-worn device relative to at least one of the three rotation axes. In this case, when the instruction set is executed by at least one processor, the wrist-worn device is caused to: analyze second sensor data to detect movement of the wrist-worn device relative to at least one of the three rotation axes, the second sensor data being received from at least one second sensor disposed on the wrist-worn device when the user wears the wrist-worn device; in response to detecting movement of at least one rotation axis of the wrist-worn device in the second sensor data, determine whether the detected movement of the wrist-worn device is consistent with and operates together with the first micro-gesture corresponding to one of a finger slide command and a finger drag command for controlling the user interface of the wrist-worn device; and based on determining that the detected movement of the wrist-worn device is consistent with and operates together with the first micro-gesture corresponding to one of a finger slide command and a finger drag command for controlling the user interface of the wrist-worn device, in response to one of the finger slide command and the finger drag command, cause the user interface of the wrist-worn device to perform one or more second functions.

[0029] In some embodiments, the at least one second sensor may include at least one of an inertial measurement unit sensor or a gyroscope-based sensor, etc. In some cases, the one or more second functions may include at least one of a scrolling function or a dragging function, etc., each of the one or more second functions along a direction based on a degree of supination or pronation of a forearm connected to a wrist of a user wearing the wrist-worn device, the direction being detected by motion of the wrist-worn device.

[0030] In various aspects described in the present application, a new type of smart watch user interface control based on finger sliding or a new type of user interface control based on micro gestures for wrist-worn devices is provided. This enables one-handed control of the user interface of the wrist-worn device through micro gestures of fingers on the side of the arm wearing the wrist-worn device, wherein the micro gestures are monitored, tracked and recognized by biosensors (in some cases, with sound / motion transducers (such as microphones, etc.) and / or IMU sensors). That is, compared with screen touch, similar gestures allow users to control the smart watch with only one hand wearing the device. Through sound / motion sensors (such as microphones, etc.) or biosensors (such as PPG or EMG) installed on smart watches, algorithms can provide more reliable input for gesture recognition, which is very challenging for solutions using only IMU data. Compared with twisting gestures, gestures are more natural and unnoticeable in public places. The solution according to various embodiments is to detect finger movements between the tip of the user's index finger and the tip of the thumb to replace the up and down sliding gestures on the touch screen for user interface functions (such as page browsing, etc.).

[0031] These and other aspects of systems and methods for implementing novel finger-swipe-based smart-watch user interface controls or novel micro-gesture-based user interface controls for wrist-worn devices are described in more detail with reference to the accompanying drawings.

[0032] The following detailed description further describes several embodiments in detail to enable those skilled in the art to implement these embodiments. The described examples are for illustrative purposes only and are not intended to limit the scope of the present invention.

[0033] In the following description, for ease of explanation, many details are set forth to provide a thorough understanding of the described embodiments. However, it will be apparent to those skilled in the art that other embodiments of the present invention may be implemented without some of the details. In other cases, some structures and devices are shown in block diagram form. Multiple embodiments are described herein, and although different embodiments have different features, it should be understood that the features described for one embodiment may also be combined with other embodiments. However, for the same reason, one or more features of any described embodiment should not be considered essential to each embodiment of the present application, as other embodiments of the present application may omit these features.

[0034] Unless otherwise stated, all numbers used herein to indicate quantities, sizes, etc., are to be understood as being modified in all cases by the term "about". In this application, unless otherwise specifically stated, the use of the singular includes the plural; unless otherwise specifically stated, the use of the terms "and" and "or" means "and / or". In addition, the use of the term "include" and other forms, such as "include" and "includes", should be considered non-exclusive. In addition, unless otherwise specifically stated, terms such as "element" or "component" include both elements and components constituting one unit and elements and components constituting multiple units.

[0035] The various embodiments described herein - when (in some cases) implemented as software products, computer-implemented methods and / or computer systems, represent tangible, concrete improvements over the prior art fields, including but not limited to wrist-worn device technology, gesture control technology, gesture control technology for wrist-worn devices, biosensor technology for gesture control, motion tracking technology for gesture control, user interface control technology, user interface control technology for wrist-worn devices, and the like. In other aspects, some embodiments may improve the functionality of a user device or system itself (e.g., a wrist-worn device, a gesture control system, a gesture control system for a wrist-worn device, a biosensor system for gesture control, a motion tracking system for gesture control, a user interface control system, a user interface control system for a wrist-worn device, etc.), for example, by analyzing first sensor data using a computing system to compare with a plurality of preset micro-gestures to identify micro-gestures of a user's fingers, the first sensor data being received from at least one first sensor disposed on the wrist-worn device when the user wears the wrist-worn device, the at least one first sensor being configured to detect micro-gestures of the user's fingers without the user's fingers contacting the wrist-worn device; in response to identifying a first micro-gesture of the user's fingers in the first sensor data, determining by a computing system whether the first micro-gesture corresponds to one of a finger sliding command and a finger dragging command for controlling a UI of the wrist-worn device; and based on determining that the first micro-gesture corresponds to one of a finger sliding command and a finger dragging command for controlling a UI of the wrist-worn device, in response to one of the finger sliding command and the finger dragging command, causing the UI of the wrist-worn device to perform one or more first functions using the computing system, etc.

[0036] In particular, in the event of any abstract concepts present in the various embodiments, these concepts can be implemented as described herein by devices, software, systems, and methods involving novel functions (e.g., steps or operations), such as identifying micro-gestures of fingers attached to a wrist of a wrist-worn device and using the identified micro-gestures as command input to control a UI on the wrist-worn device, etc., to name a few examples, which are beyond traditional computer processing operations. These functions can produce tangible results beyond the implementation of a computer system, including, by way of example only, an optimized wrist-worn device that allows single-handed control of a UI of the wrist-worn device by using micro-gestures of fingers attached to the side of the arm on which the wrist-worn device is worn, the micro-gestures being detected and identified using biosensors (in some cases, in combination with sound / motion transducers (e.g., microphones, etc.) and / or IMU sensors), i.e., by biosensors to monitor, track, and identify micro-gestures, at least some of which can be observed or measured by the user and / or the manufacturer of the wrist-worn device.

[0037] Some embodiments

[0038] We now turn to the embodiments shown in the accompanying drawings. Figure 1-Figure 5 Some features of methods, systems and devices for implementing user interface control are shown, and more specifically, as mentioned above, they relate to methods, systems and devices for implementing new finger-slide-based smart watch user interface control or new micro-gesture-based user interface control for wrist-worn devices. Figure 1-Figure 5 The methods, systems, and apparatus shown are examples of different embodiments including various components and steps, which components and steps may be considered alternatives or may be used in combination with each other in various embodiments. Figure 1-Figure 5 The descriptions of the illustrated methods, systems, and devices shown in the accompanying drawings are for illustration purposes only and should not be considered to limit the scope of the different embodiments.

[0039] With reference to the accompanying drawings, Figure 1 is a schematic diagram illustrating a system 100 for implementing a novel micro-gesture based user interface control of a wrist-worn device according to various embodiments.

[0040] exist Figure 1 In a non-limiting embodiment of the present invention, the system 100 may include a wrist-worn device 105 configured to be worn on the wrist of a user 110. In some cases, the wrist-worn device 105 may include, but is not limited to, one of a smart watch, a wrist-worn display device, or other wrist-worn user device. The wrist-worn device 105 may include, but is not limited to, a computing system 115, one or more first sensors 120, one or more second sensors 125, a display screen 130, and a communication system 135. According to some embodiments, a user interface 140 may be displayed or presented on the display screen 130.

[0041] In some embodiments, the computing system 115 may include, but is not limited to, one or more processors on the wrist-worn device (e.g., processor 115a, etc.), one or more processors on a mobile device communicatively coupled to the wrist-worn device (e.g., a processor (not shown) on the mobile device 145, which is communicatively coupled to the wrist-worn device 105 via the communication system 135 (e.g., the processor 115a, etc.), and one or more processors on a mobile device communicatively coupled to the wrist-worn device (e.g., a processor (not shown) on the mobile device 145). Figure 1 145 and the communication system 135, etc.), a signal processing system (e.g., signal processing system 115b, etc.), an artificial intelligence (AI) system (e.g., artificial intelligence system 115c, etc.), or other computing system 115d, etc. In some cases, the artificial intelligence system 115c and / or other computing system 115d include, but are not limited to, a machine learning system, a deep learning system, a neural network, a convolutional neural network (CNN), or a fully convolutional network (FCN), etc.

[0042] By way of example only, in some cases, the display device 130 includes, but is not limited to, at least one of a touch screen display device, a projection display device, a holographic display device, etc. In some cases, the communication system 135 may include a wireless communication device capable of communicating using a protocol, including but not limited to at least one of a Bluetooth communication protocol, a WiFi communication protocol or other 802.11 communication protocol, a ZigBee communication protocol, a Z-wave communication protocol or other 802.15.4 communication protocol, a cellular communication protocol (e.g., 3G, 4G, 4G LTE, 5G, etc.), or other suitable communication protocols, etc. In some cases, the mobile device 145 may include, but is not limited to, one of a smartphone, a tablet computer, a laptop computer, or a portable gaming device, etc.

[0043] According to some embodiments, the one or more first sensors 120 may each include at least one biosensor. In some cases, the at least one biosensor may include but is not limited to at least one of one or more photosensitive blood pressure sensors, one or more electromyography sensors, one or more sound transducers, one or more motion transducers, etc., each of which is configured to detect a biometric feature corresponding to a soft tissue movement in or near the wrist of a user wearing a wrist-worn device. In some cases, the soft tissue may include but is not limited to at least one of one or more muscles, one or more tendons, or one or more blood vessels in or near the wrist of the user. In some cases, one or more PPG sensors may be configured to optically detect changes in blood volume in one or more blood vessels under the skin tissue at or near the wrist of the user corresponding to the user's finger movements. In some cases, one or more EMG sensors may be configured to detect electrical activity of one or more muscles at or near the wrist of the user corresponding to the user's finger movements. In some cases, one or more sound transducers (including but not limited to microphones, etc. in some cases) may be configured to detect sounds in the body of the user corresponding to the user's finger movements. In some cases, one or more motion transducers (including, in some cases, but not limited to, microphones, etc.) can be configured to detect movement of one or more muscles, one or more tendons, and one or more blood vessels corresponding to the user's finger movements.

[0044] By way of example only, in some cases, the one or more second sensors 125 may each include, but are not limited to, at least one of an inertial measurement unit sensor or a gyroscope-based sensor or the like.

[0045] In operation, the computing system 115, the processor 115a, the signal processing system 115b, the artificial intelligence system 115c, the other computing systems 115d and / or the mobile device 145, etc. (collectively referred to as "computing systems", etc.) can analyze the first sensor data and identify the micro-gesture of the finger of the user (e.g., the user 110, etc.) by comparing with a plurality of preset micro-gestures, wherein the first sensor data is received from at least one first sensor (e.g., the first sensor or the biosensor 120, etc.) provided on the wrist-worn device (e.g., the wrist-worn device 105, etc.) when the user wears the wrist-worn device, and the at least one first sensor is configured to detect the micro-gesture of the user's finger without the user's finger contacting the wrist-worn device. In response to identifying the first micro-gesture of the user's finger in the first sensor data, the computing system can determine whether the first micro-gesture corresponds to one of a finger sliding command and a finger dragging command for controlling a user interface (e.g., the user interface 140, etc.) of the wrist-worn device. Based on determining that the first micro-gesture corresponds to one of a finger slide command and a finger drag command for controlling the UI of the wrist-worn device, the computing system may cause the UI of the wrist-worn device to perform one or more first functions in response to one of the finger slide command and the finger drag command.

[0046] In some embodiments, the finger sliding command may include but is not limited to one of an upward sliding command, a downward sliding command, a left sliding command, or a right sliding command, etc. In some cases, one or more first functions may include but are not limited to one of an upward scrolling function, a downward scrolling function, a left scrolling function, or a right scrolling function, etc., for browsing display content on a user interface that can be displayed on a display screen of a wrist-worn device. In some cases, the finger sliding command includes but is not limited to one of lifting a finger or lowering a finger, each finger sliding command relative to a relaxed posture of fingers of a user's hand and executed at a medium speed.

[0047] According to some embodiments, the finger drag command may include but is not limited to one of an upward drag command, a downward drag command, a left drag command, or a right drag command, etc. In some cases, one or more first functions may include but are not limited to one of an upward move function, a downward move function, a left move function, or a right move function, etc., for browsing display content on a UI that can be displayed on a display screen of a wrist-worn device. In some cases, the finger drag command includes but is not limited to one of lifting a finger or putting a finger down, each finger drag command relative to a relaxed posture of fingers of a user's hand and executed at a low speed.

[0048] In some embodiments, the computing system may analyze the second sensor data to detect the movement of the wrist-worn device relative to at least one of the three rotation axes (including rotation and / or linear movement, etc.), and the second sensor data is received from at least one second sensor (e.g., the second sensor or the IMU sensor 125, etc.) disposed on the wrist-worn device when the user wears the wrist-worn device. In response to detecting the movement of the wrist-worn device relative to at least one rotation axis in the second sensor data, the computing system may determine whether the detected movement of the wrist-worn device is consistent with and operates together with a first micro-gesture corresponding to one of a finger slide command and a finger drag command for controlling the UI of the wrist-worn device; and based on determining that the detected movement of the wrist-worn device is consistent with and operates together with the first micro-gesture corresponding to one of a finger slide command and a finger drag command for controlling the user interface of the wrist-worn device, in response to one of the finger slide command and the finger drag command, the computing system may cause the user interface of the wrist-worn device to perform one or more second functions.

[0049] In some cases, the one or more second functions may include at least one of a scrolling function or a dragging function, etc., each of the one or more second functions is along a direction based on the degree of supination or pronation of a forearm connected to a user's wrist on which the user wears the wrist-worn device, which direction is detected by movement of the wrist-worn device.

[0050] Compared to screen touches, micro-gestures (described herein) allow the user to control the smartwatch with only one hand wearing the wrist-worn device. Through sound / motion transducers (such as microphones, etc.) or biosensors (such as PPG or EMG) installed on the smartwatch, algorithms can provide more reliable input for gesture recognition, which is very challenging for solutions using only IMU data. Compared to twisting postures, gestures are more natural and unnoticeable in public places. The solution according to various embodiments is to detect finger movements between the tip of the user's index finger and the tip of the thumb to replace the up and down sliding gestures on the touch screen for user interface functions (such as page browsing, etc.). Various embodiments make full use of IMU, microphone (or sound / motion transducer and / or biosensor) to detect micro-movements between the index finger and the thumb, and then use micro-motion detection to output continuous values ​​to simulate the output of the up / down sliding gesture. Figure 2A-Figure 3F Describes the positions and postures of fingers and hands to perform swipe-based or drag-based gestures, and examples of how these gestures can be used to control the UI of a wrist-worn device.

[0051] The following will refer to Figure 2- Figure 4 These and other functions of system 100 (and its components) are described in further detail.

[0052] Figure 2A-2F(collectively referred to as “FIG. 2”) is a diagram illustrating a slide-based gesture (or finger slide) command (eg, Figure 2A-2C ) and drag-based gesture (or finger drag) commands (see Figure 2D-2F Schematic diagrams of various non-limiting examples 200 and 200' shown).

[0053] refer to Figure 2A-2C The non-limiting example 200 shows a swipe-based gesture (or finger swipe) command. Figure 2A As shown, the user can start with a relaxed posture of the fingers (also referred to herein as an "initial posture"), which can be a posture of the user's hand 205 with the fingers (e.g., five fingers) naturally spread out and released or relaxed (i.e., there is no muscle tension to straighten the fingers or curl them (e.g., like making a fist, etc.), etc.). For an upward slide command (such as Figure 2B ), relative to the other fingers remaining in the initial or relaxed position, from the initial or relaxed position (or Figure 2C ), the finger 210 (such as but not limited to the index finger, etc.) can be raised, for example, the finger 210 can be raised at a medium speed (such as Figure 2B In some cases, lifting the finger may be performed at a relatively fast speed (e.g., as defined by visual blurring of the finger due to movement speed, etc.). For a downward slide command (e.g., Figure 2C ), relative to the other fingers that maintain the initial or relaxed position, from the initial or relaxed position or the lifted position (as shown in Figure 2B As shown), finger 210 can move at a medium speed (as shown Figure 2C In some cases, putting down the finger can be performed at a faster speed.

[0054] According to some embodiments, starting from the initial position, if the user lifts the index finger (e.g. Figure 2B ), the computing system recognizes this micro-gesture as a swipe up command. Conversely, starting from the initial gesture, if the user pulls down the index finger (as shown in Figure 2C As shown in the figure, the computing system will recognize this micro-gesture as a swipe down command. Taking the page browsing function as an example, the user can use an activation gesture (such as a pinch command) to open a page. Then, the user can pull down the index finger to trigger the swipe down function to browse the page down. If the user lifts the index finger higher than the initial posture, the gesture will trigger the swipe up function.

[0055] refer to Figure 2D-2F The non-limiting example 200' shows a drag-based gesture (or finger drag) command. Figure 2DAs shown, the user can start with an initial or relaxed position of the fingers, similar to the above Figure 2A For the upward drag command (such as Figure 2E ), relative to the other fingers that maintain the initial or relaxed position, from the initial or relaxed position (or Figure 2F The finger 210 (such as but not limited to the index finger, etc.) can be moved at a low speed (such as Figure 2E As shown by the short curved arrow 220a, the low speed here may refer to a speed slower than the medium speed, etc.) is lifted. For the downward drag command (such as Figure 2F ), relative to other fingers that maintain an initial or relaxed posture, finger 210 can be moved from an initial or relaxed posture or a raised position (such as Figure 2E as shown) and lowered, for example at a low speed (such as Figure 3F The device is lowered as shown by the short curved arrow 220b.

[0056] In various embodiments, when worn on a user's wrist, the user may use a device configured on a wrist-worn device (e.g., Figure 1 A first sensor or biosensor (e.g., Figure 1 ), tracking, monitoring and / or identifying micro-gestures, such as a swipe up command (e.g., Figure 2B ), the downward slide command (as shown Figure 2C ), drag the command upward (as shown Figure 2E ) and / or drag the command down (as Figure 2F Micro-gestures produced or related to ).

[0057] In the user interface (for example, Figure 1 140, etc.) only provides one dimension along which the user can scroll or drag within the user interface (regardless of whether the wrist-worn device includes a second sensor or an IMU sensor (e.g., Figure 1 The second sensor or IMU sensor 125, etc.), the lifting of the finger 210 (such as Figure 2B and Figure 2E as shown) and put down (as Figure 2C and Figure 2F ) can be recognized as a micro-gesture, so that the micro-gesture causes the computing system (e.g., Figure 1 The computing system 115 in the system 115, etc.) causes the user interface to perform scrolling or dragging along a first direction (responding to Figure 2B and Figure 2E lift your finger in the middle) or perform a scroll or drag in a second direction (responding to Figure 2C and Figure 2FFor example, if the user interface only provides vertical movement, the first direction may cause the computing system to cause the user interface to perform one of a scroll up function or an upward drag function, while the second direction may cause the computing system to cause the user interface to perform one of a scroll down function or a downward drag function, etc. Similarly, if the user interface only provides horizontal movement, the first direction may cause the computing system to cause the user interface to perform one of a scroll left function or a left drag function (if the left hand is used; or if the right hand is used, a scroll right function or a right drag function is performed), while the second direction may cause the computing system to cause the user interface to perform a scroll right function or a right drag function (if the left hand is used; or if the right hand is used, a scroll left function or a left drag function is performed), etc.

[0058] In the user interface (for example, Figure 1 In the case where the user interface 140, etc.) provides a two-dimensional environment in which the user can scroll or drag within the user interface, and the wrist-worn device includes a second sensor or an IMU sensor (e.g. Figure 1 In the case of a second sensor or IMU sensor 125, the user's wrist movement (such as detected by the second sensor and the IMU sensor) and the lifting of the finger 210 (such as Figure 2B and Figure 2E as shown) and put down (as Figure 2C and Figure 2F ) can be recognized as a micro-gesture that causes the computing system (e.g., Figure 1 The computing system 115, etc.) causes the user interface to perform one of scrolling and dragging in a direction based on the degree of supination or pronation of the forearm connected to the wrist of the user wearing the wrist-worn device, as detected by a second sensor or IMU sensor that tracks the movement of the wrist-worn device. Among them, supination of the forearm can refer to the rotation of the forearm, so that when the user's arm is suspended at the user's side or when the forearm is raised from the arm-hanging position (whether the fingers are an open hand or a clenched fist), the ventral side of the forearm (or palm) is forward and / or upward relative to the user's body in a sitting or standing position; and pronation of the forearm can refer to the rotation of the forearm, so that when the user's arm is suspended at the user's side or when the forearm is raised from the arm-hanging position (whether the fingers are an open hand or a clenched fist), the ventral side of the forearm (or palm) is backward and / or downward relative to the user's body in a sitting or standing position; the mid-position of the forearm can refer to the rotation of the forearm, so that the ventral side of the forearm (or palm) faces the sagittal plane or median plane, which divides the body into left and right parts along the midline of the body and is located halfway or in the middle between the supination and pronation positions.

[0059] In one non-limiting example, if the forearm is pronated (or fully pronated), the lifting of the finger 210 (eg Figure 2B and Figure 2E as shown) and put down (as Figure 2C and Figure 2F ) can be recognized as a micro-gesture that causes a computing system (e.g. Figure 1 The computing system 115 of the embodiment of the present invention (such as the computing system 115 of the embodiment of the present invention) causes the user interface to perform one of scrolling or dragging in the vertical direction (i.e., the up-and-down direction, similar to the one-dimensional vertical motion embodiment described above), regardless of whether the user's elbow is straight, the forearm is bent beside the torso, or the forearm is bent in front of the torso. Similarly, if the forearm is in a neutral position, such that the ventral side of the forearm (or palm) faces the sagittal or median plane, then the lifting of the finger 210 (such as Figure 2B and Figure 2E as shown) and put down (as Figure 2C and Figure 2F ) can be recognized as a micro-gesture that causes a computing system (e.g. Figure 1 The computing system 115 of the embodiment of the present invention (such as the computing system 115 of the embodiment of the present invention) causes the UI to scroll or drag in the horizontal direction (i.e., in the left-right direction, similar to the one-dimensional horizontal motion embodiment described above) regardless of whether the user's elbow is straight, the forearm is bent beside the torso, or the forearm is bent in front of the torso. If the forearm is between the supinated and pronated positions (but not in the pronated or neutral position), the lifting of the finger 210 (such as Figure 2B and Figure 2E as shown) and put down (as Figure 2C and Figure 2F ) can be recognized as micro-gestures, allowing a computing system (e.g. Figure 1 The computing system 115, etc.) causes the user interface to scroll or drag in a diagonal direction based on the degree of rotation between the supinated and rotated positions, regardless of whether the user's elbow is straight, the forearm is bent beside the torso, or the forearm is bent in front of the torso.

[0060] For use with a display screen of a wrist-worn device for tracking gestures, and for scrolling or dragging in a horizontal or diagonal direction, rotating the forearm to a position other than a pronated position (possibly together with one of a pinch-based gesture command, a tap-based gesture command, or a double-tap-based gesture command, etc.) can set the direction of the scroll or drag, and then immediately lifting and lowering the finger to cause the UI to scroll or drag in the set direction. Wherein, the pinch-based gesture command can include the thumb and index finger touching or pressing together, with little or no movement of other fingers of the hand (not shown). In some embodiments, for each independent user, the function setting of the wrist-worn device may include calibration, and the calibration setting includes at least one of the following: the rotation range between the supination and pronation positions, the intermediate position, the finger-lifting feature corresponding to the upward slide command, the finger-lowering feature corresponding to the downward slide command, the finger-lifting feature corresponding to the upward drag command, the finger-lowering feature corresponding to the downward drag command, the left (or right) wrist on which the wrist-worn device is usually worn, the finger-lifting feature corresponding to the left slide (or right slide) command, the finger-lowering feature corresponding to the right slide (or left slide) command, the finger-lifting feature corresponding to the left drag (or right drag) command, the finger-lowering feature corresponding to the right drag (or left drag) command, the gesture features corresponding to the pinch-based command, the tap-based command, the double-click-based command, and the like.

[0061] In some embodiments, a machine learning system (e.g., artificial intelligence system 115c or other computing system 115d, etc.) may be used to further enhance the operation of the wrist-worn device by learning and fine-tuning the detection of micro-gestures, comparing situations in which the user intentionally performs gesture control with regular movements in which the user unintentionally performs gesture control, etc., so as to adapt to long-term use by the user, etc.

[0062] Figure 3A-Figure 3F (collectively referred to as “ FIG. 3 ”) is a schematic diagram showing various non-limiting examples 300 and 300 ′ of using swipe-based gesture (or finger swipe) commands and drag-based gesture (or finger drag) commands to control a wrist-worn device UI during implementation of a novel micro-gesture UI control for a wrist-worn device according to various embodiments.

[0063] Reference Figure 3A-3C Non-limiting example 300 of how to use swipe-based gesture (or finger swipe) commands to control the user interface of a wrist-worn device. Figure 3A-3C In the example, the user's hand 205 has a relaxed or initial position of the fingers, and the finger 210 is lifted (e.g. Figure 3B The medium length curved arrow 215a indicates the lifting of the finger at a medium speed) and the lowering of the finger 210 (as shown in FIG. Figure 3C The medium length curved arrow 215b indicates that the finger is lowered at a medium speed) and Figure 2A-2C The corresponding gesture-based commands shown and described are similar or even identical. Figure 3A-3C As shown in the non-limiting example 300, a wrist-worn device 305 (e.g., a smart watch or a watch-based wrist-worn device, etc.) may include a user interface 310 on which one or more items 315 (including but not limited to objects, icons, menu items, etc.) may be displayed, and a selection area or highlight area 320 may be displayed.

[0064] In response to a swipe up command (such as Figure 3B ), which is provided on a wrist-worn device (such as Figure 1 A first sensor or biosensor (such as a wrist-worn device 105, etc.) in or on Figure 1 The first sensor 120, etc.) monitors, tracks or identifies, and the computing system (such as Figure 1 The computing system 115 of FIG. 3 may cause the user interface 310 to perform a scroll-based function, etc. In this case, the user interface 310 may perform an upward scroll function in the vertical direction indicated by arrow 325a, wherein the menu items 320 depicted in FIG. 3 will move upward or scroll in a loop (in this case, the same as the menu items 320 in FIG. 3). Figure 3A , the menu items corresponding to “Sound,” “Bluetooth,” “Applications,” and “Settings” shown in the figure move upward or scroll in a loop, causing the menu item corresponding to “Sound” to move outside the displayed user interface 310, the menu item corresponding to “Call” to move within the displayed UI 330, the menu items corresponding to “Bluetooth,” “Applications,” and “Settings” to move upward, the selection area 320 highlights the menu item corresponding to “Applications” instead of the menu item corresponding to “Bluetooth,” etc.).

[0065] Similarly, in response to a downward swipe command (eg, Figure 3C 3 ), the computing system may cause the user interface 310 to perform a scroll-based function, etc. In this case, the user interface 310 may perform a downward scroll function in the vertical direction indicated by arrow 325 b, wherein the menu items 320 depicted in FIG. 3 will move downward or scroll in a loop (in this case, the same as Figure 3A The menu items corresponding to "Sound", "Bluetooth", "Apps" and "Settings" shown in the figure move downward or scroll in a loop, causing the menu item corresponding to "Settings" to move outside the displayed user interface 310, the menu item corresponding to "Battery" to move within the displayed UI330, the menu items corresponding to "Sound", "Bluetooth" and "Apps" to move downward, the selection area 320 highlights the menu item corresponding to "Sound" instead of the menu item corresponding to "Bluetooth", etc.).

[0066] refer to Figure 3D-Figure 3FNon-limiting example 300 ' shows how to use drag-based gesture (or finger drag) commands to control the UI of a wrist-worn device. Figure 3D-Figure 3F In the example, the user's hand 205 has a relaxed or initial position of the fingers, and the finger 210 is lifted (e.g. Figure 3E The short curved arrow 220a in FIG. 2 represents the lifting of the finger at a low speed) and the lowering of the finger 210 (as shown in FIG. Figure 3F The short curved arrow 220b in FIG. 20 represents the finger being lowered at a low speed) and Figure 2D-2F The corresponding gesture-based commands shown and described are similar or even identical. Figure 3D-Figure 3F As shown in the non-limiting example 300' of , a wrist-worn device 305 (e.g., a smart watch or a watch-based wrist-worn device, etc.) may include a user interface 310 on which an item 315 (including but not limited to an object, an icon, a menu item, etc.) within a selection or highlight area 320 may be displayed, as well as a graphical icon 330 associated with the item 315 (in this case, a volume control slider corresponding to a "sound" menu item, etc.).

[0067] In response to an upward drag command (such as Figure 3E ), which is provided on a wrist-worn device (such as Figure 1 A first sensor or biosensor (such as a wrist-worn device 105, etc.) in or on Figure 1 The first sensor 120, etc.) monitors, tracks or identifies, and the computing system (such as Figure 1 The user interface 310 may be configured to perform a drag-based function, such as a computing system 115 of the wrist-worn device 305, etc. In this case, the user interface 310 may perform an upward drag function in the vertical direction indicated by arrow 335a, wherein the graphic icon 330 is slid or dragged upward (in this case, the volume control slider corresponding to the "sound" menu item may be slid or dragged upward in the direction of arrow 335b, thereby increasing the volume output by the wrist-worn device or the volume output by a communication-coupled sound presentation device (e.g., a speaker on the wrist-worn device 305, a speaker on a communication-coupled external speaker system, or a speaker on a user device that is communication-coupled to the wrist-worn device, etc.)).

[0068] Likewise, in response to a downward drag command (e.g., Figure 3FAs shown), the computing system may cause the user interface 310 to perform a drag-based function, etc. In this case, the user interface 310 may perform a downward drag function in the vertical direction indicated by arrow 335b, wherein the graphic icon 330 is slid or dragged downward (in this case, the volume control slider corresponding to the "Sound" menu item may be slid or dragged downward in the direction of arrow 335b, thereby reducing the volume output by the wrist-worn device or the volume output by a communicatively coupled sound presentation device (e.g., a speaker on the wrist-worn device 305, a speaker on a communicatively coupled external speaker system, or a speaker on a user device communicatively coupled to the wrist-worn device, etc.)).

[0069] Although FIG. 3 depicts a one-dimensional vertical movement (similar to the movement described above with respect to FIG. 2 ), various embodiments are not limited thereto, and if the user interface provides horizontal scrolling or dragging, a one-dimensional horizontal movement may be implemented, or if the user interface provides one or more of two-dimensional horizontal scrolling or dragging, two-dimensional vertical scrolling or dragging, two-dimensional diagonal scrolling or dragging (as described above with respect to FIG. 2 ), etc., a two-dimensional movement may be implemented. Although FIG. 3 depicts a smartwatch or watch-based wrist-worn device, various embodiments are not limited thereto, and the wrist-worn device may include any suitable form factor—including a display (whether touchscreen or non-touchscreen), sensors, and functionality described according to various embodiments—that may be worn on a user's wrist. Although FIG. 3 depicts a particular user interface 310 , item 315 , selection or highlight area 320 , graphic icon 330 , slidable / scrollable / draggable object, etc., various embodiments are not limited thereto, and any suitable UI, item (including any suitable object, icon, menu item, etc.), selection or highlight area, graphic icon, slidable, scrollable, draggable object, etc. may be used as needed or appropriate.

[0070] This article refers to Figure 1 , Figure 2 and Figure 4 These and other functions of examples 300 and 300 ′ (and components thereof) are described in further detail.

[0071] Figure 4 is a flow chart illustrating a method 400 for implementing a novel micro-gesture based user interface control for a wrist-worn device according to various embodiments.

[0072] Although techniques and procedures are depicted and / or described in a particular order for purposes of illustration, it should be understood that certain procedures may be reordered and / or omitted within the scope of various embodiments. Figure 4 The method 400 shown may be performed by Figure 1 , Figure 2A-2C , Figure 2D-2F , Figure 3A-3C and Figure 3D-Figure 3FThe systems, examples or embodiments 100, 200, 200', 300 and 300' (or their components) are implemented separately or together (in some cases, as described below), but these methods can also be implemented using any suitable hardware (or software) implementation. Similarly, although Figure 1 , Figure 2A-2C , Figure 2D-2F , Figure 3A-3C and Figure 3D-Figure 3F Each system, example or embodiment 100, 200, 200', 300 and 300' (or components thereof) may be based on Figure 4 The method 400 shown is operated (e.g., by executing instructions contained on a computer-readable medium), but Figure 1 , Figure 2A-2C , Figure 2D-2F , Figure 3A-3C and Figure 3D-Figure 3F The systems, examples, or embodiments 100 - 200 , 200 ′, 300 ′ may also operate according to other operating modes and / or execute other suitable programs.

[0073] exist Figure 4 In a non-limiting embodiment of A, at step 405, method 400 may include, when a user wears the wrist-worn device, receiving, using a computing system, first sensor data from at least one first sensor disposed on the wrist-worn device.

[0074] In some embodiments, the computing system may include, but is not limited to, one or more processors on a wrist-worn device, one or more processors on a mobile device communicatively coupled to the wrist-worn device, a machine learning system, an artificial intelligence system, a deep learning system, a neural network, a convolutional neural network, or a fully convolutional network, etc. In some cases, the wrist-worn device may include one of a smart watch, a wrist-worn display device, or other wrist-worn user device, etc.

[0075] According to some embodiments, at least one first sensor may include at least one biosensor. In some cases, at least one biosensor may include but is not limited to at least one of one or more photosensitive blood pressure sensors, one or more electromyography sensors, one or more sound transducers, or one or more motion transducers, etc., each of the at least one biosensor is configured to detect a biometric feature corresponding to a soft tissue movement in or near the wrist of a user wearing a wrist-worn device. In some cases, the soft tissue may include but is not limited to at least one of one or more muscles, one or more tendons, or one or more blood vessels in or near the wrist of the user. In some cases, one or more PPG sensors are configured to optically detect changes in blood volume in one or more blood vessels under the skin tissue at or near the wrist of the user corresponding to the user's finger movements. In some cases, one or more EMG sensors are configured to detect electrical activity of one or more muscles at or near the wrist of the user corresponding to the user's finger movements. In some cases, one or more sound transducers may be configured to detect sounds in the user's body corresponding to the user's finger movements. In some cases, one or more motion sensors may be configured to detect movement of one or more muscles, one or more tendons, and one or more blood vessels corresponding to the user's finger movements.

[0076] In step 410, method 400 may include analyzing first sensor data using a computing system to compare with multiple preset micro-gestures to identify micro-gestures of a user's fingers, the first sensor data being received from at least one first sensor set on a wrist-worn device when the user wears the wrist-worn device, and the at least one first sensor being configured to detect micro-gestures of the user's fingers without the user's fingers contacting the wrist-worn device.

[0077] Method 400 may also include, at step 415, in response to identifying a first micro-gesture of a user's finger in the first sensor data, determining, using a computing system, whether the first micro-gesture corresponds to one of a finger slide command and a finger drag command for controlling a user interface of the wrist-worn device.

[0078] Method 400 further includes, based on determining that the first micro-gesture corresponds to one of a finger slide command or a finger drag command for controlling the UI of the wrist-worn device, in response to one of the finger slide command and the finger drag command, using a computing system to cause the user interface of the wrist-worn device to perform one or more first functions (step 420).

[0079] In some embodiments, the finger sliding command may include but is not limited to one of an upward sliding command, a downward sliding command, a left sliding command, or a right sliding command, etc. In some cases, one or more first functions may include but are not limited to one of an upward scrolling function, a downward scrolling function, a left scrolling function, or a right scrolling function, etc., for browsing the display content on the user interface that can be displayed on the display screen of the wrist-worn device. In some cases, the finger sliding command may include but is not limited to one of lifting a finger or putting a finger down, each finger sliding command relative to a relaxed posture of the user's hand fingers, and executed at a medium speed.

[0080] According to some embodiments, the finger drag command may include but is not limited to one of an upward drag command, a downward drag command, a left drag command, or a right drag command, etc. In some cases, one or more first functions may include but are not limited to one of an upward move function, a downward move function, a left move function, or a right move function, etc., for browsing display content on a user interface that can be displayed on a display screen of the wrist-worn device. In some cases, the finger drag command may include but is not limited to one of lifting a finger or putting a finger down, each finger drag command relative to a relaxed posture of the user's hand fingers and executed at a low speed.

[0081] In some embodiments, at step 425, the method 400 may further include, when the user wears the wrist-worn device, receiving second sensor data from at least one second sensor disposed on the wrist-worn device using a computing system. At step 430, the method 400 may include analyzing the second sensor data using a computing system to detect the movement of the wrist-worn device relative to at least one of the three rotation axes, the second sensor data being received from at least one second sensor disposed on the wrist-worn device when the user wears the wrist-worn device. At step 435, the method 400 may include, in response to detecting the movement of the wrist-worn device relative to at least one rotation axis in the second sensor data, determining, using a computing system, whether the detected movement of the wrist-worn device is consistent with and operates together with a first micro-gesture corresponding to one of a finger slide command and a finger drag command for controlling a user interface of the wrist-worn device. The method 400 may also include, based on determining that the detected movement of the wrist-worn device is consistent with and operates together with a first micro-gesture corresponding to one of a finger slide command and a finger drag command for controlling a user interface of the wrist-worn device, in response to one of the finger slide command and the finger drag command, causing the user interface of the wrist-worn device to perform one or more second functions using a computing system (step 440).

[0082] Merely by way of example, in some cases, at least one second sensor may include, but is not limited to, at least one of an inertial measurement unit sensor or a gyroscope-based sensor, etc. In some cases, the one or more second functions may include, but are not limited to, at least one of a scrolling function or a dragging function, each of the one or more second functions being performed in a direction based on a degree of supination or pronation of a forearm connected to a wrist of a user wearing the wrist-worn device, the direction being detected by motion of the wrist-worn device (as described in detail above with respect to FIG. 2 , etc.).

[0083] Examples of system and hardware implementations

[0084] Figure 5 is a block diagram illustrating an example of a computer or system hardware architecture according to various embodiments. Figure 5 A schematic diagram of one embodiment of a computer system 500 of a service provider system hardware is provided, which can perform the methods provided by various other embodiments described herein, and / or can perform the functions of the computer or hardware system (i.e., wrist-worn devices 105 and 305, computing system 115, display screen 130, and mobile device 145, etc.) described above. It should be noted that Figure 5 It is intended only to provide a general description of the various components, wherein one or more (or none) of each component may be used as desired. Figure 5 It generally describes how the various system elements can be implemented in a relatively discrete or relatively more integrated manner.

[0085] Computer or hardware system 500—combined with the above Figure 1-Figure 4 The embodiment described may represent a computer or hardware device (i.e., wrist-worn devices 105 and 305, computing system 115, display screen 130, mobile device 145, etc.) - shown, including hardware elements that can be electrically coupled (or communicate in other ways as needed) via bus 505. The hardware elements may include one or more processors 510, including but not limited to one or more general-purpose processors and / or one or more special-purpose processors (e.g., microprocessors, digital signal processing chips, graphics acceleration processors, etc.); one or more input devices 515, including but not limited to a mouse, keyboard, etc.; and one or more output devices 520, including but not limited to a display device, printer, etc.

[0086] The computer or hardware system 500 may also include (and / or communicate with) one or more storage devices 525, which may include, but are not limited to, local and / or network accessible storage, and / or may include, but are not limited to, disk drives, drive arrays, optical storage devices, solid-state storage devices, such as random access memory (RAM) and / or read-only memory (ROM), programmable, flash-updated, etc. Such storage devices may be configured to implement any appropriate data storage, including, but not limited to, various file systems, database structures, etc.

[0087] The computer or hardware system 500 may also include a communication subsystem 530, including but not limited to a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device and / or a chipset (such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a WWAN device, a cellular communication facility, etc.), etc. The communication subsystem 530 may allow data to be exchanged with a network (such as the network described below, to name just one example), with other computers or hardware systems, and / or with any other device described herein. In many embodiments, the computer or hardware system 500 also includes a working memory 535, which, as described above, may include a RAM or ROM device.

[0088] The computer or hardware system 500 may also include software elements, as shown, currently located in the working memory 535, including an operating system 540, device drivers, executable libraries and / or other code, such as one or more application programs 545, which may include computer programs provided by various embodiments (including but not limited to hypervisors, virtual machines, etc.), and / or may be designed to implement the methods and / or configuration systems provided by other embodiments described herein. By way of example only, one or more programs described with respect to the above methods may be implemented as codes and / or instructions executed by a computer (and / or a processor within a computer); in one aspect, these codes and / or instructions may be used to configure and / or adjust a general-purpose computer (or other device) to perform one or more operations according to the described methods.

[0089] The set of instructions and / or code may be encoded and / or stored on a non-transitory computer-readable storage medium, such as the storage device 525 described above. In some cases, the storage medium may be incorporated into a computer system, such as the system 500. In other embodiments, the storage medium may be separate from the computer system (i.e., removable media, such as a compact disc, etc.), and / or provided in the form of an installation package, such that the storage medium may be used to program, configure, and / or adjust the instructions / code stored thereon for a general-purpose computer. The instructions may take the form of executable code, which is executed by the computer or hardware system 500, and / or may take the form of source code and / or installable code, which takes the form of executable code after being compiled and / or installed on the computer or hardware device 500 (e.g., using any of a variety of commonly available compilers, installers, compression / decompression utilities, etc.).

[0090] It will be apparent to those skilled in the art that substantial changes may be made to specific requirements. For example, custom hardware (e.g., programmable logic controllers, field programmable gate arrays, application specific integrated circuits, etc.) may also be used, and / or specific elements that may be implemented in hardware, software (including portable software such as applets, etc.), or both. In addition, connections to other computing devices (e.g., network input / output devices) may be employed.

[0091] As described above, in one aspect, some embodiments may employ a computer or hardware system (such as computer or hardware system 500) to perform methods consistent with various embodiments of the present application. According to one set of embodiments, the computer or hardware system 500 performs some or all of the procedures of these methods in response to the processor 510 executing one or more sequences of one or more instructions contained in the working memory 535 (which may be incorporated into the operating system 540 and / or other code, such as the application 545). These instructions may be read from another computer-readable medium into the working memory 535, such as one or more storage devices 525. By way of example only, the execution of the sequence of instructions contained in the working memory 535 may cause the processor 510 to perform one or more procedures of the methods described herein.

[0092] The terms "machine-readable medium" and "computer-readable medium" as used herein refer to any medium that participates in providing data to enable a machine to operate in some manner. In embodiments implemented using a computer or hardware system 500, various computer-readable media may be involved in providing instructions / codes to the processor 510 for execution and / or may be used to store and / or carry such instructions / codes (e.g., as signals). In many embodiments, computer-readable media are non-transient, physical and / or tangible storage media. In some embodiments, computer-readable media may take a variety of forms, including but not limited to non-volatile media, volatile media, and the like. For example, non-volatile media include, optical disks and / or magnetic disks, such as storage devices 525. Volatile media include, but are not limited to, dynamic memory, such as working memory 535. In some alternative embodiments, computer-readable media may take the form of transmission media, including but not limited to coaxial cables, copper wires, and optical fibers, including wires that make up the bus 505, and various components of the communication subsystem 530 (and / or the communication subsystem 530 provides a medium for communicating with other devices). In another set of alternative embodiments, transmission media may also take the form of waves (including but not limited to radio waves, acoustic waves and / or light waves, such as those generated during radio wave and infrared data communications).

[0093] Common forms of physical and / or tangible computer readable media include, for example, floppy disks, diskettes, hard disks, magnetic tape or any other magnetic medium, CD-ROMs, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM and EPROM, FLASH-EPROM, any other memory chip or cartridge, a carrier wave as described below, or any other medium from which a computer can read instructions and / or code.

[0094] Various forms of computer-readable media may be involved in transmitting one or more sequences of instructions to the processor 510 for execution. For example, the instructions may initially be carried on a magnetic disk and / or optical disk of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions as signals through a transmission medium for receipt and / or execution by the computer or hardware system 500. According to various embodiments of the present invention, these signals may be in the form of electromagnetic signals, acoustic signals, optical signals, etc., examples of which may be encoded on a carrier wave.

[0095] The communication subsystem 530 (and / or its components) will typically receive the signal, and the bus 505 can then transfer the signal (and / or the data, instructions, etc. carried by the signal) to the working memory 535, from which the processor 505 retrieves and executes the instructions. The instructions received by the working memory 535 may optionally be stored on the storage device 525 before or after execution by the processor 510.

[0096] Although specific features and aspects have been described with respect to some embodiments, those skilled in the art will recognize that many modifications are possible. For example, the methods and processes described herein may be implemented using hardware components, software components, and / or any combination thereof. In addition, for ease of description, the various methods and processes described herein may be described with respect to specific structural and / or functional components, but the methods provided by the various embodiments are not limited to any specific structural or functional architecture, but may be implemented on any suitable hardware, firmware, and / or software configuration. Similarly, although a specific function is attributed to a specific system component, unless the context otherwise indicates, the function is not limited thereto, but may be distributed among various other system components according to several embodiments.

[0097] In addition, for the convenience of describing the procedures of the methods and processes described herein, they are described in a particular order, but unless the context dictates otherwise, different procedures may be reordered, added, and / or omitted according to different embodiments. In addition, the procedures described for a method or process may be incorporated into other described methods or processes; similarly, system components described according to a specific structural architecture and / or for a system may be organized into other structural architectures and / or incorporated into other described systems. Therefore, although various embodiments are described with or without specific features for the convenience of describing and illustrating certain aspects of these embodiments, unless the context dictates otherwise, the various components and / or features described herein in connection with a specific embodiment may be replaced, added, and / or deleted in other described embodiments. Therefore, although several embodiments are described above, it will be understood that the present invention is intended to cover all modifications and equivalents within the scope of the appended claims.

Claims

1. A method comprising: Analyzing first sensor data using a computing system to compare with a plurality of preset micro-gestures to identify a micro-gesture of a user's finger, wherein the first sensor data is received from at least one first sensor disposed on the wrist-worn device when the user wears the wrist-worn device, and the at least one first sensor is configured to detect the micro-gesture of the user's finger when the user's finger does not contact the wrist-worn device; In response to identifying a first micro-gesture of the user's finger in the first sensor data, determining, using the computing system, whether the first micro-gesture corresponds to one of a finger slide command and a finger drag command for controlling a user interface of the wrist-worn device; as well as Based on determining that the first micro-gesture corresponds to one of a finger slide command and a finger drag command for controlling the user interface of the wrist-worn device, in response to one of the finger slide command and the finger drag command, using the computing system to cause the user interface of the wrist-worn device to perform one or more first functions.

2. The method according to claim 1, characterized in that The computing system includes at least one of: one or more processors on the wrist-worn device, one or more processors on a mobile device communicatively coupled to the wrist-worn device, a machine learning system, an artificial intelligence system, a deep learning system, a neural network, a convolutional neural network, or a fully convolutional network.

3. The method according to claim 1 or 2, characterized in that: The wrist-worn device includes a smart watch, a wrist-worn display device or other wrist-worn user devices.

4. The method according to any one of claims 1 to 3, characterized in that The at least one first sensor includes at least one biosensor, and the at least one biosensor includes at least one of one or more photosensitive blood pressure sensors, one or more electromyography sensors, one or more sound transducers, and one or more motion transducers. Each of the at least one biosensor is configured to detect biological characteristics corresponding to soft tissue movement in or near a wrist of a user wearing the wrist-worn device, wherein the soft tissue includes at least one of one or more muscles, one or more tendons, and one or more blood vessels in or near the wrist of the user.

5. The method according to claim 4, characterized in that The one or more photosensitive blood pressure sensors are configured to optically detect changes in blood volume in one or more blood vessels under the skin tissue at or near the wrist of the user corresponding to the user's finger movements, wherein the one or more electromyography sensors are configured to detect electrical activities of one or more muscles at or near the wrist of the user corresponding to the user's finger movements, wherein the one or more sound transducers are configured to detect sounds within the user's body corresponding to the user's finger movements, and wherein the one or more motion transducers are configured to detect movements of one or more muscles, one or more tendons, and one or more blood vessels corresponding to the user's finger movements.

6. The method according to any one of claims 1 to 5, characterized in that The finger sliding command includes one of an upward sliding command, a downward sliding command, a left sliding command or a right sliding command, wherein the one or more first functions include one of an upward scrolling function, a downward scrolling function, a left scrolling function or a right scrolling function, for browsing the display content on the user interface displayed on the display screen of the wrist-worn device.

7. The method according to claim 6, characterized in that The finger sliding command includes one of lifting a finger or putting down a finger, each finger sliding command is relative to a relaxed posture of fingers of a user's hand and is executed at a medium speed.

8. The method according to any one of claims 1 to 7, characterized in that The finger drag command includes one of a drag up command, a drag down command, a drag left command or a drag right command, wherein the one or more first functions include one of a move up function, a move down function, a move left function or a move right function, for browsing display content on a user interface displayed on a display screen of the wrist-worn device.

9. The method according to claim 8, characterized in that The finger drag command includes one of lifting a finger or putting down a finger, each finger drag command is relative to a relaxed posture of fingers of a user's hand and is executed at a low speed.

10. The method according to any one of claims 1 to 9, characterized in that Further including: analyzing, using the computing system, second sensor data to detect movement of the wrist-worn device relative to at least one of the three rotational axes, the second sensor data being received from at least one second sensor disposed on the wrist-worn device when the user is wearing the wrist-worn device; In response to detecting movement of the wrist-worn device relative to at least one rotational axis in the second sensor data, determining, using the computing system, whether the detected movement of the wrist-worn device is consistent with and operates in conjunction with the first micro-gesture corresponding to one of a finger slide command and a finger drag command for controlling a user interface of the wrist-worn device; as well as Based on determining that the detected movement of the wrist-worn device is consistent with and operates together with the first micro-gesture corresponding to one of a finger slide command and a finger drag command for controlling a user interface of the wrist-worn device, in response to one of the finger slide command and the finger drag command, using a computing system to cause the user interface of the wrist-worn device to perform one or more second functions.

11. The method according to claim 10, characterized in that The at least one second sensor includes at least one of an inertial measurement unit sensor or a gyroscope-based sensor.

12. The method according to claim 10, characterized in that The one or more second functions include at least one of a scrolling function or a dragging function, and each of the one or more second functions is along a direction based on a degree of supination or pronation of a forearm connected to a wrist of a user wearing the wrist-worn device, and the direction is detected by movement of the wrist-worn device.

13. A wrist-worn device, comprising: at least one first sensor configured to detect micro-gestures of one or more fingers of the user without the user's fingers contacting the wrist-worn device; at least one processor; as well as a non-transitory computer-readable medium, communicatively coupled to the at least one processor, the non-transitory computer-readable medium having computer software stored thereon, the computer software comprising an instruction set, which, when executed by the at least one processor, causes the wrist-worn device to: Analyzing first sensor data to compare with a plurality of preset micro-gestures to identify micro-gestures of the user's fingers, wherein the first sensor data is received from at least one first sensor disposed on the wrist-worn device when the user wears the wrist-worn device; In response to identifying a first micro-gesture of the user's finger in the first sensor data, determining whether the first micro-gesture corresponds to one of a finger slide command and a finger drag command for controlling a user interface of the wrist-worn device; as well as Based on determining that the first micro-gesture corresponds to one of a finger slide command and a finger drag command for controlling the user interface of the wrist-worn device, in response to one of the finger slide command and the finger drag command, the user interface of the wrist-worn device is caused to perform one or more first functions.

14. The wrist-worn device according to claim 13, wherein: The wrist-worn device includes a smart watch, a wrist-worn display device or other wrist-worn user devices.

15. The wrist-worn device according to claim 13 or 14, characterized in that: The at least one first sensor includes at least one biosensor, and the at least one biosensor includes at least one of one or more photosensitive blood pressure sensors, one or more electromyography sensors, one or more sound transducers, and one or more motion transducers. Each of the at least one biosensor is configured to detect biological characteristics corresponding to soft tissue movement in or near a wrist of a user wearing the wrist-worn device, wherein the soft tissue includes at least one of one or more muscles, one or more tendons, and one or more blood vessels in or near the wrist of the user.

16. The wrist-worn device according to any one of claims 13 to 15, characterized in that: The finger sliding command includes one of an upward sliding command, a downward sliding command, a left sliding command or a right sliding command, wherein the one or more first functions include one of an upward scrolling function, a downward scrolling function, a left scrolling function or a right scrolling function, for browsing the display content on the user interface displayed on the display screen of the wrist-worn device.

17. The wrist-worn device according to any one of claims 13 to 16, characterized in that: The finger drag command includes one of a drag up command, a drag down command, a drag left command or a drag right command, wherein the one or more functions include one of a move up function, a move down function, a move left function or a move right function, for browsing display content on a user interface displayed on a display screen of the wrist-worn device.

18. The wrist-worn device according to any one of claims 13 to 17, characterized in that: Also included is at least one second sensor configured to detect movement of the wrist-worn device relative to at least one of three rotation axes, wherein when the instruction set is executed by the at least one processor, the wrist-worn device: analyzing second sensor data to detect movement of the wrist-worn device relative to at least one of the three rotation axes, the second sensor data being received from at least one second sensor disposed on the wrist-worn device when the user wears the wrist-worn device; In response to detecting movement of the wrist-worn device relative to at least one rotational axis in the second sensor data, determining, using the computing system, whether the detected movement of the wrist-worn device is consistent with and operates in conjunction with the first micro-gesture corresponding to one of a finger slide command and a finger drag command for controlling a user interface of the wrist-worn device; and Based on determining that the detected movement of the wrist-worn device is consistent with and operates together with the first micro-gesture corresponding to one of a finger slide command and a finger drag command for controlling the user interface of the wrist-worn device, in response to one of the finger slide command and the finger drag command, the user interface of the wrist-worn device is caused to perform one or more second functions.

19. The wrist-worn device according to claim 18, wherein: The at least one second sensor includes at least one of an inertial measurement unit sensor or a gyroscope-based sensor.

20. The wrist-worn device according to claim 18, wherein: The one or more second functions include at least one of a scrolling function or a dragging function, and each of the one or more second functions is along a direction based on a degree of supination or pronation of a forearm connected to a wrist of a user wearing the wrist-worn device, and the direction is detected by movement of the wrist-worn device.

Citation Information

Cited By

  • Equipment control method and electronic equipment

    CN121143623A