An interactive control method and device based on cooperation of wrist state and finger gesture

Abstract

The application discloses an interactive control method and system based on cooperation of wrist state and finger gesture. First-level coding is realized through wrist turning, and working mode or keyboard layout is selected; second-level coding is realized through finger action type and action object, and specific instructions are input. The system defaults to a free mode, and daily finger actions are not analyzed. The application realizes unified high fault tolerance, large capacity and efficient input, and can be widely applied to scenes such as intelligent finger sleeves, glove type keyboards, sign language translation, AR / VR control and the like.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction technology, specifically relating to a collaborative interaction method and wearable system that integrates wrist mode switching and fine finger control, and in particular a human-computer interaction scheme that achieves full-function input through two-level encoding of wrist and finger movements. Background Technology

[0002] In existing gesture interaction solutions, wrist interaction (such as smartwatch gestures) has a limited number of commands but is less prone to accidental triggering; finger interaction (such as data gloves) has a rich set of commands, but everyday activities are easily misinterpreted as commands. The lack of effective coordination between the two leads to either limited functionality or a high rate of accidental triggering.

[0003] In addition, existing gesture input solutions have significant limitations in character input: they either rely on virtual keyboards (which require visual feedback and cannot be used for touch typing) or each gesture corresponds to a single character (the number of commands is severely insufficient), making it difficult to achieve efficient and full-featured text input and system control on wearable devices. Summary of the Invention

[0004] Purpose of the invention: To provide a two-level collaborative interaction framework of "wrist setting mode and finger issuing command", which selects the working mode or keyboard layout by wrist rotation encoding and inputs specific characters or commands by finger movement encoding, thereby achieving a unity of high fault tolerance, large capacity of commands and efficient input.

[0005] Technical solution:

[0006] The core of this invention is a two-level coding system:

[0007] Level 1 – Wrist Coding: Mode Switching and Keyboard Layout Selection

[0008] By detecting the direction and number of wrist rotations using a wrist IMU, the following functions can be achieved:

[0009] Basic mode switching: Turning the wrist outward enters control mode, turning the wrist inward exits and returns to free mode. In the default free mode, normal finger movements do not trigger any commands.

[0010] Sub-mode / keyboard layout selection: Using the encoding method of "inward flip prefix + outward flip count" (e.g., flip inward twice consecutively, then outward with different counts), select to enter different sub-control modes. For example: 1 outward flip = letter keyboard mode; 2 outward flips = number and symbol mode; 3 outward flips = function shortcut key mode; 4 outward flips = mouse mode; 5 outward flips = handwriting input mode.

[0011] Level Two – Finger Encoding: Character and Command Input

[0012] In the current sub-control mode, the hand sensor detects the type of finger movement and the object of action to realize specific command input:

[0013] Action types include, but are not limited to, single click, double click, long press, short swipe, and long swipe.

[0014] Target: The interaction between the thumb and the index, middle, ring, and little fingers, with the interaction points including the proximal phalanx, distal phalanx, or the side of the finger.

[0015] Mapping logic: The same finger action (such as "thumb clicking the proximal knuckle of the index finger") is mapped to different output results in different sub-control modes. In alphabetic keyboard mode, it is mapped to the letter A; in numeric symbol mode, it is mapped to the number 1; in function shortcut key mode, it is mapped to Ctrl+C (copy); and in mouse mode, it is mapped to a left click.

[0016] Beneficial effects:

[0017] High fault tolerance: Everyday finger movements do not trigger any commands, fundamentally solving the problem of accidental triggering.

[0018] Large instruction capacity: The two-level coding system can generate dozens of distinguishable instructions, covering full keyboard input and complex system control.

[0019] Efficient input: Users only need to memorize a few basic finger movements to achieve "touch typing" level input efficiency in different modes.

[0020] Applicable to various scenarios: It can be used in glove keyboards, smart finger mouse / handwriting input, sign language translation gloves, AR / VR control, rehabilitation exoskeleton control and other scenarios. Attached Figure Description

[0022] Figure 1 System architecture diagram (wrist IMU + finger sensor).

[0023] Figure 2 : Diagram of a two-level coding system (first level: wrist coding for selecting modes, second level: finger coding for inputting commands).

[0024] Figure 3 : Collaborative workflow diagram (wrist rotation → mode switching → finger command → device execution).

[0025] Figure 4 : Multi-mode switching state machine. Detailed Implementation

[0027] Example 1: Finger Input Method Based on Hierarchical Coding

[0028] The user wears a glove system that integrates a wrist IMU sensor and finger flexion / pressure sensors. The system is in free mode by default, allowing the user to perform daily activities freely; finger movements are not interpreted as input commands.

[0029] When a user needs to input text, they double-flip their wrist inward twice to enter the encoding monitoring mode. Then, they flip their wrist outward once, and the system vibrates to confirm and enters the "alphabet keyboard mode." In this mode, a thumb tap on the proximal knuckle of the index finger inputs A, a double tap inputs B, a long press inputs C, and a short swipe on the side of the index finger inputs a space. A thumb tap on the middle finger inputs D, a double tap inputs E, and so on. Through combinations of the thumb and four fingers, and by differentiating between tap, double tap, long press, and swipe gestures, all 26 letters and common punctuation marks can be covered.

[0030] When numbers need to be entered, the user turns their wrist inward to exit letter mode, then turns it inward twice and outward twice to enter "number and symbol mode". At this point, a single tap with the thumb and a double tap with the index finger inputs 1, a double tap inputs 2, a long press inputs @, and so on.

[0031] When you need to use system shortcuts, switch to "Function Shortcut Key Mode". Tap your index finger with your thumb to press Ctrl+C (copy), double-tap to press Ctrl+V (paste), and long-press to press Ctrl+Z (undo).

[0032] When you need to control the cursor, switch to "mouse mode". Slide your index finger in the air to move the cursor, tap your index finger with your thumb to click the left button, tap your middle finger to click the right button, and slide your thumb on the side of your index finger to scroll the wheel.

[0033] When handwriting input is required, switch to "Handwriting Mode". Write with your index finger on any surface or in the air, and the IMU will capture the fingertip trajectory for handwriting recognition. Double-tap with your thumb to confirm the recognition result.

[0034] After inputting, turn your wrist inward to return to free mode.

[0035] Example 2: Smart Finger Wrap Mouse and Handwriting Input

[0036] Users wear smart finger sleeves with integrated IMU. Turning the wrist outward enters mouse mode; the index finger slides across any surface to control the cursor, and the thumb taps the side of the index finger to click the left button. Switching to handwriting mode, the index finger writes on the desktop or in the air, and the thumb double-clicks to confirm recognition. Turning the wrist inward exits the handwriting mode, and the finger sleeve reverts to a regular finger sleeve.

[0037] Example 3: Glove-style keyboard

[0038] The user wears gloves with integrated finger sensors. Turning the wrist inward enters control mode; the thumb then lightly touches the proximal knuckles of the index, middle, ring, and little fingers in sequence, which are mapped to A, B, C, and D respectively. Turning the wrist outward exits control mode.

[0039] Example 4: Sign Language Translation Gloves

[0040] The user wears gloves. Turning the wrist inward switches to translation mode, and sign language gestures are converted into speech or text in real time. Turning the wrist outward exits translation mode, and the gloves return to normal, avoiding mistranslation of everyday gestures.

[0041] Example 5: Two-handed coordinated AR / VR control

[0042] The left wrist IMU handles mode switching, while the right glove handles finger command input. Turning the left wrist outward enters control mode; pinching the right thumb selects, sliding the middle finger rotates, and making a fist confirms. Turning the left wrist inward exits control mode.

[0043] Example 6: Rehabilitation Exoskeleton Control

[0044] The healthy wrist IMU controls the switching of modes, while the exoskeleton of the affected hand performs grasping. This, combined with patents 1 and 3, creates a complete rehabilitation assistive system.

Claims

1. A method for interactive control based on cooperation of wrist state and finger gesture, characterized in that, include: The wrist movement is detected by a sensor worn on the wrist, and at least two working modes are defined, including at least a default free mode and a switchable control mode. In the free mode, finger movements detected by sensors worn on the hand are not interpreted as control commands to the target device; In the control mode, the preset finger gestures detected by the hand sensor are interpreted as control commands to the target device; The switching between the free mode and the control mode is triggered by a specific wrist rotation motion detected by the wrist sensor.

2. The method according to claim 1, characterized in that, The specific flipping actions include flipping outwards and flipping inwards, which correspond to entering control mode from free mode and returning to free mode from control mode, respectively.

3. The method according to claim 1, characterized in that, It also includes a menu mode, triggered by continuous inward flipping detected by the wrist sensor; in menu mode, different finger gestures are mapped to menu item selection or parameter adjustment commands.

4. The method according to claim 1, characterized in that, The preset hand gestures include at least one of the following: pinching the thumb and index finger together, bending the index finger, sliding the middle finger, making a fist with all five fingers, and spreading the five fingers.

5. The method according to claim 1, characterized in that, The control mode further includes multiple sub-control modes selected by wrist rotation encoding, each corresponding to a different device control function or input keyboard layout; wherein, the N consecutive rotations in a specific direction detected by the wrist sensor serve as a prefix, and the subsequent count of rotations in another direction is used to select the corresponding sub-control mode.

6. The method according to claim 5, characterized in that, In the same sub-control mode, different finger movement types are mapped to different output instructions. The movement types include at least one of the following: single click, double click, long press, short swipe, and long swipe. The same finger and the same movement type are mapped to different output results in different sub-control modes.

7. The method according to claim 6, characterized in that, The objects of the finger movements include the interaction between the thumb and the index, middle, ring, and little fingers, respectively, and the interaction positions include the proximal phalanx, distal phalanx, or side of each finger.

8. A wearable interactive system implementing the method of any one of claims 1 to 7, characterized in that, include: Wrist sensing unit: Worn on the user's wrist, it includes an inertial measurement unit to detect wrist rotation movements in order to define and switch working modes; Hand sensing unit: worn on the user's hand or fingers, including a bend sensor, pressure sensor or inertial measurement unit, used to detect finger gestures and movement types; Processing unit: It is communicatively connected to the wrist sensing unit and the hand sensing unit respectively, and executes the mode switching logic and the two-level gesture parsing logic.

9. The system according to claim 8, characterized in that, The hand sensing unit is a sensor array integrated into a glove or finger sleeve.

10. An interactive system comprising the wearable interactive system of claim 8 or 9, and a target device communicatively connected to the wearable interactive system.

Images