Intelligent interaction method and device based on hand bioelectric signal, terminal and medium

Abstract

The application discloses an intelligent interaction method and device based on hand bioelectric signals, a terminal and a medium, and relates to the technical field of intelligent interaction. The method comprises the following steps: capturing bioelectric signals generated when a user performs at least one interactive action through a bioelectric signal acquisition device worn on the wrist of the user; the interactive action type comprises a fingertip contact basic action and a fingertip contact derived action; matching the bioelectric signals with a plurality of bioelectric signal templates respectively, and determining the action performed by the user according to the bioelectric signal templates that are successfully matched; and triggering a corresponding function through the mapping relationship between the interactive action and the function. The application adopts the contact between the thumb and other fingers as an intelligent interaction action, and can meet the needs of a low-key and private interaction scene. On the basis of the basic fingertip contact action, a large number of independent identifiable action instruction sets such as double-click, pressure sensing and fingertip rubbing are formed, and multi-dimensional instruction output such as digital input, text entry and PPT page turning can be realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent interaction technology, and in particular to intelligent interaction methods, devices, terminals and media based on hand bioelectrical signals. Background Technology

[0002] With the rapid development of human-computer interaction technology, smart wearable devices have been widely used in many fields such as office, outdoor, and barrier-free assistance due to their advantages of convenience and discreetness. Electromyography (EMG) interaction, as a non-invasive interaction method based on human muscle electrical signals, has become a research hotspot in the field of wearable human-computer interaction because it does not rely on handheld devices and can achieve hands-free operation.

[0003] In the existing technology, the intelligent interaction method based on bioelectric signals is mainly the single-finger flexion and extension type electromyographic interaction method. It usually uses a small number of electrodes to collect continuous electrical signals of hand muscles, and realizes simple interactive control by recognizing continuous hand movements such as single-finger flexion and extension and fist clenching.

[0004] However, single-finger flexion and extension electromyographic interaction methods are easily interfered with by unconscious movements of other fingers, resulting in low accuracy in motion recognition and a high probability of misoperation. Furthermore, simple flexion and extension of a single finger makes it difficult to input multiple sets of commands or characters through diverse movements, limiting the interaction dimensions and making it difficult to adapt to complex application scenarios.

[0005] Therefore, existing technologies still need improvement and development. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide an intelligent interaction method, device, terminal and medium based on hand bioelectric signals, in order to address the above-mentioned defects of the prior art. The aim is to solve the problems of high error rate and single interaction dimension of the existing single finger flexion and extension electromyography interaction methods.

[0007] The technical solution adopted by this invention to solve the problem is as follows: In a first aspect, embodiments of the present invention provide an intelligent interaction method based on hand bioelectrical signals, the method comprising: The device captures bioelectric signals generated when a user performs at least one interactive action by wearing a bioelectric signal acquisition device on the user's wrist or arm. The types of interactive actions include: basic fingertip contact actions and fingertip contact derivative actions. The types of bioelectric signals include: electromyographic signals and / or neural electrical signals. The bioelectric signals are matched with several preset bioelectric signal templates, and the user's action is determined based on the successfully matched bioelectric signal template; each bioelectric signal template is established based on a specific interactive action. By establishing a pre-defined mapping relationship between interactive actions and functions, the function corresponding to the user's action is triggered.

[0008] In one embodiment, the bioelectric signal acquisition device has multiple electrodes disposed on its inner side; the step of capturing bioelectric signals generated when the user performs at least one interactive action through the bioelectric signal acquisition device worn on the user's wrist includes: The electrodes are used to capture the zonal bioelectrical signals of the corresponding muscle regions when the user performs at least one interactive action. Using the system clock as a reference, the sampling timestamps of the bioelectric signals of each partition are verified, and the bioelectric signals of the partitions with time offsets are interpolated to obtain the optimized bioelectric signals of each electrode. The bioelectric signals are generated based on the bioelectric signals of each optimized partition.

[0009] In one implementation, the fingertip contact basic action includes the action formed by the thumb contacting any of the other fingers.

[0010] In one embodiment, the fingertip contact derivative action includes: actions formed by changing the frequency of the action, the number of fingers involved, the contact force, and the relative movement of the fingertips based on the basic fingertip contact action, as well as palm shape switching action and wrist and arm rotation action.

[0011] In one embodiment, the step of establishing the bioelectric signal template includes: Customize the target interaction action that triggers the target function; The bioelectric signal acquisition device acquires the bioelectric signals of the user when performing the target interactive action and extracts signal features; the signal features include at least one of the following: signal amplitude, time-domain waveform, frequency-domain features, and duration. A bioelectric signal template corresponding to the target interactive action is established based on the signal characteristics.

[0012] In one implementation, the step of triggering the function corresponding to the user's action by means of a preset mapping relationship between interactive actions and functions includes: The mapping relationship between different interactive actions and operation commands is pre-defined; the operation command types include: character input operation commands, demonstration control operation commands, and shortcut operation commands; Based on the user's action, the mapping relationship is queried to obtain the corresponding target operation instruction; The target operation instruction is transmitted to the corresponding target application, and the target application executes the corresponding function based on the target operation instruction.

[0013] In one embodiment, the method further includes: The system outputs visual feedback when a function is triggered via a portable visual interaction device.

[0014] In one embodiment, the method further includes: The activation and deactivation actions of the interactive function of the bioelectric signal acquisition device are preset; When the user performs the activation action, the bioelectric signal acquisition device switches from standby mode to interactive mode; When the user performs the termination action, the bioelectric signal acquisition device exits the interactive mode and returns to the standby mode.

[0015] Secondly, embodiments of the present invention also provide an intelligent interactive device based on hand bioelectrical signals, the device comprising: A bioelectric signal acquisition device, worn on the user's wrist or arm, is used to capture bioelectric signals generated when the user performs at least one interactive action; the types of interactive actions include: basic fingertip contact actions and derivative fingertip contact actions. The signal matching module is used to match the bioelectric signals with a number of preset bioelectric signal templates, and determine the user's action based on the successfully matched bioelectric signal template; each bioelectric signal template is established based on a specific interactive action. The function triggering module is used to trigger the function corresponding to the user's action by means of a preset mapping relationship between interactive actions and functions.

[0016] In one implementation, the function triggering module includes: The storage unit is used to store the pre-defined mapping relationship between different interactive actions and operation instructions; the operation instruction types include: character input operation instructions, demonstration control operation instructions, and shortcut operation instructions; The query unit is used to query the mapping relationship based on the user's action to obtain the corresponding target operation instruction; An execution unit is used to transmit the target operation instruction to the corresponding target application, and the target application executes the corresponding function based on the target operation instruction.

[0017] In one embodiment, the device further includes: A portable visual interaction device used to output visual feedback when a function is triggered.

[0018] Thirdly, embodiments of the present invention also provide a terminal, the terminal including a memory and one or more processors; the memory stores one or more programs; the programs include instructions for executing the intelligent interaction method based on hand bioelectric signals as described above; the processor is used to execute the programs.

[0019] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having stored thereon a plurality of instructions adapted to be loaded and executed by a processor to implement the steps of the intelligent interaction method based on hand bioelectrical signals as described above.

[0020] The beneficial effects of this invention are as follows: In this embodiment, a bioelectric signal acquisition device worn on the user's wrist or arm captures bioelectric signals generated when the user performs at least one interactive action. The interactive action types include basic fingertip contact actions and derived fingertip contact actions. The bioelectric signal types include electromyographic signals and / or neural electrical signals. The bioelectric signals are matched with several preset bioelectric signal templates, and the user's action is determined based on the successfully matched template. Each bioelectric signal template is established based on a specific interactive action. Through a preset mapping relationship between interactive actions and functions, the function corresponding to the user's action is triggered. This invention uses the thumb contacting the fingertips of other fingers as an intelligent interactive action. The bioelectric signals generated by this type of action are clear and have low interference correlation. Simultaneously, the fingertip contact action has a small amplitude and is discreetly executed, meeting the needs of low-key and private interactive scenarios. Secondly, this invention divides interactive actions into two main categories: basic fingertip contact actions and derived fingertip contact actions. Based on basic fingertip contact actions, it expands to form a variety of derived actions such as double-click, triple-click, three-finger contact, pressure sensing, hand posture switching, wrist rotation, and fingertip friction. Through the combination of different actions, a massive set of independent and recognizable action instructions is formed, which can realize multi-dimensional instruction output such as number input, text input, PPT page turning, and shortcut key operation, and can adapt to complex application scenarios such as text input and multi-functional device control. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the intelligent interaction method based on hand bioelectric signals provided in an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of the multi-channel neural signal acquisition device provided in an embodiment of the present invention.

[0024] Figure 3 This is a flowchart illustrating the bioelectric signal optimization method provided in this embodiment of the invention.

[0025] Figure 4This is a schematic diagram of the basic fingertip contact action provided in an embodiment of the present invention.

[0026] Figure 5 This is a schematic diagram of fingertip contact-derived actions provided in an embodiment of the present invention.

[0027] Figure 6 This is a schematic diagram of the combination of basic fingertip contact actions and derived fingertip contact actions provided in the embodiments of the present invention.

[0028] Figure 7 This is a flowchart illustrating the bioelectric signal template generation method provided in this embodiment of the invention.

[0029] Figure 8 This is a schematic diagram of the process from fingertip contact action to function triggering provided by an embodiment of the present invention.

[0030] Figure 9 This is a schematic diagram of the bioelectric signal acquisition device provided in this embodiment of the invention being used in conjunction with smart glasses.

[0031] Figure 10 This is a flowchart illustrating the interaction mode activation and deactivation mechanism provided in this embodiment of the invention.

[0032] Figure 11 This is a schematic diagram of a module of an intelligent interactive device based on hand bioelectrical signals provided in an embodiment of the present invention.

[0033] Figure 12 This is a schematic diagram of the terminal provided in the embodiment of the present invention. Detailed Implementation

[0034] This invention discloses an intelligent interaction method, device, terminal, and medium based on hand bioelectrical signals. To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention.

[0035] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0036] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0037] To address the aforementioned deficiencies in existing technologies, this invention provides an intelligent interaction method based on hand bioelectrical signals. The method includes: capturing bioelectrical signals generated when a user performs at least one interactive action using a bioelectrical signal acquisition device worn on the user's wrist or arm; the interactive action types include: basic fingertip contact actions and fingertip contact derivative actions; the bioelectrical signal types include: electromyographic signals and / or neuroelectrical signals; matching the bioelectrical signals with several preset bioelectrical signal templates, and determining the user's action based on the successfully matched bioelectrical signal templates; each bioelectrical signal template is established based on a specific interactive action; and triggering the function corresponding to the user's action through a preset mapping relationship between interactive actions and functions. This invention uses the thumb and other fingertip contact as the intelligent interactive action, which generates clear bioelectrical signals with low interference correlation. Simultaneously, the fingertip contact action has a small amplitude and is executed discreetly, meeting the needs of low-key and private interaction scenarios. Secondly, this invention divides interactive actions into two main categories: basic fingertip contact actions and derived fingertip contact actions. Based on basic fingertip contact actions, it expands to form a variety of derived actions such as double-click, triple-click, three-finger contact, pressure sensing, hand posture switching, wrist rotation, and fingertip friction. Through the combination of different actions, a massive set of independent and recognizable action instructions is formed, which can realize multi-dimensional instruction output such as number input, text input, PPT page turning, and shortcut key operation, and can adapt to complex application scenarios such as text input and multi-functional device control.

[0038] Taking an office presentation scenario as an example, the user wears a bioelectric signal acquisition device on their wrist and establishes a communication connection with the terminal to be controlled (the presentation terminal). The terminal has pre-stored bioelectric signal templates corresponding to different types of interactive actions, as well as the mapping relationship between interactive actions and function commands. When the user's left thumb and left index finger make a single contact, the bioelectric signal acquisition device collects the bioelectric signal and successfully matches it with a certain bioelectric signal template, thus recognizing the interactive action. The mapping relationship between interactive actions and function commands triggers the PowerPoint presentation to turn the page backward, thereby realizing the control of the PowerPoint presentation.

[0039] like Figure 1 As shown, the method specifically includes the following steps: Step S100: Capture the bioelectric signals generated when the user performs at least one interactive action by using a bioelectric signal acquisition device worn on the user's wrist and arm; the types of interactive actions include: basic fingertip contact actions and derived fingertip contact actions; the types of bioelectric signals include: electromyographic signals and / or neural electrical signals.

[0040] The bioelectric signal acquisition device worn by the user can be implemented using a wristband or similar method. This device can collect bioelectric signals generated when the user performs basic fingertip contact actions and / or derived fingertip contact actions. In this embodiment, the bioelectric signals can be either electromyographic (EMG) signals or neural electrical (NED) signals, or a combination of both. This embodiment categorizes interactive actions into two types: basic fingertip contact actions and derived fingertip contact actions. The core interactive action is the contact of two fingertips. The bioelectric signals generated by this action can be considered short-term signals. Compared to the continuous signals generated by continuous actions such as single-finger flexion and extension, this method offers faster and more accurate signal acquisition and recognition, stronger determinism and reliability in signal control, more intuitive and faster interactive control, and greater concealment in action execution. It also effectively avoids the problems of misoperation and interference from other finger movements that are common with single-finger flexion and extension, and can fully utilize multiple fingers to achieve diverse and interconnected interactive actions.

[0041] In one implementation, the bioelectric signal acquisition device has multiple electrodes disposed on its inner side; the step of capturing bioelectric signals generated when the user performs at least one interactive action via the bioelectric signal acquisition device worn on the user's wrist includes: The electrodes are used to capture the zonal bioelectrical signals of the corresponding muscle regions when the user performs at least one interactive action. Using the system clock as a reference, the sampling timestamps of the bioelectric signals of each partition are verified, and the bioelectric signals of the partitions with time offsets are interpolated to obtain the optimized bioelectric signals of each electrode. The bioelectric signals are generated based on the bioelectric signals of each optimized partition.

[0042] Specifically, in this embodiment, the bioelectrical signal acquisition device worn by the user is a multi-channel neural signal acquisition device, and multiple electrodes (such as...) are arranged around the inner side of the multi-channel neural signal acquisition device. Figure 2 As shown in the image, when the user wears it on their wrist or arm, the electrodes adhere closely to the skin, stably collecting the bioelectrical signals generated when the tip of the thumb contacts the tips of other fingers. Targeted acquisition and timing calibration using multiple electrodes ensure the accuracy and usability of the bioelectrical signals. In practical applications, the bioelectrical signal acquisition device worn by the user has multiple electrodes embedded in its skin, each corresponding to a different muscle region in the wrist and arm. Each electrode works independently, specifically capturing the localized bioelectrical signals generated in the corresponding muscle region when the user performs at least one interactive action. These localized bioelectrical signals are targeted and can accurately reflect the muscle activity characteristics activated by different interactive actions. For example... Figure 3 As shown, due to slight differences in the hardware response speed of each electrode during multi-channel acquisition, time shifts can easily occur in the bioelectrical signals acquired from different channels, affecting the accuracy of signal fusion and recognition. Therefore, it is necessary to use the system clock as a unified reference to verify the sampling timestamps of the bioelectrical signals in each zone one by one, identify the bioelectrical signals in zones with time shifts, and correct the timing deviations by interpolation to synchronize the bioelectrical signals in the time dimension acquired by all electrodes, ultimately obtaining the optimized bioelectrical signals for each electrode. Then, all optimized bioelectrical signals in zones are integrated and processed to filter out redundant interference and enhance effective signal features, ultimately generating bioelectrical signals that can realistically and comprehensively represent user interaction actions.

[0043] For example, using an 8-channel bioelectric signal acquisition device (1 reference electrode + 7 detection electrodes) and a basic fingertip contact action as an example, this section explains the acquisition, timing optimization, and generation process of bioelectric signals, adapting to basic command interactions in office scenarios: The user wears the bioelectric signal acquisition device on the lower forearm. The 7 detection electrodes on the inner side of the wristband are divided into two rows along the longitudinal axis of the forearm, corresponding to the muscles controlled by the thumb and the muscles controlled by the index finger to the little finger, respectively. The distance between adjacent electrodes is 1.5cm, and all electrodes are in close contact with the skin. The reference electrode is attached to the ulnar side of the wrist where there is no muscle activity, ensuring that each detection electrode can accurately acquire signals from the corresponding muscle area. When the user performs a basic fingertip contact action, i.e., a single contact between the tips of the left thumb and index finger, each detection electrode starts acquiring signals simultaneously. Among them, the 4 detection electrodes corresponding to the radial flexor and ulnar flexor muscles capture the muscle electrical activity corresponding to this action, generating 4 channels of zoned bioelectric signals respectively. The remaining 3 detection electrodes, because the corresponding muscles are not activated, acquire signals with amplitudes below the activation threshold, which are judged as invalid signals and temporarily stored. Due to slight differences in the hardware response speed of each detection electrode during multi-channel acquisition, the sampling timestamps of the four effective zone bioelectric signals exhibit a time offset of ±1ms, affecting the accuracy of subsequent signal fusion. Therefore, using the built-in system clock of the bioelectric signal acquisition device as a unified benchmark, the sampling timestamps of the four zone bioelectric signals are verified one by one. Two signals with a time offset exceeding 0.5ms are identified, and linear interpolation is used to supplement the corresponding sampling points during the offset period, ensuring complete synchronization of the sampling time of the four zone bioelectric signals, resulting in four optimized zone bioelectric signals. Finally, spatial weighted fusion processing is performed on the four optimized zone bioelectric signals, using the muscle activation degree of each signal as the weight to remove redundant interference signals and enhance the effective signal features corresponding to the fingertip contact action, ultimately generating a bioelectric signal that can accurately characterize the basic fingertip contact action.

[0044] Taking a 6-channel bioelectric signal acquisition device (1 reference electrode + 5 detection electrodes) and fingertip contact-derived actions as an example, this paper explains the acquisition, timing optimization, and generation process of bioelectric signals under different electrode numbers and different interactive actions, adapting to quick control interactions in outdoor scenarios. The user wears the bioelectric signal acquisition device on their wrist. The 5 detection electrodes are evenly distributed on the inner side of the closed loop, corresponding to different muscle areas around the wrist. The electrodes are in close contact with the skin to ensure that the acquired regional bioelectric signals are free from significant interference. When the user performs a fingertip contact-derived action, i.e., the left thumb, index finger, and middle finger simultaneously contact and apply moderate pressure, each detection electrode synchronously acquires the regional bioelectric signals of the corresponding muscle area. Three detection electrodes capture the muscle electrical signals activated by the finger contact action, generating 3 effective regional bioelectric signals. The signals acquired by the other 2 detection electrodes are environmental noise and are filtered out. Due to a slight delay in signal transmission from the bioelectric signal acquisition device in outdoor environments, the sampling timestamps of the three effective partition bioelectric signals exhibit a time offset of ±2ms. To avoid this offset affecting the accuracy of action recognition, the sampling timestamps of the three partition bioelectric signals are verified using the system clock as a reference. Cubic spline interpolation is employed to correct the timing of signals with time offsets, filling in the sampling gaps during the offset periods and ensuring that the three partition bioelectric signals are aligned in the time dimension, resulting in three optimized partition bioelectric signals. Subsequently, the three optimized partition bioelectric signals are filtered, normalized, and integrated to retain the signal characteristics corresponding to pressure contact and eliminate outdoor environmental interference, ultimately generating bioelectric signals that can comprehensively characterize the three-finger pressure contact-derived actions.

[0045] In one implementation, the fingertip contact basic action includes the action formed by the thumb contacting any of the other fingers.

[0046] Specifically, the basic fingertip contact action refers to the fingertip contact action formed by the user's thumb as the reference point, touching and pressing against any other finger. It is the most basic and core basic action unit in the entire interactive action system (such as...). Figure 4(As shown). This type of basic movement is based on a multi-channel bioelectric signal acquisition device worn on the user's wrist and arm. Multiple electrodes arranged inside the device are in close contact with the skin of the wrist and arm, which can accurately capture the regional bioelectric signals generated in the corresponding muscle areas when the thumb and other fingertips come into contact. Compared with the continuous single-finger flexion and extension movements commonly used in existing technologies, this two-finger contact movement is a short-term movement that is completed instantaneously, outputting only clear and brief bioelectric signals. It is faster and more accurate in signal acquisition and recognition, and the determinism and reliability of signal control are stronger. The movement is executed covertly and intuitively, which can effectively avoid the problem that single-finger flexion and extension is easily interfered with by the movements of other fingers and has a high probability of misoperation. At the same time, it provides a stable basic movement support for the expansion, combination and extension of various fingertip contact derivative movements, ensuring the scalability of the entire interactive movement system.

[0047] For example, a user wears a bioelectric signal acquisition device on their wrist. When the user performs the action of touching the tips of their thumb and index finger together, this action is a basic fingertip contact action. The bioelectric signal acquisition device captures the bioelectric signal generated when the user performs this action, so as to trigger the corresponding command, such as controlling the corresponding terminal to output the number 1.

[0048] In one implementation, the fingertip contact derivative action includes: actions formed by changing the frequency of the action, the number of fingers involved, the contact force, and the relative movement of the fingertips based on the basic fingertip contact action, as well as palm shape switching action and wrist and arm rotation action.

[0049] Specifically, the fingertip contact derivative actions in this embodiment are based on the action formed by the contact between the thumb and two other fingers. Without departing from the fingertip contact interaction logic, they are extended interactive actions formed by expanding and changing the action execution form, participating fingers, and movement methods. Various derivative actions can be combined to achieve a high-density command input effect similar to single-handed nine-key text input. Fingertip contact derivative actions specifically include two main types: one type is actions formed by adjusting parameters on the basic fingertip contact action, such as double-tap, triple-tap, etc., by changing the execution frequency of the action, and three-finger or more fingertip contact actions by increasing or decreasing the number of fingers involved (e.g., double-tap, triple-tap, etc.). Figure 5 As shown), different intensities of pressure contact actions are created by adjusting the contact pressure between the fingertips, and actions such as fingertip friction and fingertip sliding are created by changing the relative motion state between the fingertips; another type is the supporting extended actions added to improve the interactive control system, specifically including palm shape switching actions consisting of clenching a fist, opening it, and quickly clenching a fist twice, and wrist and arm rotation actions consisting of wrist rotation and arm rotation. Figure 6As shown, fingertip-derived actions and basic fingertip-contact actions work together to form a complete set of interactive actions with rich types and diverse command dimensions. This significantly expands the number of recognizable actions and control command types.

[0050] For example, high-density character input can be achieved through combinations of different fingertip contact-derived actions, similar to the effect of single-handed nine-key text input. Palm shape switching actions such as clenching a fist, opening it, and quickly clenching a fist twice can be used for auxiliary control such as starting / stopping interactive modes and switching functions. Two-handed combinations further expand the interactive dimensions, while wrist and arm rotation actions can achieve functions such as viewing angle adjustment and page switching. Various derivative actions work in conjunction with basic actions to form a complete set of interactive actions with rich types and diverse command dimensions.

[0051] Step S200: Match the bioelectric signals with several preset bioelectric signal templates respectively, and determine the user's action based on the successfully matched bioelectric signal templates; each bioelectric signal template is established based on a specific interactive action.

[0052] Specifically, the system pre-establishes corresponding bioelectrical signal templates for each specific interactive action. Each bioelectrical signal template uniquely corresponds to a specific interactive action and is associated with and stored in relation to the action's unique bioelectrical signal characteristics. After completing the acquisition, timing calibration, and optimization of the user's real-time bioelectrical signals, the system compares these signals one by one with the multiple pre-set bioelectrical signal templates. When a real-time bioelectrical signal successfully matches the characteristics of one of the pre-set bioelectrical signal templates, the system can accurately determine the user's current interactive action based on the interactive action corresponding to that successfully matched bioelectrical signal template. This allows for accurate identification of the user's interactive intent and avoids action confusion and misidentification.

[0053] In one implementation, the step of establishing the bioelectric signal template includes: Customize the target interaction action that triggers the target function; The bioelectric signal acquisition device acquires the bioelectric signals of the user when performing the target interactive action and extracts signal features; the signal features include at least one of the following: signal amplitude, time-domain waveform, frequency-domain features, and duration. A bioelectric signal template corresponding to the target interactive action is established based on the signal characteristics.

[0054] This embodiment allows users to flexibly customize specific interactive actions to trigger specific functions according to their actual usage needs. Specifically, for example... Figure 7As shown, after determining the target interactive action, a bioelectric signal acquisition device worn on the user's wrist is used to collect the bioelectric signals generated when the user performs the target interactive action, and corresponding signal features are extracted from the collected and optimized bioelectric signals. These signal features include at least one of the following: signal amplitude, time-domain waveform, frequency-domain features, and signal duration. Finally, based on the extracted signal features, a bioelectric signal template corresponding to the target interactive action is established, ensuring that each bioelectric signal template corresponds one-to-one with a specific interactive action. This provides a standard reference for subsequent real-time matching and comparison of bioelectric signals and accurate identification of the user's actual actions.

[0055] For example, based on the needs of an office presentation scenario, a user customizes the target interaction action for triggering the "full-screen PPT playback" function as "rapidly double-tapping the tips of the left thumb and index finger." The user then wears the bioelectric signal acquisition device, ensuring the electrodes are in close contact with the skin, and activates the template creation mode of the device, repeatedly performing the customized "rapid double-tapping the tips of the thumb and index finger" action three times. The bioelectric signal acquisition device simultaneously collects the corresponding bioelectric signal each time the action is performed. Finally, the system calculates the mean and fuses the features of the extracted bioelectric signals from the three actions, removes abnormal fluctuations, generates a bioelectric signal template for the target interaction action "rapid double-tapping the tips of the thumb and index finger," and binds the target interaction action to the bioelectric signal template. Subsequent times the user performs this target interaction action, the "full-screen PPT playback" function can be triggered through signal matching.

[0056] Step S300: Trigger the function corresponding to the user's action by means of a preset mapping relationship between interactive actions and functions.

[0057] Specifically, the system pre-establishes and stores a mapping relationship between different interactive actions and corresponding control functions. After determining the interactive action currently being performed by the user, the system directly calls and triggers the function that uniquely corresponds to the user's action based on this mapping relationship, thereby completing the entire interactive process from bioelectrical signal acquisition and action recognition to instruction execution.

[0058] In one implementation, the step of triggering the function corresponding to the user's action by means of a preset mapping relationship between interactive actions and functions includes: The mapping relationship between different interactive actions and operation commands is pre-defined; the operation command types include: character input operation commands, demonstration control operation commands, and shortcut operation commands; Based on the user's action, the mapping relationship is queried to obtain the corresponding target operation instruction; The target operation instruction is transmitted to the corresponding target application, and the target application executes the corresponding function based on the target operation instruction.

[0059] Specifically, such as Figure 8 As shown, the entire interactive control logic in this embodiment, from action to instruction to application execution, includes: the system pre-establishes and sets the mapping relationship between different interactive actions and operation instructions. The operation instructions specifically include three types: character input operation instructions, demonstration control operation instructions, and shortcut operation instructions. During the interaction process, the system first generates exclusive bioelectric signal templates by extracting the bioelectric signal features corresponding to various interactive actions, and associates various preset operation instructions with the corresponding bioelectric signal templates. This allows users to quickly achieve stable output of instructions or characters with simple fingertip touch actions, and various instructions or characters can be flexibly pre-customized according to different application scenarios such as office work, presentations, and input. For example, the fingertip contact action of the left thumb with the left index, middle, ring, and little fingers can be defined as the character input operation command 1, 2, 3, and 4 respectively. Alternatively, the fingertip contact of the thumb with the index finger can be defined as the PPT presentation control operation command to turn the page backward, and the fingertip contact of the thumb with the middle finger can be defined as the PPT presentation control operation command to turn the page forward. After the system determines the actual interactive action performed by the user through signal acquisition and template matching, it queries the preset mapping relationship based on the action to obtain the corresponding target operation command, and transmits the target operation command to the corresponding target application. Finally, the target application executes the corresponding function based on the target operation command, completing the interactive control from fingertip contact action to command execution.

[0060] In one implementation, the method further includes: The system outputs visual feedback when a function is triggered via a portable visual interaction device.

[0061] Specifically, to provide users with immediate and intuitive operation confirmation and result feedback, this embodiment equips the bioelectric signal acquisition device with portable visual interaction devices such as smart glasses, head-mounted display devices, and near-eye display terminals (e.g., Figure 9(As shown). When the system successfully triggers the corresponding function based on the interaction action recognition result and the preset mapping relationship, it simultaneously transmits the function execution information to the portable visual interaction device. The device then outputs visual feedback information that corresponds one-to-one with the triggered function in real time. For example, it displays a page switching prompt when triggering PPT page turning, displays the corresponding input character when triggering character input, and displays a mode status prompt when triggering the start or stop of the interaction mode. This visual feedback allows users to know in real time whether the action recognition and function triggering are successful in a hands-free, concealed interaction scenario. It forms a complete interactive closed loop with the bioelectrical signal acquisition, action recognition, and function triggering process, which not only improves the certainty and reliability of interactive operations but also further optimizes the overall user experience of bioelectrical signal interaction.

[0062] In one implementation, the method further includes: The activation and deactivation actions of the interactive function of the bioelectric signal acquisition device are preset; When the user performs the activation action, the bioelectric signal acquisition device switches from standby mode to interactive mode; When the user performs the termination action, the bioelectric signal acquisition device exits the interactive mode and returns to the standby mode.

[0063] Specifically, such as Figure 10 As shown, the bioelectric signal acquisition device also features a start / stop control mechanism in its interactive mode. By pre-setting specific start and stop actions, the system can accurately switch between standby and interactive modes, ensuring controllability and low accidental touches in interactive operations. For example, clenching a fist for 2 seconds serves as the start action for interactive mode, while opening the palm serves as the stop action. Simultaneously, switching palm shapes and wrist rotation can also be used for auxiliary control such as starting / stopping interactive modes and switching functions, further enhancing the integrity, flexibility, and practicality of the entire bioelectric signal interaction solution.

[0064] In practical applications, the system pre-defines the activation action to start the interactive function and the termination action to stop the interactive function. When the user performs the preset activation action (such as clenching a fist for 2 seconds), the bioelectric signal acquisition device immediately switches from the low-power standby mode to the command recognition interactive mode, starts to start electrode signal acquisition, performs action recognition and function matching, and prepares to respond to the user's subsequent interactive actions. When the user performs the preset termination action (such as opening the palm), the bioelectric signal acquisition device immediately exits the interactive mode, stops signal acquisition and command recognition, and returns to the standby mode to reduce device power consumption and avoid accidental triggering of interactive commands by daily limb activities.

[0065] Based on the above embodiments, the present invention also provides an intelligent interactive device based on hand bioelectrical signals, such as... Figure 11 As shown, the device includes: A bioelectric signal acquisition device 01 is worn on the user's wrist and arm to capture bioelectric signals generated when the user performs at least one interactive action; the types of interactive actions include: basic fingertip contact action and derivative fingertip contact action. The signal matching module 02 is used to match the bioelectric signals with a number of preset bioelectric signal templates, and determine the user's action based on the successfully matched bioelectric signal templates; each bioelectric signal template is established based on a specific interactive action. The function triggering module 03 is used to trigger the function corresponding to the user's action by means of a preset mapping relationship between interactive actions and functions.

[0066] In one embodiment, the bioelectric signal acquisition device includes: Multiple electrodes are disposed on the inner side, each of which is used to capture the zonal bioelectrical signals of the corresponding muscle area when the user performs at least one interactive action; The signal optimization unit is used to verify the sampling timestamp of the bioelectrical signals of each partition based on the system clock, and to interpolate and supplement the bioelectrical signals of the partitions with time offsets to obtain the optimized bioelectrical signals of each electrode. A signal fusion unit is used to generate the bioelectric signal based on the bioelectric signals of each optimized partition.

[0067] In one implementation, the fingertip contact basic action includes the action formed by the thumb contacting any of the other fingers.

[0068] In one embodiment, the fingertip contact derivative action includes: actions formed by changing the frequency of the action, the number of fingers involved, the contact force, and the relative movement of the fingertips based on the basic fingertip contact action, as well as palm shape switching action and wrist and arm rotation action.

[0069] In one embodiment, the device further includes: The template creation module is used to customize the target interactive actions that trigger the target function; The bioelectric signal acquisition device acquires the bioelectric signals of the user when performing the target interactive action and extracts signal features; the signal features include at least one of the following: signal amplitude, time-domain waveform, frequency-domain features, and duration. A bioelectric signal template corresponding to the target interactive action is established based on the signal characteristics.

[0070] In one implementation, the function triggering module includes: The storage unit is used to store the pre-defined mapping relationship between different interactive actions and operation instructions; the operation instruction types include: character input operation instructions, demonstration control operation instructions, and shortcut operation instructions; The query unit is used to query the mapping relationship based on the user's action to obtain the corresponding target operation instruction; An execution unit is used to transmit the target operation instruction to the corresponding target application, and the target application executes the corresponding function based on the target operation instruction.

[0071] In one embodiment, the device further includes: A portable visual interaction device used to output visual feedback when a function is triggered.

[0072] In one embodiment, the device further includes: An interactive start / stop module is used to pre-set the start and stop actions of the interactive function of the bioelectric signal acquisition device. When the user performs the activation action, the bioelectric signal acquisition device switches from standby mode to interactive mode; When the user performs the termination action, the bioelectric signal acquisition device exits the interactive mode and returns to the standby mode.

[0073] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 12 As shown, the terminal includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an intelligent interaction method based on hand bioelectrical signals. The display screen can be a liquid crystal display (LCD) or an e-ink display.

[0074] Those skilled in the art will understand that Figure 12 The block diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0075] In one implementation, the terminal's memory stores one or more programs, and these programs are configured to be executed by one or more processors, and the programs contain instructions for performing intelligent interaction methods based on hand bioelectrical signals.

[0076] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0077] In summary, this invention discloses an intelligent interaction method, device, terminal, and medium based on hand bioelectrical signals, relating to the field of intelligent interaction technology. The method uses a bioelectrical signal acquisition device worn on the user's wrist to capture bioelectrical signals generated when the user performs at least one interactive action. The types of interactive actions include: basic fingertip contact actions and fingertip contact derivative actions; the types of bioelectrical signals include: electromyographic signals and / or neural electrical signals. The bioelectrical signals are matched with several preset bioelectrical signal templates, and the user's action is determined based on the successfully matched bioelectrical signal template. Each bioelectrical signal template is established based on a specific interactive action. Through a preset mapping relationship between interactive actions and functions, the function corresponding to the user's action is triggered. This invention uses the thumb and other fingertip contact as an intelligent interactive action. The bioelectrical signals generated by this type of action are clear and have low interference correlation. Simultaneously, the amplitude of the fingertip contact action is small and the execution is discreet, meeting the needs of low-key and private interaction scenarios. Secondly, this invention divides interactive actions into two main categories: basic fingertip contact actions and derived fingertip contact actions. Based on basic fingertip contact actions, it expands to form a variety of derived actions such as double-click, triple-click, three-finger contact, pressure sensing, hand posture switching, wrist rotation, and fingertip friction. Through the combination of different actions, a massive set of independent and recognizable action instructions is formed, which can realize multi-dimensional instruction output such as number input, text input, PPT page turning, and shortcut key operation, and can adapt to complex application scenarios such as text input and multi-functional device control.

[0078] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A smart interaction method based on hand bioelectrical signals, characterized in that, The method includes: The device captures bioelectric signals generated when a user performs at least one interactive action by wearing a bioelectric signal acquisition device on the user's wrist or arm. The types of interactive actions include: basic fingertip contact actions and fingertip contact derivative actions. The types of bioelectric signals include: electromyographic signals and / or neural electrical signals. The bioelectric signals are matched with several preset bioelectric signal templates, and the user's action is determined based on the successfully matched bioelectric signal template; each bioelectric signal template is established based on a specific interactive action. By establishing a pre-defined mapping relationship between interactive actions and functions, the function corresponding to the user's action is triggered.

2. The intelligent interaction method based on hand bioelectrical signals according to claim 1, characterized in that, The bioelectric signal acquisition device is equipped with multiple electrodes on its inner side; The steps of capturing bioelectrical signals generated when a user performs at least one interactive action using a bioelectrical signal acquisition device worn on the user's wrist include: The electrodes are used to capture the zonal bioelectrical signals of the corresponding muscle regions when the user performs at least one interactive action. Using the system clock as a reference, the sampling timestamps of the bioelectric signals of each partition are verified, and the bioelectric signals of the partitions with time offsets are interpolated to obtain the optimized bioelectric signals of each electrode. The bioelectric signals are generated based on the bioelectric signals of each optimized partition.

3. The intelligent interaction method based on hand bioelectrical signals according to claim 1, characterized in that, The basic fingertip contact action includes the action formed by the thumb contacting any other finger.

4. The intelligent interaction method based on hand bioelectrical signals according to claim 1, characterized in that, The fingertip contact derivative movements include: movements formed by changing the frequency of the movements, the number of fingers involved, the contact force, and the relative movement of the fingertips based on the basic fingertip contact movements, as well as palm shape switching movements and wrist and arm rotation movements.

5. The intelligent interaction method based on hand bioelectrical signals according to claim 1, characterized in that, The steps for establishing the bioelectric signal template include: Customize the target interaction action that triggers the target function; The bioelectric signal acquisition device acquires the bioelectric signals of the user when performing the target interactive action and extracts signal features; the signal features include at least one of the following: signal amplitude, time-domain waveform, frequency-domain features, and duration. A bioelectric signal template corresponding to the target interactive action is established based on the signal characteristics.

6. The intelligent interaction method based on hand bioelectrical signals according to claim 1, characterized in that, The steps to trigger the function corresponding to the user's action by means of a preset mapping relationship between interactive actions and functions include: The mapping relationship between different interactive actions and operation commands is pre-defined; the operation command types include: character input operation commands, demonstration control operation commands, and shortcut operation commands; Based on the user's action, the mapping relationship is queried to obtain the corresponding target operation instruction; The target operation instruction is transmitted to the corresponding target application, and the target application executes the corresponding function based on the target operation instruction.

7. The intelligent interaction method based on hand bioelectrical signals according to claim 6, characterized in that, The method further includes: The system outputs visual feedback when a function is triggered via a portable visual interaction device.

8. The intelligent interaction method based on hand bioelectrical signals according to claim 1, characterized in that, The method further includes: The activation and deactivation actions of the interactive function of the bioelectric signal acquisition device are preset; When the user performs the activation action, the bioelectric signal acquisition device switches from standby mode to interactive mode; When the user performs the termination action, the bioelectric signal acquisition device exits the interactive mode and returns to the standby mode.

9. A smart interactive device based on hand bioelectrical signals, characterized in that, The device includes: A bioelectric signal acquisition device, worn on the user's wrist or arm, is used to capture bioelectric signals generated when the user performs at least one interactive action; the types of interactive actions include: basic fingertip contact actions and derivative fingertip contact actions. The signal matching module is used to match the bioelectric signals with a number of preset bioelectric signal templates, and determine the user's action based on the successfully matched bioelectric signal template; each bioelectric signal template is established based on a specific interactive action. The function triggering module is used to trigger the function corresponding to the user's action by means of a preset mapping relationship between interactive actions and functions.

10. The intelligent interactive device based on hand bioelectrical signals according to claim 9, characterized in that, The function triggering module includes: The storage unit is used to store the pre-defined mapping relationship between different interactive actions and operation instructions; the operation instruction types include: character input operation instructions, demonstration control operation instructions, and shortcut operation instructions; The query unit is used to query the mapping relationship based on the user's action to obtain the corresponding target operation instruction; An execution unit is used to transmit the target operation instruction to the corresponding target application, and the target application executes the corresponding function based on the target operation instruction.

11. The intelligent interactive device based on hand bioelectrical signals according to claim 9, characterized in that, The device further includes: A portable visual interaction device used to output visual feedback when a function is triggered.

12. A terminal, characterized in that, The terminal includes a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the intelligent interaction method based on hand bioelectric signals as described in any one of claims 1 to 8; the processor is used to execute the programs.

13. A computer-readable storage medium storing a plurality of instructions, characterized in that, The instructions are applicable to be loaded and executed by a processor to implement the steps of the intelligent interaction method based on hand bioelectrical signals as described in any one of claims 1 to 8.

Images