Human-computer interaction-based electrooculography-controlled finger rehabilitation method, device, medium, and terminal

By collecting and classifying electrooculogram (EOG) signals to control mechanical gloves for rehabilitation training, the problems of autonomous selection and data evaluation in small-area rehabilitation training have been solved, achieving refined and efficient rehabilitation results.

CN117643537BActive Publication Date: 2026-11-10SHANGHAI INST OF MICROSYSTEM & INFORMATION TECH CHINESE ACAD OF SCI
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311606314.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-28
Publication Date
2026-11-10
Estimated Expiration
2043-11-28

AI Technical Summary

Technical Problem

Existing rehabilitation training for small areas makes it difficult for patients to choose their own rehabilitation training programs based on their actual situation, and lacks data collection, display, and tracking evaluation mechanisms, resulting in a time-consuming, labor-intensive, and highly dependent rehabilitation process.

Method used

By collecting patients' electrooculogram (EOG) signals, performing preprocessing and coarse-fine classification, and controlling the robotic glove to execute corresponding rehabilitation movements, patients can autonomously choose the direction of rehabilitation exercises, and the rehabilitation progress can be displayed and assessed in real time.

Benefits of technology

It enables patients to choose their own rehabilitation training programs, reduces reliance on manual intervention, improves the precision and efficiency of rehabilitation training, and provides real-time data display and evaluation mechanisms.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117643537B_ABST
    Figure CN117643537B_ABST
Patent Text Reader

Abstract

This application provides a method, device, terminal, and medium for electrooculography (EOG)-controlled finger rehabilitation based on human-computer interaction. It collects the patient's EOG signals and classifies the data to obtain the corresponding eye movement types. These eye movement types are then used to control a robotic glove to assist the patient in performing corresponding rehabilitation movements. This application allows the patient to form a closed-loop pathway between their own movement intentions and limb movements, enabling them to autonomously choose any eye movement direction according to their own will. Furthermore, the corresponding robotic glove movement is a more refined finger movement. This solves the problems of existing technologies, such as high reliance on manual intervention, time constraints, and the inability to conduct targeted rehabilitation training based on data, particularly for small-area rehabilitation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of intelligent rehabilitation, and in particular to a method, device, medium, and terminal for electrooculography-controlled finger rehabilitation based on human-computer interaction. Background Technology

[0002] In past research on intelligent rehabilitation technology, researchers have mostly focused on the movement of larger body parts, including the arms, hands, feet, or assistive devices such as wheelchairs. However, for smaller body parts such as limbs / fingers that are unable to move freely due to external trauma, individual differences in rehabilitation training for these smaller parts exist. Different patients have varying degrees of injury, rehabilitation progress, and physical conditions, making it difficult for them to choose their own rehabilitation exercises using existing methods. They often need to rely on manual intervention to follow a standardized rehabilitation process. This results in a long, time-constrained, and highly dependent rehabilitation process, making the entire rehabilitation process time-consuming and labor-intensive. Furthermore, existing small-part rehabilitation training often lacks mechanisms for data collection, display, and tracking of rehabilitation progress, making it impossible to adjust training plans in a timely manner, further impacting rehabilitation outcomes. Summary of the Invention

[0003] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a method, device, medium and terminal for human-computer interaction-based electrooculography-controlled finger rehabilitation, which can solve the problems of difficulty in the existing small-part rehabilitation training where it is difficult for patients to choose the corresponding rehabilitation training program according to their actual situation, and the difficulty in collecting, displaying in real time and tracking and evaluating rehabilitation training data.

[0004] To achieve the above and other related objectives, the first aspect of this application provides a human-computer interaction-based electrooculography (EOG)-controlled finger rehabilitation method, which involves collecting raw EOG data based on the patient's eye movement trajectory; preprocessing the raw EOG data and performing a coarse-fine two-layer classification operation on the preprocessed raw EOG data to obtain EOG signal categories; and sending the EOG signal category identified by the current timestamp to a mechanical glove worn on the patient's hand, so that the mechanical glove performs the corresponding rehabilitation action based on the EOG signal category.

[0005] In some embodiments of the first aspect of this application, the process of performing a coarse-fine two-layer classification operation on preprocessed raw electrooculogram (EOG) data to obtain an EOG signal category includes: extracting features from the preprocessed EOG signal to generate a feature representation of the EOG signal; and inputting the feature representation of the EOG signal into a coarse-fine two-layer classifier to obtain an EOG signal category.

[0006] In some embodiments of the first aspect of this application, the coarse-fine dual-layer classifier includes a coarse classifier and a fine classifier; the coarse classifier includes multiple first-level binary classifiers, and the output result of each first-level binary classifier is the electrooculogram (EOG) signal category corresponding to a positive sample or a negative sample; wherein, the process of the coarse-fine dual-layer classifier determining the EOG signal category is as follows: the feature representation of the EOG signal is input into each first-level binary classifier in the coarse classifier; if only one first-level binary classifier outputs a positive sample, then the EOG signal category corresponding to the positive sample is taken as the output result of the coarse classifier; if only one first-level binary classifier outputs a non-positive sample, then the feature representation of the EOG signal is input into the fine classifier for secondary classification, and the classification result of the fine classifier is taken as the EOG signal category corresponding to the feature representation of the EOG signal.

[0007] In some embodiments of the first aspect of this application, the process by which the fine classifier determines the category of electrooculogram (EOG) signal is as follows: extracting the EOG signal categories output by all first-level binary classifiers in the coarse classifier; arranging and combining the EOG signal categories output by all first-level binary classifiers in pairs to form a corresponding number of groups, and equipping each group with a second-level binary classifier for reclassification; counting the EOG signal categories output by all second-level binary classifiers, and using the EOG signal category with the largest number as the classification result of the fine classifier.

[0008] In some embodiments of the first aspect of this application, the preprocessing of the raw electrooculogram (EOG) data includes: performing peak extraction on the raw EOG data to extract eye movement signals containing valid eye activity; and performing filtering, bad conduction removal, baseline drift removal, and downsampling on the eye movement signals containing valid eye activity.

[0009] In some embodiments of the first aspect of this application, after acquiring the raw electrooculogram (EOG) data, the method further performs the following operation: sending the raw EOG data to a display device for real-time display, including: packaging the raw EOG data into raw EOG packaged data; sending the raw EOG packaged data and determining whether the transmission was successful after transmission; if successful, continuing to read the raw EOG packaged data at the next moment; otherwise, deleting the raw EOG packaged data. Upon receiving the raw EOG packaged data, the display device unpacks it and displays it in real-time.

[0010] In some embodiments of the first aspect of this application, the process of sending the electrooculogram (EOG) signal category to a mechanical glove worn on the patient's hand, so that the mechanical glove performs a corresponding rehabilitation action based on the EOG signal category, includes: issuing an operation command to the mechanical glove, so that the mechanical glove reads the EOG signal category through a serial port and judges the EOG signal category; if it is a preset category, then controlling the angle of the servo motor on the mechanical glove according to the pulse modulation width value corresponding to the EOG signal category, so as to generate a rehabilitation action corresponding to the EOG signal category; otherwise, the mechanical glove continues to receive operation commands.

[0011] To achieve the above and other related objectives, a second aspect of this application provides an electrooculography (EOG)-controlled finger rehabilitation device based on human-computer interaction. The device includes: a data acquisition module for acquiring raw EOG data based on the patient's eye movement trajectory; a coarse-fine dual-layer classification module for preprocessing the raw EOG data and performing coarse-fine dual-layer classification on the preprocessed raw EOG data to obtain EOG signal categories; a rehabilitation action execution module for sending the EOG signal category identified by the current timestamp to a mechanical glove worn on the patient's hand, so that the mechanical glove performs corresponding rehabilitation actions based on the EOG signal category; and a real-time display module for sending the raw EOG data to a display device for real-time display.

[0012] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the human-computer interaction-based electrooculography-controlled finger rehabilitation method.

[0013] To achieve the above and other related objectives, a fourth aspect of this application provides an electronic terminal, comprising: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal performs the human-computer interaction-based electrooculography-controlled finger rehabilitation method.

[0014] As described above, the electrooculography-controlled finger rehabilitation method, device, medium, and terminal based on human-computer interaction of this application have the following beneficial effects: By collecting the patient's electrooculography (EOG) signals and classifying the data, the corresponding eye movement types are obtained. These eye movement types are then used to control a robotic glove to assist the patient in performing corresponding rehabilitation actions. This application enables the patient to form a closed-loop pathway between their own movement intentions and limb movements, allowing the patient to autonomously choose any eye movement direction according to their own wishes, and the corresponding robotic glove movement mode is a more refined finger movement. This solves the problems of existing technologies, such as strong reliance on manual intervention, time constraints, and the inability to conduct targeted rehabilitation training based on data, particularly for small-area rehabilitation. Attached Figure Description

[0015] Figure 1 The diagram shows a flowchart of an embodiment of the electrooculography-controlled finger rehabilitation method based on human-computer interaction according to this application.

[0016] Figure 2 The diagram shows a schematic diagram of the connection of a 7-channel electrooculography (EOG) acquisition device in one embodiment of the human-computer interaction-based EOG-controlled finger rehabilitation method of this application.

[0017] Figure 3 The diagram shows the distribution of electrooculography (EOG) signal acquisition electrodes in one embodiment of the human-computer interaction-based electrooculography-controlled finger rehabilitation method of this application.

[0018] Figure 4 This illustration shows a schematic diagram of the eye movement paradigm during the training process of a coarse-fine two-layer classifier in one embodiment of the human-computer interaction-based electrooculography-controlled finger rehabilitation method of this application.

[0019] Figure 5 This illustration shows a flowchart of real-time reading of electrooculogram (EOG) data in one embodiment of the human-computer interaction-based electrooculogram (EOG)-controlled finger rehabilitation method of this application.

[0020] Figure 6 The diagram shows a flowchart of preprocessing, feature extraction, and classification in one embodiment of the human-computer interaction-based electrooculography-controlled finger rehabilitation method of this application.

[0021] Figure 7 The diagram shows a flowchart of a coarse-fine two-layer classifier in one embodiment of the human-computer interaction-based electrooculography-controlled finger rehabilitation method of this application.

[0022] Figure 8 The diagram shows a flowchart illustrating the process of a mechanical glove receiving instructions and driving finger movement in one embodiment of the human-computer interaction-based electrooculography-controlled finger rehabilitation method of this application.

[0023] Figure 9a This paper shows a schematic diagram of the structure of an embodiment of the electrooculography-controlled finger rehabilitation device based on human-computer interaction according to this application.

[0024] Figure 9b This illustration shows a structural schematic diagram of another embodiment of the electrooculography-controlled finger rehabilitation device based on human-computer interaction according to this application.

[0025] Figure 10 The diagram shows the structure of the electrooculography-controlled finger rehabilitation electronic terminal based on human-computer interaction according to this application. Detailed Implementation

[0026] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0027] It should be noted that in the following description, reference is made to the accompanying drawings, which illustrate several embodiments of this application. It should be understood that other embodiments may also be used, and changes in mechanical composition, structure, electrical system, and operation may be made without departing from the spirit and scope of this application. The following detailed description should not be considered limiting, and the scope of the embodiments of this application is defined only by the claims of the published patent. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. Spatially related terms, such as “upper,” “lower,” “left,” “right,” “below,” “below,” “lower part,” “above,” “upper part,” etc., may be used herein to illustrate the relationship between one element or feature shown in the figures and another element or feature.

[0028] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," "fixing," and "holding" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0029] Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, operation, element, component, item, kind, and / or group, but do not preclude the presence, occurrence, or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition arise only when combinations of elements, functions, or operations are inherently mutually exclusive in some manner.

[0030] To address the problems mentioned above, this invention provides a method, device, medium, and terminal for electrooculography-controlled finger rehabilitation based on human-computer interaction. It aims to solve the problems in existing small-area rehabilitation training where patients cannot independently select appropriate rehabilitation training programs based on their actual conditions, and where rehabilitation training data is difficult to collect, display in real-time, and track and evaluate. Furthermore, to make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention are further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention.

[0031] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0032] <1> Electrooculography (EOG): EOG is a bioelectrical signal generated by eye movements. When the eye moves, the potential difference between the retina and cornea changes; this change can be detected by electrodes on the skin's surface. EOG is commonly used to track the direction of eye movements and horizontal or vertical eye movements.

[0033] <2> Eye-tracking paradigms refer to a series of experimental designs that use eye-tracking technology to study an individual's visual attention, information processing, and decision-making processes when performing a specific task. Different paradigms are used to study different cognitive processes.

[0034] <3> ADS1299 Integrated Circuit Board: The ADS1299 is a high-performance, multi-channel bioelectrical signal acquisition chip manufactured by Texas Instruments. It is commonly used for acquiring electroencephalograms (EEGs) and other bioelectrical signals. An integrated circuit board typically includes the ADS1299 chip, along with necessary power management and signal processing components to facilitate development and research.

[0035] <4> STM32F303 Integrated Circuit Board: The STM32F303 is a 32-bit ARM Cortex-M4 microcontroller manufactured by STMicroelectronics. It is suitable for various embedded applications, such as industrial control and medical devices. Integrated circuit boards typically contain the microcontroller and its supporting hardware, enabling developers to design and implement specific applications.

[0036] <5> Qt Framework: Qt is a cross-platform C++ application framework widely used for developing graphical user interface (GUI) applications as well as non-GUI tools. Qt provides a suite of tools and libraries that enable software developers to create applications with a consistent user experience across different operating systems.

[0037] <6> Filtering: Filtering is a fundamental step in signal processing used to remove unwanted components, such as noise, from a signal, or to extract useful signal components. Filters can be low-pass, high-pass, band-pass, or band-stop, depending on the target frequency range.

[0038] <7> Bad conduction removal: In bioelectrical signal processing, bad conduction removal refers to identifying and removing abnormal signals caused by electrode detachment, cable movement, or other interference sources.

[0039] <8> Baseline drift removal: Baseline drift removal is a signal preprocessing step used to eliminate low-frequency drift in the signal. This drift is usually caused by long-term instability of the sensor or changes in physiological conditions.

[0040] <9> Kurtosis: Kurtosis is a statistical measure that describes the shape of a probability distribution. It measures the thickness of the tails and the height of the peaks of a data distribution, relative to the normal distribution. The kurtosis of a normal distribution is 3; a kurtosis greater than 3 indicates a thicker tail, while a kurtosis less than 3 indicates a thinner tail.

[0041] <10> OVR-SVM classifier (One-vs-Rest Support Vector Machine): OVR-SVM distinguishes each class from all other classes by training a binary SVM for each class and then selecting the classification result with the highest confidence.

[0042] <11> OVO-SVM (One-vs-One Support Vector Machine): OVO-SVM trains a binary SVM for each pair of classes, and finally determines the classification result through voting or other decision rules. If there are N classes, then N(N-1) / 2 classifiers need to be trained.

[0043] This invention provides a human-computer interaction-based electrooculogram (EOG)-controlled finger rehabilitation method, a system for implementing the human-computer interaction-based EOG-controlled finger rehabilitation method, and a storage medium storing an executable program for implementing the human-computer interaction-based EOG-controlled finger rehabilitation method. Regarding the implementation of the human-computer interaction-based EOG-controlled finger rehabilitation method, this invention will describe exemplary implementation scenarios.

[0044] like Figure 1 The diagram illustrates a flowchart of a human-computer interaction-based electrooculography (EOG)-controlled finger rehabilitation method according to an embodiment of the present invention. The human-computer interaction-based EOG-controlled finger rehabilitation method in this embodiment mainly includes the following steps:

[0045] Step S11: Collect raw electrooculogram data based on the patient's eye movement trajectory.

[0046] In one embodiment of the present invention, the process of collecting raw electrooculogram (EOG) data based on the patient's eye movement trajectory includes: applying conductive gel to the patient's eyes, having the patient view a visual interface and perform eye movements, and collecting data using a 7-channel EOG acquisition device placed on the patient's scalp. Exemplarily, the raw eye movement types include five eye movement paradigms: looking up, looking down, looking left, looking right, and blinking. The EOG signals are collected and the data is saved for subsequent preprocessing and coarse-to-fine two-level classification operations.

[0047] For example, such as Figure 2 As shown, the aforementioned 7-channel acquisition device includes nine electrodes, an ADS1299, and an STM32F303 integrated circuit board connected in sequence. The data acquisition process using the 7-channel EEG acquisition device placed on the patient's scalp involves the following steps: EEG signals enter the CH1-CH7 channels of the ADS1299 through the seven electrodes. A reference electrode and a ground terminal are also provided. Pins PA5, PA6, PA7, PD8, PE8, PD9, and PD10 of the ADS1299 are connected to the pins SPI_CLK, SPI_OUT, SPI_IN, SPI_CS, SPI_DRDY, SPI_START, and RESET of the STM32F303, respectively, and are transmitted to the host computer via the USB port of the STM32F303 for subsequent preprocessing and coarse / fine double-layer classification operations.

[0048] For example, Figure 3 This diagram illustrates the distribution of data acquisition electrodes on a patient's face according to one embodiment of the present invention. Figure 3 Seven electrodes were connected around the subject's eyes, with the reference electrode placed on the subject's right earlobe and the ground electrode placed in the center of the subject's forehead. During electrooculography (EOG) signal acquisition, the patient remained relaxed, with their hands resting naturally on their laps, and performed eye movements as desired. During these eye movements, the subject maintained relative stillness between their body and head.

[0049] In one embodiment of the present invention, Figure 4An eye-tracking paradigm diagram is shown during the training process of a coarse-fine two-layer classifier. When data acquisition begins, a crosshair appears in the center of the screen to prompt the patient to focus their gaze. After receiving a key input from the patient, the screen begins playing visuals, including rest and movement interfaces. The rest interface plays randomly for 0.8-1.2 seconds, followed by one of five random movements for 2 seconds; this process is repeated 20 times as one trial. The acquired electrooculogram (EOG) signals, after preprocessing and feature extraction, are input into the coarse-fine two-layer classifier for training, and the trained classifier model is saved.

[0050] In one embodiment of the present invention, after acquiring raw electrooculogram (EOG) data, the method further performs the following operation: sending the raw EOG data to a display device for real-time display, including: packaging the raw EOG data into raw EOG packaged data; sending the raw EOG packaged data and determining whether the transmission was successful after transmission; if successful, continuing to read the raw EOG packaged data at the next moment; otherwise, deleting the raw EOG packaged data. Upon receiving the raw EOG packaged data, the display device unpacks it and displays it in real-time.

[0051] Specifically, Figure 5 This illustration shows a flowchart of a process in which raw electrooculogram (EOG) data is sent to a display device for real-time display, according to an embodiment of the present invention. The display device includes a Qt-based server and a Python-based client. After the server reads the raw EOG signal via a serial port, it packages the data and sends it to the client program module. The client reads the data, unpacks it to obtain the raw EOG data, and then sends the raw EOG data to the real-time display module and the peak detection module. The real-time display module can display the unprocessed EOG signal on a computer screen in real time, facilitating intuitive observation of the signal state.

[0052] Step S12: Preprocess the raw electrooculogram (EOG) data, and perform a coarse-fine two-layer classification operation on the preprocessed raw EOG data to obtain the EOG signal category.

[0053] In one embodiment of the present invention, the preprocessing of the raw electrooculogram (EOG) data includes: performing peak extraction on the raw EOG data to extract eye movement signals containing valid eye activity; and performing filtering, bad conduction removal, baseline drift removal, and downsampling on the eye movement signals containing valid eye activity.

[0054] Specifically, raw electrooculogram (EOG) data typically corresponds to instantaneous changes in eye movement. The purpose of peak extraction on the raw EOG data is to detect the peak values ​​of the raw EOG data and determine the peak range by setting a threshold, so as to accurately locate the time range of eye movement and effectively extract data that is highly correlated with eye movement.

[0055] For example, such as Figure 6 The process of filtering, removing bad leads, removing baseline drift, and downsampling eye movement signals containing valid eye activity includes: bandpass filtering of the eye movement signal from 0.1 to 15 Hz, removing bad leads, removing baseline drift, and then downsampling to 250 Hz. Removing bad leads involves identifying and removing abnormal signals caused by external interference or equipment malfunction to improve signal quality and ensure the accuracy and reliability of subsequent analysis. Removing baseline drift removes baseline noise or drift from biomedical signals. Baseline drift refers to the slow change in the signal baseline (i.e., the signal's average value or reference level) over time. The purpose of baseline drift removal is to restore the true waveform of the signal for more accurate signal feature analysis.

[0056] In one embodiment of the present invention, the process of performing a coarse-fine two-layer classification operation on the preprocessed raw electrooculogram (EOG) data to obtain the EOG signal category includes: extracting features from the preprocessed EOG signal to generate a feature representation of the EOG signal; and inputting the feature representation of the EOG signal into a coarse-fine two-layer classifier to obtain the EOG signal category.

[0057] Furthermore, the coarse-fine dual-layer classifier includes a coarse classifier and a fine classifier; the coarse classifier includes multiple first-level binary classifiers, and the output result of each first-level binary classifier is the electrooculogram (EOG) signal category corresponding to a positive sample or a negative sample; wherein, the process of the coarse-fine dual-layer classifier determining the EOG signal category is as follows: the feature representation of the EOG signal is input into each first-level binary classifier in the coarse classifier; if only one first-level binary classifier outputs a positive sample, then the EOG signal category corresponding to the positive sample is taken as the output result of the coarse classifier; if only one first-level binary classifier outputs a non-positive sample, then the feature representation of the EOG signal is input into the fine classifier for secondary classification, and the classification result of the fine classifier is taken as the EOG signal category corresponding to the feature representation of the EOG signal.

[0058] Furthermore, the process by which the fine classifier determines the category of electrooculogram (EOG) signal is as follows: extract the EOG signal categories output by all first-level binary classifiers in the coarse classifier; arrange the EOG signal categories output by all first-level binary classifiers in pairs to form a corresponding number of groups, and equip each group with a second-level binary classifier for reclassification; count the EOG signal categories output by all second-level binary classifiers, and use the EOG signal category with the most numbers as the classification result of the fine classifier.

[0059] Figure 7 A flowchart illustrating a coarse-fine two-layer classifier in one embodiment of the present invention is shown. First, the preprocessed electrooculogram (EOG) signal is used for feature extraction using statistical parameters. Here, the channel of the EOG signal refers to the electrode configuration used to record these potential changes. Each channel consists of a pair of electrodes that can capture the potential difference generated on the skin surface during eye movement. The EOG channels in this invention include, but are not limited to, horizontal and vertical channels. Exemplarily, 10 statistical features in each channel are used as parameters to characterize the EOG signal, namely, minimum, maximum, mean, median, mode, first quartile, third quartile, interquartile range, standard deviation, and kurtosis coefficient, where the standard deviation is expressed as shown in Formula 1. S represents the standard deviation of the sample, N represents the sample data size, μ represents the arithmetic mean of the data set, and i represents the i-th sample.

[0060]

[0061] The kurtosis coefficient is expressed as shown in Formulas 2 and 3. k represents the value of the kurtosis coefficient, E represents the expected value, σ represents the standard deviation of the sample, N represents the sample size, and x... i Let represent the i-th observation, and μ represent the sample mean.

[0062]

[0063]

[0064] Furthermore, the coarse-fine two-layer classifier includes a trained and optimized OVR-SVM (One-vs-RestSupport Vector Machine) classifier and an OVO-SVM (One-vs-One Support Vector Machine) classifier. The OVR classifier creates K SVM classifiers for K categories of data, each classifier responsible for distinguishing one category from all other categories. The OVO classifier, for K categories of data, creates K*(K-1) / 2 independent SVM classifiers, each classifier responsible for distinguishing a pair of categories.

[0065] For example, the coarse-fine dual-layer classifier in this invention includes five categories of electrooculogram (EOG) signals: looking up, looking down, looking left, looking right, and blinking. This invention uses an OVR-SVM containing five SVM classifiers as the coarse classifier and an OVO-SVM containing ten SVM classifiers as the fine classifier for secondary subdivision. If the OVR-SVM classifier successfully classifies the signal, the classification result is output. If the OVR-SVM classifier fails to classify the signal, i.e., there are multiple positive classification results, the EOG signal is input into the OVO-SVM for secondary subdivision.

[0066] Specifically, the decision function of the i-th class of the OVR-SVM classifier used in this invention is shown in Equation 4.

[0067]

[0068] in, Here, k(x) is a Lagrange multiplier, and k(x) is the class label of the training sample. +1 represents the i-th class, and -1 represents all other classes. t ,x new ) is a kernel function used to map samples to a high-dimensional space, b i It is the bias term of the i-th classifier, with the superscript i∈{1,2,3,4,5}.

[0069] The binary SVM decision function between class i and class j of the OVO-SVM classifier is shown in Equation 5.

[0070]

[0071] in, Here, k(x) is a Lagrange multiplier, and k(x) is the class label of the training sample. +1 represents the i-th class, and -1 represents all other classes. t ,x new ) is a kernel function used to map samples to a high-dimensional space, b ij It is the bias term between the i-th classifier and the j-th classifier, with index i∈{1,2,3,4,5}, where i <j。

[0072] This invention has 5 categories. For each category, it is designated as the +1 class, and all samples from the other 4 categories are designated as the -1 class. Therefore, the first-layer OVR-SVM classifier constructs 5 binary classifiers. If the classification result has only one +1, the classification result is directly output. For samples with multiple +1 classification results, the OVO-SVM model is used for secondary classification. OVO constructs a binary SVM between every two classes, requiring a total of 5 classes to be built. That is, with 10 classifiers, the category with the most classification results is considered the output category, thus achieving real-time classification.

[0073] It is worth noting that the advantage of the coarse-fine two-layer classifier combining the OVR-SVM and OVO-SVM classifiers used in this invention is that, due to the possibility of complex eye movements in patients, even after processing by the OVR-SVM classifier, some raw electrooculogram (EOG) data may still be unable to be accurately classified. This invention specifically sets up a coarse-fine two-layer classifier combining the OVR-SVM and OVO-SVM classifiers to improve the classification accuracy of the raw EOG data. It should be noted that SVM is a binary classification model, and its goal is to find an optimal hyperplane that maximizes the interval between data points of different classes. The OVO and OVR classifiers are two different strategies for achieving this goal. The advantage of the OVR-SVM classifier is that the number of classifiers only increases linearly with the number of classes, which is more efficient when there are many classes. However, this invention involves only 5 classes, and using only the OVR-SVM classifier will lead to an imbalance between classifiers, especially when the number of samples in some classes is much greater than in others. Therefore, the present invention specifically incorporates an OVO-SVM classifier to improve classification accuracy.

[0074] Step S13: Send the electrooculogram signal category identified by the current timestamp to the mechanical glove on the patient's hand, so that the mechanical glove can perform the corresponding rehabilitation action based on the electrooculogram signal category.

[0075] In one embodiment of the present invention, the process of sending the electrooculogram (EOG) signal category to a mechanical glove worn on the patient's hand, so that the mechanical glove performs a corresponding rehabilitation action based on the EOG signal category, includes: issuing an operation command to the mechanical glove, so that the mechanical glove reads the EOG signal category through a serial port and judges the EOG signal category; if it is a preset category, then controlling the angle of the servo motor on the mechanical glove according to the pulse modulation width value corresponding to the EOG signal category, so as to generate a rehabilitation action corresponding to the EOG signal category; otherwise, the mechanical glove continues to receive operation commands.

[0076] like Figure 8 The diagram illustrates a flowchart of an embodiment of the present invention, showing a mechanical glove worn on a patient's hand receiving commands and driving finger movements. The process includes: first, opening a USB serial port and reading data through the USB serial port; determining whether the read data is one of five commands; if not, continuing to read commands until one of the five commands is found; if so, controlling the angle of the servo motor according to the pulse modulation width value corresponding to the command, thereby enabling the glove to drive the bending and extension of a single finger, achieving a therapeutic and rehabilitative effect; the quality of the subject's eye movement signal determines whether it can be converted into a correct movement command.

[0077] like Figure 9a The diagram illustrates a structural schematic of an eye-controlled finger rehabilitation device based on human-computer interaction, according to an embodiment of the present invention. In this embodiment, the eye-controlled finger rehabilitation device 900 based on human-computer interaction includes:

[0078] Data acquisition module 901: Used to acquire raw electrooculogram data based on the patient's eye movement trajectory.

[0079] Coarse-fine dual-layer classification module 902: used to preprocess the raw electrooculogram (EOG) data and perform coarse-fine dual-layer classification on the preprocessed raw EOG data to obtain the EOG signal category.

[0080] Rehabilitation action execution module 903: Used to send the electrooculogram signal category identified by the current timestamp to the mechanical glove worn on the patient's hand, so that the mechanical glove can perform the corresponding rehabilitation action based on the electrooculogram signal category.

[0081] Real-time display module 904: used to send the raw electrooculogram data to the display device for real-time display.

[0082] It should be noted that the above embodiments of the human-computer interaction-based electrooculogram (EOG)-controlled finger rehabilitation device are only illustrated by the division of the above-described program modules when performing human-computer interaction-based EOG-controlled finger rehabilitation. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. Furthermore, the human-computer interaction-based EOG-controlled finger rehabilitation device and the human-computer interaction-based EOG-controlled finger rehabilitation method embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0083] In one embodiment of the present invention, Figure 9bThis diagram illustrates the structure of a human-computer interaction-based electrooculography (EOG)-controlled finger rehabilitation device according to another embodiment of the present invention. It includes a 7-channel EOG acquisition device and a host computer. Data transmission between the 7-channel EOG acquisition device and the host computer is via USB. The host computer includes a real-time reading module, a real-time display module, a peak detection module, a preprocessing and classification module, and a robotic glove. The 7-channel EOG acquisition device is used to acquire the subject's EOG signals; the real-time reading module reads the EOG signals and distributes them to the real-time display module and the peak detection module; the real-time display module displays the EOG signals in real time; the peak detection module identifies the time range of eye movements by detecting the peak value of the signal, thereby extracting data related to eye movements; the preprocessing and classification module performs real-time preprocessing, feature extraction, and classification on the extracted EOG data, and the classification results are sent in the form of instructions; the robotic glove receives instructions and guides the patient to complete finger movements.

[0084] The electrooculogram-controlled finger rehabilitation method based on human-computer interaction provided in this invention can be implemented on the terminal side or the server side. For the hardware structure of the electrooculogram-controlled finger rehabilitation terminal based on human-computer interaction, please refer to [link to relevant documentation]. Figure 1 This is a schematic diagram of an optional hardware structure of an eye-controlled finger rehabilitation terminal 100 based on human-computer interaction provided in an embodiment of the present invention. The terminal 100 can be a mobile phone, computer device, tablet device, personal digital processing device, factory back-end processing device, etc. The eye-controlled finger rehabilitation terminal 100 based on human-computer interaction includes: at least one processor 101, a memory 102, at least one network interface 104, and a user interface 106. The various components in the device are coupled together through a bus system 105. It is understood that the bus system 105 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 105 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 1 The general will label all buses as bus systems.

[0085] The user interface 106 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0086] It is understood that memory 102 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0087] In this embodiment of the invention, the memory 102 is used to store various types of data to support the operation of the human-computer interaction-based electrooculogram (EOG)-controlled finger rehabilitation terminal 100. Examples of this data include any executable program for operation on the human-computer interaction-based EOG-controlled finger rehabilitation terminal 100, such as the operating system 1021 and application program 1022. The operating system 1021 includes various system programs, such as a framework layer, a core library layer, and a driver layer, for implementing various basic services and handling hardware-based tasks. The application program 1022 may include various applications, such as a media player and a browser, for implementing various application services. The implementation of the human-computer interaction-based EOG-controlled finger rehabilitation method provided in this embodiment of the invention can be included in the application program 1022.

[0088] The methods disclosed in the above embodiments of the present invention can be applied to processor 101, or implemented by processor 101. Processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 101 or by instructions in the form of software. The processor 101 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 101 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 101 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0089] In an exemplary embodiment, the human-computer interaction-based electrooculography (EOG)-controlled finger rehabilitation terminal 100 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to execute the aforementioned human-computer interaction-based EOG-controlled finger rehabilitation method.

[0090] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented using computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0091] In the embodiments provided in this application, the computer-readable and writable storage medium may include read-only memory, random access memory, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, flash memory, USB flash drive, portable hard drive, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Additionally, any connection may be appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable and writable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are intended for non-transient, tangible storage media. The disks and optical discs used in the application include compact discs (CDs), laser discs, optical discs, digital multifunction discs (DVDs), floppy disks, and Blu-ray discs, where disks typically copy data magnetically, while optical discs use lasers to copy data optically.

[0092] In summary, this application provides a method, device, terminal, and medium for electrooculography (EOG)-controlled finger rehabilitation based on human-computer interaction. By collecting the patient's EOG signals and classifying the data, the corresponding eye movement types are obtained. These eye movement types are then used to control a robotic glove to assist the patient in performing corresponding rehabilitation actions. This application allows patients to form a closed-loop pathway between their own movement intentions and limb movements, enabling them to autonomously choose any eye movement direction according to their wishes. Furthermore, the corresponding robotic glove movement is a more refined finger movement. This solves the problems of existing technologies, such as high reliance on manual intervention, time constraints, and the inability to conduct targeted rehabilitation training based on data, particularly for small-area rehabilitation. Therefore, this application effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0093] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A human-computer interaction-based electrooculography-controlled finger rehabilitation device, characterized in that, include: Data acquisition module: used to acquire raw electrooculogram data based on the patient's eye movement trajectory; Coarse and fine dual-layer classification module: used to preprocess the raw electrooculogram (EOG) data and perform coarse and fine dual-layer classification on the preprocessed raw EOG data to obtain the EOG signal category; The process includes: extracting features from the preprocessed electrooculogram (EOG) signal to generate a feature representation of the EOG signal; inputting the feature representation of the EOG signal into a coarse-fine dual-layer classifier to obtain the EOG signal category; wherein, the coarse-fine dual-layer classifier includes a coarse classifier and a fine classifier; the coarse classifier includes multiple first-level binary classifiers, and the output result of each first-level binary classifier is the EOG signal category corresponding to a positive sample or a negative sample; wherein, the process of the coarse-fine dual-layer classifier determining the EOG signal category is as follows: inputting the feature representation of the EOG signal into each first-level binary classifier in the coarse classifier; if only one first-level binary classifier outputs a positive sample, then the EOG signal category corresponding to the positive sample is taken as the output result of the coarse classifier; if only one first-level binary classifier outputs a non-positive sample, then the feature representation of the EOG signal is input into the fine classifier for secondary classification, and the classification result of the fine classifier is taken as the EOG signal category corresponding to the feature representation of the EOG signal; Rehabilitation action execution module: used to send the electrooculogram signal category identified by the current timestamp to the mechanical glove worn on the patient's hand, so that the mechanical glove can perform the corresponding rehabilitation action based on the electrooculogram signal category; Real-time display module: used to send the raw electrooculogram data to the display device for real-time display.

2. The eye-tracking finger rehabilitation device based on human-computer interaction according to claim 1, characterized in that, The process by which the fine classifier determines the category of electroocular signals is as follows: Extract the electrooculogram (EOG) signal categories output by all first-level binary classifiers in the coarse classifier; arrange the EOG signal categories output by all first-level binary classifiers in pairs to form a corresponding number of groups, and equip each group with a second-level binary classifier for reclassification; The electrooculogram (EOG) signal categories output by all second-level binary classifiers are counted, and the category with the most frequent EOG signals is used as the classification result of the fine classifier.

3. The electrooculography-controlled finger rehabilitation device based on human-computer interaction according to claim 1, characterized in that, The preprocessing of the raw electrooculogram data includes: Peak extraction is performed on the raw electrooculogram data to extract eye movement signals containing valid eye movements; The eye movement signal containing valid eye activity is subjected to filtering, bad lead removal, baseline drift removal, and downsampling.

4. The electrooculography-controlled finger rehabilitation device based on human-computer interaction according to claim 1, characterized in that, After acquiring the raw electrooculogram (EOG) data, the method further performs the following operation: sending the raw EOG data to a display device for real-time display, including: The raw electrooculogram (EOG) data is packaged to generate raw EOG packaged data; The raw electrooculogram (EOG) data is sent and the transmission status is checked after the data is sent. If the transmission is successful, the raw EOG data at the next moment is read; otherwise, the raw EOG data is deleted. After receiving the raw electrooculogram (EOG) package data, the display device unpacks the data and displays it in real time.

5. The electrooculography-controlled finger rehabilitation device based on human-computer interaction according to claim 1, characterized in that, The process of sending the electrooculogram (EOG) signal category to a mechanical glove fitted on the patient's hand, so that the mechanical glove performs corresponding rehabilitation actions based on the EOG signal category, includes: An operation command is sent to the mechanical glove to instruct it to read the electrooculogram signal category via a serial port and to determine the electrooculogram signal category. If it is a preset category, the angle of the servo motor on the mechanical glove is controlled according to the pulse modulation width value corresponding to the electrooculogram signal category to generate a rehabilitation action corresponding to the electrooculogram signal category; otherwise, the mechanical glove continues to receive operation commands.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the function of the human-computer interaction-based electrooculography-controlled finger rehabilitation device according to any one of claims 1 to 5.

7. An electronic terminal, characterized in that, include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory to enable the terminal to perform the functions of the human-computer interaction-based electrooculography-controlled finger rehabilitation device as described in any one of claims 1 to 5.

Citation Information

Patent Citations

  • Signal processing equipment and human body electrocardiosignal processing and disease prediction method

    CN108514414A

  • EEG biofeedback system based on EOG switch

    CN110037696A