Intelligent interaction and muscle force rehabilitation system based on eye movement tracking

Abstract

The invention discloses an intelligent interaction and muscle force rehabilitation system based on eye movement tracking. The method comprises the following steps: firstly, preprocessing collected real-time eye movement data; then, a watching ring in the screen can track the sight line of the patient, and eye movement tracking is achieved; interaction between the patient and a UI interface through eye movement is achieved, and the interface comprises multiple functions; then, a motor is adopted to drive a pneumatic system to control the rehabilitation gloves, and the patient can control the rehabilitation gloves through eye movement; finally, after myoelectricity data are preprocessed, feature extraction is conducted on the data through an extensible time window method in an online experiment, the data are input into the model to obtain the muscle strength of the patient, visualization is conducted on a UI interface, and the patient can control the rehabilitation gloves conveniently. According to the invention, stroke patients can actively and fully express their own demands and willingness during rehabilitation, and the efficiency of communication with others is improved; and the rehabilitation gloves are controlled to move and train, so that the relation between neural networks is enhanced, and a region for controlling movement in the brain is activated and mobilized.

Description

Technical Field

[0001] The present invention belongs to the cross - field of artificial intelligence and rehabilitation medicine, aiming to provide the ability for patients with stroke, neurogenic muscle atrophy, etc. to fully express their own needs and wishes and to actively participate in rehabilitation. In particular, it relates to an intelligent interaction and muscle strength rehabilitation system and method based on eye tracking. Background Art

[0002] Eye Tracking technology is a technology that studies visual attention, cognitive processes, and behavioral responses by capturing and analyzing eye movements. Eye tracking instruments usually use infrared light to irradiate the eyes and use the reflected light signals to calculate the position, direction, and movement characteristics of the eyes. It can record information such as the position of the eyes, fixation points, movement trajectories, and durations in a specific task or environment, and has great practical value in the fields of psychology, medicine, market research, user experience design, and wearable intelligent equipment.

[0003] Surface electromyography (sEMG) is a non - invasive method for recording and analyzing the electrical activity of muscles from the skin surface. It is a form of electromyography (EMG) technology, mainly used to evaluate and monitor muscle function, fatigue, coordination, etc. Compared with traditional needle - electrode electromyography, surface electromyography has the advantages of simple operation, non - invasiveness, and low cost. Electromyography is a non - invasive technology with high temporal and spatial resolution. Due to its high sensitivity, good repeatability, and real - time performance, it has become an ideal technology for studying muscle activity and movement control. However, electromyography also has some limitations. For example, it is affected by factors such as electrode position, muscle morphology, and muscle fatigue, and it is difficult to distinguish the activities of different muscles. Surface EMG is easily affected by motion artifacts, environmental noise, etc., and requires precise signal processing. Despite these limitations, electromyography is still a powerful tool for evaluating muscle function and diseases, monitoring human movement and posture, and designing rehabilitation programs. It has a wide range of applications in the fields of rehabilitation medicine, exercise physiology, and robotics.

[0004] Traditional eye - movement interaction systems usually can only achieve simple need expression, with relatively single functions and limited interaction methods. For example, there are many human body parts, and the ranges of the arms, legs, abdomen, back, etc. are large, and the traditional button - based interaction method cannot meet the accurate expression of the patient's feelings about a certain body part; during the patient's rehabilitation process, usually only basic needs can be provided to the patient or passive rehabilitation can be carried out, and the patient cannot actively participate in the rehabilitation of damaged muscles, making it difficult to mobilize the areas in the brain that control movement and help the patient recover better. Summary of the Invention

[0005] Aiming at the deficiencies in the above-mentioned existing technologies, the purpose of the present invention is to provide a method for stroke patients, patients with nerve atrophy, etc. to accurately express their own needs and wishes and actively participate in preventing muscle atrophy or muscle rehabilitation.

[0006] To achieve the above object, the technical solution of the present invention is: an intelligent interaction and muscle strength rehabilitation training method based on eye movement tracking, including the following steps:

[0007] Obtain real-time eye movement data and perform preprocessing, where the preprocessing includes removing noise and invalid values, deleting missing data, coordinate mapping and calibration;

[0008] Design an eye movement interaction UI interface, and combine the eye movement data after preprocessing to realize the control of each function of the UI interface by eye movement;

[0009] Manually label the categories and preprocess the collected electromyogram signal data, and use the spliced and fused feature data to train the model. The preprocessing includes band-pass filtering and normalization to eliminate noise in the signal;

[0010] For the preprocessed electromyogram signal data, use the expandable time window method to extract features to achieve online muscle strength classification;

[0011] Further classify the data after online muscle strength classification;

[0012] Realize the control of the rehabilitation glove through the way of eye movement interaction and muscle strength control.

[0013] Further, the obtaining of real-time eye movement data and the preprocessing also include the following processing: through coordinate mapping of the preprocessed real-time eye movement data, the fixation circle on the screen can accurately track the line of sight of the patient wearing the eye movement device; it is also necessary to perform coordinate calibration and standardization processing on the eye movement data, and the mean method of removing outliers is used during calibration.

[0014] Further, the design of the eye movement interaction UI interface includes: for daily needs, use the traditional button interaction method. The patient triggers the button to play the button content and display it in the demand box by staring at the button for more than 2 seconds; for physical sensations, use the marked retention method to enable the patient to accurately express the feelings of a certain part of the body. The patient stares at the picture of the body part, and when the staring time exceeds the predetermined threshold, a refreshable mark is left at the staring point, and the feelings of this part are selected through the button; for more complex requirements, input through the eye movement gaze virtual keyboard, and at the same time connect to the large model to perform intelligent question answering on the input text.

[0015] Further, the artificial labeling of the collected EMG signal data into categories and preprocessing, and the training of the model using the feature data after splicing and fusion specifically include: using a band-pass filter to remove the noise of the EMG signal; then normalizing the denoised EMG data and using the standard deviation double-threshold detection method to extract strong and continuous EMG signals as "action signals", and extracting all "action signals" to the front of the data; using a sliding window of size 116*5 and a step size of 116 to extract features from the original time series data and the EMG data processed by the standard deviation double-threshold detection method respectively, focusing on the timing and action features of the EMG data, and training the model using a support vector machine after splicing and fusing the two groups of feature data.

[0016] Further, the use of the expandable time window method to extract features for online muscle strength classification includes: inputting 116 data each time, and extracting features once when the data accumulates to 116*4 to obtain f 11 , inputting 116 data each time, and extracting features for all the data once to obtain f 12 , f 13 , f 14 , until the input data reaches 116*7, accumulating the extracted features to obtain F 1 = [f 11 , f 12 , f 13 , f 14 , and inputting it into the trained model to predict and obtain P 1 = [p 11 , p 12 , p 13 , p 14. Since the time window size used during training is 116*5, during online experiments, different-sized time windows were used for feature extraction, resulting in different signal classifications. When the time window size is smaller than that during training, due to less data, it is difficult to extract features related to action signals, or the extracted features are difficult to meet the characteristics of action signals. At this time, when the muscle strength is small, the features extracted within this time window are classified as stable signals after being input into the model. Only when the muscle strength is large, the features extracted within this time window are classified as action signals after being input into the model. For time window sizes larger than that during training, since there is more data, it is easier to extract features related to action signals, or the extracted features are easily satisfied with the characteristics of action signals. At this time, the features extracted within this time window are classified as action signals after being input into the model. The larger the time window size, the easier it is to be recognized as an action signal. Therefore, the expandable time window method proposed by the present invention solves the problem of muscle strength classification. At the same time, such muscle strength is classified through signals, rather than simply dividing the threshold of muscle strength size, which increases the robustness of muscle strength. That is, when other movements of the user rather than the actions used in the present invention affect the change of muscle signals, obvious action signals will not appear. In this way, when using the muscle strength-controlled rehabilitation glove, the situation of accidental touch will be greatly reduced.

[0017] Further, the task classification includes, after obtaining the prediction result sequence P = [P 1 , P 2 ,..., P n through online muscle strength classification, using a sliding window with a size of 5 and a step size of 1 to obtain the number of 1s within each second of the prediction result, that is, the number of action signals n act , and defining the muscle strength E Ns as a comprehensive index of muscle strength size and duration within 1s:

[0018]

[0019] Perform task classification for different muscle strengths:

[0020]

[0021] Visualize the change of this muscle strength on the UI interface and feedback it to the patient to facilitate adjusting the muscle strength and selecting the corresponding task to achieve the control of the rehabilitation glove.

[0022] Further, the realization of the control of the rehabilitation glove includes receiving signals from eye movement interaction or muscle strength control using a serial port, and then driving the motor to control the pneumatic system, and controlling the grasping and stretching actions of the rehabilitation glove through the change of air pressure to assist the patient's hands to move.

[0023] The present invention also provides an intelligent interaction and muscle strength rehabilitation system based on eye movement tracking, including

[0024] A real-time eye movement data processing module, configured to obtain real-time eye movement data and perform preprocessing, where the preprocessing includes removing noise and invalid values, deleting missing data, coordinate mapping, and calibration;

[0025] A UI interaction module, configured to combine the eye movement data after preprocessing to implement various functions of the eye movement control UI interface;

[0026] An electromyogram signal processing module, configured to manually label the categories and perform preprocessing on the collected electromyogram signal data, and use the feature data after splicing and fusion to train a model;

[0027] A muscle strength classification module, configured to use the expandable time window method to extract features from the electromyogram signal data after preprocessing to achieve online muscle strength classification;

[0028] A task classification module, configured to perform task classification on the data after online muscle strength classification;

[0029] A rehabilitation glove control module, configured to control the rehabilitation glove through the method of eye movement interaction and muscle strength control.

[0030] Finally, the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned intelligent interaction and muscle strength rehabilitation training method based on eye movement tracking are implemented.

[0031] The advantages and beneficial effects of the present invention are as follows:

[0032] The present invention provides a system for patients with stroke, nerve atrophy, etc., which enables them to accurately express their own needs and wishes and actively participate in preventing muscle atrophy or muscle rehabilitation. Through precise eye movement tracking technology, the system allows patients to control the system through eye movements, avoiding the limitation of traditional rehabilitation devices relying on hand operations. Based on the traditional eye movement interaction system, functions such as body sensation, intelligent Q&A accessing large models, and eye movement-controlled rehabilitation gloves are added to improve the comfort and operability during the rehabilitation process. The expandable time window method is used to solve the problem of muscle strength classification. At the same time, such muscle strength is classified by signals rather than simply setting a threshold for the magnitude of muscle strength, enhancing the robustness of muscle strength. That is, when the user makes other movements rather than the specified actions, which affect the change of muscle signals, no obvious action signals will appear. In this way, the situation of accidental touch will be greatly reduced when using the muscle strength-controlled rehabilitation gloves. Therefore, for some patients with muscle damage, they can adjust the muscle strength of the undamaged muscles, control the rehabilitation gloves, and promote the activation of the brain regions controlling movement through exercise training, enhancing the connection between neural networks. This not only allows patients to actively participate in rehabilitation but also enables patients to gradually regain control of muscle strength during rehabilitation, helping the damaged parts of the patients to relearn motor skills. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Flowchart of the intelligent interaction and muscle strength rehabilitation system based on eye movement tracking provided by the present invention;

[0034] Figure 2 Schematic diagram of the eye movement interaction interface;

[0035] Figure 3 Schematic diagram of accurate expression of body sensation and intelligent Q&A;

[0036] Figure 4 Schematic diagram of feature extraction by the expandable time window method in the online experiment;

[0037] Figure 5 Introduction diagram of the rehabilitation glove control interface. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0038] The technical solutions in the embodiments of the present invention will be clearly and detailedly described below with reference to the accompanying drawings in the embodiments of the present invention. The described embodiments are only a part of the embodiments of the present invention.

[0039] As Figure 1 shown, the present invention includes the following steps:

[0040] Step 1: Preprocess the real-time eye movement data. First, obtain the eye movement data by subscribing to the data of the eye movement device, and then use UDP communication and decoding to split and obtain the original eye movement data. The original eye movement data usually contains some noise and invalid values, and these interferences need to be removed through cleaning operations. At the same time, there are missing values in the eye movement data (the patient blinks or does not fixate on the screen), and these missing values need to be processed. Choose to delete the time period with missing data so that the eye movement data during this period will not affect eye movement tracking; then through coordinate mapping, the fixation circle on the screen can accurately track the patient's line of sight; due to individual differences among patients, the fixation points mapped to the screen by each person through the eye movement device are different, so coordinate calibration and standardization processing are performed on the eye movement data to ensure that patients can interact normally. When calibrating, the mean method of removing outliers is used to avoid the influence of the patient's blinking during calibration on the calibration effect.

[0041] Step 2: After preprocessing the eye movement data, implement the eye movement interaction UI design, such as Figure 2 . For simple daily needs, such as drinking water, eating, etc., use the traditional button interaction method. The patient fixes on the button for more than 2 seconds, triggers the button to play the button content and displays it in the requirement box; for body sensations, there are many human body parts, and the ranges of the arms, legs, abdomen, back, etc. are large. The traditional button interaction system cannot meet the patient's accurate expression of the feeling of a certain body part. Use the marking and leaving point method. The patient fixes on the picture of the body part. When the fixation time exceeds the predetermined threshold, the system will leave a refreshable mark at the fixation point, and select the feeling of this part through the button; for some more complex requirements, input can be made through the eye movement fixation on the virtual keyboard, and at the same time, a large model (such as the Wenxin Yiyan large model) is connected to perform intelligent question answering on the input text, such as Figure 3 .

[0042] Step 3: Use the surface electromyography sensor to collect electromyography signals and perform offline processing. The electromyography signals are often affected by motion artifacts and environmental electromagnetic interference. A band-pass filter is used to remove the noise; then, after denoising, the electromyography data is normalized and the standard deviation double-threshold detection method is used to extract relatively strong and continuous electromyography signals as "action signals", and all "action signals" are extracted to the front of the data; a sliding window with a size of 116*5 and a step size of 116 is used to extract features from the data of the original time series and the electromyography data processed by the standard deviation double-threshold detection method respectively, focusing on the timing and action features of the electromyography data respectively. After splicing and fusing the two groups of feature data, a support vector machine is used for model training. The trained model can simultaneously focus on the timing and action features in the electromyography data, so as to ensure the real-time performance of the online electromyography experiment.

[0043] Step 4: After preprocessing the electromyography data in Step 3, use the expandable time window method to extract features to achieve online muscle strength classification, such asFigure 4 In the online experiment, 116 data are input each time. When the data accumulates to 116 * 4, a feature is extracted once to obtain f 11 , each time 116 data are input, feature extraction is performed on all the data to obtain f 11 , f 12 , f 13 , f 14 , until the input data reaches 116 * 7, the extracted features are cumulatively summed to obtain F 1 = [f 11 , f 12 , f 13 , f 14 , and then it is input into the model trained in step 3 to obtain the prediction result P 1 = [p 11 , p 12 , p 13 , p 14 . The present invention uses an extensible time window method to extract features from data. The prediction result obtained by using this method can directly obtain the muscle strength magnitude according to the number of action signals. The muscle strength magnitudes in the prediction result are sorted from small to large: [0, 0, 0, 0], [0, 0, 0, 1], [0, 0, 1, 1], [0, 1, 1, 1], [1, 1, 1, 1]. There will be no prediction result like [0, 1, 1, 0] where 0 is after 1 in the prediction result; to ensure the accuracy and stability of the prediction result, in step 5, the comprehensive index of muscle strength magnitude and duration within 1 s is defined as muscle strength intensity.

[0044] Step 5: Online task classification. After the online muscle strength classification in step 4 obtains the prediction result sequence P = [P 1 , P 2 ,..., P n , a sliding window with a size of 5 and a step size of 1 is used to obtain the number of 1s in the window per second of the prediction result, that is, the number n of classified action signals act , so the muscle strength intensity (E Ns ) is defined as the comprehensive index of muscle strength magnitude and duration within 1 s:

[0045]

[0046] Furthermore, task classification is performed for different muscle strength intensities:

[0047]

[0048] And the change of this muscle strength intensity is visualized on the UI interface and fed back to the patient to facilitate adjusting the muscle strength intensity to select the corresponding task, so as to realize the control of the rehabilitation glove; then instructions are sent through the serial port to realize the control of the rehabilitation glove.

[0049] Step 6: Implement the control of the rehabilitation glove. Receive signals from eye movement interaction or muscle strength control using the serial port, and then drive the motor to control the pneumatic system. Control the grasping, stretching and other actions of the rehabilitation glove through the change of air pressure to assist the patient's hands in movement, such as Figure 5 .

[0050] The above embodiments should be understood as being only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the content recorded in the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. An intelligent interaction and muscle strength rehabilitation training method based on eye tracking, characterized in that: The following steps are involved: Acquire real-time eye movement data and perform preprocessing, wherein the preprocessing includes removing noise and invalid values, deleting missing data, coordinate mapping and calibration; Design eye-movement interaction UI interface, and combine the pre-processed eye-movement data to realize the functions of eye-movement control UI interface; Manually label and preprocess the collected electromyographic signal data, and use the spliced ​​and fused feature data to train the model; For the preprocessed EMG data, the scalable time window method is used to extract features to achieve online muscle force classification; Further perform task classification on the data after online muscle strength classification; The rehabilitation gloves are controlled through eye movement interaction and muscle strength control.

2. The method of intelligent interaction and muscle strength rehabilitation training based on eye tracking according to claim 1, characterized in that: The acquisition of real-time eye movement data and preprocessing also includes the following processing: coordinate mapping of the preprocessed real-time eye movement data so that the fixation circle on the screen can accurately track the line of sight of the patient wearing the eye movement device; the eye movement data also needs to be coordinate calibrated and standardized, and the mean method of removing outliers is used during calibration.

3. The method of intelligent interaction and muscle strength rehabilitation training based on eye tracking according to claim 1, characterized in that: The designed eye-movement interaction UI interface includes: for daily needs, using the traditional button interaction method, the patient gazes at the button for more than 2 seconds, triggering the button to play the button content and display it in the demand box; for body feelings, using the mark-point method to enable the patient to accurately express his feelings about a certain part of the body, the patient gazes at the picture of the body part, and when the gaze time exceeds a predetermined threshold, a refreshable mark is left on the gaze point, and the feeling of the part is selected through the button; for more complex requirements, input is performed by gazing at the virtual keyboard through eye movement, and at the same time, the large model is connected to perform intelligent question and answer on the input text.

4. The method of intelligent interaction and muscle strength rehabilitation training based on eye tracking according to claim 1, characterized in that: The method of manually marking the categories and preprocessing the collected electromyographic signal data, and using the spliced ​​and fused feature data to train the model specifically includes: using a bandpass filter to remove the noise of the electromyographic signal; then normalizing the denoised electromyographic data, using the standard deviation double threshold detection method to extract strong and continuous electromyographic signals as "action signals", and extracting all "action signals" to the front column of the data; using a sliding window with a size of 116*5 and a step size of 116 to extract features from the original time series data and the electromyographic data processed by the standard deviation double threshold detection method, and splicing and fusing the two sets of feature data and using a support vector machine to train the model.

5. The method of intelligent interaction and muscle strength rehabilitation training based on eye tracking according to claim 1, characterized in that: The method of using the expandable time window method to extract features to achieve online muscle strength classification includes: inputting 116 data each time, extracting a feature once when the data accumulates to 116*4 to obtain f 11 , each time 116 data are input, feature extraction is performed on all the data to obtain f 12 ,f 13 ,f 14 , until the input data reaches 116*7, the extracted features are accumulated to obtain F1=[f 11 ,f 12 ,f 13 ,f 14 ], input into the trained model to predict P1 = [p 11 ,p 12 ,p 13 ,p 14 ].

6. The method of intelligent interaction and muscle strength rehabilitation training based on eye tracking according to claim 1, characterized in that: The task classification includes online muscle strength classification to obtain a prediction result sequence P = [P1, P2, ..., P n ], a sliding window with a size of 5 and a step size of 1 is used to obtain the number of prediction results 1 per second, that is, the number of action signals n act , define muscle strength E Ns It is a comprehensive indicator of muscle strength and duration within 1 second: Classification of tasks for different muscle strengths: The changes in muscle strength are visualized on the UI interface and fed back to the patient, making it easier to adjust the muscle strength and select the corresponding task to achieve control of the rehabilitation gloves.

7. The method of intelligent interaction and muscle strength rehabilitation training based on eye tracking according to claim 1, characterized in that: The control of the rehabilitation gloves includes using a serial port to receive signals from eye movement interaction or muscle strength control, and then driving a motor to control a pneumatic system, thereby controlling the grasping and stretching movements of the rehabilitation gloves through changes in air pressure to assist the patient in exercising both hands.

8. An intelligent interaction and muscle strength rehabilitation system based on eye tracking for implementing the method according to any one of claims 1 to 7, characterized in that: include A real-time eye movement data processing module, used to obtain real-time eye movement data and perform preprocessing, wherein the preprocessing includes removing noise and invalid values, deleting missing data, coordinate mapping and calibration; UI interaction module, which is used to combine the pre-processed eye movement data to realize the eye movement control of various functions of the UI interface; The electromyographic signal processing module is used to manually label and pre-process the collected electromyographic signal data, and use the spliced ​​and fused feature data to train the model; The muscle force classification module is used to extract features from the pre-processed electromyographic signal data using the scalable time window method to achieve online muscle force classification; The task classification module is used to classify the data after online muscle strength classification; The rehabilitation glove control module is used to control the rehabilitation gloves through eye movement interaction and muscle strength control.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the intelligent interaction and muscle strength rehabilitation training method based on eye tracking are implemented.

Images