Method for individualized motor imagery of multi-stage tasks based on differential actions

Abstract

The application provides a method for multi-stage task personalized motor imagery based on different actions, comprising an encoding stage, a retrieval stage and a reinforcement stage; in the encoding stage, the retrieval stage and the reinforcement stage, action information is presented in different forms; the presented action information is encoded by a subject and triggers memory, and the influence of the action on the subject is simultaneously reinforced. The application has the beneficial effects that the method for multi-stage task personalized motor imagery based on different actions visualizes the influence of the action factor in MI on the electroencephalogram through multi-stage tasks of different actions, and effectively improves the efficiency and stability of the MI-BCI system.

Description

Technical Field

[0001] This invention belongs to the field of brain-computer interface in biomedical engineering, and in particular relates to a method for personalized motor imagery for multi-stage tasks based on differential movements. Background Technology

[0002] Imagining specific motion scenarios plays a crucial role in brain-computer interface (BCI) systems based on motor imagery (MI). Due to its unique spontaneous nature, motor imagery makes irreplaceable contributions and holds great promise in fields such as control and rehabilitation. Cursors and robotic arms are common controlled objects in motor imagery BCI systems. However, visually guided MI based on traditional arrows suffers from low recognition accuracy and poor stability in MI-based BCI systems.

[0003] Currently, brain-computer interfaces for motor imagery can guide motor imagery potentials using traditional motor imagery evoked paradigms, but the accuracy of categorizing the evoked motor images is low. While motion observation methods can induce motor imagery, they cannot deeply explore the effects of differences in the movements used in the video on motor imagery; furthermore, neither motion observation nor traditional motor imagery evoked paradigms have the ability to be personalized for individual subjects.

[0004] How differences in imagined actions affect the effectiveness of motor imagery in behavioral observation, and how to optimize the induction of motor imagery potentials for higher recognition rates by the BCI system based on the individual characteristics of the subjects, are urgent problems to be solved. Summary of the Invention

[0005] In view of this, the present invention aims to propose a method for personalized motor imagery for multi-stage tasks based on differential movements, which addresses the problems of poor personalization ability and low intention recognition rate of existing motor imagery brain-computer interfaces, and is used to realize multi-stage tasks with differential movements.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0007] A multi-stage task-based personalized motor imagery method based on differential movements includes an encoding stage, a retrieval stage, and a reinforcement stage;

[0008] Action information is presented in different forms during the encoding, retrieval, and reinforcement phases;

[0009] Subjects encode the presented action information and trigger memory, while simultaneously amplifying the impact of the action on the subjects.

[0010] Furthermore, in the encoding, retrieval and reinforcement phases, X daily actions from a first-person perspective are established, where each action contains 2Y action postures, including Y left-hand action postures and Y right-hand action postures.

[0011] Subjects need to perform a multi-stage task with X differential actions to complete all X actions.

[0012] Furthermore, in the encoding stage, the subject is provided with action videos to memorize the actions. Multiple videos are randomly played in the center of the screen, including the same number of left and right hand action videos. Each video is shown only once and lasts for 'a' seconds. The subject is required to memorize as many of these N postures as possible. Subsequent tasks will prompt the subject to carefully observe the actions and encode the action information for memorization, where N∈2Y.

[0013] Furthermore, during the retrieval phase, M pose images are displayed as visual stimuli. The pose images are screenshots of action videos, which include videos that appeared during the encoding phase and videos that did not appear during the encoding phase. During the retrieval phase, subjects are required to determine whether the image appeared during the encoding phase by pressing a key.

[0014] Furthermore, during the retrieval phase, the motor imagery was guided by displaying an arrow and a video that had already been played during the encoding phase. During the reinforcement phase, the participants' impressions and memories of various postures under that action were deepened by repeatedly playing the action video, and the participants were guided to perform similar actions, thereby strengthening their motor imagery under that action.

[0015] Furthermore, the experiment was based on a brain-computer interface system, including a 64-electrode cap, an EEG acquisition device, a signal amplifier, a computer, and the subjects.

[0016] The 64-conductive electrode cap is worn on the subject's head and connected to the EEG acquisition device to transmit the subject's EEG signals.

[0017] One end of the EEG acquisition device is used to acquire the EEG signals of the 64-conducting electrode cap, and the other end is connected to a signal amplifier to acquire the EEG signals of the subject.

[0018] One end of the signal amplifier is connected to the EEG acquisition device, and the other end is connected to the computer. Its function is to amplify the EEG signal and transmit it to the computer.

[0019] The computer receives signals from the amplifier and also serves as a tool to guide experiments and induce EEG signals in the subjects.

[0020] The delayed matching action task provides visual stimuli to participants via computer.

[0021] An electronic device includes a processor and a memory communicatively connected to the processor and used to store processor-executable instructions, the processor being used to perform a method for personalized motion visualization for a multi-stage task based on differential actions.

[0022] A server includes at least one processor and a memory communicatively connected to the processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the processor to cause the at least one processor to perform a method for personalized motion visualization of a multi-stage task based on differential actions.

[0023] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements a method for personalized motion visualization of multi-stage tasks based on differential actions.

[0024] Compared with existing technologies, the method for personalized motor imagery based on differential actions in multi-stage tasks described in this invention has the following beneficial effects:

[0025] The method for personalized motor imagery based on multi-stage tasks involving differential actions described in this invention reveals the influence of motor factors on EEG output in motor imagery (MI) through multi-stage tasks involving differential actions. This provides an effective method for identifying personalized actions for subjects, thereby improving the accuracy of MI recognition. Ultimately, it effectively enhances the efficiency and stability of the MI-BCI system. Attached Figure Description

[0026] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0027] Appendix Figure 1 These are three experimental scenarios selected for the multi-stage task with differentiated actions;

[0028] Appendix Figure 2 It is a multi-stage task sequence diagram with differentiated actions;

[0029] Appendix Figure 3 This is a sequence diagram of the reinforcement phase. Detailed Implementation

[0030] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0031] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0032] To address the shortcomings of existing brain-computer interfaces for motor imagery, such as poor personalization capabilities and low intent recognition rates, this invention aims to provide a method for mining personalized imagined movements within motor imagery. The technical solution to achieve this objective is a multi-stage task focusing on differentiated movements.

[0033] The multi-stage task for differentiated actions establishes X everyday actions from a first-person perspective. Three actions are used as examples: cutting a ball with a knife (CK), catching a ball with fingers (GF), and drawing lines on paper with a pen (WP). All actions have a black background, and the moving hands are wearing white gloves. Each action includes five left-hand gestures and five right-hand gestures. All actions and gestures are played four times at a constant speed within 6 seconds.

[0034] CK: The action of cutting a white rubber ball with a ceramic knife. CK's five poses are based on five directions: cutting the ball from the front, cutting the ball vertically from above, cutting the ball horizontally from above, cutting the ball from the left, and cutting the ball from the right.

[0035] GF: The action of grasping a white rubber ball with your fingers. The five GF gestures are based on finger gestures: five-finger grasp, four-finger grasp, index finger poking the ball, two-finger pinching the ball, and thumb pressing the ball.

[0036] WP: The action of holding a pen and writing strokes. The five WP postures are based on strokes: horizontal, vertical, left-falling, right-falling, and round.

[0037] The multi-stage task of differential motor skills consists of an encoding stage, a retrieval stage, and a reinforcement stage. These three stages present motor information in different forms so that participants can encode the motor information and trigger memory, while simultaneously reinforcing the impact of the motor on the participants.

[0038] The experiment was based on a brain-computer interface system, mainly consisting of a 64-electrode cap, an EEG acquisition device, a signal amplifier, a computer, and the participants. The 64-electrode cap was worn on the participants' heads and connected to the EEG acquisition device to transmit the participants' EEG signals. One end of the EEG acquisition device was used to acquire the EEG signals from the 64-electrode cap, and the other end was connected to the signal amplifier to acquire the participants' EEG signals. One end of the signal amplifier was connected to the EEG acquisition device, and the other end was connected to the computer; its function was to amplify the EEG signals and transmit them to the computer. The computer both received the signals returned from the amplifier and served as a tool to guide the experiment and induce the participants' EEG signals. A delayed matching action task was performed, providing visual stimulation to the participants via the computer.

[0039] This experiment was conducted in a quiet, soundproof laboratory. Participants sat in comfortable chairs, 70cm from the computer screen. Participants were in good mental condition, with neat hair, and the experimental impedance was required to be below 5KΩ. Before the experiment, participants wore 64 conductive electrode caps on their heads, and the experimental procedure was explained to them until they fully understood it before the experiment began.

[0040] (1) Encoding Phase: The encoding phase provides participants with video demonstrations of the movements to help them memorize them. Multiple videos, including an equal number of videos showing left and right hand movements, are randomly played in the center of the screen. Each video is shown only once and lasts for *a* seconds. Participants are required to memorize as many of these *N* movements as possible. Subsequent tasks will encourage participants to carefully observe the movements and encode the movement information for memorization. (Where *N* ∈ 2Y)

[0041] (2) Retrieval Phase: The retrieval phase tests the subject's memory of actions by displaying visual image stimuli. This phase begins by displaying M posture images as visual stimuli. The posture images are screenshots from the action video. The action video includes videos that appeared in the encoding phase and those that did not. In this phase, the subject is required to determine whether the image appeared in the encoding phase by pressing keys. Subjects are required to respond to the visual stimuli by pressing the keyboard: the "y" key indicates a positive probe, and the "n" key indicates a negative probe. Each image is displayed for b seconds in this phase. The following example uses b=9. In this phase, each image lasts for 6 seconds, allowing the subject to make a judgment. If no result is given within 6 seconds, it is considered a missed judgment. After the image is displayed, a cross appears in the center of the screen for 3 seconds to alert the subject to the next stimulus. Then the next image appears. Each image is displayed once, therefore the total duration of the visual stimulus is 25 times. In this phase, the subject needs to retrieve information from their brain to match the visual stimulus with their memory.

[0042] (3) Reinforcement Phase: The guidance for motor imagery involves displaying an arrow and a video that was played during the encoding phase. During the reinforcement phase, the video of the action is repeatedly played to deepen the subject's impression and memory of various postures under that action, and the subject is guided to perform similar actions, thereby strengthening their motor imagery under that action. Each trial in this phase lasts c seconds. Taking c = 9 seconds as an example: In the initial 0-2 seconds, the screen is blank for rest. In 2-3 seconds, a white gaze cross appears in the center of the screen. From the next 3-9 seconds, motor imagery is performed, and an arrow and a video played during the encoding phase are displayed. The arrow indicates the direction of the imagery. The video guides the subject in imagining the action. Each motor imagery trial is defined as 6 seconds long. There are n trials in this phase.

[0043] Subjects need to perform a multi-stage task involving differential movements X times to complete all X movements. There is a 15-minute rest period between each movement. The multi-stage task involving differential movements induces motor imagery and event-related potentials, which are collected by an EEG acquisition device and transmitted to a signal amplifier. The signal amplifier then amplifies the EEG signals and transmits them to a computer.

[0044] The characteristic EEG patterns of the three movements were used for subsequent feature extraction and classification to obtain the classification accuracy for different subjects. Furthermore, by combining factors such as the activation amplitude of event-related synchronization and desynchronization, the amplitude and latency of event-related potentials, the most suitable MI movement for the individual was selected.

[0045] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0046] In the several embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the division of units described above is merely a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The aforementioned units may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention according to actual needs.

[0047] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

[0048] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for individualized motor imagery of multi-stage tasks based on differential actions, characterized by: The method comprises an encoding stage, a retrieval stage and a reinforcement stage. In the encoding stage, the retrieval stage and the reinforcement stage, the action information is presented in different forms. The subject encodes the presented action information and triggers memory, and at the same time, the action reinforces the influence on the subject. In the encoding stage, the retrieval stage and the reinforcement stage, X daily actions in the first-person perspective are established, wherein each action contains 2Y action poses, including Y left-hand action poses and Y right-hand action poses. The subject needs to complete all X actions by performing a multi-stage task of X difference actions; In the encoding phase, the encoding phase provides the subject with action videos to memorize actions, randomly playing a plurality of videos in the center of the screen, including the same number of left and right hand action videos, each video being displayed only once, lasting a seconds, asking the subject to memorize as many of these N poses as possible, the subsequent tasks will prompt the subject to carefully observe the actions while encoding the action information for memory, wherein N ∈2Y; In the retrieval phase, the subject is shown M a posture image as a visual stimulus, the posture image is a screenshot of a motion video, the motion video contains videos that appeared in the encoding phase and videos that did not appear in the encoding phase, and the subject is required to judge whether the picture appeared in the encoding phase by pressing a key; In the retrieval stage, the guiding mode of motor imagery is to display an arrow and a video that has been played in the encoding stage, and in the reinforcement stage, the action video is repeatedly played to deepen the subject's impression and memory of various poses in the action, and guide the subject to perform similar actions, thereby reinforcing the subject's motor imagery in the action; The experiment is based on a brain-computer interface system, which includes a 64-electrode cap, an electroencephalogram acquisition instrument, a signal amplifier, a computer and a subject; The 64-electrode cap is worn on the head of the subject and is connected to the electroencephalogram acquisition instrument for transmitting the electroencephalogram signals of the subject; The electroencephalogram acquisition instrument is used to acquire the electroencephalogram signals of the 64-electrode cap and is connected to the signal amplifier for acquiring the electroencephalogram signals of the subject; The signal amplifier is connected to the electroencephalogram acquisition instrument at one end and to the computer at the other end, and functions to amplify the electroencephalogram signals and transmit them to the computer; The computer receives the signals transmitted back by the amplifier on one hand, and serves as a tool for guiding the experiment to induce the electroencephalogram signals of the subject on the other hand; The delay matching action task provides visual stimulation for the subject through the computer.

2. An electronic device, comprising a processor and a memory connected to the processor in communication, and used to store executable instructions of the processor, characterized in that: The processor is used to execute the method for individualizing motor imagery based on a multi-stage task of difference actions according to claim 1.

3. A server, characterized by: The processor is used to execute the method for individualizing motor imagery based on a multi-stage task of difference actions according to claim 1.

4. A computer readable storage medium storing a computer program, characterized in that: The computer program is executed by the processor to realize the method for individualizing motor imagery based on a multi-stage task of difference actions according to claim 1.

Images