Method for realizing hand fine action motor imagery by combining electroencephalogram and picture coding
Patent Information
- Application Number
- CN202410956937.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2026-10-30
- Estimated Expiration
- 2044-07-17
AI Technical Summary
[0005]为了解决背景技术中面向同一物体的多种手势较好区分，而较难区分的手势往往面向不同的物体的技术问题，使脑控机械臂能够执行更加符合人类期望，解决实际需求的操作，本发明主要为如何基于无创脑机接口技术解码手部精细动作M I帮助功能受损的用户自如地操控五指灵巧手抓取物体提供了一种方法
[0030]上述一种脑电与图片联合编码实现手部精细动作运动想象的方法，预设脑电预训练模型，该模型包括脑电编码器和脑电解码器，对脑电编码器进行训练并采用脑电解码器计算损失，得到训练后的脑电编码器；采集操作空间的视频并播放，通过分析用户观看视频期间的注意力，得到用户期望抓取的目标物体对应的目标图像，利用CLIP模型对目标图像进行处理，提取出目标图像的嵌入向量；采集用户观看视频期间想象采用预设的多种手部精细动作抓取目标物体时的EEG信号并预处理，得到预处理后的EEG信号，通过训练后的脑电编码器对预处理后的EEG信号进行编码，得到预处理后的EEG信号的嵌入向量；将目标图像的嵌入向量与预处理后的EEG信号的嵌入向量进行拼接，对拼接后的嵌入向量进行面向预设的多种手部精细动作的EEG信号解码，将手部精细动作运动想象直接映射到操控五指灵巧手上。通过上述方法，用户可以在不进行任何动作的情况下通过运动想象直接控制五指灵巧手用其想象的动作抓取目标，为用户提供自然的操作体验，辅助手部精细运动康复，建立自然高效的BCI系统，使得运动障碍患者能够自如地控制五指灵巧手采取符合其需求的方式与目标交互，具有重大的实际意义。
Smart Images
Figure CN118820856B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of cognitive engineering and machine vision technology, and in particular to a method for realizing fine motor imagery of hand movements through joint encoding of electroencephalogram (EEG) and images. Background Technology
[0002] BCI (brain-computer interface) is an interactive technology that bridges the gap between humans and machines. MI (motor imagery) technology does not rely on external stimuli; it uses endogenous, spontaneous EEG signals. Subjects only need to imagine performing a specific motor task, without actually moving, to decode related motor patterns in scalp EEG signals, thus determining the subject's intention. When a subject imagines the movement of a limb, it triggers changes in the sensorimotor cortex's SMRs (sensor imotor rhythms). Different limb imaginings result in SMRs with different spatiotemporal distribution characteristics, making their EEG signals highly distinguishable. Therefore, pattern recognition methods can be used to decode MI patterns in EEG signals and translate them into control commands, thereby enabling the manipulation of external devices. The development of this technology is of great significance in helping people who cannot control devices through traditional methods due to motor impairments or muscle injuries.
[0003] However, because EEG signals are collected from the scalp, they suffer from low signal-to-noise ratio and low spatial resolution. Past brain-computer interface (MI) technologies primarily addressed movements with significant spatial differences. Currently, hands, feet, and tongue can be classified relatively well. However, decoding fine hand movements using EEG signals remains a significant challenge in the field of brain-computer interfaces (BCI). Therefore, current brain-controlled robotic arms primarily employ a method where the user selects a target via a brain-computer interface, and the robotic arm, equipped with a two-finger gripper, autonomously grasps it. However, this fusion-based operation mode is rigid, only enabling simple grasping and failing to address specific user needs. This is because users often use different hand shapes to grasp objects of different configurations to achieve the most stable grasp or meet specific requirements. Even when facing the same object, users have different expectations regarding the grasping position and grip shape depending on the purpose. For example, when a user's need is to drink water, a sideways five-finger grip on the bottle body is suitable, while a three-finger pinch on the cap is not; when the user's need is to open a bottle cap or take a bottle from a case of water, a three-finger pinch on the cap is more suitable. Adopting different hand shapes to meet different needs is the result of people's experience accumulated over a long period of life.
[0004] Previous research on EEG decoding of fine motor imagery of the hands (including five-finger grasping, two-finger pinching, three-finger pinching, and five-finger releasing) found that multiple gestures facing the same object were relatively easy to distinguish, while gestures that were more difficult to distinguish often faced different objects. For example, five-finger grasping, five-finger releasing, and two-finger or three-finger pinching all performed well in classification, but two-finger pinching was easily confused with three-finger pinching. When faced with actual tasks, two-finger pinching often faced card-shaped objects such as leaves or tissues, while three-finger pinching often faced smaller balls or cylindrical objects such as chess pieces, ping-pong balls, or bottle caps. Five-finger grasping, five-finger releasing, and two-finger pinching can be combined to complete tasks such as opening bottle caps with both hands, and five-finger grasping, five-finger releasing, and two-finger pinching can be combined to complete tasks such as pulling tissues with both hands. Summary of the Invention
[0005] To address the technical challenge of distinguishing between multiple gestures facing the same object, while gestures that are difficult to distinguish often face different objects, and to enable brain-controlled robotic arms to perform operations that better meet human expectations and solve practical needs, this invention provides a method for decoding fine motor skills (MI) of the hand based on non-invasive brain-computer interface technology to help users with impaired function freely control their five-finger dexterity to grasp objects.
[0006] To achieve the above objectives, the present invention provides a method for realizing fine motor imagery of hand movements through joint encoding of electroencephalogram (EEG) and images, the method comprising the following steps:
[0007] S1. Pre-set EEG pre-training model, which includes an EEG encoder and an EEG decoder. The EEG encoder is trained using a pre-set training set and the loss is calculated using the EEG decoder to obtain the trained EEG encoder. The trained EEG encoder can establish the correlation between EEG signals and various pre-set fine motor imagery of the hands.
[0008] S2. Collect and play the video of the operation space. By analyzing the user's attention during the video viewing, obtain the target image corresponding to the target object that the user expects to grab. Use the CLIP model to process the target image and extract the embedding vector of the target image.
[0009] S3. Collect and preprocess the EEG signals when the user imagines grasping the target object using a variety of preset fine hand movements while watching the video. The preprocessed EEG signal is then encoded by a trained EEG encoder to obtain the embedding vector of the preprocessed EEG signal.
[0010] S4. The embedding vector of the target image is concatenated with the embedding vector of the preprocessed EEG signal. The concatenated embedding vector is then decoded for various preset fine hand movements using EEG signals. The fine hand movement imagery is then directly mapped onto the dexterous hand that controls the five fingers.
[0011] Preferably, in S1, the EEG encoder is trained using a preset training set and the loss is calculated using an EEG decoder to obtain the trained EEG encoder, specifically including the following:
[0012] S11 includes a pre-set EEG pre-training model, various fine motor skills, training videos, and tags;
[0013] S12. Collect EEG signals when the user imagines using preset fine hand movements to grasp a target object while watching a training video.
[0014] S13. The EEG encoder in the EEG pre-training model encodes the EEG signal obtained in S12 to obtain an encoded embedding vector that reflects the relationship between the EEG signal and fine motor imagery of the hand.
[0015] S14. The EEG decoder in the EEG pre-training model receives the encoded embedding vector and decodes it to obtain the mapping result. It calculates the loss between the mapping result and the preset label and trains the EEG encoder to obtain the trained EEG encoder.
[0016] Preferably, S2 specifically includes the following:
[0017] S21. Capture and play video of the operating space through a camera, and capture the user's visual trajectory while watching the video through an eye tracker and obtain the coordinates of the user's eye movement observation point;
[0018] S22. Extract the coordinates of the gaze sampling point from the coordinates of the eye movement observation point using the velocity threshold filtering algorithm and add them to the gaze buffer. Filter the coordinates of the gaze sampling point in the gaze buffer using the low-pass filtering algorithm to obtain the gaze coordinates that fall within the target.
[0019] S23. Input the gaze coordinates and video frame images falling within the target into the SAM model for processing, and segment the target image corresponding to the target object that the user expects to capture from the video frame images.
[0020] S24. Encode the target image using the CLIP model to obtain the embedding vector of the target image.
[0021] Preferably, in step S3, the EEG signals of the user imagining grasping a target object using various preset fine hand movements during video viewing are collected and preprocessed to obtain the preprocessed EEG signal, specifically including the following:
[0022] S31. The user looks at the target object and imagines selecting the desired fine hand movements from a variety of preset fine hand movements to grasp or release the target object.
[0023] S32. Collect EEG signals from the user's imagined fine hand movements to grasp or release a target object using an EEG signal acquisition device and preprocess them to obtain preprocessed EEG signals.
[0024] Preferably, S32 specifically includes the following:
[0025] S321. Preset sampling duration: During the period when the user imagines using the desired fine hand movements to grasp or release the target object, the EEG signal is collected by the EEG signal acquisition device within the preset sampling duration.
[0026] S322. Bandpass filtering and standardization preprocessing are performed on the EEG signals collected within the preset sampling time to obtain the preprocessed EEG signals.
[0027] Preferably, the encoder in S1 includes a convolutional module and a multi-head self-attention module connected in sequence, and the decoder in the pre-trained model is a classifier module implemented by a multilayer perceptron.
[0028] Preferably, the preset fine hand movements in S1 include: two-finger pinching when facing a sheet-like object, three-finger pinching when facing a knob-shaped circular object, and five-finger grasping and five-finger spreading when facing a cylindrical object.
[0029] Preferably, the characteristics of the target object that the user expects to grasp in step S2 include card-shaped, spherical, and cylindrical.
[0030] The aforementioned method for achieving fine motor imagery through EEG and image co-encoding involves a pre-trained EEG model comprising an EEG encoder and an EEG decoder. The EEG encoder is trained, and the loss is calculated using the EEG decoder to obtain the trained EEG encoder. Video of the operating space is acquired and played back. By analyzing the user's attention during video viewing, the target image corresponding to the target object the user intends to grasp is obtained. The target image is processed using a CLIP model to extract its embedding vector. EEG signals from when the user imagines grasping the target object using various pre-defined fine motor imagery during video viewing are acquired and pre-processed to obtain pre-processed EEG signals. These pre-processed EEG signals are then encoded using the trained EEG encoder to obtain their embedding vectors. The embedding vectors of the target image and the pre-processed EEG signals are concatenated. The concatenated embedding vectors are then decoded using EEG signals for various pre-defined fine motor imagery, directly mapping the fine motor imagery to the dexterous hand controlling the five fingers. Through the above methods, users can directly control their five-finger dexterity hand to grasp targets by imagining movements without performing any actions. This provides users with a natural operating experience, assists in the rehabilitation of fine motor skills in the hand, and establishes a natural and efficient BCI system. This enables patients with motor disorders to freely control their five-finger dexterity hand to interact with targets in a way that meets their needs, which has significant practical implications. Attached Figure Description
[0031] Figure 1 This is a schematic diagram of a method for jointly encoding EEG and decoding fine motor imagery of the hand in one embodiment of the present invention;
[0032] Figure 2 This is the feature distribution result of four types of fine motor imagery of the hand in one embodiment of the present invention, wherein, Figure 2 (a) The feature distribution of fine motor imagery of the hand directly decoded by the EEG pre-training model. Figure 2 (b) Feature distribution of motor imagery for recognizing fine hand movements using EEG and image co-coding. Detailed Implementation
[0033] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.
[0034] See Figure 1 A method for achieving fine motor imagery of hand movements through joint encoding of electroencephalogram (EEG) and images, characterized by the following steps:
[0035] S1. Pre-set EEG pre-training model, which includes an EEG encoder and an EEG decoder. The EEG encoder is trained using a pre-set training set and the loss is calculated using the EEG decoder to obtain the trained EEG encoder. The trained EEG encoder can establish the correlation between EEG signals and various pre-set fine motor imagery of the hands.
[0036] Furthermore, in S1, the EEG encoder is trained using a pre-set training set and the loss is calculated using an EEG decoder to obtain the trained EEG encoder, specifically including the following:
[0037] S11 includes a pre-set EEG pre-training model, various fine motor skills for the hands, training videos, and tags.
[0038] Specifically, the EEG pre-training model includes an EEG encoder and an EEG decoder. The EEG encoder includes a convolutional module and a multi-head self-attention module connected in sequence, and the EEG decoder is a classifier module implemented by a multilayer perceptron.
[0039] In this embodiment, four fine hand movements are preset, including: two-finger pinching facing a sheet-like object, three-finger pinching facing a knob-like object, five-finger grasping facing a columnar object (that is, five-finger lateral grasping), and five-finger opening (that is, releasing the hand).
[0040] The labels specifically refer to the labels set for the four preset fine hand movements mentioned above. For example, the label for five-finger grasping is [1,0,0,0], the label for two-finger pinching is [0,1,0,0], the label for three-finger pinching is [0,0,1,0], and the label for five-finger opening (that is, releasing the hand) is [0,0,0,1].
[0041] S12. Collect EEG signals when the user imagines using preset fine hand movements to grasp a target object while watching a training video.
[0042] S13. The EEG encoder in the EEG pre-training model encodes the EEG signal obtained in S12 to obtain an encoded embedding vector that reflects the relationship between the EEG signal and fine motor imagery of the hand.
[0043] S14. The EEG decoder in the EEG pre-training model receives the encoded embedding vector and decodes it to obtain the mapping result. It calculates the loss between the mapping result and the preset label and trains the EEG encoder to obtain the trained EEG encoder.
[0044] Specifically, the EEG decoder is a classifier module implemented by a multilayer perceptron. It classifies the encoded embedding vectors through MLP (Multilayer Perceptron) and performs EEG decoding on the encoded embedding vectors for four types of fine hand movements.
[0045] Once trained, the EEG encoder can encode the input EEG signals to obtain the corresponding embedding vector (corresponding to...) of the input EEG signals. Figure 1 The EEG embedding vector reflects the relationship between the input EEG signal and the corresponding fine motor imagery of the hand.
[0046] S2. Acquire and play the video in the operation space. By analyzing the user's attention during the video viewing, obtain the target image corresponding to the target object that the user expects to grab. Encode the target image using the CLIP model and extract the embedding vector of the target image.
[0047] Furthermore, S2 specifically includes the following:
[0048] S21. Capture and play video of the operating space using a camera, and capture the user's visual trajectory while watching the video using an eye tracker and obtain the coordinates of the user's eye movement observation point.
[0049] Specifically, an eye tracker is used to collect the visual trajectory of the user on the display when selecting the target object to be grasped from card-shaped, spherical, and cylindrical objects while watching a video. Points with coordinates less than 0 in the visual trajectory are removed, thereby obtaining the coordinates [x,y] of the user's eye movement observation point.
[0050] S22. Extract the coordinates of the gaze sampling point from the coordinates of the eye movement observation point using a velocity threshold filtering algorithm and add them to the gaze buffer. Filter the coordinates of the gaze sampling point in the gaze buffer using a low-pass filtering algorithm to obtain the gaze coordinates that fall within the target.
[0051] Specifically, the velocity threshold filtering algorithm is a method for eye movement data analysis that distinguishes between fixation and saccades based on eye movement velocity. The core idea of the velocity threshold filtering algorithm is that if the eye movement velocity exceeds a preset threshold, then that point is defined as a saccade point; if the velocity is below this threshold, it is considered a fixation point. The eye movement velocity is calculated as follows:
[0052]
[0053] Where v represents eye movement velocity, x1 represents the x-coordinate of the eye movement observation point, x2 represents the x-coordinate of the adjacent eye movement observation point, y1 represents the y-coordinate of the eye movement observation point, y2 represents the y-coordinate of the adjacent eye movement observation point, t1 represents the observation point time, and t2 represents the adjacent observation point time.
[0054] By comparing the calculated eye movement speed with the preset speed threshold, the saccade point and fixation point can be distinguished.
[0055] The eye saccharidation component in the eye movement is removed from the coordinates of the user's eye movement observation point by a velocity threshold filtering algorithm. The coordinates of the fixation sampling point are extracted and added to the fixation buffer to obtain the coordinates of the fixation sampling point in the fixation buffer. The coordinate jitter of the fixation sampling point in the fixation buffer is reduced by a low-pass filtering algorithm. The fixation coordinates [x', y'] that fall within the target are obtained by averaging the coordinate data in the fixation buffer.
[0056] The coordinates of the gaze sampling points in the gaze buffer are filtered using a low-pass filtering algorithm to obtain the gaze coordinates that fall within the target. The specific process is as follows:
[0057] 1) Sort the gaze sampling points in the gaze buffer extracted by velocity threshold filtering according to their coordinate size;
[0058] 2) Select fixation sampling points based on the number of sorted fixation sampling points and calculate the average value of the selected fixation sampling points:
[0059] When the number of gaze sampling points in the gaze buffer is even, the average of the coordinates of the two middle sampling points after sorting is calculated.
[0060] When the number of gaze sampling points in the gaze buffer is odd, the average value is taken between the middle sampling point after sorting and the two sampling points adjacent to the middle sampling point.
[0061] 3) Use the average value as the gaze coordinate within the target.
[0062] By applying the above processing to the gaze sampling points in the gaze buffer using a low-pass filtering algorithm, the jitter of the gaze sampling points can be reduced.
[0063] S23. Input the gaze coordinates and video frame images falling within the target into the SAM model for processing, and segment the target image corresponding to the target object that the user expects to capture from the video frame images.
[0064] Specifically, SAM is an advanced image segmentation technique that can automatically identify and segment objects in an image. The gaze coordinates [x', y'] falling within the target are provided as cues to the SAM model. A binary mask of the gazed target image can be extracted from the video frame images of the operating space acquired in S2, where 1 represents an object inside the mask and 0 represents an object outside the mask. Multiplying the video frame images of the operating space acquired in S2 with the binary mask by position allows the target object to be segmented from the environment, resulting in a target image containing only the gazed target object.
[0065] S24. Encode the target image using the CLIP model to obtain the embedding vector of the target image.
[0066] Specifically, through CLIP (Contrast Language-Image Pre-training, based on pre-trained contrastive text-image pairs) models (corresponding to...) Figure 1 The image pre-trained model in the image extracts the image feature embedding vector containing only the target object after masking (corresponding to...). Figure 1 The image embedding vector is encoded as a one-dimensional feature tensor.
[0067] S3. Collect and preprocess the EEG signals when the user imagines grasping a target object using various preset fine hand movements while watching the video. The preprocessed EEG signal is then encoded by the encoder of the pretrained EEG model to obtain the embedding vector of the preprocessed EEG signal.
[0068] Furthermore, S3 collects and preprocesses the EEG signals when the user imagines grasping a target object using various preset fine hand movements while watching the video, resulting in the preprocessed EEG signal, which specifically includes the following:
[0069] S31. The user looks at the target object and imagines selecting the desired fine hand movements from a variety of preset fine hand movements to grasp or release the target object.
[0070] S32. Collect EEG signals from the user's imagined fine hand movements to grasp or release a target object using an EEG signal acquisition device and preprocess them to obtain preprocessed EEG signals.
[0071] Furthermore, S32 specifically includes the following:
[0072] S321. Preset sampling duration: During the period when the user imagines using the desired fine hand movements to grasp or release the target object, the EEG signal is collected by the EEG signal acquisition device within the preset sampling duration.
[0073] S322. Bandpass filtering and standardization preprocessing are performed on the EEG signals collected within the preset sampling time to obtain the preprocessed EEG signals.
[0074] Specifically, the user gazes at the target object and imagines selecting a desired fine motor gesture from multiple preset fine motor gestures to grasp or release it. During this process, EEG signals are simultaneously acquired by an EEG signal acquisition device. This device collects EEG signals from 32 electrodes according to the 10-20 system standard, with the reference electrode TP10 and ground located on the forehead, and a sampling frequency of 1000Hz. The EEG signals are segmented with a step size of 200ms and a window length of 3000ms, resulting in 32*3000 EEG signal data.
[0075] The EEG signal data is preprocessed, including Butterworth bandpass filtering and normalization preprocessing in the range of 0.1-30Hz, to obtain the preprocessed EEG signal.
[0076] The preprocessed EEG signal is encoded by a trained EEG encoder. Specifically, the encoding is performed by adding a multi-head self-attention module to the convolution module of the trained EEG encoder to obtain the embedding vector of the preprocessed EEG signal. This embedding vector can reflect the relationship between the EEG signal and the fine motor imagery of the hand.
[0077] S4. The embedding vector of the target image is concatenated with the embedding vector of the preprocessed EEG signal. The concatenated embedding vector is then decoded for various preset fine hand movements using EEG signals. The fine hand movement imagery is then directly mapped onto the dexterous hand that controls the five fingers.
[0078] Specifically, the preprocessed EEG signal is encoded using a trained EEG encoder to obtain the embedding vector of the preprocessed EEG signal, which is a one-dimensional feature tensor. The target image of the target object is encoded using a CLIP model to obtain the embedding vector of the target image, which is also a one-dimensional feature tensor. The embedding vector of the preprocessed EEG signal and the embedding vector of the target image are concatenated, that is, the two one-dimensional feature tensors are concatenated to form the embedding vector of the concatenated EEG signal and the target image, which is the concatenated one-dimensional feature tensor.
[0079] The spliced one-dimensional feature tensor is classified by a classifier, such as MLP (Multilayer Perceptron). The spliced one-dimensional feature tensor is then subjected to EEG decoding for four types of fine hand movements, and the fine hand movement imagination is directly mapped to the dexterous hand that manipulates the five fingers.
[0080] The following experiment further illustrates the beneficial effects of the method proposed in this invention, which combines EEG and image encoding to achieve fine motor imagery of the hand.
[0081] See Figure 2 , Figure 2 This is the feature distribution result of four types of fine motor imagery of the hand in one embodiment of the present invention, wherein, Figure 2 (a) The feature distribution of fine motor imagery of the hand directly decoded by the EEG pre-training model. Figure 2 (b) Feature distribution of motor imagery for recognizing fine hand movements using EEG and image co-coding.
[0082] By combining image processing with brain-computer interface motor imagery technology, fine motor imagery decoding of the hand is achieved, allowing users to control the hand shape of a dexterous hand robotic arm simply by gazing and imagining, without taking any physical action. In an embodiment of this invention, the distribution of feature points for four types of fine hand movements (five-finger grasping, three-finger pinching, two-finger pinching, and five-finger releasing) is shown in the figure below. Figure 2 As shown in (a). From Figure 2 As shown in (a), the distribution of feature points corresponding to the five fingers being released and the five fingers being grasped is relatively good, while the feature points corresponding to the three fingers being pinched and the two fingers being pinched are less distinguishable. The overall classification accuracy is 66%.
[0083] By concatenating the embedding vector of the EEG signal with the embedding vector of the target image, the classifier was fine-tuned, and the distribution of feature points was significantly improved. Figure 2 As shown in (b), the overall classification accuracy also increased from 66% to 89%. Therefore, this invention supplements the decoding method for fine motor imagery of the hands in non-invasive brain-computer interfaces, greatly improving decoding accuracy and having significant implications for controlling the dexterous hand of a robotic arm to perform fine tasks via brain-computer interface technology.
[0084] The aforementioned method for achieving fine motor imagery through EEG and image co-encoding involves a pre-trained EEG model comprising an EEG encoder and an EEG decoder. The EEG encoder is trained, and the loss is calculated using the EEG decoder to obtain the trained EEG encoder. Video of the operating space is acquired and played back. By analyzing the user's attention during video viewing, the target image corresponding to the target object the user intends to grasp is obtained. The target image is processed using a CLIP model to extract its embedding vector. EEG signals from when the user imagines grasping the target object using various pre-defined fine motor imagery during video viewing are acquired and pre-processed to obtain pre-processed EEG signals. These pre-processed EEG signals are then encoded using the trained EEG encoder to obtain their embedding vectors. The embedding vectors of the target image and the pre-processed EEG signals are concatenated. The concatenated embedding vectors are then decoded using EEG signals for various pre-defined fine motor imagery, directly mapping the fine motor imagery to the dexterous hand controlling the five fingers. Through the above methods, users can directly control their five-finger dexterity hand to grasp targets by imagining movements without performing any actions. This provides users with a natural operating experience, assists in the rehabilitation of fine motor skills in the hand, and establishes a natural and efficient BCI system. This enables patients with motor disorders to freely control their five-finger dexterity hand to interact with targets in a way that meets their needs, which has significant practical implications.
[0085] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for realizing fine motor imagery of hand movements through joint encoding of electroencephalogram (EEG) and images, characterized in that, The method includes the following steps: S1. Pre-set EEG pre-training model, which includes an EEG encoder and an EEG decoder. The EEG encoder is trained using a pre-set training set and the loss is calculated using the EEG decoder to obtain the trained EEG encoder. The trained EEG encoder can establish the correlation between EEG signals and various pre-set fine motor imagery of the hands. S2. Collect and play the video of the operation space. By analyzing the user's attention during the video viewing, obtain the target image corresponding to the target object that the user expects to grab. Use the CLIP model to process the target image and extract the embedding vector of the target image. S3. Collect and preprocess the EEG signals when the user imagines grasping the target object using a variety of preset fine hand movements while watching the video. The preprocessed EEG signal is then encoded by a trained EEG encoder to obtain the embedding vector of the preprocessed EEG signal. S4. Concatenate the embedding vector of the target image with the embedding vector of the preprocessed EEG signal, and then perform EEG signal decoding on the concatenated embedding vector for various preset fine hand movements. ， The fine motor skills of the hand are directly mapped onto the dexterous hands that control the five fingers.
2. The method for realizing fine motor imagery of the hand through joint encoding of EEG and images according to claim 1, characterized in that, In S1, the EEG encoder is trained using a pre-set training set, and the loss is calculated using an EEG decoder to obtain the trained EEG encoder, which specifically includes the following: S11 includes a pre-set EEG pre-training model, various fine motor skills, training videos, and tags; S12. Collect EEG signals when the user imagines using preset fine hand movements to grasp a target object while watching a training video. S13. The EEG encoder in the EEG pre-training model encodes the EEG signal obtained in S12 to obtain an encoded embedding vector that reflects the relationship between the EEG signal and fine motor imagery of the hand. S14. The EEG decoder in the EEG pre-training model receives the encoded embedding vector and decodes it to obtain the mapping result. It calculates the loss between the mapping result and the preset label and trains the EEG encoder to obtain the trained EEG encoder.
3. The method for realizing fine motor imagery of the hand through joint encoding of EEG and images according to claim 2, characterized in that, S2 specifically includes the following: S21. Capture and play video of the operating space through a camera, and capture the user's visual trajectory while watching the video through an eye tracker and obtain the coordinates of the user's eye movement observation point; S22. Extract the coordinates of the gaze sampling point from the coordinates of the eye movement observation point using the velocity threshold filtering algorithm and add them to the gaze buffer. Filter the coordinates of the gaze sampling point in the gaze buffer using the low-pass filtering algorithm to obtain the gaze coordinates that fall within the target. S23. Input the gaze coordinates and video frame images that fall within the target into the SAM model for processing, and segment the target image corresponding to the target object that the user expects to capture from the video frame images; S24. Encode the target image using the CLIP model to obtain the embedding vector of the target image.
4. The method for realizing fine motor imagery of the hand through joint encoding of EEG and images according to claim 3, characterized in that, S3 collects and preprocesses EEG signals from users imagining grasping a target object using various preset fine hand movements while watching a video, resulting in preprocessed EEG signals, specifically including the following: S31. The user looks at the target object and imagines selecting the desired fine hand movements from a variety of preset fine hand movements to grasp or release the target object. S32. Collect EEG signals from the user's imagined fine hand movements to grasp or release a target object using an EEG signal acquisition device and preprocess them to obtain preprocessed EEG signals.
5. The method for realizing fine motor imagery of the hand through joint encoding of EEG and images according to claim 4, characterized in that, S32 specifically includes the following: S321. Preset sampling duration: During the period when the user imagines using the desired fine hand movements to grasp or release the target object, the EEG signal is collected by the EEG signal acquisition device within the preset sampling duration. S322. Bandpass filtering and standardization preprocessing are performed on the EEG signals collected within the preset sampling time to obtain the preprocessed EEG signals.
6. The method for realizing fine motor imagery of the hand through joint encoding of EEG and images according to claim 1, characterized in that, The encoder in S1 consists of a convolutional module and a multi-head self-attention module connected in sequence, and the decoder in the pre-trained model is a classifier module implemented by a multilayer perceptron.
7. The method for realizing fine motor imagery of the hand through joint encoding of EEG and images according to claim 1, characterized in that, The S1 includes a variety of preset fine hand movements, such as two-finger pinching when facing a flat object, three-finger pinching when facing a knob-shaped object, and five-finger grasping and five-finger spreading when facing a cylindrical object.
8. The method for realizing fine motor imagery of the hand through joint encoding of EEG and images according to claim 1, characterized in that, In step S2, the characteristics of the target object that the user expects to grab include card-shaped, spherical, and cylindrical.