A brain-computer interface system based on fine joint movement imagery of one upper limb
By combining EEG and functional near-infrared spectroscopy signals in a dual-stream multimodal manifold neural network, the intention of fine joint movement of one upper limb is decoded, achieving efficient control of fine joint movement of one upper limb and overcoming the shortcomings of existing MI-BCI systems in fine control.
Patent Information
- Application Number
- CN202311606125.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-28
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-11-28
AI Technical Summary
Existing brain-computer interface systems based on motor imagery (MI-BCI) struggle to achieve precise control over different joints of the same limb, resulting in insufficient system control flexibility.
A brain-computer interface system based on unilateral upper limb fine joint movement imagination is adopted. By combining EEG and functional near-infrared spectroscopy signals, a dual-stream multimodal manifold neural network is constructed to extract and decode fine joint movement intentions, generate upper limb fine movement intention signals, and trigger corresponding movement patterns or stimulation patterns.
It achieves efficient recognition and control of fine joint movements of one upper limb, solves the shortcomings of existing MI-BCI systems in fine movement control, and improves the system's flexibility and accuracy.
Smart Images

Figure CN117814813B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of brain-computer interface applications, and more specifically, to a brain-computer interface system, method, electronic device, and computer-readable storage medium based on fine joint movement imagery of one upper limb. Background Technology
[0002] Motor imagery refers to the act of intending to move without actually performing the action. Motor imagery-based brain-computer interfaces (MI-BCIs) decode motor intentions and enable external communication control by recognizing changes in brain electrical signals induced by motor imagery. MI-BCI does not require external stimuli and directly reflects the user's subjective motor awareness. Its operating mode best matches the state of normal brain activity, truly "turning thought into action." The brain electrical oscillation patterns induced by MI are closest to the body's original motor activation patterns, promoting the reorganization of damaged brain regions and neural pathways. MI-BCI has become an important tool in motor rehabilitation therapy. However, most MI-BCI research focuses on recognizing brain electrical patterns induced by single limb movements (such as left hand, right hand, and foot movements). This only enables the control output of a single target limb movement and cannot meet the requirements for fine control of different joints of the same limb (such as shoulder, elbow, wrist, and fingers). This severely affects the flexibility of the MI-BCI system's external control.
[0003] Therefore, one or more methods are needed to solve the above problems.
[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0005] The purpose of this disclosure is to provide a brain-computer interface system, method, electronic device, and computer-readable storage medium based on fine joint movement imagination of a unilateral upper limb, thereby overcoming at least to some extent one or more problems caused by the limitations and defects of related technologies.
[0006] According to one aspect of this disclosure, a brain-computer interface system based on unilateral upper limb fine joint movement imagination is provided, comprising a fine joint movement intention induction module, a bimodal signal acquisition module, a bimodal signal processing module, and a control command response module, wherein:
[0007] The fine joint motion intention induction module is used to induce experimenters to generate fine joint motion intentions based on a preset fine joint motion intention induction paradigm.
[0008] The dual-modal signal acquisition module is used to acquire EEG signals and near-infrared spectral signals generated by the experimental personnel during the induction of the preset fine joint movement intention evoked paradigm, based on the EEG acquisition system and the continuous wave near-infrared system, respectively.
[0009] The dual-modal signal processing module is used to extract the proprietary features and shared features of the EEG signal and the near-infrared spectral signal respectively, construct a dual-stream multimodal manifold neural network, and then calculate manifold attention based on the dual-stream multimodal manifold neural network. Based on the tensor manifold features, the proprietary features and shared features are fused and decoded to generate upper limb fine motor intention signals.
[0010] The control command response module is used to trigger a movement mode or stimulation mode corresponding to the upper limb fine motor intention signal based on the upper limb fine motor intention signal.
[0011] In one exemplary embodiment of this disclosure, the preset fine joint movement intention induction paradigm in the fine joint movement intention induction module of the system further includes four stages: task preparation, task prompting, task triggering, and rest, wherein:
[0012] During the task preparation phase, a fixed cross shape is displayed for 2 seconds.
[0013] The task prompting phase is used to present video and text of the corresponding joint movements on a preset screen to prompt the experimenters about the content of this experiment, lasting for 2 seconds.
[0014] The task triggering phase is used to prompt the experimenter to perform a motion imagination task according to the displayed task, lasting 4 seconds;
[0015] The duration of the rest phase is a random value between 10 and 12 seconds.
[0016] In one exemplary embodiment of this disclosure, the system further includes:
[0017] Fine motor skills of the upper limbs include shoulder joint movements, elbow joint movements, wrist joint movements, and hand movements;
[0018] Based on the preset fine joint movement intention induction paradigm, multiple groups of experimental subjects were tested, with each group including multiple induction tests of shoulder joint movement, elbow joint movement, wrist joint movement and hand movement.
[0019] In one exemplary embodiment of this disclosure, the dual-mode signal acquisition module of the system further includes:
[0020] The Neuroscan 64-lead EEG acquisition system was used to collect EEG signals generated by the experimenters during the induction of the preset fine joint movement intention paradigm.
[0021] The near-infrared spectral signal generated by the experimenter during the induction of the preset fine joint movement intention paradigm was acquired using the NIRX24 channel continuous wave near-infrared system.
[0022] The EEG acquisition system and the continuous wave near-infrared system are based on the same acquisition cap for the arrangement of electrodes and optical poles.
[0023] In one exemplary embodiment of this disclosure, the dual-mode signal processing module of the system further includes:
[0024] The unique feature of the EEG signal is the covariance matrix of the multi-channel brain power signal;
[0025] The covariance matrix of multichannel concentration change sequences of near-infrared spectral signals, which is more relevant to motion imagery, is selected based on specific features.
[0026] The shared feature of EEG and near-infrared spectral signals is the shared spatial feature obtained by arranging the EEG and near-infrared spectral data at each time point into a spatial matrix according to the lead position and merging them.
[0027] In one exemplary embodiment of this disclosure, the dual-mode signal processing module of the system further includes:
[0028] The EEG signal features after multi-band bandpass filtering are combined along the channel axis, the covariance matrix of the two modal features is calculated separately, and the features are mapped to a symmetric positive definite manifold.
[0029] The extraction of proprietary and shared features is achieved based on the bilinear mapping layer, the manifold batch normalization layer, and the eigenvalue rectifier layer, respectively.
[0030] Based on the manifold attention method, proprietary features and shared features are integrated respectively to achieve multi-branch feature weight allocation;
[0031] Tensor manifold features are introduced, proprietary features and shared features are fused and decoded, and the motion intent is output.
[0032] In one aspect of this disclosure, a manifold attention generation method based on a dual-modal signal processing module of the system is provided, comprising:
[0033] For each branch's feature map, calculate the geometric mean of the features;
[0034] The geometric mean is used to generate the tangent space of a symmetric positive definite manifold, and the mapping function maps the input features to the tangent space. The arithmetic mean of the features in the tangent space is calculated, and the attention weight of each branch feature is calculated.
[0035] In one aspect of this disclosure, an electronic device is provided, comprising:
[0036] Processor; and
[0037] A memory storing computer-readable instructions that, when executed by the processor, implement the method according to any one of the preceding claims.
[0038] In one aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method according to any one of the preceding claims.
[0039] An exemplary embodiment of this disclosure discloses a brain-computer interface system based on unilateral upper limb fine joint movement imagery. The system includes: a fine joint movement intention induction module for inducing fine joint movement intentions in experimental subjects based on a preset fine joint movement intention induction paradigm; a dual-modal signal acquisition module for acquiring electroencephalogram (EEG) signals and near-infrared spectral signals from experimental subjects; a dual-modal signal processing module for extracting specific and shared features from the EEG and near-infrared spectral signals respectively, fusing and merging the specific and shared features based on tensor manifold features, and decoding to generate an upper limb fine movement intention signal; and a control command response module for triggering a movement pattern or stimulation pattern corresponding to the upper limb fine movement intention signal. This disclosure achieves efficient identification of fine joint movement intentions through joint decoding of dual-modal brain signals, providing realistic feedback on the user's movement intentions, and thus achieving the goal of fine motor control.
[0040] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0041] The above and other features and advantages of this disclosure will become more apparent from the detailed description of exemplary embodiments thereof with reference to the accompanying drawings.
[0042] Figure 1 A schematic block diagram of a brain-computer interface system based on fine joint movement imagery of one upper limb according to an exemplary embodiment of the present disclosure is shown.
[0043] Figure 2 A schematic diagram of the EEG and functional near-infrared spectroscopy lead arrangement of a brain-computer interface system based on fine joint motion imagery of one upper limb according to an exemplary embodiment of the present disclosure is shown.
[0044] Figure 3 A schematic diagram of a two-stream network structure based on proprietary and shared features is shown for a brain-computer interface system based on unilateral upper limb fine joint movement imagination according to an exemplary embodiment of the present disclosure.
[0045] Figure 4A flowchart is shown of a manifold attention generation method based on a dual-modal signal processing module of the system according to an exemplary embodiment of the present disclosure;
[0046] Figure 5 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown schematically;
[0047] Figure 6 The illustration shows a schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure. Detailed Implementation
[0048] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.
[0049] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of the specific details described, or other methods, components, materials, apparatuses, steps, etc., can be employed. In other instances, well-known structures, systems, methods, implementations, materials, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.
[0050] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, or in one or more software-hardened modules, or in different network and / or processor devices and / or microcontroller devices.
[0051] In this example embodiment, a brain-computer interface system based on fine joint movement imagery of a unilateral upper limb is first provided; see reference. Figure 1 As shown, this brain-computer interface system based on unilateral upper limb fine joint movement imagination may include a fine joint movement intention induction module, a bimodal signal acquisition module, a bimodal signal processing module, and a control command response module, wherein:
[0052] The fine joint motion intention induction module is used to induce experimenters to generate fine joint motion intentions based on a preset fine joint motion intention induction paradigm.
[0053] The dual-modal signal acquisition module is used to acquire EEG signals and near-infrared spectral signals generated by the experimental personnel during the induction of the preset fine joint movement intention evoked paradigm, based on the EEG acquisition system and the continuous wave near-infrared system, respectively.
[0054] The dual-modal signal processing module is used to extract the proprietary features and shared features of the EEG signal and the near-infrared spectral signal respectively, construct a dual-stream multimodal manifold neural network, and then calculate manifold attention based on the dual-stream multimodal manifold neural network. Based on the tensor manifold features, the proprietary features and shared features are fused and decoded to generate upper limb fine motor intention signals.
[0055] The control command response module is used to trigger a movement mode or stimulation mode corresponding to the upper limb fine motor intention signal based on the upper limb fine motor intention signal.
[0056] An exemplary embodiment of this disclosure discloses a brain-computer interface system based on unilateral upper limb fine joint movement imagery. The system includes: a fine joint movement intention induction module for inducing fine joint movement intentions in experimental subjects based on a preset fine joint movement intention induction paradigm; a dual-modal signal acquisition module for acquiring electroencephalogram (EEG) signals and near-infrared spectral signals from experimental subjects; a dual-modal signal processing module for extracting specific and shared features from the EEG and near-infrared spectral signals respectively, fusing and merging the specific and shared features based on tensor manifold features, and decoding to generate an upper limb fine movement intention signal; and a control command response module for triggering a movement pattern or stimulation pattern corresponding to the upper limb fine movement intention signal. This disclosure achieves efficient identification of fine joint movement intentions through joint decoding of dual-modal brain signals, providing realistic feedback on the user's movement intentions, and thus achieving the goal of fine motor control.
[0057] The following will further describe a brain-computer interface system based on fine joint movement imagination of one upper limb in this example embodiment.
[0058] Example 1:
[0059] The brain-computer interface system based on unilateral upper limb fine joint movement imagery includes a fine joint movement intention induction module 110, a bimodal signal acquisition module 120, a bimodal signal processing module 130, and a control command response module 140, wherein:
[0060] The fine joint motion intention induction module 110 is used to induce experimental personnel to generate fine joint motion intentions based on a preset fine joint motion intention induction paradigm.
[0061] The dual-modal signal acquisition module 120 is used to acquire the EEG signals and near-infrared spectral signals generated by the experimental personnel when induced based on the preset fine joint movement intention evoked paradigm, respectively, using the EEG acquisition system and the continuous wave near-infrared system.
[0062] The dual-modal signal processing module 130 is used to extract the specific features and shared features of the EEG signal and the near-infrared spectral signal respectively, construct a dual-stream multimodal manifold neural network, and then calculate manifold attention based on the dual-stream multimodal manifold neural network. Based on the tensor manifold features, the specific features and shared features are fused and decoded to generate upper limb fine motor intention signals.
[0063] The control command response module 140 is used to trigger a movement mode or stimulation mode corresponding to the upper limb fine motor intention signal based on the upper limb fine motor intention signal.
[0064] In this example embodiment, the preset fine joint movement intention induction paradigm in the fine joint movement intention induction module 110 of the system further includes four stages: task preparation, task prompting, task triggering, and rest, wherein:
[0065] During the task preparation phase, a fixed cross shape is displayed for 2 seconds.
[0066] The task prompting phase is used to present video and text of the corresponding joint movements on a preset screen to prompt the experimenters about the content of this experiment, lasting for 2 seconds.
[0067] The task triggering phase is used to prompt the experimenter to perform a motion imagination task according to the displayed task, lasting 4 seconds;
[0068] The duration of the rest phase is a random value between 10 and 12 seconds.
[0069] In this example embodiment, the system further includes:
[0070] Fine motor skills of the upper limbs include shoulder joint movements, elbow joint movements, wrist joint movements, and hand movements;
[0071] Based on the preset fine joint movement intention induction paradigm, 8 groups of experimental subjects were tested, with each group including 10 induction tests of shoulder joint movement, elbow joint movement, wrist joint movement and hand movement.
[0072] In this example embodiment, the dual-mode signal acquisition module 120 of the system further includes:
[0073] The Neuroscan 64-lead EEG acquisition system was used to collect EEG signals generated by the experimenters during the induction of the preset fine joint movement intention paradigm.
[0074] The near-infrared spectral signal generated by the experimenter during the induction of the preset fine joint movement intention paradigm was acquired using the NIRX24 channel continuous wave near-infrared system.
[0075] The EEG acquisition system and the continuous wave near-infrared system are based on the same acquisition cap for the arrangement of electrodes and optical poles.
[0076] In this example embodiment, the dual-mode signal processing module 130 of the system further includes:
[0077] The unique feature of the EEG signal is the covariance matrix of the multi-channel brain power signal;
[0078] The covariance matrix of multichannel concentration change sequences of near-infrared spectral signals, which is more relevant to motion imagery, is selected based on specific features.
[0079] The shared feature of EEG and near-infrared spectral signals is the shared spatial feature obtained by arranging the EEG and near-infrared spectral data at each time point into a spatial matrix according to the lead position and merging them.
[0080] In this example embodiment, the dual-mode signal processing module 130 of the system further includes:
[0081] The EEG signal features after multi-band bandpass filtering are combined along the channel axis, the covariance matrix of the two modal features is calculated separately, and the features are mapped to a symmetric positive definite manifold.
[0082] The extraction of proprietary and shared features is achieved based on the bilinear mapping layer, the manifold batch normalization layer, and the eigenvalue rectifier layer, respectively.
[0083] Based on the manifold attention method, proprietary features and shared features are integrated respectively to achieve multi-branch feature weight allocation;
[0084] Tensor manifold features are introduced, proprietary features and shared features are fused and decoded, and the motion intent is output.
[0085] In this example embodiment, a novel external control output channel is constructed for the user, reconstructing the patient's limb motor control pathway and achieving motor function compensation. This invention addresses the problem of insufficient system control flexibility caused by the inconsistency between the decoded motor intention and the actual action output in current MI-BCI systems. It constructs a fine joint movement imagery paradigm for unilateral upper limbs and integrates functional near-infrared spectral signals with EEG signals. Through dual-modal brain signal joint decoding, it efficiently identifies fine joint movement intentions and outputs commands via external effectors, providing realistic feedback on the user's movement intentions and achieving the goal of fine motor control.
[0086] Example 2:
[0087] In this example embodiment, the system disclosed herein includes a fine joint movement intention induction module 110, a dual-modal signal acquisition module 120 (EEG and functional near-infrared spectroscopy signals), a dual-modal signal processing module 130, and a control command response module 140. This system designs a single-joint movement intention induction paradigm for the shoulder, elbow, wrist, and fingers, simultaneously acquiring the user's EEG and functional near-infrared spectroscopy dual-modal brain signals. Utilizing dual-modal signal feature extraction and recognition, it achieves accurate decoding of unilateral upper limb fine joint movement intentions and outputs control commands to external response devices. This system establishes a novel MI-BCI paradigm based on unilateral upper limb fine joint movements, overcoming the mismatch between MI-BCI movement intentions and action outputs, and exploring new avenues and providing new methods for promoting the practical application of MI-BCI in rehabilitation engineering.
[0088] In the embodiment of this example, existing MI-BCI can only achieve control output for a single movement of the target limb (such as left hand, right hand, or foot movements), and cannot meet the requirements for fine control of different joints of the same limb (such as shoulder, elbow, wrist, and finger joints). This will seriously affect the flexibility of the MI-BCI system in external control. The present invention designs a brain-computer interface system and device based on unilateral upper limb fine joint movement imagination. By analyzing the dual-modal brain signals of electroencephalography (EEG) and functional near-infrared spectroscopy, it can accurately decode the intention of unilateral upper limb fine joint movements such as shoulder, elbow, wrist, and fingers, and complete the output of control commands to external devices, which helps to achieve fine motor control.
[0089] In this example embodiment, the fine joint movement intention induction module 110 includes:
[0090] Each trial (18-20 seconds) of the experimental paradigm consisted of four phases. First, a fixed cross was displayed in the center of the screen for 2 seconds. Second, the screen displayed text describing the limbs and actions corresponding to the task, accompanied by a demonstration video, for 2 seconds. Then, the demonstration video disappeared, leaving only the text describing the limbs and actions, and displayed "Start Imagining." Participants then imagined the limb movements prompted on the screen for 4 seconds. Finally, the screen displayed "Rest," lasting 10-12 seconds (randomized).
[0091] There are four categories of fine joint movements in the upper limbs: shoulder joint movements (e.g., shoulder flexion / extension), elbow joint movements (e.g., elbow flexion / extension), wrist joint movements (e.g., wrist flexion / extension), and hand movements (e.g., finger grasping / spreading). Each experiment consists of 8 blocks, and each block contains 40 trials. Each of the four movements has 10 trials, and each trial lasts approximately 18-20 seconds. The order of the tasks within each block is randomized. A 5-10 minute rest period is provided between each block.
[0092] In this example embodiment, the dual-mode signal acquisition module 120 includes:
[0093] The experiment simultaneously acquired electroencephalogram (EEG) and functional near-infrared spectroscopy (FIR) signals. EEG was acquired using a Neuroscan 64-lead EEG acquisition system, with the left mastoid process (M1) as the reference electrode. The sampling rate was 1000 Hz, and the signals were filtered through a 0.5 Hz high-pass filter, a 100 Hz low-pass filter, and a 50 Hz notch filter. FIR signals were acquired using a NIRX 24-channel continuous-wave near-infrared system with a sampling rate of 7.8125 Hz and electrode arrangement according to a 10⁻⁵ system. The same acquisition cap was used for both EEG and FIR signals. The electrode distribution for both EEG leads and FIR signals is shown below. Figure 2 As shown in the diagram, white circles represent EEG leads, while dark gray and light gray circles represent the light source and detector for functional near-infrared spectroscopy (FIR) signals, respectively. To better extract information from the right upper limb, this scheme places all FIR signal electrodes at the locations corresponding to the left motor cortex of the brain.
[0094] In this example embodiment, the dual-mode signal processing module 130 includes:
[0095] To address the challenge of constructing high-performance multimodal decoding models from EEG and functional near-infrared spectroscopy signals, this invention proposes a two-stream network that simultaneously extracts modality-specific and modality-shared features, thereby improving multimodal coding performance. This invention constructs a two-stream network and performs fusion decoding from both modality-specific and shared feature perspectives, such as... Figure 3 As shown.
[0096] The specific feature of the EEG signal is the covariance matrix of the multi-channel EEG signal. The specific feature of the functional near-infrared spectroscopy signal is the covariance matrix of the multi-channel concentration change sequence of oxyhemoglobin (HbO), which is more related to motor imagery. The shared feature is the Common Spatial Feature (CSF) proposed in this project. Spatial domain refers to arranging the EEG signal data and HbO signal data at each time step into a spatial matrix according to the lead position (the elements of non-signal data in the matrix are set to 0). Then, the spatial matrices of T time steps are merged to form a tensor of dimension H×W×T (H and W are the width and height of the spatial matrix, respectively, and T is the number of signal sampling points), which is the shared feature.
[0097] Subsequently, using covariance pooling as shown in equation (1), the CSF is flattened into a one-dimensional vector f of length H*W. t (t∈T) is mapped onto a symmetric positive definite manifold to form a multimodal shared spatial domain feature in the manifold space, thereby promoting the full expression of multimodal spatiotemporal information.
[0098]
[0099] in, λ is the regularization parameter, and tr represents the trace operation.
[0100] After obtaining the specific and shared features on the manifold space through the above process, a two-stream multimodal manifold neural network can be constructed. First, the EEG signal features after multi-band bandpass filtering are merged along the channel axis, and the covariance matrix of the two modal features is calculated separately, mapping the features to a symmetric positive definite manifold. Then, the specific and shared features are further extracted using a bilinear mapping layer, a manifold batch normalization layer, and an eigenvalue rectification layer, respectively. Next, the manifold attention method is combined to integrate the two types of features and highlight the role of key features, achieving a more effective attention allocation mechanism than existing Euclidean space attention methods. Finally, tensor manifold features are introduced to fuse and merge the two types of features and perform decoding.
[0101] The manifold attention method is as follows. The feature maps of the G branches are divided into G groups (G=4, indicating that attention is assigned to the feature maps corresponding to the four frequency bands). For the g-th group (g∈{1,…,G}), the geometric mean of all features is calculated. Since directly calculating the weighted geometric mean is very time-consuming, this project proposes to use an iterative method for fast approximation. Let the input data be... (N represents the number of trials, X is an E x T matrix, E represents the number of leads, and T represents time), then the geometric mean The nth approximation M n It can be calculated using the iterative method shown in equation (2).
[0102]
[0103] Then, take the geometric mean. Construct the tangent space of a symmetric positive definite manifold M All input features are mapped to the tangent space using the mapping function shown in equation (3), and the arithmetic mean of the tangent space is calculated. Here, logm(·) represents the logarithm of a matrix. The method for calculating logm(A) for any diagonalizable matrix A is shown in equation (4), where D represents a diagonal matrix, and log(D) represents the natural logarithm of the diagonal elements in D. Calculating the matrix The attention A for the g-th group is calculated by averaging all elements and then applying the sigmoid function. (g) .
[0104]
[0105]
[0106] Finally, A (g) Multiplying with all inputs in group g allows for the allocation of different attention weights to different frequency bands based on manifold features, thereby further improving the model's performance.
[0107] In this example embodiment, the control command response module 140 includes:
[0108] This module acts as a command receiving effector, including but not limited to mechanical exoskeletons, mechanical prostheses, and functional electrical stimulation. It receives the upper limb fine motor intention signal output by the dual-modal signal processing module 130 and triggers the corresponding movement mode or stimulation mode.
[0109] It should be noted that although several modules or units of a brain-computer interface system based on unilateral upper limb fine joint movement imagination are mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.
[0110] Furthermore, in this example embodiment, a manifold attention generation method based on the dual-modal signal processing module 130 of the system is also provided. (Refer to...) Figure 4 As shown, the manifold attention generation method based on the dual-modal signal processing module 130 of the system includes:
[0111] Step S410: For each branch's feature map, calculate the geometric mean of the features;
[0112] Step S420: Take the geometric mean to generate the tangent space of the symmetric positive definite manifold, and use the mapping function to map the input features to the tangent space to calculate the arithmetic mean on the tangent space, and calculate the attention weight of each branch feature.
[0113] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.
[0114] Furthermore, in an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.
[0115] Those skilled in the art will understand that various aspects of the present invention can be implemented as or program products. Therefore, various aspects of the present invention can be specifically implemented as: a completely hardware embodiment, a completely software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."
[0116] The following reference Figure 5 To describe an electronic device 500 according to such an embodiment of the present invention. Figure 5 The electronic device 500 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.
[0117] like Figure 5 As shown, the electronic device 500 is manifested in the form of a general-purpose computing device. The components of the electronic device 500 may include, but are not limited to: at least one processing unit 510, at least one storage unit 520, a bus 530 connecting different system components (including storage unit 520 and processing unit 510), and a display unit 540.
[0118] The storage unit stores program code that can be executed by the processing unit 510, causing the processing unit 510 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 510 can perform actions such as... Figure 1 Steps S410 to S420 are shown in the diagram.
[0119] Storage unit 520 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 5201 and / or cache memory 5202, and may further include a read-only memory (ROM) 5203.
[0120] Storage unit 520 may also include a program / utility 5204 having a set (at least one) program module 5203, such program module 5205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.
[0121] Bus 530 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.
[0122] Electronic device 500 can also communicate with one or more external devices 570 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 500, and / or with any device that enables electronic device 500 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 550. Furthermore, electronic device 500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 560. As shown, network adapter 560 communicates with other modules of electronic device 500 via bus 530. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0123] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.
[0124] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section above.
[0125] refer to Figure 6 As shown, a program product 600 for implementing the above-described method according to an embodiment of the present invention is described. It may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.
[0126] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0127] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.
[0128] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.
[0129] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java and C++, and conventional procedural programming languages such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0130] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.
[0131] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.
[0132] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.
Claims
1. A brain-computer interface system based on unilateral upper limb fine joint movement imagination, characterized in that, The system includes a fine joint movement intention induction module, a dual-modal signal acquisition module, a dual-modal signal processing module, and a control command response module, wherein: The fine joint motion intention induction module is used to induce experimenters to generate fine joint motion intentions based on a preset fine joint motion intention induction paradigm. The dual-modal signal acquisition module is used to acquire EEG signals and near-infrared spectral signals generated by the experimental personnel during the induction of the preset fine joint movement intention evoked paradigm, based on the EEG acquisition system and the continuous wave near-infrared system, respectively. The dual-modal signal processing module is used to: use the covariance matrix of the multi-channel EEG signal as the proprietary feature of the EEG signal; use the covariance matrix of the multi-channel concentration change sequence of oxyhemoglobin related to motor imagery as the proprietary feature of the near-infrared spectral signal; arrange the EEG signal data and near-infrared spectral signal data at each time point into a spatial matrix according to the lead position and merge them to obtain a shared spatial feature; merge the EEG signal features after multi-band bandpass filtering along the channel axis, calculate the covariance matrix of the EEG signal features and the near-infrared spectral signal features respectively, and map the features to a symmetric positive definite manifold; extract the proprietary features and shared features based on a bilinear mapping layer, a manifold batch normalization layer, and an eigenvalue rectification layer; integrate the proprietary features and shared features according to the manifold attention method and perform multi-branch feature weight allocation; fuse and merge the proprietary features and shared features based on tensor manifold features and perform decoding to generate an upper limb fine motor intention signal; The control command response module is used to trigger a movement mode or stimulation mode corresponding to the upper limb fine motor intention signal based on the upper limb fine motor intention signal.
2. The system as described in claim 1, characterized in that, The preset fine joint movement intention induction paradigm in the fine joint movement intention induction module of the system also includes four stages: task preparation, task prompting, task triggering, and rest. During the task preparation phase, a fixed cross shape is displayed for 2 seconds. The task prompting phase is used to present video and text of the corresponding joint movements on a preset screen to prompt the experimenters about the content of this experiment, lasting for 2 seconds. The task triggering phase is used to prompt the experimenter to perform a motion imagination task based on the joint movement video and text shown in the task prompting phase, lasting 4 seconds. The duration of the rest phase is a random value between 10 and 12 seconds.
3. The system as described in claim 2, characterized in that, The system also includes: Fine motor skills of the upper limbs include shoulder joint movements, elbow joint movements, wrist joint movements, and hand movements; Based on the preset fine joint movement intention induction paradigm, multiple groups of experimental subjects were tested, with each group including multiple induction tests of shoulder joint movement, elbow joint movement, wrist joint movement and hand movement.
4. The system as described in claim 1, characterized in that, The dual-mode signal acquisition module of the system further includes: The Neuroscan 64-lead EEG acquisition system was used to collect EEG signals generated by the experimenters during the induction of the preset fine joint movement intention paradigm. The near-infrared spectral signal generated by the experimenter during the induction of the preset fine joint movement intention paradigm was acquired using the NIRX24 channel continuous wave near-infrared system. The EEG acquisition system and the continuous wave near-infrared system are based on the same acquisition cap for the arrangement of electrodes and optical poles.
5. A method for generating manifold attention based on a dual-modal signal processing module of the system according to any one of claims 1-4, characterized in that, The method includes: For each branch's feature map, calculate the geometric mean of the features; The geometric mean is used to generate the tangent space of a symmetric positive definite manifold, and the mapping function maps the input features to the tangent space. The arithmetic mean of the features in the tangent space is calculated, and the attention weight of each branch feature is calculated.
6. An electronic device, characterized in that, include Processor; and A memory storing computer-readable instructions that, when executed by the processor, implement the method according to claim 5.
7. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method according to claim 5.
Citation Information
Patent Citations
Self-training brain-computer interface system and related training method
CN110123313A
Motor imagery electroencephalogram signal classification method based on deep learning and mixed noise data enhancement
CN113269048A
