Hand motion intention recognition method based on priori knowledge and data driving

Abstract

The invention relates to the field of hand motion intention recognition, in particular to a hand motion intention recognition method based on priori knowledge and data driving. According to the method, on a signal level, an EEG channel and an sEMG channel are regarded as different types of nodes respectively, the edge of the sEMG-sEMG type is determined through muscle cooperation features, the edge of the EEG-EEG type is determined according to a phase locking value, multi-type edges are formed by calculating cross-channel correlation (EEG-sEMG), a heterogeneous graph structure based on prior cognition and data driving is constructed, and the heterogeneous graph structure based on prior cognition and data driving is obtained. The brain-muscle multi-modal association is embodied; the weights of nodes and edges are distributed more flexibly in the message passing process of the heterogeneous graph neural network by using a multilayer convolution and attention mechanism designed for the heterogeneous graph, and efficient fusion and characterization of multi-modal information are realized; compared with a traditional homogeneous graph or single-modal processing mode, cross-modal complex information interaction can be reserved and deeply mined.

Description

Technical Field

[0001] The present invention relates to the field of hand movement intention recognition, and in particular to a hand movement intention recognition method based on prior knowledge and data driving. Background Art

[0002] Stroke is one of the most common causes of disability in the world. Every year, about 15 million people suffer from stroke worldwide, of which 5 million suffer from lifelong disability. Stroke patients often suffer from motor disorders due to damaged neural pathways, especially in fine hand movements and daily operations. In stroke rehabilitation, the use of graph neural networks to process signal fusion based on electroencephalogram (EEG) and (sEMG) is of great significance for accurately identifying hand movements. The fusion of EEG and sEMG signals can make full use of the complementary advantages of the two, using EEG to understand the brain's instructions and sEMG to monitor muscle reactions, accurately reflecting the patient's movement intentions and helping to overcome the difficulties encountered by stroke patients in identifying movement intentions.

[0003] In the existing studies on motor intention recognition based on EEG and sEMG, most homogeneous graphs are used as the input of the graph neural network model, that is, all signal channels are simply regarded as nodes of the same type, and the weights of the edges are defined only based on temporal correlation. When researchers use homogeneous graphs to process EEG and sEMG signals, they model the signal channels as nodes, build edges based on the Pearson coefficients between different channels, convert EEG and sEMG into graph structures, and then input them into the graph neural network. However, homogeneous graphs cannot effectively distinguish the characteristics of different modal signals, ignore the heterogeneity between EEG and sEMG, and the synergistic features related to biological characteristics, resulting in insufficient ability to model the relationship between signals and limiting the decoding of motor intention. Some studies have shown that when processing EEG-sEMG fusion signals of stroke patients, the coherence of EEG and EMG signals needs to be considered, which is crucial for the rehabilitation of patients with movement disorders. Therefore, treating EEG and sEMG as nodes of the same type will ignore the different characteristics of the signal sources and fail to fully explore the heterogeneous relationships between different signals. Summary of the invention

[0004] In order to solve the problem that the heterogeneous relationship between different signals cannot be fully mined, the present invention provides a hand movement intention recognition method based on prior knowledge and data-driven. At the signal level, EEG channels and sEMG channels are innovatively regarded as different types of nodes, the sEMG-sEMG type of edge is determined by muscle synergy characteristics, the EEG-EEG type of edge is determined according to the phase locking value, and multi-type edges are formed by calculating the cross-channel correlation (EEG-sEMG). A heterogeneous graph structure based on prior cognition and data-driven is constructed to reflect the brain-muscle multimodal association; by utilizing multi-layer convolution and attention mechanisms designed for heterogeneous graphs, the weights of nodes and edges are more flexibly allocated in the message passing process of heterogeneous graph neural networks, so as to achieve efficient fusion and representation of multimodal information; compared with traditional homogeneous graphs or single modality processing methods, it can retain and deeply mine complex cross-modal information interactions.

[0005] A hand movement intention recognition method based on prior knowledge and data-driven mainly includes the following steps:

[0006] S1: Acquire synchronized EEG and sEMG data;

[0007] S2: Correlations between EEG-EEG, sEMG-sEMG, and EEG-sEMG channels are evaluated separately to determine different types of edges;

[0008] The EEG-EEG type of edge was determined based on the phase locking value, which measures the degree of phase synchronization between two EEG channels within a specific frequency and time range;

[0009] The sEMG-sEMG type edge is determined by calculating the similarity between the synergy vectors of each channel according to the muscle synergy characteristics;

[0010] The nonlinear relationship between nodes of different modalities is calculated using mutual information to construct EEG-sEMG type edges;

[0011] S3: Based on EEG-EEG type edges, sEMG-sEMG type edges, and EEG-sEMG type edges, a heterogeneous graph containing multiple types of nodes and multiple types of edges is constructed based on prior knowledge and data-driven;

[0012] S4: Input the heterogeneous graph into the heterogeneous graph neural network to perform hand movement intention recognition, and obtain a recognition result of the hand movement intention.

[0013] A computer device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0014] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above method.

[0015] A computer program product comprises a computer program or instructions, and when the program or instructions are executed by a processor, the steps of the above method are implemented.

[0016] The beneficial effects brought by the technical solution provided by the present invention are:

[0017] 1. By constructing a heterogeneous graph based on prior knowledge and data-driven, the present invention can more comprehensively capture the complex correlation between EEG and sEMG, significantly reduce the misjudgment caused by single signal noise or instability, and thus achieve higher accuracy in hand movement intention recognition;

[0018] 2. After the present invention introduces the attention mechanism, it can automatically assign higher weights to key nodes or edges (such as the relationship between specific brain areas and muscle groups), effectively suppressing the interference of noise and outliers; even if the signal quality of some channels is poor, it can rely on other strongly related nodes to provide compensation, and the overall recognition performance is more stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0020] Figure 1 It is a framework diagram of a method for identifying hand movement intention based on prior knowledge and data-driven in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0022] Example 1

[0023] Please refer to Figure 1 , Figure 1 is a framework diagram of a method for identifying hand movement intention based on prior knowledge and data-driven in an embodiment of the present invention, specifically including:

[0024] S1: Multi-channel electrodes are placed on the scalp of stroke patients to collect EEG signals, and surface electromyography electrodes are placed on the forearm, upper arm or other muscle groups to obtain synchronized EEG and sEMG data;

[0025] S2: After preprocessing each channel, the correlation between EEG-EEG, sEMG-sEMG, and EEG-sEMG channels is evaluated to determine different types of edges;

[0026] The EEG-EEG type of edge was determined based on the phase locking value, which measures the degree of phase synchronization between two EEG channels within a specific frequency and time range;

[0027] The sEMG-sEMG type edge is determined by calculating the similarity between the synergy vectors of each channel according to the muscle synergy characteristics;

[0028] The nonlinear relationship between nodes of different modalities is calculated using mutual information to construct EEG-sEMG type edges, and the channel pairs with significant correlation (exceeding the specified threshold) are connected as edges in the heterogeneous graph;

[0029] S3: According to EEG-EEG type edges, sEMG-sEMG type edges, and EEG-sEMG type edges, a heterogeneous graph containing multiple types of nodes and multiple types of edges (EEG-EEG, sEMG-sEMG, EEG-sEMG) is constructed based on prior knowledge and data-driven; this heterogeneous graph can not only capture the functional coupling of different brain areas, but also reflect the synergy between muscle groups and the cross-modal association between brain and muscle.

[0030] S4: Input the heterogeneous graph into the heterogeneous graph neural network for hand movement intention recognition, and obtain the recognition result of hand movement intention. Specifically, the present invention introduces an attention mechanism in two layers of heterogeneous graph convolution layers (HConv1, HConv2), more flexibly allocates the weights of nodes and edges in the message passing process of the heterogeneous graph neural network, and uses ReLU as an activation function after each layer to enhance the nonlinear expression ability; after the output of the convolution layer, a fully connected layer (FC) and a dropout mechanism are added to suppress overfitting in training and extract more discriminative features, and finally the recognition result of hand movement intention is output through a Softmax classifier. Compared with traditional single modality or simple parallel processing methods, the present invention can more accurately integrate cerebral cortical activity and muscle contraction information, which not only improves the accuracy and robustness of movement intention recognition, but also can provide more effective neural feedback support for real-time rehabilitation training and external equipment control of stroke patients, such as rehabilitation robots, prostheses, etc.

[0031] The present invention is suitable for effectively fusing EEG and sEMG for stroke patients using heterogeneous graphs and using heterogeneous graph neural networks to identify hand movement intentions. It can fully tap the multimodal information of brain regions and muscle systems, significantly improve the accuracy of accurately capturing and identifying the movement intentions of stroke patients, and thus better assist in the formulation and implementation of rehabilitation training programs, and promote neural remodeling and functional recovery. Existing studies use homogeneous graphs to fuse EEG and sEMG, ignoring the different characteristics of the two signals of EEG and sEMG. In contrast to homogeneous graphs, heterogeneous graphs allow the inclusion of multiple types of nodes and edges, and can provide corresponding representations for different types of signals and channel relationships, reflecting the complexity between multimodal data.

[0032] Example 2

[0033] A computer device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0034] Example 3

[0035] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above method.

[0036] Example 4

[0037] A computer program product comprises a computer program or instructions, and when the program or instructions are executed by a processor, the steps of the above method are implemented.

[0038] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for hand movement intention recognition based on prior knowledge and data-driven, characterized in that: The steps include: S1: Acquire synchronized EEG and sEMG data; S2: Correlations between EEG-EEG, sEMG-sEMG, and EEG-sEMG channels are evaluated separately to determine different types of edges; The EEG-EEG type of edge was determined based on the phase locking value, which measures the degree of phase synchronization between two EEG channels within a specific frequency and time range; The sEMG-sEMG type edge is determined by calculating the similarity between the synergy vectors of each channel according to the muscle synergy characteristics; The nonlinear relationship between nodes of different modalities is calculated using mutual information to construct EEG-sEMG type edges; S3: Based on EEG-EEG type edges, sEMG-sEMG type edges, and EEG-sEMG type edges, a heterogeneous graph containing multiple types of nodes and multiple types of edges is constructed based on prior knowledge and data-driven; S4: Input the heterogeneous graph into the heterogeneous graph neural network to perform hand movement intention recognition, and obtain a recognition result of the hand movement intention.

2. The method for identifying hand movement intention based on prior knowledge and data-driven according to claim 1, characterized in that: In S1, EEG data is obtained by placing multi-channel electrodes on the patient's scalp, and surface electromyography (sEMG) data is obtained by placing surface electromyography (EMG) electrodes on the forearm, upper arm or other muscle groups.

3. The method for identifying hand movement intention based on prior knowledge and data-driven according to claim 1, characterized in that: In S3, when calculating the nonlinear relationship between nodes of different modalities, the significantly correlated channel pairs exceeding the specified threshold are connected as edges in the heterogeneous graph.

4. The method for identifying hand movement intention based on prior knowledge and data-driven according to claim 1, characterized in that: In S4, the heterogeneous graph neural network includes two layers of heterogeneous graph convolution layers HConv1 and HConv2, a fully connected layer FC and a Softmax classifier connected in sequence; the activation function ReLU is used after each heterogeneous graph convolution layer to enhance the nonlinear expression ability; the fully connected layer FC and the dropout mechanism are added after the output of the heterogeneous graph convolution layer to suppress overfitting and extract more discriminative features during training. The Softmax classifier is used to output the recognition results of hand movement intention.

5. The method for identifying hand movement intention based on prior knowledge and data-driven according to claim 4, characterized in that: Attention mechanisms are introduced in both heterogeneous graph convolutional layers to more flexibly allocate the weights of nodes and edges in the message passing process of heterogeneous graph neural networks.

6. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the hand movement intention recognition method based on prior knowledge and data-driven as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that: A computer program is stored, and when the program is executed by a processor, the steps of the hand movement intention recognition method based on prior knowledge and data-driven as described in any one of claims 1 to 5 are implemented.

8. A computer program product, characterized in that It includes a computer program or instruction, which, when executed by a processor, implements the steps of the hand movement intention recognition method based on prior knowledge and data-driven as described in any one of claims 1 to 5.

Images