A cross-modal hand action evaluation method

Abstract

A cross-modal hand action evaluation method comprises the following steps: acquiring a multi-channel motion execution brain electromyogram and preprocessing the brain electromyogram; extracting action-related electroencephalogram signals through electroencephalogram independent component analysis and cortical muscle coherence analysis; electroencephalogram source imaging, calculating the electroencephalogram signals to obtain their mapping in the electroencephalogram source space, and calculating the activity distribution of each brain region according to the Brodmann partition; establishing a relationship model between the source space action-related electroencephalogram and the action mode through a dynamic graph convolutional neural network, extracting electroencephalogram time sequence features and electroencephalogram position space features; and establishing an information fusion model between the source space action-related electroencephalogram signals and the electromyogram signals to complete the recognition of four action modes; the two modal signals of the brain electromyogram are analyzed, the fusion information model is used for recognition and classification, the amount of electroencephalogram signal data is reduced, the analysis accuracy is improved, and the calculation efficiency and classification precision are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of gesture action pattern recognition, and particularly relates to a cross-modal hand action evaluation method for brain and myoelectric signal pattern recognition of different motor execution. BACKGROUND

[0002] Brain Computer Interface (BCI) technology refers to establishing a direct connection channel between the human brain and external devices to realize information interaction and functional integration between the nervous system and external devices. By extracting and classifying the collected relevant electroencephalogram (EEG) or magnetoencephalogram signals, a relevant control signal is generated to control the external device, thereby realizing the interaction between our brain and the external device. Motor execution, as a popular field of BCI, has been widely applied in medical rehabilitation. The collected EEG signals are classified, and the classification results are fed back to the external device to help the subject to perform limb movement. Therefore, in the field of medical rehabilitation, a real-time and high-accuracy motor intention recognition algorithm is of great significance.

[0003] The existing patent application "Stroke patient upper limb and hand rehabilitation training action recognition method (Patent application number: CN201911394850.3)" proposes a stroke patient upper limb and hand rehabilitation training action recognition method, which mainly includes: using a non-negative matrix factorization model to perform blind source separation on the myoelectric signal data, removing non-stationary muscle activation information, and obtaining stable time-varying blind source separation results; applying the decomposed time-varying blind source separation result data to further pattern recognition, improving the stability and accuracy of recognition; and through the CNN-RNN model, the learned features simultaneously maintain time and spatial characteristics. However, this method has some shortcomings. First, brain electrical data is not collected for comprehensive analysis. Brain electrical data is a signal that truly reflects whether the relevant brain region of a stroke patient has activation capability. Using only myoelectric data cannot reflect the user's motor intention. Second, the method uses a non-negative matrix factorization model to decompose the preprocessed myoelectric signal data to obtain multiple blind source separation result matrices, and iteratively trains the CNN-RNN model. In this process, there are fewer myoelectric signal channels related to the action. After correlation analysis of the myoelectric data, the data with higher correlation is used as input to obtain more accurate results. Third, the CNN-RNN model has a large amount of data, and the iterative training time is long, which requires a high-performance computer.

[0004] The hand action pattern recognition algorithm based on EEG mainly includes feature extraction and classification of the electroencephalogram signal. At present, the algorithm for feature extraction of the electroencephalogram signal mainly includes time domain, frequency domain and time-frequency analysis method; and the classifier includes linear discriminant analysis and support vector machine. In recent years, one highlight in machine learning is the proposal of convolutional neural network (CNN), and especially in the computer vision task, the recognition accuracy of CNN has been greatly improved compared with the traditional machine learning method. Therefore, people begin to study the potential of convolutional neural network in brain signal decoding. Although the CNN model has been proved to be very powerful in processing classification problems, it is worth noting that the previous CNN application pays more attention to learning local features from images, videos and speech, in which the data points of the signal are constantly changing. For another feature learning problem, such as feature learning from traffic networks and brain networks, the traditional CNN method may not be suitable because the signal is discrete and discontinuous in the spatial domain. In this case, the graph-based description method will provide a more effective method.

[0005] Compared with the invasive brain-computer interface technology requiring implantation of electrode arrays, the non-invasive brain-computer interface technology has the characteristics of safety, convenience and low cost, but since the neuron discharge needs to be transmitted to the scalp through the cerebral cortex before it can be detected and recorded by the scalp surface electrode, the signal will be attenuated to a certain extent during the transmission process, and more noise will be introduced, and the electrical signals of the neurons in the adjacent area will interfere with each other, bringing greater challenges to the decoding work. SUMMARY

[0006] In order to improve the accuracy of single-hand multi-class action pattern recognition classification based on physiological signals, the purpose of the present application is to provide a cross-modal hand action evaluation method, which establishes an information fusion model between the source space action-related electroencephalogram signal and electromyogram signal, not only realizes the cross-modal analysis of the electromyogram and electroencephalogram signals, but also realizes the multi-dimensional fusion of one-dimensional neural network and multi-dimensional neural network, reduces the amount of electroencephalogram signal data, and improves the accuracy of analysis.

[0007] In order to achieve the above purpose, the technical scheme of the present application is as follows:

[0008] A cross-modal hand action evaluation method, comprising the following steps:

[0009] Step 1: Obtain multi-channel motion execution brain electromyogram and pre-process the brain electromyogram;

[0010] Step 2: Extract the action-related electroencephalogram signal from the pre-processed brain electromyogram through electroencephalogram independent component analysis and cortical muscle coherence analysis;

[0011] Step three: electroencephalography, calculate the mapping of the action-related electroencephalogram extracted in step two in the electroencephalography space, and calculate the activity distribution of each brain region according to the Brodmann partition;

[0012] Step four: establish the relationship model of the action-related electroencephalogram in the electroencephalography space and the action mode through the dynamic graph convolutional neural network, extract the electroencephalogram time sequence feature and the electroencephalogram position space feature;

[0013] Step five: based on the electroencephalogram time sequence feature and the electroencephalogram position space feature, establish the information fusion model between the action-related electroencephalogram signal in the source space and the electromyogram signal, and complete the recognition of the four action modes.

[0014] In the process of acquiring the multi-channel motor execution brain electromyogram in step one, the acquired electroencephalogram is represented as Wherein, m represents the number of subjects, Noc indicates the brain electroencephalogram signal collected by each subject, n represents the action category, m is the number of Noc lead electroencephalogram signals collected in the clinical experiment of the mth stroke patient, and the nth action of the mth subject; Indicates a real matrix, There are Noc rows and t columns in the matrix, t is the number of sampling points, and it is assumed that the sampling time is t t seconds and the sampling frequency is F, then t=t t ×F, the acquired electromyogram is represented as Indicates the electromyogram collected by the mth subject in the nth action, which is obtained by bipolar differential measurement, Noc a Indicates the channel number of the collected electromyogram data, the ath channel number is Noc a , and two leads form a channel.

[0015] The preprocessing of the brain electromyogram in step one includes filtering, baseline calibration and deletion of signals with obvious electroencephalogram drift, and the preprocessed electroencephalogram data is The preprocessed electromyogram data is

[0016] The specific process in step two is as follows:

[0017] (1) Independent component analysis is performed on the preprocessed electroencephalogram data Y mm，n , and the Noc lead electroencephalogram data Y mm，n collected in the clinical experiment of the stroke patient is decomposed into a series of time-independent and spatially fixed components, and the formula is represented as:

[0018]

[0019] In the formula, n is the number of independent components, Qi It is matrix Q Noc×n The element in the i-th column represents the spatial topographic map of the i-th independent component, T i It is matrix T n×t The elements in the i-th row represent the time series of the i-th independent component. W is a weight diagonal matrix, with weight values ​​existing only on the diagonal. i The weight represents the unique non-zero value in the i-th column vector of W.

[0020] (2) Cortical-muscle coherence analysis was performed using electromyographic signals, and the Y-values ​​of each EEG data point were calculated. m Time series T of independent components i Compared with the multi-channel electromyography signals collected in the above clinical trials The formula for calculating the cortical-muscular coherence coefficient between the two layers is as follows:

[0021]

[0022] In the formula, f represents the frequency. Represents each EEG data point Y m，n Time series T of independent components i Compared with the multi-channel electromyography signals collected in the above clinical trials Cross-power spectral density between Represents each EEG data point Y m，n Time series T of independent components i The self-power spectral density, This refers to the multichannel electromyography signals collected in the above clinical trial. The self-power spectral density, This represents each calculated EEG data point Y. m，n Time series T of independent components i Compared with the multi-channel electromyography signals collected in the above clinical trials The cortical-muscle coherence coefficient between the two is used to obtain independent components that are closely related to action-related EEG.

[0023] (3) Extraction of action-related EEG signals: The selected independent components are subjected to inverse independent component analysis to reconstruct the action-related EEG signals. The reconstruction formula is as follows:

[0024]

[0025] In the formula, Y m，n，back The reconstructed action-related EEG signals, n ok Q represents the number of independent components associated with brain electrical activity. i For this independent component, the corresponding spatial topographic map, T iFor the time series corresponding to the independent components, the weight diagonal matrix W remains unchanged.

[0026] The specific process in step three is as follows:

[0027] (1) Using the action-related EEG signals reconstructed in step two, the leading field is calculated using a real boundary cell head model based on magnetic resonance imaging (MRI). The mathematical model for the generation of scalp EEG signals is established as follows:

[0028] Y m，n，back =A×X+noise (4)

[0029] Y mm，n，back The signal represents the action-related EEG signal obtained after step two; A represents the lead field, which is a zero-lead matrix that simulates the distribution of neurons in the cerebral cortex and the changes in their postsynaptic potentials; X represents the dipole moment of the cortical current dipole, which simulates the distribution of neurons in the cerebral cortex and the changes in their postsynaptic potentials (source activity); noise represents the measurement noise.

[0030] (2) The inverse problem is solved by using the minimum norm estimation method;

[0031]

[0032] L2 regularization is used to further constrain the solution in order to find the solution X that minimizes both the residual and energy of the source space; this additional constraint is represented by the second term in equation (5); the regularization parameter λ is obtained by using the L-curve method to balance the influence of these two terms. In the case of L2 norm, the analytical solution of equation (5) yields the optimal estimate of the source distribution X, as shown in equation (6).

[0033]

[0034] When W is only the identity matrix, this solution Known as Minimum-norm Estimation (MNE), it is the mapping of EEG signals in the brain's energy space. T The flip matrix represents the leading field;

[0035] In step three, the number of dipoles is categorized according to Brodmann partitions. The measurements of dipoles within a region at the same time are summed and considered as the overall output of that brain region, thus obtaining the source distribution matrix for a single subject. Where n∈{1, 2, 3, 4} represents the category label of the motion task, and d={1, 2, 3, …, N m}, N mN represents the number of times the experiment was conducted for the m-th action; c Y represents motor-related EEG signals m，n，back The number of channels mapped into the source space (i.e., the number of regions of interest after Brodmann partitioning); N s This represents the number of sampling points.

[0036] Step four, which involves establishing a relationship model between source space action-related EEG and action patterns using a dynamic graph convolutional neural network, and extracting EEG time-series features and EEG location-space features, includes the following:

[0037] (1) The source distribution matrix of a single subject in the source space obtained in step three. A brain network graph (Graph) is constructed using a dynamic graph convolutional neural network; the brain network graph is an undirected, weighted graph with N nodes. c A graph with n vertices can be expressed by the following formula:

[0038] Graph = {V, E, W} G} (7)

[0039] In the formula, N c W represents the number of leads in the EEG. G For N c ×N c The adjacency matrix describes the connection between any two nodes in V, where W is the adjacency matrix in the i-th row and j-th column. G The term is denoted as w ij , representing the importance of the connection between the i-th node and the j-th node, and the final vertex set V contains N c Each vertex and its corresponding feature are represented as follows: For point, For each feature vector, there are as many elements as there are features; E is the set of edges, containing brain connectivity information and the weights of those connections, which are expressed through W. G And V to determine;

[0040] (2) Dynamically learn the adjacency matrix W representing the relationships between each vertex node. G ,make This represents the optimal adjacency matrix to be learned, which is the source distribution matrix of a single subject in the source space. Separate the data, and extract a vector from each row. n c ∈{1, 2, 3, ..., N c} represents motor-related EEG signals Y m，n，back The channel number mapped in the source space. Spatial filtering g(L) * ) Determined The graph convolution of a vector is represented as:

[0041]

[0042] In the above formula, L * L represents the Laplacian matrix of a brain network graph. * It can be represented as:

[0043]

[0044] It is a diagonal matrix, and the i-th diagonal element can be obtained from... Calculations show that express The element in the i-th row and j-th column; for a given Its graphical Fourier transform is represented as:

[0045]

[0046] U represents the vector of EEG-related signals after transformation in the frequency domain. * It is an orthogonal matrix that can be obtained by the Laplacian operator matrix L of the singular value decomposition graph. * get:

[0047]

[0048] U * The list Constructing Fourier bases, It is a diagonal matrix. From equation (10), the inverse representation of the graph Fourier transform is:

[0049]

[0050] In equation (8), g(·) is the filtering function. As can be seen from equation (11), after g(L) * Filtered signal Represented as And among them g(Λ) * ) is represented as:

[0051]

[0052] According to formulas (9)-(13), formula (8) is written as formula (14), where ⊙ represents the Hadamard product divided by elements:

[0053]

[0054] let Represents Λ *The largest element in the diagonal terms, and using Normalization Λ * , making The diagonal elements of lie in the interval [-1, 1], where I N It is N c ×N c The identity matrix, within the framework of the K-order Chebyshev polynomial, yields g(Λ). * Approximately:

[0055]

[0056] In the above formula, θ k The coefficients of the Chebyshev polynomial are... The following recursive expressions (16) and (17) are used for recursive calculation:

[0057]

[0058]

[0059] According to equation (15), the graph convolution operation defined in equation (8) can be rewritten as:

[0060]

[0061] Equation (18) represents the calculation The graph convolution can be represented as The convolution result is a combination of the components of the Chebyshev polynomial, where... Based on Equation (18), a model of the relationship between the brain network and action patterns of source space action-related EEG is established. The framework consists of four main layers: graph data filtering layer, convolutional layer, ReLU activation layer, and fully connected layer. Each channel in the source space is represented as a node in the model. Next is a 1×1 convolutional layer. The ReLU activation function is used to ensure that the output of the graph data filtering layer is non-negative. Finally, the output of the activation function is sent to the multi-layer fully connected network.

[0062] The network parameters are iteratively updated using the backpropagation algorithm until an optimal or near-optimal solution is obtained. Specifically, a cross-entropy loss function is used, with the following form:

[0063] LOSS = cross_entropy(I, I p )+α||Θ|| (19)

[0064] I and I p Let Θ represent the training data and the predicted actual label vectors, respectively; let Θ represent all model parameters; and let α be the weight for the tradeoff regularization. Calculate the loss function relative to... The partial derivative of is expressed as:

[0065]

[0066] in, express The element in the i-th row and j-th column; according to the chain rule, The calculation formula can be expressed as:

[0067]

[0068] After calculating Then, the optimal adjacency matrix is ​​updated using the following rules.

[0069]

[0070] In equation (21), ρ represents the learning rate of the network.

[0071] Step five involves establishing an information fusion model between source space action-related EEG and EMG signals to complete the recognition process of the four action patterns, including:

[0072] (1) Training the model:

[0073] Electromyography (EMG) signals are processed through a two-layer one-dimensional convolutional network to extract action-related EMG features. Then, the one-dimensional convolutional output of EMG and the model of the relationship between the brain network and action patterns of action-related EEG in the source space are output. After information fusion through a fully connected layer, the classification result is output. The model is trained by the backpropagation algorithm, which automatically extracts features from EMG and EEG networks and automatically selects the most prominent features to achieve the classification of four action attempts.

[0074] (2) Model Evaluation:

[0075] Using the M-fold cross-validation method, a total of K data points were collected for each subject for each action. n Next, after each attempt, T vault The data in seconds is used as a time window, and one graph feature is extracted for each window. There are four actions in total, totaling 4×K. n 1 feature sample; randomly select from each action Samples, total One sample is used as the training set, and the remaining samples are used as the test set. The training set is divided into two parts: 80% is used to train the model and 20% is used to validate the model to see if overfitting occurs. The above operation is repeated M times, and the average recognition accuracy is finally calculated to evaluate the ability of the proposed new algorithm and existing algorithms to distinguish and recognize different actions.

[0076] Compared with the prior art, the present invention has the following advantages:

[0077] (1) In step two, the brain-muscle electroencephalogram (BME) signals are executed for multi-channel motion, and independent component analysis is performed on the EME data. The cortical-muscle coherence analysis is performed in conjunction with the EME signals, so that the brain electrical activity generated in the relevant areas of the user's brain is simultaneously transmitted to the relevant cortical muscles and generates EME activity. The action-related EME is selected and the action-related EME data is reconstructed, which reduces the amount of EME signal data and improves the accuracy of the analysis.

[0078] (2) In step three, based on the theoretical basis of neurophysiology, the present invention performs spatial imaging of brain power sources and establishes a temporal positional relationship model of source space action-related EEG and action patterns, which is beneficial to reduce the volume conduction effect during EEG signal transmission and noise interference during the acquisition process.

[0079] (3) In step four, a model of the relationship between the brain network of source space action-related EEG and action patterns is established by using a dynamic graph convolutional neural network. This realizes the dynamic interconnection between the various leads of the EEG and establishes the graph network relationship between source space action-related EEG and motor execution actions. Without increasing the computational cost, the amount of input information of the model is increased, which improves the classification accuracy.

[0080] (4) In step five, a cross-modal and cross-dimensional deep learning fusion model is used to classify the four types of action attempts of the brain multichannel motor execution electromyography signals. This not only realizes the cross-modal analysis of the two major signals of electromyography, but also realizes the multi-dimensional fusion of one-dimensional neural network and multi-dimensional neural network, achieving a high recognition accuracy. Attached Figure Description

[0081] Figure 1 This is a flowchart of the algorithm used in this invention.

[0082] Figure 2(a) is a schematic diagram of the scalp EEG acquisition device used in the experiment of this invention, and Figure 2(b) is a diagram of the EEG channel distribution after acquisition.

[0083] Figure 3 A diagram illustrating the hand movement pattern designed for this invention.

[0084] Figure 4 A schematic diagram of the experimental protocol for acquiring electroencephalogram (EEG) and electromyogram (EMG) signals.

[0085] Figure 5 This is a flowchart of the extraction process for action-related EEG signals.

[0086] Figure 6 This is a diagram of the dynamic graph convolutional network structure used in this invention.

[0087] Figure 7 This is an accuracy table for testing on thirteen subjects. Detailed Implementation

[0088] The process and advantages of the present invention will be described in detail below with reference to the accompanying drawings.

[0089] The specific experiments of this invention were conducted under the Windows 10 (64-bit) operating system. EEG preprocessing, data augmentation, and source imaging were performed using Matlab R2019b. Network training was based on the TensorFlow 2.6.0 framework, with Keras as the front-end API and Python 3.9 as the compiler virtual environment for training. EEG source localization was implemented using the Brainstorm toolbox based on Matlab 2019b.

[0090] Figure 1 This is a brief overview of the entire algorithm, which consists of five modules. It is a cross-modal hand motion evaluation method, including the following steps:

[0091] Step 1: Acquire multi-channel electromyographic (EMG) signals of the brain during motion execution and preprocess the EMG signals;

[0092] Step 2: Extract action-related EEG signals from the preprocessed electromyography (EMG) signals using independent component analysis of EEG and cortico-muscle coherence analysis.

[0093] Step 3: Brain power imaging. The action-related EEG signals extracted in Step 2 are calculated to obtain their mapping in the brain power space, and the activity distribution of each brain region is calculated according to the Brodmann partition.

[0094] Step 4: Establish a model of the relationship between brain power spatial movement-related EEG and movement patterns through a dynamic graph convolutional neural network, and extract EEG time series features and EEG location spatial features;

[0095] Step 5: Based on the time series features and spatial features of EEG, establish an information fusion model between source space action-related EEG signals and electromyographic signals to complete the recognition of four action patterns.

[0096] Figure 2(a) is a schematic diagram of the scalp EEG acquisition device used in the experiment of this invention, and Figure 2(b) is a distribution diagram of the EEG channels after acquisition, showing the distribution of leads for the scalp EEG acquisition, using the 32-lead EEG signal of the neuroscan EEG system.

[0097] Experimental protocols for acquiring electroencephalogram (EEG) and electromyogram (EMG) signals are as follows: Figure 4As shown, electroencephalogram (EEG) and electromyographic (EMG) signals were collected from each subject. EEG signals were acquired using the Neuroscan EEG system. A total of 32 leads of EEG signals were selected for this experiment. Electrode positions were chosen according to the international 10-20 standard system, with the reference electrode at A2 of the right mastoid process and the ground electrode at AFz in the parietal lobe. Conductive fluid was injected into each electrode hole of the electrode cap. To ensure signal acquisition quality, the impedance of all electrodes was checked at least twice to ensure that the impedance of each electrode was less than 5 kΩ. EMG signals were also acquired synchronously with the EEG using the Neuroscan system. Two pairs of surface differential electrodes with four leads were used, with the distance between each pair of electrodes not exceeding 20 mm. Two surface EMG signals were simultaneously acquired from the extensor digitorum and palmaris longus muscles of the forearm, respectively, with the reference electrode placed on the wrist. Before the experiment, the skin surface of the subject's forearm was wiped with exfoliating cream to reduce interference from skin impedance during the experiment. During simultaneous electroencephalography (EEG) acquisition, the signal sampling frequency was set to 1 kHz. Each patient also underwent MRI imaging. Patients presented with unilateral cortical or subcortical injuries due to localized ischemia or hemorrhage (assessed using MRI images), without cerebellar injury. All participants were voluntary and clearly informed of the experiment's purpose, research content and protocol, potential risks and benefits, fees and remuneration, injury compensation, confidentiality of personal information, and the right to withdraw. The experiment included four hand movements: clenching a fist, opening a fist, pinching with two fingers, and pinching with three fingers. Figure 3 As shown, the experimental signals were collected from 13 participants, including four types of actions, with each action recorded 60 times. Each participant was required to observe the action to be performed during the 0-2s cue phase, perform the corresponding action with their right hand according to the cue during the 2-4s execution phase, and return their right hand to a relaxed state during the 4-5s rest phase, waiting for the start of the next experimental trial.

[0098] Based on the above-mentioned motor execution EEG / EMG dataset, the following was adopted: Figure 1 The algorithm flow shown, taking one subject as an example, the specific implementation steps of the present invention are as follows:

[0099] In step one, during the acquisition of multi-channel motion execution electroencephalogram (EEG) signals, the acquired EEG signals are represented as follows: Where m represents the subject number, and Noc-guided EEG signals were collected for each subject (Noc = 32 in this experiment), n represents the action category, m is the Noc-guided EEG signal number collected in the clinical experiment for the m-th stroke patient, and n is the n-th action for the m-th subject (four types were collected in the experiment, and a total of four action patterns were collected in the experiment, namely clenching fist, opening fist, two-finger pinching and three-finger pinching, so the maximum value of n is 4). Represents a real matrix. There is a Noc row and a t column, where t is the number of sampling points, and the sampling time is assumed to be t.t If the sampling frequency is F, then t = t t ×F. The acquired electromyographic signals are represented as This represents the electromyographic signal obtained by attempting the nth action on the m-th subject (four types were collected in the experiment, with n having a maximum value of 4), measured using bipolar differential measurement. Noc a This indicates the channel number of the acquired electromyography data, with the a-th channel number being Noc. a The two leads form a channel.

[0100] The preprocessing of the EEG / EMG signals in step one includes filtering, baseline calibration, and removal of signals exhibiting significant EEG drift (if removal is performed, the entire segment is deleted to ensure data consistency across all EEG channels). EEG preprocessing typically uses a 0.05-35Hz bandpass filter, while EMG preprocessing uses a 5-500Hz bandpass filter. Based on the timing information from the signal acquisition experimental protocol, the preprocessed signals are segmented into four different single-handed action attempts: clenching a fist, opening a fist, pinching with two fingers, and pinching with three fingers, with the same number of attempts for each single-handed action. The resulting preprocessed EEG data is as follows: The electromyography data obtained after preprocessing are

[0101] like Figure 5 As shown, the specific process in step two is as follows:

[0102] (1) Preprocessed EEG data Y mm，n Independent component analysis was performed on the Noc-guided EEG data Y collected from clinical trials of stroke patients. mm，n It can be decomposed into a series of time-independent and spatially fixed components, and its formula can be expressed as:

[0103]

[0104] In the formula, n is the number of independent components, Q i It is matrix Q Noc×n The element in the i-th column represents the spatial topographic map of the i-th independent component, T i It is matrix T n×t The elements in the i-th row represent the time series of the i-th independent component. W is a weight diagonal matrix, with weight values ​​existing only on the diagonal. i The weight represents the unique non-zero value in the i-th column vector of W.

[0105] (2) Cortical-muscle coherence analysis was performed using electromyographic signals, and the Y-values ​​of each EEG data point were calculated. m Time series T of independent components i Compared with the multi-channel electromyography signals collected in the above clinical trials The formula for calculating the cortical-muscular coherence coefficient between the two layers is as follows:

[0106]

[0107] In the formula, f represents the frequency. Represents each EEG data point Y m，n Time series T of independent components i Compared with the multi-channel electromyography signals collected in the above clinical trials Cross-power spectral density between Represents each EEG data point Y m，n Time series T of independent components i The self-power spectral density, This refers to the multichannel electromyography signals collected in the above clinical trial. The self-power spectral density, This represents each calculated EEG data point Y. m，n Time series T of independent components i Compared with the multi-channel electromyography signals collected in the above clinical trials The cortical-muscular coherence coefficients between the brain and motor activity were analyzed. Cortical-muscular coherence coefficients in motor-related frequency bands were selected for analysis. Ignoring independent EEG components with low coherence coefficients removes independent components unrelated to the brain electrical activity of interest, thus obtaining independent components closely related to motor-related EEG activity.

[0108] (3) Extraction of action-related EEG signals. The selected independent components are subjected to inverse independent component analysis (ICA) to reconstruct the action-related EEG signals. The reconstruction formula is as follows:

[0109]

[0110] In the formula, Y mm，n，back The reconstructed action-related EEG signals, n ok Q represents the number of independent components related to the brain electrical activity we are interested in. i For this independent component, the corresponding spatial topographic map, T i For the time series corresponding to the independent components, the weight diagonal matrix W remains unchanged.

[0111] The specific process in step three is as follows:

[0112] (1) Solving the problem of EEG, namely, establishing a head model that can explain how the activity of neurons in the cerebral cortex is projected onto the scalp through the brain volume effect and the conduction of the skull. The main parts of the head model include the geometry of the brain and the conductivity characteristics of the different tissues (gray matter, white matter, cerebrospinal fluid, skull, and skin) through which the propagation of neural electrical activity occurs. Using the action-related EEG signals reconstructed in step two, a real boundary cell head model based on magnetic resonance imaging (MRI) is used to calculate the lead field. The mathematical model for the generation of scalp EEG signals is established as follows:

[0113] Y m，n，back =A×X+noise (4)

[0114] Y mm，n，back The signal represents the action-related EEG signal obtained after step two; A represents the lead field, which is a zero-lead matrix that simulates the distribution of neurons in the cerebral cortex and the changes in their postsynaptic potentials; X represents the dipole moment of the cortical current dipole, which simulates the distribution of neurons in the cerebral cortex and the changes in their postsynaptic potentials (source activity); noise represents the measurement noise.

[0115] (2) The inverse problem is solved by using the minimum norm estimation method.

[0116]

[0117] Theoretically, the optimal solution for X is the one that minimizes the residual between the estimate and the recorded scalp EEG in the L2 norm least squares sense. Considering that the number of dipoles is often greater than the number of electrodes, and many types of dipole distributions can produce similar results on the scalp, L2 regularization is used to further constrain the solution in order to find the solution X that minimizes both the residual and energy of the source space. This additional constraint is represented by the second term in equation (5); the regularization parameter λ is obtained using the L-curve method to balance the influence of these two terms. In the L2 norm case, the analytical solution of equation (5) yields the optimal estimate of the source distribution X, as shown in equation (6).

[0118]

[0119] When W is only the identity matrix, this solution is called the Minimum-norm Estimation (MNE), A T This represents the flip matrix of the leading field.

[0120] In step three, the number of dipoles is categorized according to Brodmann partitions. The measurements of dipoles within a region at the same time are summed and considered as the overall output of that brain region, thus obtaining the source distribution matrix for a single subject. (i.e., the EEG signal mapped from the sensor space to the source space), where n∈{1,2,3,4} represents the category label of the motion task, and d={1,2,3,…,N m}, N m N represents the number of times the experiment was conducted for the m-th action; c Y represents motor-related EEG signals mm，n，back The number of channels mapped into the source space (i.e., the number of regions of interest after Brodmann partitioning); N s This represents the number of sampling points.

[0121] like Figure 6 As shown, step four, which involves establishing a relationship model between source space action-related EEG and action patterns using a dynamic graph convolutional neural network, and extracting EEG time-series features and EEG location-space features, includes:

[0122] (1) Regarding action-related EEG signals Y mm，n，back Source distribution matrix of a single subject obtained after brain power imaging analysis. The detailed process is explained in step three.

[0123] (2) The source distribution matrix of a single subject in the source space obtained in the previous step. A brain network graph (Graph) is constructed using a dynamic graph convolutional neural network. The brain network graph is an undirected, weighted graph with N nodes. c A graph with n vertices can be expressed by the following formula:

[0124] Graph = {V, E, W} G} (7)

[0125] In the formula, N c W represents the number of leads in the EEG. G For N c ×N c The adjacency matrix describes the connection between any two nodes in V, where W is the adjacency matrix in the i-th row and j-th column. G The term is denoted as w ij Let represent the importance of the connection between the i-th node and the j-th node, and let W be the adjacency matrix representing the relationships between the vertex nodes. G It is dynamically learned, not predetermined. The dynamic learning process is described in part (3) of this step. V is the set of vertices, and V contains N. cEach vertex has N vertices, and each vertex has corresponding features. These features can be extracted from the action-related EEG signals of the leads corresponding to each vertex. The features extracted in the experiment include linear features such as absolute value integral, peak detection, number of zero crossings, variance, power spectral energy, autoregressive model coefficients, short-time Fourier transform energy, and wavelet transform coefficients, as well as nonlinear features such as approximate entropy, sample entropy, fuzzy entropy, information entropy, complexity, higher-order statistics, fractal dimension, and Lyapunov exponent. The final vertex set V contains N vertices. c Each vertex and its corresponding feature are represented as follows: For point, For each feature vector, there are as many elements as there are features. E is the set of edges, containing brain connectivity information and the weights of those connections, which can be represented by W. G The weight of V indicates a close connection between the two EEG leads, while a lower weight indicates a loose connection.

[0126] (3) Dynamically learn the adjacency matrix W representing the relationships between each vertex node. G .make This represents the optimal adjacency matrix to be learned. It represents the source distribution matrix of a single subject in the source space. Separate the data, and extract a vector from each row. n c ∈{1, 2, 3, ..., N c} represents motor-related EEG signals Y m，n，back The channel number mapped in the source space. Spatial filtering g(L) * ) Determined The graph convolution of a vector can be represented as:

[0127]

[0128] In the above formula, L * L represents the Laplacian matrix of a brain network graph. * It can be represented as:

[0129]

[0130] It is a diagonal matrix, and the i-th diagonal element can be obtained from... Calculations show that express The element in the i-th row and j-th column. For a given... Its graphical Fourier transform is represented as:

[0131]

[0132] U represents the vector of EEG-related signals after transformation in the frequency domain. * It is an orthogonal matrix that can be obtained by the Laplacian operator matrix L of the singular value decomposition graph. * get:

[0133]

[0134] U * The list Constructing Fourier bases, It is a diagonal matrix. From equation (10), the inverse of the graph Fourier transform can be expressed as:

[0135]

[0136] In equation (8), g(·) is the filtering function. As can be seen from equation (11), after g(L) * Filtered signal It can be represented as And among them g(Λ) * This can be represented as:

[0137]

[0138] According to formulas (9)-(13), formula (8) is written as formula (14), where ⊙ represents the Hadamard product divided by elements:

[0139]

[0140] Because it is difficult to calculate g(Λ) directly * The expression for g(Λ) is used in this invention. * The polynomial expansion of ... Represents Λ * The largest element in the diagonal terms, and using Normalization Λ * , making The diagonal elements of lie in the interval [-1, 1], where I N It is N c ×N c The identity matrix. Within the framework of the K-order Chebyshev polynomial, we can obtain g(Λ * Approximately:

[0141]

[0142] In the above formula, θ k The coefficients of the Chebyshev polynomial are... The following recursive expressions (16) and (17) can be used to calculate the following:

[0143]

[0144]

[0145] According to equation (15), we obtain that the graph convolution operation defined in equation (8) can be rewritten as:

[0146]

[0147] Equation (18) represents the calculation The graph convolution can be represented as The convolution result is a combination of the components of the Chebyshev polynomial, where... Based on equation (18), this invention proposes a model of the relationship between brain networks and action patterns in source space action-related EEG, the framework of which is as follows: Figure 6 As shown, the framework consists of four main layers: a graph data filtering layer, a convolutional layer, a ReLU activation layer, and a fully connected layer. Each channel in the source space is represented as a node in the model. Next, a 1×1 convolutional layer is used to learn discriminative features between different frequency domains. Furthermore, to achieve the network's non-linear mapping capability, a ReLU activation function is used to ensure the output of the graph data filtering layer is non-negative. Finally, the output of the activation function is sent to the multi-layer fully connected network.

[0148] The network parameters are iteratively updated using the backpropagation algorithm until an optimal or suboptimal solution is obtained. This invention employs a cross-entropy loss function, which has the following form:

[0149] LOSS = cross_entropy(I, I p )+α||Θ|| (19)

[0150] I and I p Let represent the training data and the predicted actual label vectors, respectively. Θ represents all model parameters, and α is the weighting for regularization. The cross-entropy function measures the difference between the actual and predicted action labels, while regularization aims to prevent overfitting of the model parameters. The loss function is calculated relative to... The partial derivative of is expressed as:

[0151]

[0152] in express The element in the i-th row and j-th column. According to the chain rule, The calculation formula can be expressed as:

[0153]

[0154] After calculating Subsequently, the present invention uses the following rules to update the optimal adjacency matrix.

[0155]

[0156] In equation (21), ρ represents the learning rate of the network.

[0157] Step five, which involves establishing an information fusion model between source space motion-related EEG and EMG signals to complete the recognition of four motion patterns, includes:

[0158] (1) Training the model:

[0159] Electromyography (EMG) signals are processed through a two-layer one-dimensional convolutional network to extract action-related EMG features. Then, the one-dimensional convolutional output of the EMG signals and the output of the brain network model relating action-related EEG signals from the source space are fused through a fully connected layer before the classification result is output. Training this model using the backpropagation algorithm automatically extracts features from the EMG and EEG networks and automatically selects the most prominent features to classify four action attempts. This deep learning fusion model of brain-EMG uses the cross-entropy loss function as the cost function, has three fully connected layers, a four-node output layer, and the softmax activation function of the last fully connected layer, outputting the probability value P corresponding to the four action attempts. n ,0 <P n <1. The output with the highest probability value represents the action attempt type identified by the model. The optimizer used during network training is the Adaptive Moment Estimator (Adam), with a learning rate set to 0.001, beta1 to 0.9, and beta2 to 0.999. To prevent overfitting in the entire deep learning fusion model, a dropout layer is added to the network design, placed between the last two fully connected layers, with a dropout rate set to 0.5. Early stopping is also employed to improve the model's generalization ability; the training set is split twice into training and validation sets, and training stops if the validation set accuracy does not increase after 10 consecutive iterations.

[0160] (2) Model Evaluation:

[0161] This invention employs M-fold cross-validation to evaluate a proposed cross-modal hand movement assessment method. A total of K data points were collected for each subject for each movement. n Next, after each attempt, T vault seconds (T in the experiment) vaultData (2 seconds) is used as a time window, and one graph feature is extracted for each window. There are four actions, totaling 4×K. n 10 feature samples. Randomly select 10 features from each action. Samples, total One sample is used as the training set, and the remaining samples are used as the test set. The training set is divided into two parts: 80% is used to train the model, and 20% is used to validate the model to check for overfitting. This process is repeated M times, and the average recognition accuracy is calculated to evaluate the ability of the proposed new algorithm and existing algorithms to distinguish and recognize different action attempts (clenched fist, open fist, two-finger pinch, and three-finger pinch).

[0162] Figure 7 The accuracy rate is obtained by six-fold cross-validation on a thirteen-subject test set using a specific implementation method of the present invention.

Claims

1. A method for evaluating cross-modal hand movements, characterized in that, Includes the following steps: Step 1: Acquire multi-channel electromyographic (EMG) signals of the brain during motion execution and preprocess the EMG signals; Step 2: Extract action-related EEG signals from the preprocessed electromyography (EMG) signals using independent component analysis of EEG and cortico-muscle coherence analysis. Step 3: Brain power imaging, calculate the mapping of the EEG signals in the brain power space, and calculate the activity distribution of each brain region according to the Brodmann partition; Step 4: Establish a relationship model between source space action-related EEG and action patterns using a dynamic graph convolutional neural network, and extract EEG time series features and EEG location spatial features; Step 5: Based on the time series features and spatial features of EEG, establish an information fusion model between EEG signals and electromyographic signals related to spatial movement of brain power sources, and complete the recognition of four movement patterns; Step four, which involves establishing a relationship model between source space action-related EEG and action patterns using a dynamic graph convolutional neural network, and extracting EEG time-series features and EEG location-space features, includes the following: (1) The source distribution matrix of a single subject in the source space obtained in step three. A brain network graph (Graph) is constructed using a dynamic graph convolutional neural network; the brain network graph is an undirected, weighted graph with N nodes. c A graph with n vertices can be expressed by the following formula: Graph={V,E,W G } (7) In the formula, N c W represents the number of leads in the EEG. G For N c ×N c The adjacency matrix describes the connection between any two nodes in V, where W is the adjacency matrix in the i-th row and j-th column. G The term is denoted as w ij , representing the importance of the connection between the i-th node and the j-th node, and the final vertex set V contains N c Each vertex and its corresponding feature are represented as follows: For point, For each feature vector, there are as many elements as there are features; E is the set of edges, containing brain connectivity information and the weights of those connections, which are expressed through W. G And V to determine; (2) Dynamically learn the adjacency matrix W representing the relationships between each vertex node. G ,make This represents the optimal adjacency matrix to be learned, which is the source distribution matrix of a single subject in the source space. Separate the data, and extract a vector from each row. n c ∈{1,2,3,…,N c } represents motor-related EEG signals Y m,n,back The channel number mapped in the source space. Spatial filtering g(L) * ) Determined U * g(Λ * )U *T The graph convolution of a vector is represented as: In the above formula, L * L represents the Laplacian matrix of a brain network graph. * It can be represented as: It is a diagonal matrix, and the i-th diagonal element can be obtained from... Calculations show that express The element in the i-th row and j-th column; for a given Its graphical Fourier transform is represented as: U represents the vector of EEG-related signals after transformation in the frequency domain. * It is an orthogonal matrix, which can be obtained by the Laplacian operator matrix L of the singular value decomposition graph. * get: L * =U * ×Λ * ×U *T (11) U * The list Constructing Fourier bases, It is a diagonal matrix. From equation (10), the inverse representation of the graph Fourier transform is: In equation (8), g(·) is the filtering function. As can be seen from equation (11), after g(L) * Filtered signal Represented as And among them g(Λ) * ) is represented as: According to formulas (9)-(13), formula (8) is written as formula (14), where ⊙ represents the Hadamard product divided by elements: let Represents Λ * The largest element in the diagonal terms, and using Normalization Λ * , making The diagonal elements of lie in the interval [-1, 1], where I N It is N c ×N c The identity matrix, within the framework of the K-order Chebyshev polynomial, yields g(Λ * Approximately: In the above formula, θ k The coefficients of the Chebyshev polynomial are... The following recursive expressions (16) and (17) are used for recursive calculation: According to equation (15), the graph convolution operation defined in equation (8) can be rewritten as: Equation (18) represents the calculation The graph convolution can be represented as The convolution result is a combination of the components of the Chebyshev polynomial, where... Based on Equation (18), a model of the relationship between the brain network and action patterns of source space action-related EEG is established. The framework consists of four main layers: graph data filtering layer, convolutional layer, ReLU activation layer, and fully connected layer. Each channel in the source space is represented as a node in the model. Next is a 1×1 convolutional layer. The ReLU activation function is used to ensure that the output of the graph data filtering layer is non-negative. Finally, the output of the activation function is sent to the multi-layer fully connected network. The network parameters are iteratively updated using the backpropagation algorithm until an optimal or near-optimal solution is obtained. Specifically, a cross-entropy loss function is used, with the following form: LOSS=cross_entropy(I,I p )+α||Θ|| (19) I and I p Let Θ represent the training data and the predicted actual label vectors, respectively; let Θ represent all model parameters; and let α be the weight for the tradeoff regularization. Calculate the loss function relative to... The partial derivative of is expressed as: in express The element in the i-th row and j-th column; according to the chain rule, The calculation formula can be expressed as: After calculating Then, the optimal adjacency matrix is ​​updated using the following rules. (21) In the formula, ρ represents the learning rate of the network; Step five, which involves establishing an information fusion model between source space motion-related EEG and EMG signals to complete the recognition of four motion patterns, includes: (1) Training the model: Electromyography (EMG) signals are processed through a two-layer one-dimensional convolutional network to extract action-related EMG features. Then, the one-dimensional convolutional output of EMG and the output of the brain network and action pattern relationship model of action-related EEG in the source space are fused through a fully connected layer to output the classification result. The model is trained by backpropagation algorithm, which automatically extracts features of EMG and EEG networks and automatically selects the most prominent features to achieve classification of four action attempts. (2) Model Evaluation: Using the M-fold cross-validation method, a total of K data points were collected for each subject for each action. n Next, after each attempt, T vault The data in seconds is used as a time window, and one graph feature is extracted for each window. There are four actions in total, totaling 4×K. n 1 feature sample; randomly select from each action Samples, total One sample is used as the training set, and the remaining samples are used as the test set. The training set is divided into two parts: 80% is used to train the model and 20% is used to validate the model to see if overfitting occurs. The above operation is repeated M times, and the average recognition accuracy is finally calculated to evaluate the ability of the proposed new algorithm and existing algorithms to distinguish and recognize different actions.

2. The cross-modal hand movement evaluation method according to claim 1, characterized in that, In step one, during the acquisition of multi-channel motion execution electroencephalogram (EEG) signals, the acquired EEG signals are represented as follows: Where m represents the subject number, and Noc-guided EEG signals are collected for each subject, n represents the action category, and m is the Noc-guided EEG signal number collected in the clinical trial for the m-th stroke patient, and the n-th action for the m-th subject. Represents a real matrix. There is a Noc row and a t column, where t is the number of sampling points, and the sampling time is assumed to be t. t If the sampling frequency is F, then t = t t ×F, the acquired electromyographic signal is represented as This represents the electromyographic signal obtained from the nth action of the mth subject, measured using bipolar difference. Noc a This indicates the channel number of the acquired electromyography data, with the a-th channel number being Noc. a The two leads form a channel.

3. The cross-modal hand movement evaluation method according to claim 1, characterized in that, The preprocessing of the electroencephalogram (EEG) signals in step one includes filtering, baseline calibration, and removal of signals exhibiting significant EEG drift. The resulting EEG data is as follows: The electromyography data obtained after preprocessing are 4. The cross-modal hand movement evaluation method according to claim 1, characterized in that, The specific process in step two is as follows: (1) Preprocessed EEG data Y m,n Independent component analysis was performed on the Noc-guided EEG data Y collected from clinical trials of stroke patients. m,n It can be decomposed into a series of time-independent and spatially fixed components, as expressed by the following formula: In the formula, n is the number of independent components, Q i It is matrix Q Noc×n The element in the i-th column represents the spatial topographic map of the i-th independent component, T i It is matrix T n×t The elements in the i-th row represent the time series of the i-th independent component. W is a weight diagonal matrix, with weight values ​​existing only on the diagonal. i The weight represents the unique non-zero value in the i-th column vector of W. (2) Cortical-muscle coherence analysis was performed using electromyographic signals, and the Y-values ​​of each EEG data point were calculated. m Time series T of independent components i Compared with the multi-channel electromyography signals collected in the above clinical trials The formula for calculating the cortical-muscular coherence coefficient between the two layers is as follows: In the formula, f represents the frequency. Represents each EEG data point Y m,n Time series T of independent components i Compared with the multi-channel electromyography signals collected in the above clinical trials Cross-power spectral density between Represents each EEG data point Y m,n Time series T of independent components i The self-power spectral density, This refers to the multichannel electromyography signals collected in the above clinical trial. The self-power spectral density, This represents each calculated EEG data point Y. m,n Time series T of independent components i Compared with the multi-channel electromyography signals collected in the above clinical trials The cortical-muscle coherence coefficient between the two is used to obtain independent components that are closely related to action-related EEG. (3) Extraction of action-related EEG signals: The selected independent components are subjected to inverse independent component analysis to reconstruct the action-related EEG signals. The reconstruction formula is as follows: In the formula, Y m,n,back The reconstructed action-related EEG signals, n ok Q represents the number of independent components associated with brain electrical activity. i For this independent component, the corresponding spatial topographic map, T i For the time series corresponding to the independent components, the weight diagonal matrix W remains unchanged.

5. The cross-modal hand movement evaluation method according to claim 1, characterized in that, The specific process in step three is as follows: (1) Using the action-related EEG signals reconstructed in step two, the leading field is calculated using a real boundary cell head model based on magnetic resonance imaging. The mathematical model for generating scalp EEG signals is established as follows: Y m,n,back < A×X+noise (4) Y m,n,back The signal represents the action-related EEG signal obtained after step two; A represents the lead field, which is a zero-lead matrix that simulates the distribution of neurons in the cerebral cortex and the changes in their postsynaptic potentials; X represents the dipole moment of the cortical current dipole, which simulates the distribution of neurons in the cerebral cortex and the changes in their postsynaptic potentials; noise represents the measurement noise. (2) The inverse problem is solved by using the minimum norm estimation method; L2 regularization is used to further constrain the solution in order to find the solution X that minimizes both the residual and energy of the source space; this additional constraint is represented by the second term in equation (5); the regularization parameter λ is obtained by using the L-curve method to balance the influence of these two terms. In the case of L2 norm, the analytical solution of equation (5) yields the optimal estimate of the source distribution X, as shown in equation (6). When W is only the identity matrix, this solution Known as the minimum norm estimate, it is the mapping of EEG signals in the brain's energy space. A T This represents the flip matrix of the leading field.

6. The cross-modal hand movement evaluation method according to claim 1, characterized in that, In step three, the number of dipoles is categorized according to Brodmann partitions. The measurements of dipoles within a region at the same time are summed and considered as the overall output of that brain region, thus obtaining the source distribution matrix for a single subject. Where n∈{1,2,3,4} represents the category label of the motion task, and d={1,2,3,…,N m }, N m N represents the number of times the experiment was conducted for the m-th action; c Y represents motor-related EEG signals m,n,back The number of channels mapped in the source space; N s This represents the number of sampling points.

Images