Upper limb rehabilitation training method and device based on brain imaging and electroencephalogram fusion

Abstract

The invention discloses an upper limb rehabilitation training method and device based on brain imaging and electroencephalogram fusion. The device comprises a signal acquisition module, a signal processing module and a training control module. The signal acquisition module is used for acquiring a brain signal set of a user in an upper limb rehabilitation training process; the brain signal set comprises a brain electrical signal set and a near-infrared brain image information set; the signal processing module is respectively connected with the signal acquisition module and the training control module and is used for processing the brain signal set to obtain a brain characteristic signal set; and the training control module is used for performing control calculation on the brain characteristic signal set to obtain training intensity control information.

Description

Technical Field

[0001] The present invention relates to the field of medical rehabilitation technology, and in particular to an upper limb rehabilitation training method and device based on brain imaging and electroencephalography fusion. Background Art

[0002] With the aging of society and the increasing incidence of various neurological diseases and trauma, the number of patients with upper limb dysfunction is increasing. Upper limb function is essential for daily life activities, such as eating, dressing, writing, etc. At present, there are many upper limb rehabilitation training methods, including traditional physical therapy, occupational therapy, etc., but these methods have certain limitations.

[0003] Traditional rehabilitation training lacks accurate monitoring and feedback of brain neural activity. Electroencephalogram (EEG) can reflect the electrical activity of neurons in the cerebral cortex and can be used to detect the functional state of the brain, but its spatial resolution is low. Near-infrared brain imaging technology (fNIRS) can reflect local hemodynamic changes in the brain by detecting changes in hemoglobin concentration in the cerebral cortex. It has a high spatial resolution, but its temporal resolution is relatively low.

[0004] Using EEG signals or near-infrared brain imaging technology alone cannot fully and accurately obtain the brain's neural activity information during rehabilitation training, making it difficult to achieve personalized and precise upper limb rehabilitation training, resulting in unsatisfactory rehabilitation effects and a long rehabilitation cycle. Summary of the invention

[0005] The present invention mainly solves the problem of how to comprehensively and accurately obtain the neural activity information of the brain during rehabilitation training to achieve personalized and precise upper limb rehabilitation training. The present invention discloses an upper limb rehabilitation training method and device based on brain imaging and electroencephalography fusion.

[0006] In a first aspect of the embodiment of the present application, an upper limb rehabilitation training device based on brain imaging and EEG fusion is disclosed, comprising: a signal acquisition module, a signal processing module, and a training control module;

[0007] The signal acquisition module is used to acquire a set of brain signals of the user during upper limb rehabilitation training; the set of brain signals includes an electroencephalogram signal set and a near-infrared brain image information set;

[0008] The signal processing module is connected to the signal acquisition module and the training control module respectively, and is used to process the brain signal set to obtain a brain feature signal set;

[0009] The training control module is used to perform control calculations on the brain characteristic signal set to obtain training intensity control information.

[0010] The signal acquisition module includes an electroencephalogram signal acquisition unit and a near-infrared brain imaging signal acquisition unit;

[0011] The EEG signal acquisition unit is used to acquire a set of EEG signals of the user during upper limb rehabilitation training;

[0012] The near-infrared brain imaging signal acquisition unit is used to acquire a near-infrared brain image information set of the user during the upper limb rehabilitation training.

[0013] The signal processing module is used to process the brain signal set to obtain a brain characteristic signal set, including:

[0014] The signal processing module is used to preprocess the brain signal set to obtain a preprocessed signal set;

[0015] Feature extraction is performed on the preprocessed signal set to obtain a brain feature signal set.

[0016] In a second aspect of the embodiment of the present invention, a method for upper limb rehabilitation training based on brain imaging and electroencephalography fusion is disclosed, which is implemented by using the upper limb rehabilitation training device based on brain imaging and electroencephalography fusion, and includes:

[0017] S1, using the signal acquisition module to acquire a set of brain signals of a user during upper limb rehabilitation training; the set of brain signals includes a set of electroencephalogram signals and a set of near-infrared brain image information;

[0018] S2, using the signal processing module to process the brain signal set to obtain a brain feature signal set;

[0019] S3, using the training control module to perform control calculation on the brain characteristic signal set to obtain training intensity control information.

[0020] The using the signal processing module to process the brain signal set to obtain a brain characteristic signal set includes:

[0021] S21, using the signal processing module to preprocess the brain signal set to obtain a preprocessed signal set;

[0022] S22, performing feature extraction on the preprocessed signal set to obtain a brain feature signal set.

[0023] The preprocessing of the brain signal set to obtain a preprocessed signal set includes:

[0024] S211, performing data cleaning processing on the preprocessed signal set to obtain a first data set;

[0025] S212, performing category detection processing on the first data set to obtain a second data set;

[0026] S213, performing time registration processing on the second data set to obtain a third data set;

[0027] S214, performing a pattern check on the third data set to obtain a preprocessing signal set.

[0028] The performing a pattern check on the third data set to obtain a preprocessed signal set includes:

[0029] S2141, for each data attribute of the third data set, taking the data collection information of the data as an independent variable and the data value of the data as a dependent variable, performing cluster analysis processing to obtain clustering result information of the data attribute of the class; the clustering result information includes the cluster category to which all data of the data attribute of the class belongs;

[0030] S2142, determining the number of data included in each cluster category; setting a data volume threshold; deleting the data included in the cluster category whose number of data is less than the data volume threshold from the third data set;

[0031] S2143, execute S2141 and S2142 for all class data attributes of the third data set to obtain a preprocessed signal set.

[0032] The step of extracting features from the preprocessed signal set to obtain a brain feature signal set includes:

[0033] Performing EEG feature analysis on the EEG signal set in the preprocessed signal set to obtain an EEG feature value sequence;

[0034] Using the trained feature extraction network, the near-infrared brain image information set is processed to obtain image feature values;

[0035] A brain characteristic signal set is constructed using the EEG characteristic value sequence and the image characteristic value.

[0036] The performing EEG feature analysis processing on the EEG signal set in the preprocessed signal set to obtain an EEG feature value sequence includes:

[0037] The EEG signal set in the preprocessed signal set is represented as a signal matrix; the row vector in the signal matrix is ​​the EEG signal sequence in the EEG signal set;

[0038] Performing principal component analysis on the signal matrix to obtain a coefficient matrix and a principal component matrix;

[0039] Performing cross-correlation calculation on the signal matrix to obtain a cross-correlation matrix;

[0040] The mutual correlation matrix and the principal component matrix are fused and calculated to obtain a fused feature matrix; the expression of the fusion calculation process is:

[0041]

[0042] Among them, M is the fusion feature matrix, S XX is the cross-correlation matrix, S XY is the principal component matrix;

[0043] Performing singular value calculation on the fused feature matrix to obtain a singular value set;

[0044] The singular value set is determined to be an EEG eigenvalue sequence.

[0045] The feature extraction network includes: a first processing unit, a second processing unit and a fully connected unit;

[0046] The output end of the first processing unit is connected to the input end of the second processing unit; the output end of the second processing unit is connected to the input end of the fully connected layer;

[0047] The input of the first processing unit is a near-infrared brain image sequence;

[0048] The first processing unit includes a convolutional layer, a pooling layer and a fully connected layer;

[0049] The convolution layer uses a two-dimensional convolution kernel w∈R 3×3 Feature extraction is performed on the near-infrared brain image sequence to obtain a feature matrix C n ;

[0050] The pooling layer uses the maximum pooling method to extract features. The calculation expression of the maximum pooling method is:

[0051] p u =Max 2×2 [C n ],

[0052] Among them, u represents the number of pooling times, Max 2×2 represents the maximum pooling operation method of a 2×2 matrix, p u is the extracted features;

[0053] The fully connected layer, for the feature p u Performing dimension transformation processing to obtain a variable that matches the input dimension of the second processing unit; the output end of the fully connected layer is connected to the input end of the second processing unit;

[0054] The second processing unit is used to perform time series attribute extraction processing on the input data to obtain an output value, and input the output value into the fully connected unit;

[0055] The fully connected unit is used to calculate the output value of the second processing unit to obtain a picture feature value.

[0056] The beneficial effects of the present invention are:

[0057] Improve the accuracy of rehabilitation training: By synchronously collecting and analyzing near-infrared brain imaging signals and EEG signals, and making full use of their complementary advantages in spatial resolution and temporal resolution, it is possible to more comprehensively and accurately obtain information on the brain's neural activity during rehabilitation training, thereby providing more precise guidance for upper limb rehabilitation training and realizing the formulation and adjustment of personalized rehabilitation training programs.

[0058] Enhance the effect of rehabilitation training: Adjust the rehabilitation training plan in real time according to the brain's neural activity, so that the training tasks match the functional state of the patient's brain. This can effectively stimulate the plasticity of the brain, promote the recovery of neural function, improve the effect of upper limb rehabilitation training, and shorten the rehabilitation cycle.

[0059] Improve patients' enthusiasm for training: The real-time feedback and incentive mechanism provided by the system can enable patients to intuitively understand their training progress and the response of their brain, enhance patients' sense of participation and confidence in rehabilitation training, and improve patients' enthusiasm and initiative in training. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 1 is a flow chart for implementing the method of the present invention;

[0061] Figure 2 It is a block diagram of the composition of the device of the present invention. DETAILED DESCRIPTION

[0062] In order to better understand the content of the present invention, an embodiment is given here.

[0063] Figure 1 It is a flow chart for implementing the method of the present invention. Figure 2 It is a block diagram of the composition of the device of the present invention.

[0064] In a first aspect of the embodiment of the present application, an upper limb rehabilitation training device based on brain imaging and EEG fusion is disclosed, comprising: a signal acquisition module, a signal processing module, and a training control module;

[0065] The signal acquisition module is used to acquire a set of brain signals of the user during upper limb rehabilitation training; the set of brain signals includes an electroencephalogram signal set and a near-infrared brain image information set;

[0066] The signal processing module is connected to the signal acquisition module and the training control module respectively, and is used to process the brain signal set to obtain a brain feature signal set;

[0067] The training control module is used to perform control calculations on the brain characteristic signal set to obtain training intensity control information.

[0068] The signal acquisition module includes an electroencephalogram signal acquisition unit and a near-infrared brain imaging signal acquisition unit;

[0069] The EEG signal acquisition unit is used to acquire a set of EEG signals of the user during upper limb rehabilitation training;

[0070] The near-infrared brain imaging signal acquisition unit is used to acquire a near-infrared brain image information set of the user during the upper limb rehabilitation training;

[0071] The signal processing module is used to process the brain signal set to obtain a brain characteristic signal set, including:

[0072] The signal processing module is used to preprocess the brain signal set to obtain a preprocessed signal set;

[0073] Feature extraction is performed on the preprocessed signal set to obtain a brain feature signal set.

[0074] The preprocessing of the brain signal set to obtain a preprocessed signal set includes:

[0075] S221, performing data cleaning processing on the preprocessed signal set to obtain a first data set;

[0076] S222, performing category detection processing on the first data set to obtain a second data set;

[0077] S223, performing time registration processing on the second data set to obtain a third data set;

[0078] S224, performing a pattern check on the third data set to obtain a preprocessing signal set.

[0079] The data cleaning process includes: filling missing values, smoothing noise data, smoothing or deleting outliers; the smoothed noise data is firstly determined to obtain noise data, and then the noise data is smoothed according to the data before and after the noise data; the noise data is a value whose value is less than the detection sensitivity of the sensor of the observed data, or greater than the measurement upper limit of the sensor of the observed data. The outlier point can be determined by the Kalman filter method. The filling value of the missing value can be determined by averaging the measured values ​​within a certain sampling interval before and after the missing value.

[0080] The time registration process is to unify different types of data to the same time reference; the time registration process can adopt the internal push / extrapolation method, the Lagrange three-point interpolation method, etc.;

[0081] The category detection process is to detect whether the data category of each type of data in the first data set is consistent with the preset category, and delete the inconsistent data from the first data set.

[0082] The performing a pattern check on the third data set to obtain a preprocessed signal set includes:

[0083] S2241, for each data attribute of the third data set, taking the data collection information of the data as an independent variable and the data value of the data as a dependent variable, performing cluster analysis processing to obtain clustering result information of the data attribute of each class; the clustering result information includes the cluster category to which all data of the data attribute of the class belongs;

[0084] S2242, determining the number of data included in each cluster category; setting a data volume threshold; deleting the data included in the cluster category whose number of data is less than the data volume threshold from the third data set;

[0085] S2243, executing S2241 and S2242 for all data of the third data set to obtain a preprocessed signal set;

[0086] The data collection information includes time information and space information corresponding to the data collection; when performing data fitting, any of the above information can be used as an independent variable, and the corresponding data value can be used as a dependent variable.

[0087] The step of extracting features from the preprocessed signal set to obtain a brain feature signal set includes:

[0088] Performing EEG feature analysis on the EEG signal set in the preprocessed signal set to obtain an EEG feature value sequence;

[0089] Using the trained feature extraction network, the near-infrared brain image information set is processed to obtain image feature values;

[0090] A brain characteristic signal set is constructed using the EEG characteristic value sequence and the image characteristic value.

[0091] The near-infrared brain image information set includes a near-infrared brain image sequence;

[0092] The performing EEG feature analysis processing on the EEG signal set in the preprocessed signal set to obtain an EEG feature value sequence includes:

[0093] The EEG signal set in the preprocessed signal set is represented as a signal matrix; the row vector in the signal matrix is ​​the EEG signal sequence in the EEG signal set;

[0094] Performing principal component analysis on the signal matrix to obtain a coefficient matrix and a principal component matrix;

[0095] Performing cross-correlation calculation on the signal matrix to obtain a cross-correlation matrix; the elements of the i-th row and j-th column of the cross-correlation matrix are cross-correlation values ​​of the i-th row vector and the j-th row vector of the signal matrix;

[0096] The mutual correlation matrix and the principal component matrix are fused and calculated to obtain a fused feature matrix; the expression of the fusion calculation process is:

[0097]

[0098] Among them, M is the fusion feature matrix, S XX is the cross-correlation matrix, S XY is the principal component matrix;

[0099] Performing singular value calculation on the fused feature matrix to obtain a singular value set;

[0100] Determine the singular value set as an EEG eigenvalue sequence;

[0101] Determining the singular value set as an EEG eigenvalue sequence includes arranging the elements in the singular value set from large to small according to their values ​​to obtain the EEG eigenvalue sequence.

[0102] The expression of the principal component analysis process is:

[0103] Y=CX,

[0104] Among them, Y is the principal component matrix, C is the coefficient matrix, and X is the signal matrix. The principal component analysis process can be implemented by the PCA algorithm, and the principal component matrix and the coefficient matrix are both determined by the principal component analysis process;

[0105] The feature extraction network includes: a first processing unit, a second processing unit and a fully connected unit;

[0106] The output end of the first processing unit is connected to the input end of the second processing unit; the output end of the second processing unit is connected to the input end of the fully connected layer;

[0107] The input of the first processing unit is a near-infrared brain image sequence;

[0108] The first processing unit includes a convolutional layer, a pooling layer and a fully connected layer;

[0109] The convolution layer uses a two-dimensional convolution kernel w∈R 3×3 Feature extraction is performed on the near-infrared brain image sequence to obtain a feature matrix C n ;

[0110] The pooling layer uses the maximum pooling method to extract features. The calculation expression of the maximum pooling method is:

[0111] p u =Max 2×2 [C n ],

[0112] Among them, u represents the number of pooling times, Max 2×2 represents the maximum pooling operation method of a 2×2 matrix, p u is the extracted features;

[0113] The fully connected layer, for the feature p u Performing dimension transformation processing to obtain a variable that matches the input dimension of the second processing unit; the output end of the fully connected layer is connected to the input end of the second processing unit;

[0114] The second processing unit is used to perform time series attribute extraction processing on the input data to obtain an output value, and input the output value into the fully connected unit;

[0115] The fully connected unit is used to calculate the output value of the second processing unit to obtain a picture feature value;

[0116] The convolution layer is a structure in which 13 layers of convolution modules are cascaded; the maximum pooling layer is a structure in which 5 layers of maximum pooling modules are cascaded; the fully connected layer is a structure in which 3 layers of fully connected modules are cascaded; the convolution layer is connected to the pooling layer, and the pooling layer is connected to the fully connected layer;

[0117] The calculation expression of the convolutional layer is:

[0118]

[0119] Among them, n represents the number of convolution operations, m represents the number of convolution kernels, and p i represents the i-th feature matrix obtained, f represents the nonlinear activation function, · represents the corresponding operation of the shared weight of the convolution kernel and the feature matrix, w represents the weight of the convolution kernel, b represents the bias value, R 3×3 represents a 3×3 real matrix;

[0120] The dimensional transformation process can be implemented by using the Reshape function.

[0121] The fully connected unit can be implemented using the FC layer.

[0122] The processing process of the second processing unit is expressed as:

[0123] o t =g(V st +V s ' T′ ),

[0124]

[0125] Among them, s t represents the output of the positive sequential input of the second processing unit at time t, s' t represents the output of the reverse sequential input of the second processing unit at time t, U Xt Represents the initial input of the second processing unit in the forward direction, U′ Xt represents the initial input quantity of the reverse sequence of the second processing unit, Indicates the positive sequential input quantity of the second processing unit at the previous moment, is the reverse time series input quantity at the next moment, o t represents the standard output of the second processing unit at time t, where g and f are the activation function and sigmoid function respectively.

[0126] The output of the second processing unit includes s' t 、s t , o t .

[0127] The input of the forward time series is the extracted feature p u ;

[0128] The input of the reverse time series is the extracted feature p u The reverse arrangement of

[0129] The training process of the feature extraction network includes:

[0130] Initialize the number of training iterations;

[0131] Input the training data in the training data set as input data into the feature extraction network;

[0132] Using the feature extraction network, the input data is processed to obtain a predicted value;

[0133] Performing difference calculation processing on the obtained predicted value and the label information corresponding to the input data to obtain a difference value;

[0134] Determine whether the difference value satisfies a convergence condition, and obtain a first determination result;

[0135] When the first judgment result is no, judging whether the training iteration number value is equal to the training number threshold, and obtaining a second judgment result;

[0136] When the second judgment result is no, determining that the model training state does not meet the training termination condition;

[0137] When the second judgment result is yes, determining that the model training state satisfies the training termination condition;

[0138] When the first judgment result is yes, determining that the model training state satisfies a training termination condition;

[0139] When the model training state does not meet the termination training condition, the parameter update model is used to update the parameters of the feature extraction network, the training iteration number value is increased by 1, and the execution is triggered to input the training data in the training data set as input data into the feature extraction network;

[0140] When the model training state satisfies the termination training condition, the training process of the feature extraction network is completed to obtain a trained feature extraction network.

[0141] The difference value satisfies the convergence condition, which means that the difference value is less than a preset convergence threshold; the difference value does not satisfy the convergence condition, which means that the difference value is not less than the preset convergence threshold.

[0142] The difference calculation process may be implemented using a loss function.

[0143] The loss function may adopt a cross entropy loss function.

[0144] The parameter updating model is:

[0145]

[0146] θ←θ+v;

[0147] In the formula, is the difference value calculated for the i-th training data in the training data set, v is the parameter update value, θ is the parameter of the feature extraction, η is the initial parameter learning rate, α is the momentum angle parameter, 0≤α≤π / 4, It means to find partial derivatives with respect to variable θ;

[0148] The control calculation includes:

[0149] Using the EEG characteristic value sequences at all moments in the brain characteristic signal set, an EEG characteristic matrix is ​​constructed; a row vector of the EEG characteristic matrix is ​​a sequence of EEG characteristic values ​​at a moment;

[0150] Using the image feature values ​​at all moments in the brain feature signal set, constructing a brain feature vector;

[0151] The EEG feature matrix and the brain feature vector are calculated to obtain training intensity control information; the calculation expression of the training intensity control information is:

[0152] cg=(PTV+aI) -1 P T t,

[0153] V=QR -1 ,

[0154]

[0155] Among them, cg i is the i-th item of the eigenvector, cg is the eigenvector, V is the eigenmatrix, P is the EEG feature matrix, Q and R are the Q matrix and R matrix obtained by QR decomposition of the EEG feature matrix P, a is the maximum eigenvalue of the EEG feature matrix P, M is the number of elements contained in the eigenvector, t is the brain feature vector, I represents the unit matrix, and F is the training intensity control information.

[0156] The EEG signal acquisition unit includes multiple electrodes placed according to the international 10-20 system standard, which are used to collect EEG signals on the surface of the scalp. The electrodes are connected to the EEG amplifier through lead wires, and the amplifier performs pre-processing such as amplification and filtering on the collected weak EEG signals to meet the requirements of subsequent signal processing.

[0157] The near-infrared brain imaging signal acquisition unit is composed of multiple near-infrared light sources and detectors. The light source emits near-infrared light of a specific wavelength (such as 700-900nm), and the detector receives the near-infrared light signal after being scattered by brain tissue to obtain a near-infrared brain image information set. The light source and the detector are distributed on the head-mounted device according to a certain geometric layout to ensure that the brain movement-related areas, such as the primary motor cortex, supplementary motor area, etc., can be covered.

[0158] The signal processing module: uses a high-performance microprocessor or digital signal processor (DSP). This module receives the pre-processed EEG signals and near-infrared brain imaging signals from the signal acquisition module, and further extracts and analyzes the signals. For EEG signals, the power spectrum features of different frequency bands such as α waves, β waves, and γ waves, event-related potentials (ERP) and other features are extracted; for near-infrared brain imaging signals, the changes in hemoglobin concentrations in different brain regions are calculated, and the location, intensity and other features of the activated regions are extracted.

[0159] The training control module generates corresponding training control instructions based on the brain nerve activity characteristics obtained by the signal processing module, combined with the patient's rehabilitation training stage and the preset rehabilitation training program. For example, when it is detected that the activation level of the brain movement-related area is low, the exercise difficulty of the rehabilitation training equipment is automatically reduced or the training assistance strength is increased; when the brain nerve activity characteristics show that the patient adapts well to the current training task, the training difficulty is appropriately increased.

[0160] The upper limb rehabilitation training device based on brain imaging and EEG fusion also includes a rehabilitation training execution module, which includes upper limb rehabilitation training equipment, such as an intelligent rehabilitation robot, a wearable rehabilitation training device, etc. The module receives the training intensity control information sent by the training control module, and executes corresponding upper limb rehabilitation training actions, such as arm flexion and extension, rotation, grasping, etc., to provide personalized rehabilitation training for patients, and the torque value during the execution of the action is proportional to the training intensity control information.

[0161] In a second aspect of the embodiment of the present application, a method for upper limb rehabilitation training based on brain imaging and electroencephalography fusion is disclosed, which is implemented by using the upper limb rehabilitation training device based on brain imaging and electroencephalography fusion, including:

[0162] S1, using the signal acquisition module to acquire a set of brain signals of a user during upper limb rehabilitation training; the set of brain signals includes a set of electroencephalogram signals and a set of near-infrared brain image information;

[0163] S2, using the signal processing module to process the brain signal set to obtain a brain feature signal set;

[0164] S3, using the training control module to perform control calculation on the brain characteristic signal set to obtain training intensity control information.

[0165] The using the signal processing module to process the brain signal set to obtain a brain characteristic signal set includes:

[0166] S21, using the signal processing module to preprocess the brain signal set to obtain a preprocessed signal set;

[0167] S22, performing feature extraction on the preprocessed signal set to obtain a brain feature signal set.

[0168] The preprocessing of the brain signal set to obtain a preprocessed signal set includes:

[0169] S211, performing data cleaning processing on the preprocessed signal set to obtain a first data set;

[0170] S212, performing category detection processing on the first data set to obtain a second data set;

[0171] S213, performing time registration processing on the second data set to obtain a third data set;

[0172] S214, performing a pattern check on the third data set to obtain a preprocessing signal set.

[0173] The performing a pattern check on the third data set to obtain a preprocessed signal set includes:

[0174] S2141, for each data attribute of the third data set, taking the data collection information of the data as an independent variable and the data value of the data as a dependent variable, performing cluster analysis processing to obtain clustering result information of the data attribute of the class; the clustering result information includes the cluster category to which all data of the data attribute of the class belongs;

[0175] S2142, determining the number of data included in each cluster category; setting a data volume threshold; deleting the data included in the cluster category whose number of data is less than the data volume threshold from the third data set;

[0176] S2143, executing S2141 and S2142 for all data of the third data set to obtain a preprocessed signal set;

[0177] The data collection information includes time information and space information corresponding to the data collection; when performing data fitting, any of the above information can be used as an independent variable, and the corresponding data value can be used as a dependent variable.

[0178] The step of extracting features from the preprocessed signal set to obtain a brain feature signal set includes:

[0179] Performing EEG feature analysis on the EEG signal set in the preprocessed signal set to obtain an EEG feature value sequence;

[0180] Using the trained feature extraction network, the near-infrared brain image information set is processed to obtain image feature values;

[0181] A brain characteristic signal set is constructed using the EEG characteristic value sequence and the image characteristic value.

[0182] The near-infrared brain image information set includes a near-infrared brain image sequence;

[0183] The performing EEG feature analysis processing on the EEG signal set in the preprocessed signal set to obtain an EEG feature value sequence includes:

[0184] The EEG signal set is represented as a signal matrix; the row vector in the signal matrix is ​​the EEG signal sequence in the EEG signal set;

[0185] Performing principal component analysis on the signal matrix to obtain a coefficient matrix and a principal component matrix;

[0186] Performing cross-correlation calculation on the signal matrix to obtain a cross-correlation matrix; the elements of the i-th row and j-th column of the cross-correlation matrix are cross-correlation values ​​of the i-th row vector and the j-th row vector of the signal matrix;

[0187] The mutual correlation matrix and the principal component matrix are fused and calculated to obtain a fused feature matrix; the expression of the fusion calculation process is:

[0188]

[0189] Among them, M is the fusion feature matrix, S XX is the cross-correlation matrix, S XY is the principal component matrix;

[0190] Performing singular value calculation on the fused feature matrix to obtain a singular value set;

[0191] Determine the singular value set as an EEG eigenvalue sequence;

[0192] The expression of the principal component analysis process is:

[0193] Y=CX,

[0194] Among them, Y is the principal component matrix, C is the coefficient matrix, and X is the signal matrix. The principal component analysis process can be implemented by the PCA algorithm, and the principal component matrix and the coefficient matrix are both determined by the principal component analysis process;

[0195] The feature extraction network includes: a first processing unit, a second processing unit and a fully connected unit;

[0196] The output end of the first processing unit is connected to the input end of the second processing unit; the output end of the second processing unit is connected to the input end of the fully connected layer;

[0197] The input of the first processing unit is a near-infrared brain image sequence;

[0198] The first processing unit includes a convolutional layer, a pooling layer and a fully connected layer;

[0199] The convolution layer uses a two-dimensional convolution kernel w∈R 3×3 Feature extraction is performed on the near-infrared brain image sequence to obtain a feature matrix C n ;

[0200] The pooling layer uses the maximum pooling method to extract features. The calculation expression of the maximum pooling method is:

[0201] p u =Max 2×2 [C n ],

[0202] Among them, u represents the number of pooling times, Max 2×2 represents the maximum pooling operation method of a 2×2 matrix, p u is the extracted features;

[0203] The fully connected layer, for the feature p u Performing dimension transformation processing to obtain a variable that matches the input dimension of the second processing unit; the output end of the fully connected layer is connected to the input end of the second processing unit;

[0204] The second processing unit is used to perform time series attribute extraction processing on the input data to obtain an output value, and input the output value into the fully connected unit;

[0205] The fully connected unit is used to calculate the output value of the second processing unit to obtain a picture feature value;

[0206] The control calculation includes:

[0207] Using the EEG characteristic value sequences at all moments in the brain characteristic signal set, an EEG characteristic matrix is ​​constructed; a row vector of the EEG characteristic matrix is ​​a sequence of EEG characteristic values ​​at a moment;

[0208] Using the image feature values ​​at all moments in the brain feature signal set, constructing a brain feature vector;

[0209] The EEG feature matrix and the brain feature vector are calculated to obtain training intensity control information; the calculation expression of the training intensity control information is:

[0210] cg=(P T V+aI) -1 P T t,

[0211] V=QR -1 ,

[0212]

[0213] Among them, cg iis the i-th item of the eigenvector, cg is the eigenvector, V is the eigenmatrix, P is the EEG feature matrix, Q and R are the Q matrix and R matrix obtained by QR decomposition of the EEG feature matrix P, a is the maximum eigenvalue of the EEG feature matrix P, M is the number of elements contained in the eigenvector, t is the brain feature vector, I represents the unit matrix, and F is the training intensity control information.

[0214] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An upper limb rehabilitation training device based on brain imaging and EEG fusion, characterized in that: include: Signal acquisition module, signal processing module, training control module; The signal acquisition module is used to acquire a set of brain signals of the user during upper limb rehabilitation training; The brain signal set includes an electroencephalogram signal set and a near-infrared brain image information set; The signal processing module is connected to the signal acquisition module and the training control module respectively, and is used to process the brain signal set to obtain a brain feature signal set; The training control module is used to perform control calculations on the brain characteristic signal set to obtain training intensity control information.

2. The upper limb rehabilitation training device based on brain imaging and EEG fusion as claimed in claim 1, characterized in that: The signal acquisition module includes an electroencephalogram signal acquisition unit and a near-infrared brain imaging signal acquisition unit; The EEG signal acquisition unit is used to acquire a set of EEG signals of the user during upper limb rehabilitation training; The near-infrared brain imaging signal acquisition unit is used to acquire a near-infrared brain image information set of the user during the upper limb rehabilitation training.

3. The upper limb rehabilitation training device based on brain imaging and EEG fusion as claimed in claim 2, characterized in that: The signal processing module is used to process the brain signal set to obtain a brain characteristic signal set, including: The signal processing module is used to preprocess the brain signal set to obtain a preprocessed signal set; Feature extraction is performed on the preprocessed signal set to obtain a brain feature signal set.

4. An upper limb rehabilitation training method based on brain imaging and EEG fusion, characterized in that: The method is implemented by using the upper limb rehabilitation training device based on brain imaging and brain-electroencephalography fusion according to any one of claims 1 to 3, comprising: S1, using the signal acquisition module to acquire a set of brain signals of a user during upper limb rehabilitation training; the set of brain signals includes a set of electroencephalogram signals and a set of near-infrared brain image information; S2, using the signal processing module to process the brain signal set to obtain a brain feature signal set; S3, using the training control module to perform control calculation on the brain characteristic signal set to obtain training intensity control information.

5. The upper limb rehabilitation training method based on brain imaging and EEG fusion as claimed in claim 4, characterized in that: The using the signal processing module to process the brain signal set to obtain a brain characteristic signal set includes: S21, using the signal processing module to preprocess the brain signal set to obtain a preprocessed signal set; S22, performing feature extraction on the preprocessed signal set to obtain a brain feature signal set.

6. The upper limb rehabilitation training method based on brain imaging and EEG fusion as claimed in claim 5, characterized in that: The preprocessing of the brain signal set to obtain a preprocessed signal set includes: S211, performing data cleaning processing on the preprocessed signal set to obtain a first data set; S212, performing category detection processing on the first data set to obtain a second data set; S213, performing time registration processing on the second data set to obtain a third data set; S214, performing a pattern check on the third data set to obtain a preprocessing signal set.

7. The upper limb rehabilitation training method based on brain imaging and EEG fusion as claimed in claim 6, characterized in that: The performing a pattern check on the third data set to obtain a preprocessed signal set comprises: S2141, for each data attribute of the third data set, taking the data collection information of the data as an independent variable and the data value of the data as a dependent variable, performing cluster analysis processing to obtain clustering result information of the data attribute of the class; the clustering result information includes the cluster category to which all data of the data attribute of the class belongs; S2142, determining the number of data included in each cluster category; setting a data volume threshold; deleting the data included in the cluster category whose number of data is less than the data volume threshold from the third data set; S2143, execute S2141 and S2142 for all class data attributes of the third data set to obtain a preprocessed signal set.

8. The upper limb rehabilitation training method based on brain imaging and EEG fusion as claimed in claim 7, characterized in that: The step of extracting features from the preprocessed signal set to obtain a brain feature signal set includes: Performing EEG feature analysis on the EEG signal set in the preprocessed signal set to obtain an EEG feature value sequence; Using the trained feature extraction network, the near-infrared brain image information set is processed to obtain image feature values; A brain characteristic signal set is constructed using the EEG characteristic value sequence and the image characteristic value.

9. The upper limb rehabilitation training method based on brain imaging and EEG fusion as claimed in claim 8, characterized in that: The performing EEG feature analysis processing on the EEG signal set in the preprocessed signal set to obtain an EEG feature value sequence includes: The EEG signal set in the preprocessed signal set is represented as a signal matrix; the row vector in the signal matrix is ​​the EEG signal sequence in the EEG signal set; Performing principal component analysis on the signal matrix to obtain a coefficient matrix and a principal component matrix; Performing cross-correlation calculation on the signal matrix to obtain a cross-correlation matrix; The mutual correlation matrix and the principal component matrix are fused and calculated to obtain a fused feature matrix; the expression of the fusion calculation process is: Among them, M is the fusion feature matrix, S XX is the cross-correlation matrix, S XY is the principal component matrix; Performing singular value calculation on the fused feature matrix to obtain a singular value set; The singular value set is determined to be an EEG eigenvalue sequence.

10. The upper limb rehabilitation training method based on brain imaging and EEG fusion as claimed in claim 8, characterized in that: The feature extraction network comprises: a first processing unit, a second processing unit and a fully connected unit; The output end of the first processing unit is connected to the input end of the second processing unit; the output end of the second processing unit is connected to the input end of the fully connected layer; The input of the first processing unit is a near-infrared brain image sequence; The first processing unit includes a convolutional layer, a pooling layer and a fully connected layer; The convolution layer uses a two-dimensional convolution kernel w∈R 3×3 Feature extraction is performed on the near-infrared brain image sequence to obtain a feature matrix C n ; The pooling layer uses the maximum pooling method to extract features. The calculation expression of the maximum pooling method is: p u =Max 2×2 [C n ], Among them, u represents the number of pooling times, Max 2×2 represents the maximum pooling operation method of a 2×2 matrix, p u is the extracted features; The fully connected layer, for the feature p u Performing dimension transformation processing to obtain a variable that matches the input dimension of the second processing unit; the output end of the fully connected layer is connected to the input end of the second processing unit; The second processing unit is used to perform time series attribute extraction processing on the input data to obtain an output value, and input the output value into the fully connected unit; The fully connected unit is used to calculate the output value of the second processing unit to obtain a picture feature value.

Images