Multi-model fusion motor imagery intention recognition system

Abstract

The invention discloses a multi-model fusion motor imagery intention recognition system which comprises a training fusion strategy module and an application fusion strategy module. The training fusion strategy module obtains an optimal model-parameter group and the output weight of each model-parameter pair in the optimal model-parameter group according to the offline training data set, wherein the model-parameter pair is a combination of a feature extraction method, a classifier and corresponding parameters; the application fusion strategy module is used for training each model-parameter pair in the optimized model-parameter group by utilizing an offline clinical data set based on the acquired offline clinical data set of the user and real-time clinical online data, inputting the clinical online data into the trained optimized model-parameter group, and outputting the clinical online data into the optimized model-parameter group; and based on the output weight of each model-parameter pair, carrying out weighted fusion on the output of all the model-parameter pairs to obtain a motor imagery intention recognition result. According to the method, the motor imagery intention recognition precision can be improved without complicated hyper-parameter optimization under the conditions of a small data set and finite time.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of brain-computer interface, and particularly relates to a multi-model fusion motor imagery intention recognition system. BACKGROUND

[0002] The brain-computer interface technology is a control mode independent of peripheral and muscle, and a user can directly send a control instruction to the outside world through brain activity. The motor imagery is a common brain-computer interface paradigm, and a user only needs to imagine that he or she attempts to move rather than actually implements the movement. The brain-computer interface system captures the brain activity before the movement, recognizes the user's intention, and controls the external device. The motor imagery brain activity has time-varying characteristics, and the brain activity characteristics are different for different users or the same user at different times to perform the motor imagery paradigm. Before using the brain-computer interface based on the motor imagery, the user needs to complete a training link. After the brain-computer interface system collects sufficient training data, the motor intention recognition can be performed in real time.

[0003] The brain-computer interface system based on the motor imagery can be used in the stroke rehabilitation field. In an actual application scenario, the training stage is short and the training data are limited in consideration of the experience of a patient. The limited training data makes the brain-computer interface system unable to use the recognition method such as deep learning which needs a large amount of training data. In the traditional machine learning field, the recognition accuracy is reduced due to too few training data. The recognition model is sensitive to hyperparameters, and different hyperparameters can greatly affect the model performance. Under the conditions of a small data set and limited processing time, it is difficult to optimize the hyperparameters of the recognition model, which further affects the system recognition accuracy. SUMMARY

[0004] The technical purpose of the application is to solve the technical problem that the hyperparameters of the recognition model are difficult to be optimized and the system recognition accuracy is affected under the conditions of a small data set and limited processing time for the current motor imagery intention recognition technology, and a multi-model fusion motor imagery intention recognition system is provided.

[0005] To achieve the above technical purpose, the embodiments of the application adopt the following technical solutions.

[0006] The multi-model fusion motor imagery intention recognition system provided by the embodiments of the application comprises: The training fusion strategy module is configured to obtain an optimal model-parameter group and output weights of each model-parameter pair in the optimal model-parameter group according to an offline training data set, the model-parameter pair being a combination of a feature extraction method, a classifier and corresponding parameters. The application fusion strategy module is used for training each model-parameter pair in the preferred model-parameter group based on the collected user offline clinical data set and real-time clinical online data, inputting the clinical online data into the trained preferred model-parameter group, weighting and fusing outputs of all model-parameter pairs based on output weights of each model-parameter pair, and obtaining a motor imagery intention recognition result.

[0007] Compared with the prior art, the multi-model fusion motor imagery intention recognition system provided by the embodiments of the application has the following technical effects: the system selects a preferred model-parameter group from a large number of model-parameter pairs by training the fusion strategy module, and calculates output weights according to model performance. In the application of the fusion strategy module, the clinical data is processed by using the preferred model-parameter pairs and the weights, the outputs of multiple models are weighted and fused, the advantages of different models are integrated, the limitations of a single model are avoided, and thus the accuracy of motor imagery intention recognition is improved. For example, in an actual application scenario, for motor imagery intention recognition of different users, the system can more accurately judge the user's intention, and provide a reliable basis for subsequent action execution.

[0008] The offline training data set used for training the fusion strategy module covers users with different characteristics and contains quality difference data, which enables the system to learn diversified data characteristics in the training process, and enhances the adaptability of the model to different users and different data quality. Whether it is a case with good data quality or data with certain interference or noise, the system can better perform motor imagery intention recognition, and improve the generality of the system.

[0009] It should be understood that the summary section is not intended to identify key or essential features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0010] The drawings described herein are for illustrative purposes only and are not intended to limit the scope of the present disclosure in any way. In addition, the shapes and scale sizes of the components in the drawings are only illustrative and are used to help understand the present application, and are not specific limitations on the shapes and scale sizes of the components of the present application. Those skilled in the art can select various possible shapes and scale sizes to implement the present application according to specific circumstances under the teaching of the present application. In the drawings: Figure 1 A principle schematic diagram of the multi-model fusion motor imagery intention recognition system provided by the embodiments of the application. DETAILED DESCRIPTION

[0011] In order to enable a person skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person skilled in the art without creative work should belong to the protection scope of the present application.

[0012] The present application provides a multi-model fusion motor imagery intention recognition system, which is suitable for the application scene of clinical motor imagery, and can improve the system recognition accuracy in a small sample set and limited time.

[0013] The present application will be further described below in combination with the drawings and specific embodiments of the specification.

[0014] A multi-model fusion motor imagery intention recognition system includes a training fusion strategy module and an application fusion strategy module.

[0015] The training fusion strategy module is used to obtain an optimal model-parameter group and output weights of each model-parameter pair in the optimal model-parameter group according to an offline training data set, the model-parameter pair being a combination of a feature extraction method, a classifier and corresponding parameters.

[0016] The application fusion strategy module is used to train each model-parameter pair in the optimal model-parameter group based on an offline clinical data set collected from a user and real-time clinical online data, input the clinical online data into the trained optimal model-parameter group, weight fuse the outputs of all model-parameter pairs based on the output weights of each model-parameter pair, and obtain a motor imagery intention recognition result.

[0017] In the embodiments, the EEG of the scalp position of the user can be collected based on the motor imagery hand function rehabilitation equipment, the intention of the user to move the hand is recognized, then the mechanical exoskeleton is controlled to perform the corresponding gripping action, the neural feedback is formed, and the brain motor function remodeling is promoted.

[0018] A schematic diagram of the principle of the multi-model fusion motor imagery intention recognition system is shown in Figure 1 The system involves two stages of training fusion strategy and application fusion strategy. In the training fusion strategy stage, the training fusion strategy module obtains an optimal model-parameter group and model-parameter pair output weights for fusing the model-parameter group result according to an offline training data set.

[0019] In the clinical application, the stage of rehabilitation training using the device is the application fusion strategy stage, and the user completes the training link to obtain the offline clinical data set. The application fusion strategy module uses the offline clinical data set to train the optimal model-parameter group. Subsequently, the user enters the online link, and the system collects the user's brain activity signals in real time. The collected online clinical data set is sent to the optimal model-parameter group to obtain a set of optimal model-parameter pair outputs. The recognition result is obtained by weighting the model-parameter pair output weights.

[0020] The system clearly divides the training fusion strategy module and the application fusion strategy module. In the training fusion strategy stage, comprehensive model screening and weight calculation are performed. In the application fusion strategy stage, efficient real-time recognition is performed based on the training results. This design makes the training and recognition process more clear and reasonable, improving the system's running efficiency. At the same time, in the training process, the model-parameter pairs are subjected to cluster analysis and screening, reducing unnecessary calculation resource consumption and improving the training speed. In the recognition stage, fast weighted fusion calculation can realize real-time recognition of motor imagination intention, meeting the needs of actual application.

[0021] In some embodiments, the training fusion strategy module includes a data set preparation unit, a model construction unit, a performance evaluation unit, an optimal screening unit, and a weight calculation unit.

[0022] The data set preparation unit is used to prepare an offline training data set containing multiple data. The data sources cover users with different characteristics and contain quality difference data. The model construction unit is used to construct a model-parameter group including multiple model-parameter pairs. The performance evaluation unit is used to match and evaluate each model-parameter pair in the model-parameter group with each data in the offline training data set to form a model-parameter pair performance feature group. The optimal screening unit is used to perform cluster analysis on the model-parameter pair performance feature group and select a preset number of model-parameter pairs from each cluster category to form an optimal model-parameter group. The weight calculation unit is used to calculate the model-parameter pair output weight based on the performance indicators of each model-parameter pair in the optimal model-parameter group.

[0023] In an embodiment, the training fusion strategy stage includes the following steps: 1. Prepare the offline training data set: Prepare the offline training data set for training fusion strategy. Specifically, the data set includes N0 data. In some embodiments, N0≥N K (N K is the lower limit of the data amount in the data set, such as N K = 100, and contains N k1 %-N k2 % (Nk1 N k2 are respectively lower quality data quantity proportion lower limit and proportion upper limit, such as 10%-20% of low quality data, the low quality data is determined by lead standard deviation.

[0024] The data sources are preferably from clinical data, and include a small amount of poor quality data to ensure the scalability of the classifier. Each user's data should not be excessive, and the same user's data should not be collected on the same day. The users should preferably include different age groups, different genders, and different affected limbs.

[0025] For example, in a system device for hand function rehabilitation, the offline training data set includes 20 clinical patients, each providing 5 data, totaling 100, of which 10 data is of poor quality (lead standard deviation is large). There are 10 male patients and 10 female patients. In the offline training data set, the patient age distribution, affected side distribution, and patient overall data distribution are similar.

[0026] 2. Prepare a model-parameter group: Specifically, the model-parameter group is a combination of different models and specific parameters, where each element is a model-parameter pair; the model-parameter pair is a combination of a feature extraction method, a classifier, and corresponding parameters.

[0027] The "feature extraction method" extracts effective features from electroencephalogram (EEG) signals, the "classifier" classifies the extracted features, and the "parameters" are "adjusting items" that allow the "feature extraction method" and the "classifier" to adapt to the data and achieve accurate recognition. The three together form a complete "model-parameter pair" that can be directly used for training and identification.

[0028] For example, for a hand function rehabilitation system, the recognition model needs to extract features from EEG signals for binary classification to determine whether the patient wants to hold the affected side or is in an idle state, where the affected limb is the left hand or the right hand.

[0029] In the embodiment, the feature extraction method in the model-parameter pair can include at least one of common spatial pattern, filter bank common spatial pattern; the classifier includes at least one of support vector machine, decision tree classifier, K nearest neighbor classifier, and EEGNet general model.

[0030] As an example, the recognition model for motor imagery intention can include Common Spatial Patterns (CSP) for feature extraction and Support Vector Machine (SVM) for feature classification, hereinafter referred to as CSP+SVM. Other methods for feature extraction and classifiers also include Filter-Bank Common Spatial Pattern (FBCSP), Decision Tree Classifier, K-Nearest Neighbors Classifier (KNN), EEGNet general model, etc., which can be selected according to actual needs and added to the model-parameter group.

[0031] In the embodiment, the optional parameters include two kinds of pre-processing parameters and model hyperparameters. The pre-processing parameters include the start point of the time window, the length of the time window, the filter bandwidth, etc. The model parameters include the number of CSP features, whether the CSP needs to be regularized, the type of SVM kernel function, and the SVM penalty parameter C.

[0032] As an example, the model-parameter group is shown in Table 1 as follows: Table 1 Model-parameter pair in model-parameter group in the embodiment

[0033] In the above embodiment, the model-parameter group of CSP-SVM has a total of 2880 model-parameter pairs.

[0034] 3. Model-parameter group training and evaluation. This step evaluates each model in the model-parameter group with each data in the offline training data set to form the performance characteristics of the model-parameter pairs.

[0035] Specifically, each element in the model-parameter group is a specific model-parameter pair. Each piece of data in the offline training dataset includes both offline and online data. During evaluation, the offline data is used to train the model-parameter pair, and then the online data is used to evaluate the performance of the model-parameter pair, forming a model-parameter pair performance feature group, where each element is a model-parameter pair performance feature. The total number of elements in the model-parameter pair performance feature group is the same as the total number of elements in the model-parameter group. Each element in the model-parameter pair performance feature group is a vector, and each element in the vector is the accuracy of the model-parameter pair on the corresponding data, that is, the performance of the model-parameter pair on each piece of data. The length of the vector is consistent with the number of offline training datasets. The performance data range can be 0-1, indicating the accuracy of the model-parameter pair on the online part of the data. If the classifier cannot be trained and converged on a certain data, it is considered abnormal and marked as -1.

[0036] 4. Screening the preferred model-parameter group, clustering the model-parameter pair performance feature group to form N 1 category. Select the top N 2 model-parameter pairs in each category with the highest average accuracy to form the preferred model-parameter group.

[0037] As an example, the average accuracy of the i-th model-parameter pair is calculated as follows: ; Where is the average accuracy of the i-th model-parameter pair in the model-parameter group, is the k-th value of the i-th model-parameter pair performance feature group, and N0 is the total number of elements in the i-th model-parameter pair performance feature group.

[0038] The average accuracy of all model-parameter pairs in the same cluster category is averaged to obtain the average accuracy of each cluster, expressed as: ; V j is the average accuracy of the model-parameter pair in the j-th category, N j is the total number of elements in the j-th category, is the average accuracy of the k1-th model-parameter pair in the j-th category.

[0039] The average accuracy of each cluster is used as the performance indicator of the model-parameter pair in the category, so all model-parameter pairs in the same category share the same performance indicator.

[0040] Assuming that different model-parameter pairs have specific advantages for different types of data, after clustering the model-parameter pair performance feature group, the model-parameter pair suitable for specific data will be clustered into a category. The total number of elements in the preferred model-parameter group is The number will affect the response speed of the application fusion strategy stage. As an example, the total number of elements of the preferred model-parameter group can be between 8-15. If the total number of elements of the preferred model-parameter group is 15, the values of N1 and N2 are (15, 1), (5, 3), (3, 5), etc. can directly filter the model-parameter pairs. If the value of N1 is 10, 2 model-parameter pairs with the highest average accuracy are selected from the 5 categories with the highest average accuracy, and 1 model-parameter pair with the highest average accuracy is selected from the other 5 categories.

[0041] The clustering method can be K-means clustering, and other methods can also be used.

[0042] The output weight of the model-parameter pair is calculated, and the accuracy of the selected model-parameter pair in each category is normalized to obtain the corresponding weight coefficient.

[0043] As an example, in the weight calculation unit, the model-parameter pairs come from different categories, and the average accuracy of the model-parameter pairs in the same category forms the performance indicator b, and the performance indicators of all model-parameter pairs in the same category are the same.

[0044] In an embodiment, the output weight of each model-parameter pair in the preferred model-parameter group is calculated: Where a j is the output weight of the jth model-parameter pair in the preferred model-parameter group, b j is the performance indicator of the jth model-parameter pair in the preferred model-parameter group, b o is the performance indicator of the oth model-parameter pair in the preferred model-parameter group, and N o is the total number of elements of the preferred model-parameter group.

[0045] The training fusion strategy stage is completed, and the preferred model-parameter group and the output weight of the model-parameter pair are output.

[0046] The application fusion strategy stage includes the following steps: 1. The user completes the clinical offline training link, and the system collects the offline clinical data set. The form of clinical offline training, the category and parameter of data collection, and the data collection method of the offline training data set are the same.

[0047] 2. After completing the clinical offline training link, train each model-parameter pair in the preferred model-parameter group using the clinical data set.

[0048] The clinical online link starts, the system collects clinical online data in real time, and after single data collection, the data is sent to each model-parameter pair in the preferred model-parameter group to obtain a group of model-parameter pair outputs. ​

[0049] 3. The online classification results are fused by using the model-parameter pair output weight to obtain the fusion result O, which is expressed as: ; wherein o j is the output of the jth model-parameter pair, a j is the output weight of the jth model-parameter pair in the preferred model-parameter group.

[0050] In some embodiments, the result fusion unit further comprises a class conversion subunit, which is configured to convert the fusion result O obtained by the weighted fusion into a class coefficient c t of the n-class task, and the conversion formula is:

[0051] wherein int() is an integer function, n is the number of classification classes, and n≥2. According to the class coefficient c t , the motor imagery intention recognition result c is obtained, which is expressed as: .

[0052] When there are K classes of motor imagery intention recognition, the corresponding relationship between the class coefficient c t and the recognition result c is expressed as: the output range interval (0, K] of the fusion result O is equally divided into K small intervals (i-1, i] (i=2,..., K), if the class coefficient c t falls within the interval (i-1, i], then c=i.

[0053] As an example, when performing a 3-class task, when n=3, the value output range of O is 0-2, and the corresponding relationship between the class coefficient c t and the motor imagery intention recognition result c is: If 2<c t ≤3, then c=3. If 1<c t ≤2, then c=2. If 0<c t ≤1, then c=1.

[0054] The system pre-constructs model-parameter groups containing a large number of model-parameter pairs (such as 2880 combinations of the CSP-SVM model in the example) in the training fusion strategy stage, and filters out preferred model-parameter groups (such as 8-15 in total) through cluster analysis and performance evaluation. This "pre-filtering" mechanism avoids optimization of all possible hyperparameter combinations in the application stage: the preferred model-parameter groups have been verified for generalization ability through offline training data, and the number is small (such as 8-15), which can be quickly adapted in a small data set and limited time, without the need to traverse all hyperparameters for optimization, greatly reducing the computational cost.

[0055] A small data set is prone to single model overfitting or hyperparameter sensitivity (different hyperparameters have a large impact on accuracy). The system integrates the advantages of different models by weighting the outputs of multiple preferred model-parameter pairs (based on the weight coefficients calculated in the training stage). The amount of clinical offline training data in the application stage is limited (consistent with the actual needs of short patient training time), but the system only needs to train the "preferred model-parameter groups" (not all models), which significantly reduces the training amount and can be completed in a limited time.

[0056] At the same time, the offline training data set contains "a small amount of poor quality data" and diverse users (different ages, genders, and affected sides), so the preferred model-parameter groups have certain anti-interference ability and generalization, reducing overfitting to specific data in a small sample.

[0057] It should be fully understood that the user information (including but not limited to user physiological information, user personal information, etc.) involved in the present application is information and data authorized by the user or fully authorized by all parties. The use of user information should comply with privacy policies and practices that are generally considered to meet or exceed industry or government requirements for maintaining user privacy. The collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards, and provide corresponding operation portals for users to choose authorization or refusal.

[0058] The system, device, module or unit illustrated in the above embodiments can be specifically implemented by a computer chip or entity, or by a product with certain functions. A typical implementation device is a computer. Specifically, the computer may, for example, be a personal computer, a laptop computer, a tablet computer, or a combination of any of these devices.

[0059] The multi-model fusion motor imagery intention recognition system provided by the present application is described in detail above, and the principles and implementation modes of the present application are described in this paper. The above description of the embodiments is only used to help understand the concept of the present application and should not be understood as limiting the scope of protection of the present application.

Claims

1. Multi-model fusion motion imagery intention recognition system, characterized by: include: A training fusion strategy module is used to obtain an optimal model-parameter group and output weights of each model-parameter pair in the optimal model-parameter group based on the offline training data set, wherein the model-parameter pair is a combination of a feature extraction method, a classifier and corresponding parameters; A fusion strategy module is applied to train each model-parameter pair in the preferred model-parameter group based on the collected user offline clinical data set and real-time clinical online data, input the trained preferred model-parameter group with the clinical online data, and perform weighted fusion on the outputs of all model-parameter pairs based on the output weights of each model-parameter pair to obtain the motor imagery intention recognition result.

2. The multi-model fusion motion imagery intention recognition system according to claim 1 is characterized in that: The training fusion strategy module includes: A data set acquisition unit is used to acquire an offline training data set including multiple data, the data sources including users with different characteristics and including data with different quality; A model building unit, configured to build a model-parameter group including a plurality of model-parameter pairs; a performance evaluation unit, configured to perform matching evaluation on each model-parameter pair in the model-parameter group and each piece of data in the offline training data set to form a model-parameter pair performance feature group; A preferred screening unit is used to perform cluster analysis on the performance characteristic groups of model-parameter pairs, and screen a preset number of model-parameter pairs from each cluster category to form a preferred model-parameter group; The weight calculation unit is used to calculate the output weight of the model-parameter pair based on the performance index of each model-parameter pair in the preferred model-parameter group.

3. The multi-model fusion motion imagery intention recognition system according to claim 2, characterized in that: The number of data copies in the offline training dataset is N 0， N0≥N K , and contains N k1 %-N k2 % of low-quality data, N K is the lower limit of the amount of data in the dataset, N k1 、N k2 They are the lower limit and upper limit of the proportion of low-quality data respectively, and the low-quality data is determined by the lead standard deviation.

4. The multi-model fusion motion imagery intention recognition system according to claim 1, characterized in that: The feature extraction method in the model-parameter pair includes at least one of a co-space mode and a filter bank co-space mode; the classifier includes at least one of a support vector machine, a decision tree classifier, a K-nearest neighbor classifier, and an EEGNet general model.

5. The multi-model fusion motion imagery intention recognition system according to claim 1 is characterized in that: The parameters include preprocessing parameters and model hyperparameters; the preprocessing parameters include the time window starting point, time window length and filter bandwidth; the model hyperparameters include the number of features, regularization parameters, kernel function type and penalty coefficient.

6. The multi-model fusion motion imagery intention recognition system according to claim 2, characterized in that: Each element in the model-parameter pair performance feature group is the accuracy of the model-parameter pair on the corresponding data; if the model-parameter pair fails to converge, it is marked as abnormal.

7. The multi-model fusion motion imagery intention recognition system according to claim 2, characterized in that: In the preferred screening unit, cluster analysis uses the K-means clustering algorithm, the number of clusters is N1, and N2 model-parameter pairs with the highest average accuracy are screened from each cluster. The total number of elements in the preferred model-parameter group is .

8. The multi-model fusion motor imagery intention recognition system according to claim 2, characterized in that: In the weight calculation unit, the performance index is obtained by averaging the accuracy of the model-parameter pairs in the same cluster category, and calculating the output weight of each model-parameter pair in the preferred model-parameter group: ； in, is the output weight of the jth model-parameter pair in the optimal model-parameter group, b j is the performance index of the jth model-parameter pair in the optimal model-parameter group, is the performance index of the oth model-parameter pair in the optimal model-parameter group, is the total number of elements in the preferred model-parameter group.

9. The multi-model fusion motion imagery intention recognition system according to claim 1, characterized in that: The application fusion strategy module also includes a module for converting the fusion result O obtained by weighted fusion into a category coefficient c of n classification tasks. t , the conversion formula is: ； Where n is the number of classification categories, int() is the rounding function, and n≥2; According to the category coefficient c t , we get the motor imagery intention recognition result c, which is expressed as: 。 10. The multi-model fusion motion imagery intention recognition system according to claim 9, characterized in that: When there are K types of motor imagery intentions to be recognized, the category coefficient c t The corresponding relationship with the recognition result c is expressed as: Divide the output range interval (0,K] of the fusion result O into K small intervals (i-1,i] (i=1,2,...,K), if the category coefficient c t If it falls within the interval (i-1,i], then c=i.

Images