Multi-model fusion motor imagery intention recognition system

Abstract

The application discloses a multi-model fusion motor imagery intention recognition system, comprising a training fusion strategy module and an application fusion strategy module; the training fusion strategy module obtains an optimal model-parameter pair and output weights of the optimal model-parameter pair from an offline training data set, the model-parameter pair being a combination of a feature extraction method, a classifier and corresponding parameters; the application fusion strategy module is used for training each model-parameter pair in the optimal model-parameter pair based on an offline clinical data set and real-time online clinical data, inputting the online clinical data into the trained optimal model-parameter pair, weighting and fusing outputs of all model-parameter pairs based on the output weights of the model-parameter pairs, and obtaining a motor imagery intention recognition result. The application can improve motor imagery intention recognition accuracy without complex hyperparameter optimization under small data sets and limited time.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of brain-computer interface, and particularly relates to a multi-model fusion motor imagery intention recognition system. BACKGROUND

[0002] The brain-computer interface technology is a control mode independent of peripheral and muscle, and a user can directly send a control instruction to the outside world through brain activity. The motor imagery is a common brain-computer interface paradigm, and a user only needs to imagine that he or she attempts to move rather than actually implements the movement. The brain-computer interface system captures the brain activity before the movement, recognizes the user's intention, and controls the external device. The motor imagery-based brain activity has time-varying characteristics, and the brain activity characteristics are different for different users or the same user at different times to perform the motor imagery paradigm. Before using the brain-computer interface based on the motor imagery, the user needs to complete a training link. After the brain-computer interface system collects sufficient training data, the motor intention recognition can be performed in real time.

[0003] The brain-computer interface system based on the motor imagery can be used in the stroke rehabilitation field. In an actual application scenario, the training stage is short and the training data are limited in consideration of the experience of a patient. The limited training data makes the brain-computer interface system unable to use the recognition method such as deep learning which needs a large amount of training data. In the traditional machine learning field, the recognition accuracy is reduced due to too few training data. The recognition model is sensitive to hyperparameters, and different hyperparameters can greatly affect the model performance. Under the limitation of small data set and limited processing time, it is difficult to optimize the hyperparameters of the recognition model, which further affects the system recognition accuracy. SUMMARY

[0004] The technical purpose of the application is to solve the technical problem that the hyperparameters of the recognition model are difficult to be optimized and the system recognition accuracy is affected under the condition of small data set and limited processing time in the current motor imagery intention recognition technology, and a multi-model fusion motor imagery intention recognition system is provided.

[0005] To achieve the above technical purpose, the embodiments of the application adopt the following technical solutions.

[0006] The multi-model fusion motor imagery intention recognition system provided by the embodiments of the application comprises:

[0007] The training fusion strategy module is configured to obtain an optimal model-parameter group and output weights of each model-parameter pair in the optimal model-parameter group according to an offline training data set, the model-parameter pair being a combination of a feature extraction method, a classifier and corresponding parameters.

[0008] The application fusion strategy module is used for training each model-parameter pair in the preferred model-parameter group based on the collected user offline clinical data set and real-time clinical online data, inputting the clinical online data into the trained preferred model-parameter group, weighting and fusing outputs of all model-parameter pairs based on output weights of each model-parameter pair, and obtaining a motor imagery intention recognition result.

[0009] Compared with the prior art, the multi-model fusion motor imagery intention recognition system provided by the embodiments of the present application has the following technical effects: the system selects a preferred model-parameter group from a large number of model-parameter pairs by training the fusion strategy module, and calculates output weights according to model performance. In the application of the fusion strategy module, the clinical data is processed by using the preferred model-parameter pairs and the weights, the outputs of multiple models are weighted and fused, the advantages of different models are integrated, the limitations of a single model are avoided, and thus the accuracy of motor imagery intention recognition is improved. For example, in an actual application scenario, for motor imagery intention recognition of different users, the system can more accurately judge the user's intention, and provide a reliable basis for subsequent action execution.

[0010] The offline training data set used for training the fusion strategy module covers users with different characteristics and contains quality difference data, which enables the system to learn diversified data characteristics in the training process, and enhances the adaptability of the model to different users and different data quality. Whether it is a case with good data quality or data with certain interference or noise, the system can better perform motor imagery intention recognition, and improve the generality of the system.

[0011] It should be understood that the summary section is not intended to identify key or essential features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0012] The drawings described herein are for illustrative purposes only and are not intended to limit the scope of the present disclosure in any way. In addition, the shapes and scale sizes of the components in the drawings are only illustrative and are used to help understand the present application, and are not specific limitations on the shapes and scale sizes of the components of the present application. Those skilled in the art can select various possible shapes and scale sizes to implement the present application according to specific circumstances under the teaching of the present application. In the drawings:

[0013] Figure 1 The principle diagram of the multi-model fusion motor imagery intention recognition system provided by the embodiments of the present application. DETAILED DESCRIPTION

[0014] In order to enable a person skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person skilled in the art without creative work should belong to the protection scope of the present application.

[0015] The present application provides a multi-model fusion motor imagery intention recognition system, which is suitable for the application scene of clinical motor imagery, and can improve the system recognition accuracy in a small sample set and limited time.

[0016] The present application will be further described below in combination with the drawings and specific embodiments of the specification.

[0017] A multi-model fusion motor imagery intention recognition system comprises a training fusion strategy module and an application fusion strategy module.

[0018] The training fusion strategy module is used to obtain an optimal model-parameter group and output weights of each model-parameter pair in the optimal model-parameter group according to an offline training data set, the model-parameter pair being a combination of a feature extraction method, a classifier and corresponding parameters.

[0019] The application fusion strategy module is used to train each model-parameter pair in the optimal model-parameter group based on an offline clinical data set collected from a user and real-time clinical online data, input the clinical online data into the trained optimal model-parameter group, weight fuse the outputs of all model-parameter pairs based on the output weights of each model-parameter pair, and obtain a motor imagery intention recognition result.

[0020] In the embodiments, the EEG of the scalp position of the user can be collected based on the motor imagery hand function rehabilitation equipment, the intention of the user to move the hand is recognized, then the mechanical exoskeleton is controlled to perform the corresponding gripping action, the neural feedback is formed, and the brain motor function remodeling is promoted.

[0021] A schematic diagram of the principle of the multi-model fusion motor imagery intention recognition system is shown in Figure 1 The system involves two stages of training fusion strategy and application fusion strategy. In the training fusion strategy stage, the training fusion strategy module obtains an optimal model-parameter group and model-parameter pair output weights for fusing the model-parameter group result according to an offline training data set.

[0022] In the clinical application, the stage of rehabilitation training using the device is the application fusion strategy stage, and the user completes the training link to obtain the offline clinical data set. The application fusion strategy module uses the offline clinical data set to train the optimal model-parameter group. Subsequently, the user enters the online link, and the system collects the user's brain activity signals in real time. The collected clinical online data set is sent to the optimal model-parameter group to obtain a set of optimal model-parameter pair outputs. The recognition result is obtained by weighting the model-parameter pair output weights.

[0023] The system clearly divides the training fusion strategy module and the application fusion strategy module. In the training fusion strategy stage, comprehensive model screening and weight calculation are performed. In the application fusion strategy stage, efficient real-time recognition is performed based on the training results. This design makes the training and recognition process more clear and reasonable, improving the running efficiency of the system. At the same time, in the training process, the model-parameter pairs are subjected to cluster analysis and screening, reducing unnecessary calculation resource consumption and improving the training speed. In the recognition stage, fast weighted fusion calculation can realize real-time recognition of motor imagination intention, meeting the needs of actual application.

[0024] In some embodiments, the training fusion strategy module includes a data set preparation unit, a model construction unit, a performance evaluation unit, an optimal screening unit, and a weight calculation unit.

[0025] The data set preparation unit is used to prepare an offline training data set containing multiple data. The data sources cover users with different characteristics and contain quality difference data.

[0026] The model construction unit is used to construct a model-parameter group including multiple model-parameter pairs.

[0027] The performance evaluation unit is used to match and evaluate each model-parameter pair in the model-parameter group with each data in the offline training data set to form a model-parameter pair performance feature group.

[0028] The optimal screening unit is used to perform cluster analysis on the model-parameter pair performance feature group, and select a preset number of model-parameter pairs from each cluster category to form an optimal model-parameter group.

[0029] The weight calculation unit is used to calculate the model-parameter pair output weight based on the performance indicators of each model-parameter pair in the optimal model-parameter group.

[0030] In an embodiment, the training fusion strategy stage includes the following steps:

[0031] 1. Prepare the offline training data set: Prepare the offline training data set for training fusion strategy. Specifically, the data set includes N0 pieces of data. In some embodiments, N0≥N K (N KN K 100)100, and includes N k1 %-N k2 %(N k1 ,N k2 are the lower and upper limits of the proportion of low-quality data, respectively, such as 10%-20% of low-quality data, which is determined by the standard deviation of the lead.

[0032] The data sources should come from the clinic as much as possible, and include a small amount of poor-quality data to ensure the scalability of the classifier. Each user's data should not be too much, and the same user's data should not be collected on the same day. The user should include limbs of different ages, different genders, and different affected sides as much as possible.

[0033] For example, in a system device for hand function rehabilitation, the offline training data set includes 20 clinical patients, each providing 5 data, totaling 100, of which 10 data is of poor quality (large lead standard deviation). There are 10 male patients and 10 female patients. In the offline training data set, the patient age distribution, affected side distribution, and patient overall data distribution are similar.

[0034] 2. Prepare the model-parameter group:

[0035] Specifically, the model-parameter group is a combination of different models and specific parameters, where each element is a model-parameter pair; the model-parameter pair is a combination of a feature extraction method, a classifier, and corresponding parameters.

[0036] The "feature extraction method" extracts effective features from electroencephalogram (EEG) signals, the "classifier" classifies the extracted features, and the "parameters" are "adjusting items" that allow the "feature extraction method" and the "classifier" to adapt to the data and achieve accurate recognition. The three together form a complete "model-parameter pair" that can be directly used for training and recognition.

[0037] For example, for a hand function rehabilitation system, the recognition model needs to extract features from EEG signals for binary classification to determine whether the patient wants to hold the affected side or is in an idle state, where the affected limb is the left hand or the right hand.

[0038] In the embodiment, the feature extraction method in the model-parameter pair can include at least one of common spatial patterns, filter bank common spatial patterns; the classifier includes at least one of support vector machines, decision tree classifiers, K-nearest neighbor classifiers, and EEGNet general models.

[0039] As an example, the recognition model for motor imagery intention can include Common Spatial Patterns (CSP) for feature extraction and Support Vector Machine (SVM) for feature classification, hereinafter referred to as CSP+SVM. Other methods for feature extraction and classifiers also include Filter-Bank Common Spatial Pattern (FBCSP), Decision Tree Classifier, K-Nearest Neighbors Classifier (KNN), EEGNet general model, etc., which can be selected according to actual needs and added to the model-parameter group.

[0040] In the embodiment, the optional parameters include two kinds of pre-processing parameters and model hyperparameters. The pre-processing parameters include the start point of the time window, the length of the time window, the filter bandwidth, etc. The model parameters include the number of CSP features, whether the CSP needs to be regularized, the type of SVM kernel function, and the SVM penalty parameter C.

[0041] As an example, the model-parameter group is shown in Table 1 as follows:

[0042] Table 1: Model-parameter pair in the model-parameter group in the embodiment

[0043]

[0044] In the above embodiment, the model-parameter group of CSP-SVM has a total of 2880 model-parameter pairs.

[0045] 3. Model-parameter group training and evaluation: this step evaluates each model in the model-parameter group with each data in the offline training data set to form the performance characteristics of the model-parameter pair.

[0046] Specifically, each element in the model-parameter group is a specific model-parameter pair. Each piece of data in the offline training dataset includes both offline and online data. During evaluation, the offline data is used to train the model-parameter pair, and then the online data is used to evaluate the performance of the model-parameter pair, forming a model-parameter pair performance feature group, where each element is a model-parameter pair performance feature. The total number of elements in the model-parameter pair performance feature group is the same as the total number of elements in the model-parameter group. Each element in the model-parameter pair performance feature group is a vector, and each element in the vector is the accuracy of the model-parameter pair on the corresponding data, that is, the performance of the model-parameter pair on each piece of data. The length of the vector is consistent with the number of offline training datasets. The performance data range can be 0-1, indicating the accuracy of the model-parameter pair on the online part of the data. If the classifier cannot be trained and converged on a certain data, it is considered abnormal and marked as -1.

[0047] 4. Screening the preferred model-parameter group, clustering the model-parameter pair performance feature group to form N 1 category. Select the top N 2 model-parameter pairs in each category with the highest average accuracy to form the preferred model-parameter group.

[0048] As an example, the average accuracy of the i-th model-parameter pair is calculated as follows:

[0049] ;

[0050] where is the average accuracy of the i-th model-parameter pair in the model-parameter group, is the k-th value of the i-th model-parameter pair performance feature group, and N0 is the total number of elements in the i-th model-parameter pair performance feature group.

[0051] The average accuracy of all model-parameter pairs in the same cluster category is averaged to obtain the average accuracy of each cluster, expressed as: ;

[0052] V j is the average accuracy of the model-parameter pair in the j-th category, N j is the total number of elements in the j-th category, is the average accuracy of the k1-th model-parameter pair in the j-th category.

[0053] The average accuracy of each cluster is used as the performance indicator of the model-parameter pair in the category, so all model-parameter pairs in the same category share the same performance indicator.

[0054] Assuming that different model-parameter pairs have specific advantages for different types of data, after clustering the performance characteristics of the model-parameter pairs, the model-parameter pairs suitable for specific data will be clustered into a class. The total number of elements of the preferred model-parameter group is preferably The number will affect the response speed in the application fusion strategy stage. As an example, the total number of elements of the preferred model-parameter group can be between 8-15. If the total number of elements of the preferred model-parameter group is 15, N1 and N2 take values such as (15, 1), (5, 3), (3, 5), etc. can directly select the model-parameter pairs. If N1 takes the value of 10, in the 10 categories, 2 model-parameter pairs with the highest average accuracy are selected from the 5 categories with the highest average accuracy, and 1 model-parameter pair with the highest average accuracy is selected from the other 5 categories.

[0055] The clustering method can be K-means clustering, and other methods can also be used.

[0056] The output weight of the model-parameter pair is calculated, and the accuracy of the selected model-parameter pair in each category is normalized to obtain the corresponding weight coefficient.

[0057] As an example, in the weight calculation unit, the model-parameter pairs come from different categories, and the average accuracy of the model-parameter pairs in the same category forms the performance indicator b, and the performance indicators of all model-parameter pairs in the same category are the same.

[0058] In the embodiment, the output weight of each model-parameter pair in the preferred model-parameter group is calculated:

[0059] Where a j is the output weight of the jth model-parameter pair in the preferred model-parameter group, b j is the performance indicator of the jth model-parameter pair in the preferred model-parameter group, b o is the performance indicator of the oth model-parameter pair in the preferred model-parameter group, and N o is the total number of elements of the preferred model-parameter group.

[0060] At this point, the training fusion strategy stage is completed, and the preferred model-parameter group and the output weight of the model-parameter pair are output.

[0061] The application fusion strategy stage includes the following steps:

[0062] 1. The user completes the clinical offline training link, and the system collects the offline clinical data set. The form of clinical offline training, the type and parameter of collected data, and the data collection method of the offline training data set are the same.

[0063] 2. After completing the clinical offline training link, train each model-parameter pair in the preferred model-parameter group using the clinical data set.​

[0064] The clinical online link starts, the system collects clinical online data in real time, after single data collection, the data is sent to each model-parameter pair in the preferred model-parameter group, and a set of model-parameter pair outputs is obtained.

[0065] 3. The online classification result is fused by using the model-parameter pair output weight to obtain the fusion result O, which is expressed as:

[0066] ;

[0067] Wherein, o j is the output of the jth model-parameter pair, a j is the output weight of the jth model-parameter pair in the preferred model-parameter group.

[0068] In some embodiments, the result fusion unit further includes a class conversion subunit, which is used to convert the fusion result O obtained by weighted fusion into the class coefficient c t of the n-class task, and the conversion formula is:

[0069]

[0070] Wherein, int( ) is the integer function, n is the number of classification categories, and n≥2;

[0071] According to the class coefficient c t , the motor imagination intention recognition result c is obtained, which is expressed as:

[0072] .

[0073] When there are K classes of motor imagination intention recognition, the corresponding relationship between the class coefficient c t and the recognition result c is expressed as: the output range interval (0, K] of the fusion result O is equally divided into K small intervals (i-1, i] (i=2,..., K), if the class coefficient c t falls in the interval (i-1, i], then c=i.

[0074] As an example, when performing a 3-class task, when n=3, the value output range of O is 0-2, and the corresponding relationship between the class coefficient c t and the motor imagination intention recognition result c is:

[0075] If 2<c t ≤3, then c=3;

[0076] If 1<c t ≤2, then c=2;

[0077] If 0<c t≤ 1, then c = 1.

[0078] The system pre-constructs model-parameter groups containing a large number of model-parameter pairs (such as 2880 combinations of the CSP-SVM model in the example) in the training fusion strategy stage, and filters out preferred model-parameter groups (such as 8-15 in total) through clustering analysis and performance evaluation. This "pre-filtering" mechanism avoids optimization of all possible hyperparameter combinations in the application stage: the preferred model-parameter groups have been verified for generalization ability through offline training data, and the number is small (such as 8-15), which can be quickly adapted in a small data set and limited time, without the need to traverse and optimize all hyperparameters, greatly reducing the computational cost.

[0079] Small data sets are prone to single model overfitting or hyperparameter sensitivity (different hyperparameters have a large impact on accuracy). The system integrates the advantages of different models by weighting the outputs of multiple preferred model-parameter pairs (based on the weight coefficients calculated in the training stage). The amount of clinical offline training data in the application stage is limited (consistent with the actual needs of short patient training time), but the system only needs to train the "preferred model-parameter groups" (not all models), which significantly reduces the training amount and can be completed in a limited time.

[0080] At the same time, the offline training data set contains "a small amount of poor quality data" and diverse users (different ages, genders, and affected sides), so that the preferred model-parameter groups have certain anti-interference ability and generalization, reducing overfitting to specific data in small samples.

[0081] It should be fully understood that the user information (including but not limited to user physiological information, user personal information, etc.) involved in the present application is information and data authorized by the user or fully authorized by all parties. The use of user information should follow privacy policies and practices that are generally considered to meet or exceed industry or government requirements for maintaining user privacy. The collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards, and provide corresponding operation portals for users to choose authorization or refusal.

[0082] The system, device, module or unit illustrated in the above embodiments can be specifically implemented by a computer chip or entity, or by a product with certain functions. A typical implementation device is a computer. Specifically, the computer may, for example, be a personal computer, a laptop computer, a tablet computer, or a combination of any of these devices.

[0083] The multi-model fusion motor imagery intention recognition system provided by the application is described in detail above, and the principles and implementation manners of the application are described by applying specific examples. The above description of the embodiments is only used to help understand the concept of the application, and should not be understood as a limitation on the protection scope of the application.

Claims

1.A multi-model fusion motor imagery intention recognition system, characterized in that, The application comprises: A training fusion strategy module is configured to obtain an optimal model-parameter group and output weights of each model-parameter pair in the optimal model-parameter group from an offline training data set, wherein the model-parameter pair is a combination of a feature extraction method, a classifier and corresponding parameters; the optimal model-parameter group is obtained by constructing a model-parameter group comprising a plurality of model-parameter pairs, performing matching evaluation on each model-parameter pair in the model-parameter group and each data in the offline training data set to form a model-parameter pair performance feature group, performing cluster analysis on the model-parameter pair performance feature group, and selecting a preset number of model-parameter pairs from each cluster category to form the optimal model-parameter group; and the output weights are calculated based on performance indicators of each model-parameter pair in the optimal model-parameter group. An application fusion strategy module is configured to train each model-parameter pair in the optimal model-parameter group based on an offline clinical data set and real-time clinical online data, input the clinical online data into the trained optimal model-parameter group, perform weighted fusion on the outputs of all model-parameter pairs based on the output weights of each model-parameter pair, and obtain a motor imagery intention recognition result. 2.The multi-model fusion motor imagery intention recognition system according to claim 1, characterized in that, The data quantity in the offline training data set is N0, N0≥N K , and contains N k1 %-N k2 % low-quality data, N K is the lower limit of the data quantity in the data set, N k1 and N k2 are respectively the lower limit and the upper limit of the low-quality data quantity ratio, and the low-quality data is determined by lead standard deviation. 3.The multi-model fusion motor imagery intention recognition system according to claim 1, characterized in that, The feature extraction method in the model-parameter pair comprises at least one of common spatial patterns and filter bank common spatial patterns; and the classifier comprises at least one of a support vector machine, a decision tree classifier, a K nearest neighbor classifier and an EEGNet universal model. 4.The multi-model fusion motor imagery intention recognition system according to claim 1, characterized in that, The parameters comprise preprocessing parameters and model hyperparameters; the preprocessing parameters comprise a time window starting point, a time window length and a filter bandwidth; and the model hyperparameters comprise a feature number, a regularization parameter, a kernel function type and a penalty coefficient. 5.The multi-model fusion motor imagery intention recognition system according to claim 1, wherein, Each element in the model-parameter pair performance feature group is an accuracy rate of a model-parameter pair on corresponding data; if the model-parameter pair cannot converge, it is marked as abnormal. 6.The multi-model fusion motor imagery intention recognition system according to claim 1, wherein, The cluster analysis adopts a K-means clustering algorithm, the number of clusters is N1, N2 model-parameter pairs with the highest average accuracy rate are selected from each cluster, and the total number of elements of the optimal model-parameter group is N1×N2. 7.The multi-model fusion motor imagery intention recognition system according to claim 1, characterized in that, The performance indicators are obtained by averaging the accuracy rates of model-parameter pairs in the same cluster category, and the output weights of each model-parameter pair in the optimal model-parameter group are calculated as follows: ； where a j is the output weight of the jth model-parameter pair in the preferred model-parameter set, b j is the performance indicator of the jth model-parameter pair in the preferred model-parameter set, b o is the performance indicator of the oth model-parameter pair in the preferred model-parameter set, N o is the total number of elements of the preferred model-parameter set. 8.The multi-model fusion motor imagery intention recognition system according to claim 1, wherein, The application fusion strategy module further comprises a conversion formula for converting the fusion result O obtained by the weighted fusion into class coefficients c of the n classification tasks t , and the conversion formula is: ； wherein n is the number of classification categories, int() is an integer function, and n≥2. According to the category coefficient c t , the motion imagination intention recognition result c is obtained, which is expressed as: 。 9.The multi-model fusion motor imagery intention recognition system according to claim 8, characterized in that, When the common K-class motor imagery intention is recognized, the category coefficient c t The correspondence relationship with the recognition result c is represented as: The output range interval (0, K] of the fusion result O is equally divided into K small intervals (i-1, i] (i=1, 2, …, K), and if the category coefficient c falls in the interval (i-1, i], then c=i. t The output range interval (0, K] of the fusion result O is equally divided into K small intervals (i-1, i] (i=1, 2, …, K), and if the category coefficient c falls in the interval (i-1, i], then c=i.

Images