Asynchronous motor imagery recognition method integrating multi-model output
By integrating the outputs of multiple models, the EEGNet deep learning framework was used to train multiple motor imagery classification models in parallel and introduce resting-state data to construct a weight matrix and perform dynamic optimization. This solved the problems of insufficient adaptive capability and recognition accuracy of asynchronous motor imagery brain-computer interfaces, and achieved stable and secure user interaction.
Patent Information
- Application Number
- CN202511756652.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-27
- Publication Date
- 2026-03-03
AI Technical Summary
Existing asynchronous motion imagery brain-computer interface technology has shortcomings in terms of adaptability and recognition accuracy, making it difficult to achieve stable and secure user interaction in practical applications.
A method integrating multiple model outputs is adopted, which trains multiple motion image classification models in parallel using the EEGNet deep learning framework, and introduces resting state data as independent categories. The initial weight matrix is constructed using the category conditional probability matrix, and unsupervised weight updates are performed in combination with the information maximization criterion, thereby realizing the weighted fusion and dynamic optimization of multiple models.
It improves the recognition accuracy and stability of the system in asynchronous environments, enhances its adaptability to individual differences and signal fluctuations, reduces the risk of misjudgment, improves the security and practicality of interaction, and reduces the reliance on human experience and labeled data.
Smart Images
Figure CN121598199A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of brain-computer interface recognition technology, specifically to an asynchronous motor imagery recognition method that integrates multiple model outputs. Background Technology
[0002] Motor imagery brain-computer interfaces, as an important development direction of human-computer interaction, have shown broad application prospects in the fields of rehabilitation medicine and intelligent control. Current technology has made significant progress in the synchronous paradigm, which can realize the recognition of motor intentions within a fixed time window. With the continuous expansion of application scenarios, the research focus is gradually shifting to the asynchronous interaction paradigm, which is more in line with actual needs. This paradigm allows users to initiate control commands autonomously, which is closer to the natural interaction method.
[0003] In recent years, multi-model fusion methods have attracted attention for their ability to improve system stability. They enhance recognition robustness by integrating the outputs of multiple classifiers. At the same time, accurate discrimination of uncontrolled states has also become a research focus, which is of great significance for ensuring system security.
[0004] Existing research has actively explored model fusion strategies and resting-state recognition, laying a solid foundation for the development of asynchronous brain-computer interface technology. Against this backdrop, further improving the system's adaptive capability and recognition accuracy in asynchronous environments has become the key to promoting the practical application of this technology. Summary of the Invention
[0005] The purpose of this invention is to address the problems existing in the background technology by proposing an asynchronous motion image recognition method that integrates multiple model outputs.
[0006] The technical solution of this invention: an asynchronous motion image recognition method integrating multiple model outputs, comprising the following specific implementation steps:
[0007] S1. Collect EEG signals for multiple types of motor imagery tasks and resting state tasks through synchronous experiments;
[0008] S2. Preprocess the acquired raw EEG signals, including bandpass filtering, artifact removal, segmentation, and normalization.
[0009] S3. Based on the EEGNet deep learning framework, several independent motion imagery classification models are trained in parallel by setting different network structure parameters. Each model introduces resting state data as an independent category during training and outputs a probability distribution that includes all motion imagery categories and resting state.
[0010] S4. During the synchronous debugging phase, the prediction results of each model are obtained according to the known system instructions, the category conditional probability matrix of each model is calculated, and this matrix is used as the initial weight matrix for the fusion of the outputs of each model.
[0011] S5. In the online recognition stage, real-time EEG data is acquired through a sliding window. The prediction probabilities of multiple models are weighted and fused using the initial weight matrix to obtain a comprehensive prediction result. A loss function is constructed based on the information maximization criterion. The fusion weights are dynamically updated using an unsupervised gradient descent method to achieve stable recognition and adaptive optimization in an asynchronous environment.
[0012] Preferably, the preprocessing in step S2 also includes standardizing the number of channels and the length of sampling points for all samples to ensure that the data format of different subjects is consistent.
[0013] Preferably, the frequency range of the bandpass filter in step S2 is from 0.5 Hz to 40 Hz.
[0014] Preferably, the construction of the motion imagery classification model in step S3 is achieved by adjusting the kernel size of the temporal convolutional layer and the separable convolutional layer in the EEGNet model;
[0015] Several motion imagery classification models include at least three models with different parameter configurations, wherein the first model has a temporal convolutional layer kernel size of 1×8 and a separable convolutional layer kernel size of 1×8, the second model has a temporal convolutional layer kernel size of 1×64 and a separable convolutional layer kernel size of 1×16, and the third model has a temporal convolutional layer kernel size of 1×128 and a separable convolutional layer kernel size of 1×32.
[0016] Preferably, in step S4, the element P(i|j) in the category conditional probability matrix represents the probability that the model actually belongs to category i when it predicts that the category is j, reflecting the model's discriminative ability and stability under each category.
[0017] Preferably, the specific process of weighted fusion in step S5 is as follows: for each sliding window of data, each model outputs the predicted probability distribution of the category, and the predicted probabilities of each model are weighted and fused based on the initial weight matrix to calculate the comprehensive predicted probability. Finally, the predicted category is determined by taking the maximum value of the comprehensive predicted probability.
[0018] Preferably, the specific implementation of the information maximization criterion in step S5 is as follows: take multiple sliding window data as a sliding batch, calculate the average entropy of the comprehensive prediction distribution and the sample prediction entropy within the batch, and construct the information maximization objective loss function based on this.
[0019] Preferably, the fusion weights of each model are updated unsupervised by minimizing the information-maximizing target loss function using gradient descent, and the updated weights are used for fusion prediction in the next time step.
[0020] Preferably, step S5 further includes:
[0021] When the final prediction category of the comprehensive prediction result is resting state, no control command is output;
[0022] When the final predicted category is the motion imagination category, the corresponding control command is generated and output.
[0023] The technical solution of this invention: An asynchronous motion imagery recognition system integrating multiple model outputs, used to execute the aforementioned asynchronous motion imagery recognition method integrating multiple model outputs, comprising:
[0024] The signal acquisition module is used to acquire and cache the subject's motor imagery and resting-state EEG signals in real time.
[0025] The preprocessing module is used to call filtering, segmentation and normalization algorithms to process the raw EEG signals and output standardized data;
[0026] The multi-model prediction module is used to load several pre-trained motion imagery classification models, receive standardized data, and output the class prediction probability distribution of each model.
[0027] The synchronous debugging and weight fusion module is used to execute the synchronous test process, calculate the initial weight matrix of each model, and perform weighted fusion of the online output based on the matrix to generate a comprehensive prediction result.
[0028] The information maximization weight fine-tuning module is used to calculate the average entropy within the sliding batch and the sample prediction entropy, construct the information maximization target loss function, and minimize the function through gradient descent to achieve adaptive update of the fusion weights.
[0029] The asynchronous output module is used to receive the comprehensive prediction results. It does not generate control commands when the final prediction category is resting state, and generates and outputs the corresponding commands when the final prediction category is motion imagination.
[0030] Compared with the prior art, the above-mentioned technical solution of the present invention has the following beneficial technical effects:
[0031] This invention designs an asynchronous motion imagery recognition method that integrates multiple model outputs. By training multiple structurally differentiated EEGNet models in parallel and introducing the resting state as an independent category, the coverage and discriminative ability of motion imagery recognition are effectively improved. Utilizing the complementarity of multiple model outputs significantly enhances the system's adaptability and robustness to individual differences and signal fluctuations among subjects. By constructing a category conditional probability matrix as initial fusion weights during the synchronous debugging phase, a reliable fusion foundation is provided for online recognition, effectively reducing the risk of misjudgment due to the limitations of a single model. In the online phase, a sliding window is used for real-time processing, combining the most relevant information... The unsupervised dynamic weight update mechanism of the maximization criterion enables the system to continuously optimize the fusion strategy in an asynchronous environment, adapting to the non-stationary characteristics of EEG signals and changes in user state, thereby achieving stable control of the recognition process and consistency of output. At the same time, by automatically distinguishing between motor imagery state and resting state, this invention avoids erroneous command outputs due to non-intentional control, improving the safety and practicality of brain-computer interface interaction. The entire process achieves fully automatic weight learning and adjustment, reducing reliance on human experience and large amounts of labeled data, lowering the complexity of system deployment and maintenance, and effectively improving the recognition accuracy and stability of asynchronous motor imagery brain-computer interface. Attached Figure Description
[0032] Figure 1 This is a flowchart illustrating an asynchronous motion image recognition method that integrates multiple model outputs proposed in this invention.
[0033] Figure 2 This is a schematic diagram of the synchronous debugging process proposed in this invention;
[0034] Figure 3 This is a schematic diagram of the architecture of an asynchronous motion image recognition system that integrates multiple model outputs proposed in this invention.
[0035] Figure labels: Label 1, labeln: different motion image category labels; Model1, Modelm: multiple motion image classification models running in parallel; Result 1, Result m: the predicted output of each model on the input data. Detailed Implementation
[0036] Example 1: This invention proposes an asynchronous motion imagery recognition method that integrates multiple model outputs, such as... Figure 1 As shown, the specific implementation steps include the following:
[0037] S1. Conduct a synchronous motor imagery experiment on the subjects and collect EEG data; the experiment includes several motor imagery tasks (e.g., left hand, right hand, foot, tongue, etc.) and resting state tasks; each task is set with a fixed duration and cue stimulus; EEG signals are acquired through standard EEG acquisition equipment (e.g., a 16-channel EEG system).
[0038] S2. Preprocess the collected raw EEG signals, including bandpass filtering (e.g., 0.5-40Hz), artifact removal, segmentation, and normalization; to ensure that the data format of different subjects is consistent, the number of channels and the length of sampling points are standardized for all samples.
[0039] S3. Based on the EEGNet deep learning framework, multiple independent motion image classification models are trained in parallel by setting different network structure parameters, initialization parameters, or hyperparameters. Resting-state data is introduced as an independent category during training, enabling the model to recognize resting states. Specifically:
[0040] Each EEGNet model includes the following layers:
[0041] Input layer: Receives preprocessed, standardized data; the input dimension is C×T.
[0042] Where C is the number of EEG channels; T is the number of sampling points per trial.
[0043] Temporal convolutional layer: A one-dimensional temporal convolutional kernel (size 1×k1) is used to extract the temporal features of the EEG signal, where k1 is the kernel length;
[0044] Batch normalization layer: Normalizes the convolutional output to stabilize the training process;
[0045] Depth-separable spatial convolutional layer: using a convolutional kernel size of 1×C to extract spatial features of EEG signals;
[0046] Average pooling layer: downsamples the time dimension to reduce the feature dimension;
[0047] Dropout layer: Randomly discards some neurons during the training phase to prevent overfitting;
[0048] Separable convolutional layers: using a convolutional kernel size of 1×k2 to further fuse temporal and spatial features;
[0049] Fully connected layer: Uses the Softmax activation function to output the category probability distribution, which includes multiple motor imagery categories (such as left hand, right hand, foot, tongue) and resting state;
[0050] Train models with at least three different parameter configurations by adjusting the kernel size of temporal convolutional layers and separable convolutional layers;
[0051] For example, EEGNet-A: temporal convolutional layer kernels are 1×8, and separable convolutional layer kernels are 1×8; EEGNet-B: temporal convolutional layer kernels are 1×64, and separable convolutional layer kernels are 1×16; EEGNet-C: temporal convolutional layer kernels are 1×128, and separable convolutional layer kernels are 1×32.
[0052] Each model is trained independently using preprocessed, standardized data (including resting state categories), and the loss function (such as cross-entropy loss) is optimized so that the model can output the probability distribution of each motion image category and resting state.
[0053] S4. In the synchronous testing phase, the prediction results of each model are obtained according to the known instructions. The class conditional probability matrix of each model is calculated. This matrix represents the degree of correlation between the model prediction and the actual class, and is used as the initial weight matrix for the fusion of the outputs of each model, such as... Figure 2 As shown, specifically:
[0054] Subjects were instructed to complete various motion imagery and resting tasks in sequence according to system instructions, while the predicted output and actual labels of each model were recorded.
[0055] For each model, calculate its class conditional probability matrix P(i|j), where P(i|j) represents the probability that the model actually belongs to class i when it predicts class j. This matrix reflects the model's discriminative ability and stability in each class.
[0056] The class conditional probability matrix of each model is used as its initial weight matrix, i.e., the weight matrix w of the k-th model in subsequent output fusion. k This is used for weighted fusion in the subsequent online phase.
[0057] S5. In the online phase, real-time EEG data is acquired through a sliding window. The predicted probabilities of multiple models are weighted and fused using initial weights to obtain a comprehensive result. A loss function is constructed based on the information maximization criterion, and the fusion weights are updated adaptively and unsupervised to achieve dynamic optimization and stable output control during the asynchronous recognition process. Specifically:
[0058] The continuously acquired EEG signals were segmented using the sliding window method, with each window having the same data dimension as the training data (C×T); the total number of sliding windows is B, and the index of each window is t (t=1,2,...,B);
[0059] Each window of data is input into multiple trained motion imagery classification models, and each model outputs a predicted probability distribution p for the category. k That is, the probability vector of the k-th model;
[0060] Based on the initial weight matrix w obtained during the synchronous debugging phase kThe predicted probabilities of each model are weighted and fused to calculate the comprehensive predicted probability P. t (i):
[0061] Where M represents the total number of models; n is the number of categories (including resting states);
[0062] Final prediction category Determined by taking the maximum probability value:
[0063] If the fusion result is a resting state, the system does not output control commands; otherwise, it outputs control commands corresponding to the motion imagery category.
[0064] To optimize the fusion weights, an information maximization criterion is introduced:
[0065] Multiple sliding window data are treated as a single sliding batch (with B windows), and the average entropy of the comprehensive prediction distribution and the sample prediction entropy within the batch are calculated to construct the information maximization objective loss function, Loss.
[0066]
[0067] Where H(·) represents the information entropy calculation function; This represents the average value of the comprehensive prediction distribution within the batch;
[0068] The loss function Loss is minimized using gradient descent, and the fusion weights w for each model are calculated. k Perform unsupervised updates:
[0069] Where η represents the learning rate;
[0070] Updated weight w′ k Fusion predictions for the next time step enable dynamic optimization.
[0071] Example 2, as Figure 3 As shown, the present invention proposes an asynchronous motion image recognition system that integrates multiple model outputs. It is used to execute the asynchronous motion image recognition method that integrates multiple model outputs proposed in Embodiment 1. The system includes: a signal acquisition module, a preprocessing module, a multi-model prediction module, a synchronous debugging and weight fusion module, an information maximization weight fine-tuning module, and an asynchronous output module.
[0072] The signal acquisition module acquires and caches the subject's motor imagery and resting-state EEG signals in real time;
[0073] The preprocessing module calls filtering, segmentation, and normalization algorithms to process the raw EEG signals and output standardized data.
[0074] The multi-model prediction module loads several pre-trained motion imagery classification models, receives standardized data, and outputs the predicted probability distribution of each model category.
[0075] The synchronous debugging and weight fusion module executes the synchronous test process, calculates the initial weight matrix of each model, and performs weighted fusion of the online output based on the matrix to generate a comprehensive prediction result.
[0076] The information maximization weight fine-tuning module calculates the average entropy within the sliding batch and the sample prediction entropy, constructs the information maximization target loss function, minimizes the function through gradient descent, and achieves adaptive updating of the fused weights.
[0077] The asynchronous output module receives the comprehensive prediction results, does not generate control commands in the "resting state", and generates and outputs corresponding commands in the motion imagination category.
[0078] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.
Claims
1. An asynchronous motion imagery recognition method integrating multiple model outputs, characterized in that, The specific implementation steps include the following: S1. Collect EEG signals for multiple types of motor imagery tasks and resting state tasks through synchronous experiments; S2. Preprocess the acquired raw EEG signals, including bandpass filtering, artifact removal, segmentation, and normalization. S3. Based on the EEGNet deep learning framework, several independent motion imagery classification models are trained in parallel by setting different network structure parameters. Each model introduces resting state data as an independent category during training and outputs a probability distribution that includes all motion imagery categories and resting state. S4. During the synchronous debugging phase, the prediction results of each model are obtained according to the known system instructions, the category conditional probability matrix of each model is calculated, and this matrix is used as the initial weight matrix for the fusion of the outputs of each model. S5. In the online recognition stage, real-time EEG data is acquired through a sliding window. The prediction probabilities of multiple models are weighted and fused using the initial weight matrix to obtain a comprehensive prediction result. A loss function is constructed based on the information maximization criterion. The fusion weights are dynamically updated using an unsupervised gradient descent method to achieve stable recognition and adaptive optimization in an asynchronous environment.
2. The asynchronous motion imagery recognition method based on integrated multi-model outputs according to claim 1, characterized in that, The preprocessing in step S2 also includes standardizing the number of channels and the length of sampling points for all samples to ensure that the data format of different subjects is consistent.
3. The asynchronous motion imagery recognition method based on integrated multi-model outputs according to claim 2, characterized in that, The frequency range of the bandpass filter in step S2 is from 0.5 Hz to 40 Hz.
4. The asynchronous motion imagery recognition method based on integrated multi-model outputs according to claim 3, characterized in that, In step S3, the construction of the motion imagery classification model is achieved by adjusting the kernel size of the temporal convolutional layer and the separable convolutional layer in the EEGNet model; Several motion imagery classification models include at least three models with different parameter configurations, wherein the first model has a temporal convolutional layer kernel size of 1×8 and a separable convolutional layer kernel size of 1×8, the second model has a temporal convolutional layer kernel size of 1×64 and a separable convolutional layer kernel size of 1×16, and the third model has a temporal convolutional layer kernel size of 1×128 and a separable convolutional layer kernel size of 1×32.
5. The asynchronous motion imagery recognition method based on integrated multi-model outputs according to claim 4, characterized in that, In step S4, the element P(i|j) in the category conditional probability matrix represents the probability that the model actually belongs to category i when it predicts that the category is j, reflecting the model's discriminative ability and stability under each category.
6. The asynchronous motion imagery recognition method based on integrated multi-model outputs according to claim 5, characterized in that, The specific process of weighted fusion in step S5 is as follows: For each sliding window of data, each model outputs the predicted probability distribution of the category. The predicted probabilities of each model are weighted and fused based on the initial weight matrix to calculate the comprehensive predicted probability. Finally, the predicted category is determined by taking the maximum value of the comprehensive predicted probability.
7. The asynchronous motion imagery recognition method based on integrated multi-model outputs according to claim 6, characterized in that, The specific implementation of the information maximization criterion in step S5 is as follows: take multiple sliding window data as a sliding batch, calculate the average entropy of the comprehensive prediction distribution and the sample prediction entropy within the batch, and construct the information maximization objective loss function based on this.
8. The asynchronous motion imagery recognition method based on integrated multi-model outputs according to claim 7, characterized in that, The objective loss function is minimized by using gradient descent to minimize information. The fusion weights of each model are then updated unsupervised, and the updated weights are used for fusion prediction in the next time step.
9. The asynchronous motion imagery recognition method based on integrated multi-model outputs according to claim 8, characterized in that, Step S5 also includes: When the final prediction category of the comprehensive prediction result is resting state, no control command is output; When the final predicted category is the motion imagination category, the corresponding control command is generated and output.
10. An asynchronous motion imagery recognition system integrating multiple model outputs, used to execute the asynchronous motion imagery recognition method integrating multiple model outputs as described in any one of claims 1 to 9, characterized in that, include: The signal acquisition module is used to acquire and cache the subject's motor imagery and resting-state EEG signals in real time. The preprocessing module is used to call filtering, segmentation and normalization algorithms to process the raw EEG signals and output standardized data; The multi-model prediction module is used to load several pre-trained motion imagery classification models, receive standardized data, and output the class prediction probability distribution of each model. The synchronous debugging and weight fusion module is used to execute the synchronous test process, calculate the initial weight matrix of each model, and perform weighted fusion of the online output based on the matrix to generate a comprehensive prediction result. The information maximization weight fine-tuning module is used to calculate the average entropy within the sliding batch and the sample prediction entropy, construct the information maximization target loss function, and minimize the function through gradient descent to achieve adaptive update of the fusion weights. The asynchronous output module is used to receive the comprehensive prediction results. It does not generate control commands when the final prediction category is resting state, and generates and outputs the corresponding commands when the final prediction category is motion imagination.