A test-time adaptation method and system for calibration-free brain-computer interface

Abstract

The present invention discloses a test-time adaptation method for a calibration-free brain-computer interface, the method comprising the following steps: obtaining labeled training data from multiple source domain users to establish classification models for multiple categories; establishing the classification model, performing Euclidean alignment on the training data of each user, then merging all source domains, training multiple classification models using traditional cross-entropy loss, and obtaining multiple trained source domain models; obtaining the next test sample on the target domain data stream of the current user, performing incremental Euclidean alignment, and inputting the data into the trained multiple source domain models to obtain predicted probability values based on the multiple source domain models on the target domain data; using a spectral meta-learner method to perform integrated prediction of multiple models to obtain the predicted value of the current test sample. Based on the predicted probability value, each model is optimized separately by minimizing conditional entropy and using adaptive marginal distribution constraints within the batch to obtain optimized models adapted to the target domain; the present invention takes into account the real-time application of the cross-user brain-computer interface system, and adaptively adjusts the model without adding a calibration link.

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface recognition, and in particular to a test-time adaptation method and system for a calibration-free brain-computer interface. Background Art

[0002] The brain, as a complex organ in the human central nervous system, is closely related to human physiological functions and psychological activities. A brain-computer interface (BCI) is a direct interaction pathway between the brain and external devices that can be used to study, assist, enhance, or repair human cognitive or sensorimotor functions. Based on the signal acquisition method, BCI systems can be divided into non-invasive, invasive, and semi-invasive types. Of these three acquisition methods, non-invasive BCI is safer, less expensive, and more convenient to use.

[0003] Electroencephalogram (EEG) is a multi-channel time series and the most commonly used non-invasive BCI input signal, with the characteristics of direct measurement and high temporal resolution. BCI can be divided into active and passive types according to the signal generation method. Active BCI identifies patterns from brain activity, which is directly and consciously controlled by the subject, and the control of the device is independent of external events. Passive BCI extracts the response output to external stimuli from brain activity, and the subject indirectly adjusts the external stimuli to control the application. Common active BCI paradigms include emotional BCI, etc. Common passive BCI paradigms include motor imagery, event-related potentials, and steady-state visual evoked potentials. Motor imagery is the imagination of movements of hands, feet, direction, etc. to control external wheelchairs, drones, etc.

[0004] In practical applications, BCI systems rely on accurate signal decoding, making intelligent and precise analysis of brain signals crucial for their success. However, brain signal analysis presents numerous challenges. For example, for non-invasive BCIs based on scalp EEG, individual variability is a major challenge. This manifests as differences in physiological signals collected from different individuals during the same task. This problem makes it difficult to design algorithms and systems that are uniformly applicable to all individuals. Algorithm models built using training data often perform significantly worse on new users. Currently, a calibration process is typically required for each new subject before use, collecting personalized user data to calibrate the algorithm. This time-consuming and labor-intensive calibration process significantly reduces the usability of BCI systems. The non-stationarity of EEG signals, as well as paradigm-specific issues such as subject fatigue, external interference, and lack of training in motor imagery paradigms, can also complicate decoding. These issues currently hinder the development and implementation of BCI systems. Summary of the Invention

[0005] In order to solve the technical problem of poor compatibility caused by large individual differences in EEG subjects, high calibration cost, and difficulty in real-time decoding during brain-computer interface testing, the present invention proposes a test-time adaptation method for a calibration-free brain-computer interface, the method comprising:

[0006] S1. Obtaining labeled training data from users in multiple source domains to establish classification models for multiple categories;

[0007] S2. Build a classification model. After performing Euclidean alignment on the training data of each user, merge all source domains and train multiple classification models using traditional cross-entropy loss to obtain multiple trained source domain models.

[0008] S3. Obtain the next test sample on the target domain data stream of the current user, perform incremental Euclidean alignment, and input it into multiple trained source domain models to obtain predicted probability values based on the multiple source domain models on the target domain data.

[0009] S4. Use the spectral meta-learner method to perform integrated prediction of multiple models to obtain the predicted value of the current test sample.

[0010] S5. Based on the predicted probability values, optimize each model separately by minimizing conditional entropy and using adaptive marginal distribution constraints within the batch to obtain optimized models adapted to the target domain;

[0011] S6. Repeat steps S3-S5 until the test data flow of the current target domain user is completed.

[0012] The beneficial effects provided by the present invention are: taking into account the real-time application of the cross-user brain-computer interface system, adapting to the target user during testing, and adaptively adjusting the model without adding a calibration link. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the process of the method of the present invention;

[0014] Figure 2 Schematic diagram of the loss function for updating the target domain model in the method of the present invention;

[0015] Figure 3 This chart compares the scenarios of online migration adapted for testing and traditional offline migration.

[0016] Figure 4 This is a performance comparison chart of transfer learning methods in offline and online scenarios on the BNCI2014001 motor imagery EEG dataset;

[0017] Figure 5 This is a performance comparison chart of transfer learning methods in offline and online scenarios on the BNCI2014002 motor imagery EEG dataset;

[0018] Figure 6 This is a performance comparison chart of transfer learning methods in offline and online scenarios on the BNCI2015001 motor imagery EEG dataset.

[0019] Figure 7 This is a performance comparison chart of transfer learning methods under imbalanced label distribution offset on three motor imagery EEG datasets.

[0020] Figure 8 This is a performance comparison chart of continuous domain adaptation scene transfer learning methods on three motor imagery EEG datasets. DETAILED DESCRIPTION

[0021] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0022] Please refer to Figure 1 , Figure 1 It is a process diagram of the method of the present invention.

[0023] The present invention provides a test-time adaptation method for a calibration-free brain-computer interface, comprising the following steps:

[0024] S1. Obtaining labeled training data from users in multiple source domains to establish classification models for multiple categories;

[0025] S2. Build a classification model. After performing Euclidean alignment on the training data of each user, merge all source domains and train multiple classification models using traditional cross-entropy loss to obtain multiple trained source domain models.

[0026] S3. Obtain the next test sample on the target domain data stream of the current user, perform incremental Euclidean alignment, and input it into multiple trained source domain models to obtain predicted probability values based on the multiple source domain models on the target domain data.

[0027] S4. Use the spectral meta-learner method to perform integrated prediction of multiple models to obtain the predicted value of the current test sample.

[0028] S5. Based on the predicted probability values, optimize each model separately by minimizing conditional entropy and using adaptive marginal distribution constraints within the batch to obtain optimized models adapted to the target domain;

[0029] S6. Repeat steps S3-S5 until the test data flow of the current target domain user is completed.

[0030] It should be noted that the implementation scenarios of the present invention are as follows:

[0031] Consider using labeled training data from L source domain users to build a classification model for K categories. The lth source domain (l=1,...,l) includes n s,l Annotated samples Each sample is the two-dimensional matrix of the ith EEG trial, whose dimension is the number of electrode channels ch multiplied by the time sampling point ta, that is, is its corresponding label. n from the target domain user t Test EEG trials The target domain will be sent to the system one by one, and the goal is to correctly predict its corresponding label in an unsupervised and online system. The goal of the test-time adaptation method is to calibrate the model in a completely unsupervised and online manner and correctly predict the corresponding labels of the test data. Specifically, at the test time s, for Prediction Must be completed immediately, using only unlabeled target domain data visible up to the current moment information to make predictions and adjust the model.

[0032] The proposed method maximizes information integration during testing without using any target domain labeled samples, and only uses the predictions of the source domain model on the test data to calibrate the target domain model.

[0033] 1. Source domain model training

[0034] Euclidean alignment is used for each source domain, and all L source domain data that have been aligned are merged as a large source domain, and a total of M source domain models are trained.

[0035] For the l-th source domain (l=1,...,L), the Euclidean alignment calculates the mean of the covariance matrix of all EEG trials of this user:

[0036]

[0037] Then perform the transformation:

[0038]

[0039] All Merged into and is considered as a large source domain, where The corresponding tags are also merged into

[0040] Based on the labeled training data in the source domain, the traditional cross entropy classification loss is optimized through gradient descent back propagation. Train M EEGNet models with randomly initialized parameters

[0041] 2. Incremental Euclidean alignment of target data

[0042] The target domain EEG trials arrive at the system one by one online and are incrementally aligned in an Euclidean manner.

[0043] Specifically, when When it arrives, first update the mean covariance matrix:

[0044]

[0045] Then Perform the transformation:

[0046]

[0047] Then use replace As actual input to the model The data used for classification.

[0048] 3. Target Domain Label Prediction

[0049] The target domain model is based on the above The model is initialized and on each test sample Make predictions upon arrival.

[0050] Assumptions The models were updated to Sample. The next EEG trial When it arrives, it is first converted to Then input into each model f m , obtain M predicted probability value vectors and merge them into Then, ensemble learning is used to obtain the ensemble prediction classification category

[0051] 1) When a≤M, for each predicted value Average to integrate predictions, predict categories That is the category with the largest average probability.

[0052] 2) When a>M, use the spectral meta-learner to weight the integrated prediction value use To represent the predicted value of M models on category k, where δ k Output of the normalized exponential function The spectral meta-learner first calculates the sample covariance matrix of M classifiers

[0053]

[0054]

[0055] Q k be approximately a rank-one matrix, whose principal eigenvector v k will be proportional to the balanced classification accuracy of the M models. Subsequently, a meta-learner is constructed for prediction by weighted summation of the predicted probabilities of the M models:

[0056]

[0057] where v k,m refers to the m-th element of v k .

[0058] 4. Target domain model update

[0059] The target domain model is updated in sliding batches of size B. When the target data is not sufficient to form a batch, i.e., a < B, the target domain model is fixed as When a ≥ B, the model updates for each test sample .

[0060] When arrives at the system, it is first transformed into by incremental Euclidean alignment, and then the model f is updated through the loss function m :

[0061]

[0062] where and represent conditional entropy minimization and adaptive marginal distribution constraint respectively, as follows:

[0063] 1) Conditional entropy minimization: Conditional entropy minimization can reduce the model prediction uncertainty within a single sample, forcing the model to obtain higher-confidence predictions to adapt to the target data. Specifically, the loss function is based on Shannon information entropy to measure the conditional entropy of the predicted probabilities and is averaged within the batch:

[0064]

[0065] 2) Adaptive marginal distribution constraint: Only conditional entropy minimization cannot avoid two situations: an invalid solution where all target domain samples are classified into a fixed class, or the model becomes overconfident about misclassified samples. Therefore, an adaptive marginal distribution constraint within the batch is also added to alleviate the above phenomena. Specifically, the average predicted probability for class k within the batch is:

[0066]

[0067] The estimated target domain category frequency for category k at test time a is:

[0068]

[0069] Where τ is the pseudo-label confidence threshold hyperparameter, τ∈[0.5,1.0].

[0070] The average predicted probability within the recalibration batch is:

[0071]

[0072] Where c is a small positive integer to avoid the denominator being 0 and to avoid an overly unbalanced estimated category ratio when the source domain model is first applied to the target domain.

[0073] Then for q k Normalization:

[0074]

[0075] And used to calculate the adaptive marginal distribution constraints:

[0076]

[0077] According to the above process, the detailed operation process of the present invention is as follows:

[0078] Operation S1: Input labeled training data from L source domain users Initialize M artificial neural network models Parameters; input hyperparameters τ, c, B; initial z k 0; enter the target user's test data

[0079] Operation S2: Training data for L source domain users Perform Euclidean alignment and calculate Then calculate Transform the training data into for in The corresponding tags are also merged into As input to the actual input model;

[0080] Operation S3: Train M artificial neural network models Each model is Optimize the traditional cross entropy loss term until convergence;

[0081] Operation S4: At the current test time a, the currently visible test data Perform incremental Euclidean alignment, computing Transformed into Then use replace As input to the actual input model;

[0082] Operation S5: When a≤M, for each predicted value Average to integrate predictions, predict categories That is, the category with the largest average probability. When a>M, the spectral meta-learner is used to weight the integrated prediction value. use To represent the predicted value of M models on category k, where δ k Output of the normalized exponential function The kth element of . First calculate the sample covariance matrix of M classifiers O k The principal eigenvector v k The value of will be proportional to the balanced classification accuracy of the M models. The meta-learner is then constructed by weighted summation of the prediction probabilities of the M models to make predictions: where v k,m Refers to v k Output the result;

[0083] Operation S6: Update the target domain category frequency z for category k estimated at test time a k for

[0084] Operation S7: If a ≥ B, calculate the conditional entropy within the batch Calculate the average predicted probability for class k within the batch The average predicted probability within the recalibration batch is calculated as Then for q k Normalization And used to calculate the adaptive marginal distribution constraints Then through the loss function Update model f m ;

[0085] Operation S8: Enter the next test time a+1, repeat steps S4 to S7 until a=n t , output all results and end the operation.

[0086] The solution of this application is Test-Time Information Maximization Ensemble (T-TIME), which is mainly based on the source domain EEGNet model trained based on traditional cross-entropy loss after merging multiple Euclidean aligned data. The ensemble prediction is tested by the spectral meta-learner when the target domain is tested online, and the joint loss term calibration optimization model with conditional entropy minimization and adaptive marginal distribution constraints is used to complete real-time prediction and online calibration.

[0087] refer to Figure 3 , Figure 3 A comparison of online migration using test-time adaptation and traditional offline migration methods is presented. In the standard TL scenario, the unsupervised domain adaptation method considers using data from both the source and target domains for offline iterative training to build the model; considering the protection of source domain data privacy, the source-free domain adaptation method uses the source domain model to avoid directly accessing or storing source domain data, but still requires offline training; to meet the needs of online real-time scenarios, the test-time adaptation method uses online updates to adjust the model to adapt to the target domain distribution, and completes real-time decision feedback and update calibration for online test samples / batches.

[0088] Figure 4-6 The results of cross-user (leave-one-user-out-of-the-box validation) experiments using this patented solution on the BNCI2014001, BNCI2014002, and BNCI2015001 datasets are presented. These experiments cover three scenarios: no migration, offline migration, and online migration. These experiments encompass traditional methods, deep learning methods, data alignment methods, unsupervised domain adaptation methods, passive domain adaptation methods, and test-time adaptation methods. The best offline TL accuracy is marked with an *, the best online TL accuracy is bolded, and the second-best is underlined. The experimental results show that T-TIME outperforms other cutting-edge test-time adaptation methods in online scenarios and can approach the migration performance of offline scenarios when integrated. Figure 7 The performance comparison of the transfer learning method of this patented solution under the imbalanced label distribution offset on the above three motor imagery EEG datasets is shown. The best accuracy is marked in bold and the second best is underlined. Figure 8 The patented solution is compared with the continuous domain adaptation method performance on the three motor imagery EEG datasets mentioned above. It shows that T-TIME has a stable improvement over cutting-edge methods in various scenarios and is suitable for complex practical application scenarios of brain-computer interface systems.

[0089] The beneficial effects of the present invention are: taking into account the real-time application of the cross-user brain-computer interface system, adapting to the target user during testing, and adaptively adjusting the model without adding a calibration link.

[0090] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A test-time adaptation method for a calibration-free brain-computer interface, characterized by: The following steps are involved: S1. Obtain the labeled training data from L source domain users to build a classification model for K categories. The lth source domain (l=1,...,L) includes n s,l Annotated samples Each sample is the two-dimensional matrix of the ith EEG trial, whose dimension is the number of electrode channels ch multiplied by the time sampling point ts, that is, is its corresponding label; S2. Build a classification model, perform Euclidean alignment on the training data of each user, merge all source domains, and train multiple EEGNet classification models using traditional cross-entropy loss to obtain multiple trained source domain models. S3. Obtain the next test sample on the target domain data stream of the current user, perform incremental Euclidean alignment, and input it into multiple trained source domain models to obtain predicted probability values based on the multiple source domain models on the target domain data. S4. Use the spectral meta-learner method to perform integrated prediction of multiple models to obtain the predicted value of the current test sample; S5. Based on the predicted probability values, optimize each model separately by minimizing conditional entropy and using adaptive marginal distribution constraints within the batch to obtain optimized models adapted to the target domain; S6. Repeat steps S3-S5 until the test data flow of the current target domain user is completed.

2. The test-time adaptation method for a calibration-free brain-computer interface according to claim 1, characterized in that: Euclidean alignment is applied to each source domain, and all L aligned source domain data are merged into a large source domain; step S2 is as follows: S21. For the l-th source domain (l=1,...,L), calculate the mean of the covariance matrix of all EEG trials of this user by Euclidean alignment: S22, then transform: S23, all Merged into and is considered as a large source domain, where The corresponding tags are also merged into 3. The test-time adaptation method for a calibration-free brain-computer interface according to claim 2, characterized in that: Based on the labeled training data in the source domain, the cross entropy loss is optimized by gradient descent back propagation. Train M EEGNet artificial neural network classification models with randomly initialized parameters 4. The test-time adaptation method for a calibration-free brain-computer interface according to claim 3, characterized in that: Step S4 is specifically as follows: S41, the target domain model is based on the above The model is initialized and on each test sample Make predictions after arrival; assume The models were updated to Sample; the next EEG trial When it arrives, first use the above incremental EA to convert to Then input into each model f m , obtain M predicted probability value vectors and merge them into Then, ensemble learning is used to obtain the ensemble prediction classification category S42. When a≤M, for each predicted value Average to integrate predictions, predict categories That is the category with the largest average probability; S43. When a>M, use the spectral meta-learner to weight the integrated prediction value use To represent the predicted value of M models on category k, where δ k Output of the normalized exponential function The kth element of ; the spectral meta-learner first calculates the sample covariance matrix of M classifiers Q k will approximate a rank-one matrix whose principal eigenvector v k The value of will be proportional to the balanced classification accuracy of the M models; the meta-learner is then constructed by weighted summation of the prediction probabilities of the M models to make predictions: where v k,m Refers to v k The mth element of .

5. The test-time adaptation method for a calibration-free brain-computer interface according to claim 4, characterized in that: Step S5 is specifically as follows: S51. The target domain model is updated in sliding batches of size B; when the target data is not sufficient to form a batch, i.e., a < B, the target domain model is fixed as When a ≥ B, the model processes each test sample and updates it; when it arrives at the system, it is first converted to S52, Conditional Entropy Minimization: The loss function is based on Shannon information entropy to measure the conditional entropy of the predicted probability and averaged within the batch: S53, Adaptive marginal distribution constraint: The average predicted probability for category k within a batch is: The estimated target domain category frequency for category k at test time a is: Where τ is the pseudo-label confidence threshold hyperparameter, τ∈[0.5,1.0]; The average predicted probability within the recalibration batch is: Where c is a small positive integer to avoid the denominator being 0 and to avoid an overly unbalanced estimated category ratio when the source domain model is first applied to the target domain; Then for q k Normalization: And used to calculate the adaptive marginal distribution constraints: S54, then update the model f through the loss function m : in and denote conditional entropy minimization and adaptive marginal distribution constraints, respectively.

Images