Motion imagery training and online decoding methods and devices

Abstract

This disclosure provides a method and apparatus for training and online decoding of motor imagery. The method includes: dynamically determining a task difficulty threshold for the current trial based on the subject's task completion rate in previous trials; obtaining a prediction result for the corresponding motor imagery category using a decoding model for classifying motor imagery categories based on EEG data collected by the subject during the current trial; determining whether the current trial corresponds to valid motor imagery based on a comparison between the prediction result and the task difficulty threshold; providing feedback corresponding to valid motor imagery in response to determining that the current trial corresponds to valid motor imagery; updating the decoding model after the current trial ends; repeating the above steps to complete a current training session including multiple trials, and performing trial-level updates to the decoding model based on the collected corresponding EEG data. The method according to embodiments of this disclosure can avoid performance fluctuations and catastrophic forgetting caused by blind updates.

Description

Technical Field

[0001] This disclosure relates to the field of brain-computer interface technology, specifically to a method and apparatus for motor imagery training and online decoding. Background Technology

[0002] Brain-computer interfaces (BCIs) establish a communication bridge between the human brain and external devices. Motor imagery (MI), an important BCI paradigm, refers to the process of mentally rehearsing movement without actually performing physical actions; the brain activation patterns during motor imagery are similar to those during actual movement. Significant differences exist in individual motor imagery abilities, and some subjects struggle to perform motor imagery successfully and effectively. Furthermore, due to the low signal-to-noise ratio and non-stationarity of electroencephalogram (EEG) signals, it is difficult to design suitable models for accurate decoding.

[0003] The execution of the motor imagery task can be viewed as a process of close collaboration between the subject and the decoding model, and its effectiveness is greatly influenced by two key aspects: first, an effective MI training method to help subjects learn to generate separable MI EEG data; and second, a robust decoding model to correctly classify EEG samples.

[0004] To improve online decoding accuracy, various MI training methods have been proposed to help subjects learn to generate separable MI EEG data. Currently, two main types of training methods are often used to enhance MI performance: First, designing MI guidance strategies to enhance activation of relevant brain regions during MI. For example, providing electrical or vibrational stimulation before or during MI for guidance. Second, introducing feedback into MI training to help subjects self-regulate their neural responses. When the subject's motor intention is recognized or their MI performance reaches a certain threshold, different types of feedback are provided, such as visual feedback, tactile feedback based on electrical / vibrational stimulation, and proprioceptive feedback based on exoskeleton assistance, to promote self-regulation.

[0005] However, in most feedback-based MI training methods, the difficulty of the MI task is pre-set. Because MI performance varies significantly between different subjects, and even within the same subject at different time points, the pre-defined task difficulty often does not match the subject's actual performance. This inevitably makes it difficult for subjects to effectively engage in training, and may even lead to MI task failure.

[0006] On the other hand, researchers have designed a variety of decoding models. In practical applications, decoding models are typically trained during the calibration phase, and the calibrated parameters remain fixed during subsequent decoding. Since the subjects' states inevitably change, decoding models with fixed parameters often perform poorly in practice. Summary of the Invention

[0007] One of the purposes of this disclosure is to provide a method for motor imagery training and online decoding that can increase the number of effective trials.

[0008] One of the purposes of this disclosure is to provide a method for motor imagery training and online decoding that can avoid performance fluctuations and catastrophic forgetting caused by blind updates.

[0009] According to one aspect of this disclosure, a method for training and online decoding of motor imagery is provided, applied to a brain-computer interface, and includes: dynamically determining a task difficulty threshold for the current trial based on the subject's task completion level in previous trials; obtaining a prediction result for the corresponding motor imagery category using a decoding model for classifying motor imagery categories based on EEG data collected by the subject in the current trial; determining whether the current trial corresponds to valid motor imagery based on a comparison between the prediction result and the task difficulty threshold; providing feedback corresponding to valid motor imagery in response to determining that the current trial corresponds to valid motor imagery; updating the decoding model after the current trial ends; repeating the above steps to complete a current training session including multiple trials, wherein, after each trial of the current training session, the decoding model is updated at the trial level based on the collected corresponding EEG data.

[0010] Optionally, after the current training session ends, the decoding model can be updated at the session level based on the EEG data of the current training session and the EEG data of the historical training sessions.

[0011] Optionally, the step of dynamically determining the task difficulty threshold for the current trial based on the subject's task completion rate in previous trials may include: obtaining the predicted probabilities of the subject in previous trials output by the decoding model; determining the candidate task difficulty threshold based on the predicted probabilities; and using an exploration-exploitation strategy to determine whether to update the task difficulty threshold for the current trial to the candidate task difficulty threshold.

[0012] Optionally, the step of using an exploration-exploitation strategy to determine whether to update the current task difficulty threshold to a candidate task difficulty threshold may include: calculating the average task completion rate of the subject in previous trials and setting the average as the exploration probability; generating a random number between zero and one; updating the task difficulty threshold of the current trial to a candidate task difficulty threshold in response to the random number being less than the exploration probability; and maintaining the task difficulty threshold unchanged in response to the random number being greater than or equal to the exploration probability.

[0013] Optionally, in response to the fact that the task completion rate in multiple consecutive trials in the previous multiple trials is lower than the preset lower limit, the task difficulty threshold of the current trial can be reset to a preset safe initial value, which is between 0.4 and 0.6.

[0014] Optionally, the trial-level update of the decoding model may include: constructing a current category training set using EEG data collected in the current trial, the current category training set containing all samples in the current trial and their true category labels; constructing a center-guided training set for each category other than the current motor imagery category from historical conversation data, the center-guided training set containing multiple historical samples whose features are closest to the category center in the corresponding category; supervising the decoding model based on the current category training set to optimize its ability to discriminate the current category; applying knowledge distillation constraints to the decoding model based on the center-guided training set to retain its ability to discriminate other historical categories; and weighting and summing the supervised training loss and the knowledge distillation loss as the overall optimization objective for the trial-level update, used to fine-tune the decoding model, wherein the supervised training loss is calculated based on the difference between the true category labels of each sample in the current category training set and the prediction results of the decoding model; and the knowledge distillation loss is calculated based on the distribution difference between the softened output generated by the decoding model for each sample in the center-guided training set before the update and the output of the model to be updated for the same sample.

[0015] Optionally, the session-level update of the decoding model may include: using the EEG data corresponding to the current training session as the target domain data and the EEG data corresponding to the historical training sessions as the source domain data; constructing a representative prototype for each motor imagery category based on the source domain data; calculating the contrast loss, domain alignment loss, and classification loss using the representative prototype, target domain data, and source domain data; and weighting and merging the contrast loss, domain alignment loss, and classification loss as the overall optimization objective for the session-level update, which is used to further fine-tune the decoding model.

[0016] Optionally, the representative prototype can be obtained by averaging the feature vectors of all samples belonging to the same motion image category in the source domain data in the feature space of the decoding model; the contrast loss can be generated by calculating the matching degree between the target domain data and each representative prototype, which encourages the features of similar samples to be close to the corresponding prototype and the features of dissimilar samples to be far away from the non-corresponding prototype; the domain alignment loss can be generated by measuring the overall distribution difference between the source domain data and the target domain data in the feature space; the classification loss can be calculated based on the difference between the true category label of the target domain data and the prediction result of the decoding model.

[0017] Optionally, the motor imagery training and online decoding method may include providing the subject with feedback corresponding to the comparison results.

[0018] According to a second aspect of this disclosure, a motor imagery training and online decoding device is provided, which is applied to a brain-computer interface and includes a memory storing a program or instructions that, when executed by a processor, cause the processor to perform the motor imagery training and online decoding method as described above.

[0019] The motion imagery training and online decoding method according to embodiments of this disclosure can effectively suppress cross-session performance fluctuations and improve long-term stability.

[0020] The motion imagery training and online decoding method according to embodiments of this disclosure requires no additional annotation or user intervention and can achieve fully online adaptation. Attached Figure Description

[0021] The above and other objects and features of exemplary embodiments of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings, which exemplarily illustrate the embodiments, wherein:

[0022] Figure 1 This is a flowchart illustrating a motion imagery training and online decoding process according to an exemplary embodiment of the present disclosure;

[0023] Figure 2 The complete process of motion imagery training and online decoding according to an example embodiment of this disclosure is shown;

[0024] Figure 3 This is a flowchart of a motion imagery training and online decoding method according to an exemplary embodiment of the present disclosure;

[0025] Figure 4 An online update process for a decoding model according to an example embodiment of this disclosure is illustrated;

[0026] Figure 5 A flowchart illustrating a trial-level update of a decoding model according to an example embodiment of the present disclosure is shown;

[0027] Figure 6 A flowchart illustrating a session-level update of a decoding model according to an example embodiment of this disclosure is shown;

[0028] Figure 7 The accuracy comparison results of the scheme according to the example embodiment of this disclosure and the comparative scheme are shown;

[0029] Figure 8 The effect of threshold adaptive adjustment in a motion imagery training method according to an example embodiment of the present disclosure is shown;

[0030] Figure 9 The average accuracy of different methods varies with the number of trials; and

[0031] Figure 10 The average accuracy for each category of different methods is shown as a function of trials. Detailed Implementation

[0032] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, examples of which are illustrated in the drawings, wherein the same reference numerals always refer to the same parts. The embodiments will now be described with reference to the accompanying drawings in order to explain this disclosure.

[0033] The following detailed description is provided to aid in obtaining a full understanding of the methods, apparatus, and / or systems described herein. However, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein; equivalent substitutions or changes may be made, except for operations that must occur or be performed in a specific order. Furthermore, for clarity and conciseness, descriptions of content well-known in the art will be omitted or simplified.

[0034] Unless otherwise specified, the same reference numerals generally refer to the same elements (e.g., components, steps, and methods). Reference numerals described in previous embodiments that reappear in later embodiments may be omitted. Furthermore, technical features described in different or the same embodiments can be combined in any way, as long as the combined embodiment or technical solution is complete and can solve the technical problems of this application or achieve the technical effects described or not described in this disclosure but which can be determined based on the complete technical solution described above.

[0035] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner.

[0036] Subjects: Users (e.g., humans) who participate in brain-computer interface experiments or use BCI systems, whose brain activity (e.g., EEG signals) is collected and used for intent decoding.

[0037] Trial: The subject performs an independent and complete motor imagery task, usually guided by the experimental system through visual / auditory cues, accompanied by a fixed duration of EEG signal acquisition.

[0038] A session is the set of all trials completed by a subject during a single, continuous experiment, representing a complete training or use cycle under relatively stable physiological and cognitive conditions.

[0039] To encourage subjects to generate more separable EEG data, this disclosure models the motor imagery training process as a zero-sum minimax game and adaptively adjusts the difficulty of the training task by solving the minimax problem, thereby improving the motor imagery training effect of the subjects.

[0040] This disclosure provides an online update method for decoding models based on knowledge distillation and paradigm-guided domain adaptation techniques. This ensures that the decoding model can be updated stably in real time using new data generated by the subjects, thereby improving the accuracy of online decoding.

[0041] The motor imagery training and online decoding method disclosed herein can effectively improve the online performance of MI-BCI. This method can be used to enhance neurorehabilitation and motor assistance effects; however, its application is not limited thereto.

[0042] Figure 1 This is a flowchart illustrating a motion imagery training and online decoding process according to an exemplary embodiment of the present disclosure; Figure 2 The complete process of motion imagery training and online decoding according to an example embodiment of this disclosure is shown; Figure 3 This is a flowchart of a motion imagery training and online decoding method according to an exemplary embodiment of the present disclosure; Figure 4 An online update process for a decoding model according to an example embodiment of this disclosure is illustrated; Figure 5 A flowchart illustrating a trial-level update of a decoding model according to an example embodiment of the present disclosure is shown; Figure 6 A flowchart of a session-level update of a decoding model according to an example embodiment of this disclosure is shown.

[0043] Reference Figure 1 The currently collected EEG data can be input into a decoding model used for motor imagery category classification to obtain the predicted probability of the corresponding category (e.g., the average predicted probability P). avg (For example, 0.4).

[0044] In addition, the task difficulty threshold P for the current trial can be dynamically calculated based on the subject's task completion rate in previous trials (such as the proportion of effective trials, response stability, etc.). thr (e.g., 0.5) to achieve personalized adaptation. Increase the threshold to increase challenge when subjects perform well, and decrease the threshold to maintain training confidence when subjects perform poorly.

[0045] The predicted probability can be compared with a dynamic threshold. If the average predicted probability P avg Greater than the task difficulty threshold P thr If the result is positive, the attempt is considered a valid motor imagery (or corresponds to a valid motor imagery); otherwise, it is considered invalid.

[0046] As an example, subsequent actions are triggered only when the condition is deemed valid, such as performing corresponding interactive feedback (e.g., moving objects, playing prompts) in a virtual reality scene, thus forming a neural feedback loop.

[0047] The above mechanism not only ensures the quality of data used for model updates and avoids interference from noise labels, but also maintains efficient learning of subjects within the "zone of proximal development" through adaptive difficulty adjustment, significantly improving the stability of online training and user experience.

[0048] Specifically, the training process of motor imagery can be mathematically modeled. Motor imagery can be viewed as a close human-computer collaboration process in which the subject and the decoding model share a common goal: to achieve higher classification accuracy. From the perspective of the decoding model, the recognition process can be defined by the following formula (1):

[0049] (1)

[0050] in, This represents the electroencephalogram (EEG) data generated by the subjects. Indicate category The predicted probability. This probability is determined by the model. and its set of learnable parameters It is estimated that, for example, learnable parameters may include parameters that can be updated through backpropagation. The higher the value, the better the input sample. Classified as a category The greater the likelihood, the higher the probability.

[0051] From the participants' perspective, when asked to perform category... The EEG data generated during the motor imagery task It can be described as:

[0052] (2)

[0053] in, Representing the relevant parameters of the subjects, here Implicit integrated parameters that influence EEG generation during motor imagery can be used, such as neural activation patterns, attention, imagery strategies, and proficiency. In research based on human-computer co-learning, the objective function can be defined as:

[0054] , (3)

[0055] To achieve this goal, participants need to learn to modulate neural signals to generate more separable EEG data, thereby improving prediction probabilities. Simultaneously, the decoding model should also be adjusted to better reflect the distribution of EEG data.

[0056] In this disclosure, to adaptively adjust the task difficulty, the motor imagery training process can be modeled as a two-player zero-sum minimax game, which can be described as follows:

[0057] (4)

[0058] Among them, threshold The difficulty of the task representing the movement and imagination. The threshold is set to prevent participants from being unable to reach the target due to an excessively high threshold. The motor imagery task is considered complete when the predicted probability reaches or exceeds this threshold.

[0059] Specifically, during training, subjects strive to generate more separable EEG data to increase the probability value. This minimizes the distance between the probability and the threshold. Conversely, to incentivize participants to generate more separable EEG data, this distance can be maximized by adjusting the threshold. Furthermore, setting an unattainable threshold can severely damage participants' confidence and even lead to task failure. The value can be maximized within the range that the subject is most likely to reach.

[0060] As an example, The determination of the value can be based on the subject's historical performance. This minimax problem can be further described as:

[0061] (5)

[0062] in, Represents historical probability records. It is a scaling factor. The Nash equilibrium of this game can be expressed as:

[0063] (6)

[0064] This shows that when the subject's historical records are obtained... Then, the difficulty of the new task can be determined by solving this minimax problem. This indicates the optimal target threshold for task completion dynamically set by the current subject during the current training phase. This indicates the target value of the implicit integrated parameter that the subject influences the generation of electroencephalograms during motor imagery when the game reaches Nash equilibrium.

[0065] This allows for the automatic calculation of a new threshold best suited to a subject's current abilities based on their historical performance, through a theoretically grounded optimization process (e.g., minimax game), thereby achieving a balance between "avoiding frustration" and "promoting progress."

[0066] For details, please refer to Figure 3 The motion imagery training and online decoding method according to the embodiments of the present disclosure may include steps S110, S120, S130 and S140.

[0067] In step S110, the task difficulty threshold for the current trial is dynamically determined based on the subject's task completion rate in previous trials.

[0068] As an example, minimax game problems can be solved based on historical predicted probabilities. By combining an exploration-exploitation strategy to adaptively adjust the threshold, the difficulty can be both challenging and achievable, thereby improving training effectiveness and online decoding performance.

[0069] Specifically, the predicted probabilities output by the decoding model in previous trials can be obtained. After each trial of motor imagery, the collected EEG data can be input into the current decoding model to obtain the predicted probability for the corresponding category. This probability reflects the model's confidence in recognizing the subject's intention. The values ​​of the most recent trials (e.g., the past 10–20 valid trials) can be continuously cached to form a historical predicted probability set (i.e., "historical record").

[0070] Next, the difficulty threshold of the candidate task can be determined based on the predicted probability.

[0071] As an example, virtual reality (VR) scenarios can be used to reflect participants' motor imagery performance and task difficulty, where motor imagery is combined with a goal-oriented drinking task. However, the feedback methods disclosed herein are not limited to this.

[0072] When participants generate EEG data with a high predictive probability, virtual characters in a virtual scenario are more likely to complete the task. The F-index is used to quantitatively assess task completion, and its calculation formula is as follows:

[0073] (7)

[0074] in, The average value representing the predicted probability is calculated using the following formula:

[0075] (8)

[0076] in, Indicates the first The probability of a motion imagery task corresponding to each sample. This represents the number of samples obtained using the sliding window method in one trial. In one trial, whenever a new sample is obtained, The value will be updated and recorded in the historical probability record. In the middle. Meanwhile, the threshold It can be limited to the range [0,1] and can remain fixed in a trial.

[0077] The state of a virtual character in a virtual scenario can be updated based on the F-score. When the participant performs well... When the value approaches 1, the virtual character tends to complete the task.

[0078] Furthermore, an exploration-exploitation strategy can be employed to determine whether to update the task difficulty threshold of the current trial to the candidate task difficulty threshold. An ε-greedy exploration-exploitation mechanism can be used to strike a balance between stability and adaptability: increasing the update probability to enhance difficulty when completion is high, and maintaining the threshold to avoid frustration when completion is low, thus achieving personalized and robust adaptive training.

[0079] For example, the average task completion rate of the subject in the previous multiple trials can be calculated first, and the average value can be set as the exploration probability; a random number between zero and one can be generated; in response to the random number being less than the exploration probability, the task difficulty threshold of the current trial can be updated to the candidate task difficulty threshold; in response to the random number being greater than or equal to the exploration probability, the task difficulty threshold can be kept unchanged.

[0080] Specifically, it adopts an ε-greedy ( The strategy of (-greedy) is used to decide whether to adopt the new task difficulty calculated by the Nash equilibrium in formula (6).

[0081] Specifically, it can be based on the previous The average completion index is calculated from the task completion indices recorded in each trial. and will The value is set to equal to .

[0082] Then With variables randomly generated in the range [0,1] The comparison is performed to determine whether to adopt the new task difficulty. If... The value exceeds If the new task difficulty is not adopted, the threshold will not be used. Keep it unchanged; otherwise, adopt the new task difficulty and... Set to be based on historical EEG samples The calculation result of the maximum value is merely an example, and the way the threshold is set in this disclosure is not limited to this. Therefore, the threshold is more likely to increase when the subject performs well, and may remain unchanged when they perform poorly. If in multiple consecutive trials... If the value remains consistently low, the threshold will be set to a predefined value to avoid subject frustration. For all subjects, the scaling factor... The value can be set to 1.

[0083] In step S120, the decoding model for classifying motor imagery categories is used to obtain the prediction result of the corresponding motor imagery category based on the EEG data generated by the subject collected in the current trial.

[0084] During a single motor imagery task, the electroencephalogram (EEG) signals generated by the subject can be collected and input into a decoding model for motor imagery intention recognition.

[0085] The aforementioned decoding model has acquired the ability to distinguish EEG patterns of different movement categories (such as left hand, right hand, etc.) through prior calibration or continuous online learning. The model performs feature extraction and classification calculations on the received EEG data for the current trial, outputting the predicted probability of each preset motor imagery category. The prediction result for the category required for the trial (specified by the experimental prompts) can be used as a core indicator to measure the quality of the task performance.

[0086] In step S130, based on the comparison between the prediction result and the task difficulty threshold, it is determined whether the current trial corresponds to valid motor imagery.

[0087] For example, the prediction result output by the decoding model can be compared with the currently set task difficulty threshold. If the prediction result reaches or exceeds the threshold, the current trial is considered a valid motion imagery; otherwise, it is considered an invalid trial. However, this is just an example; the validity of the current trial can also be determined by using a ratio.

[0088] In step S140, in response to determining that the current trial corresponds to a valid motor image, feedback corresponding to the valid motor image is provided.

[0089] When the current trial is determined to correspond to a valid motor imagery (i.e., the predicted probability output by the decoding model reaches or exceeds the currently set task difficulty threshold), an external feedback mechanism matching the intention of that motor imagery can be triggered to strengthen the subject's neuromodulation behavior and form a closed-loop training. Specifically, corresponding feedback devices can be activated based on the cue category (e.g., "left hand" or "right hand"): for example, driving a robotic arm to perform a movement in the corresponding direction, triggering electrical stimulation (FES) to activate the target limb muscles, or presenting hand movement animations in a virtual reality environment. This feedback can last for several seconds (e.g., 5 seconds), has a clear spatiotemporal correspondence, and allows the subject to intuitively perceive that their brain electrical activity has been successfully translated into control commands. Immediate, concrete, and intentional. Figure 1 Positive feedback can enhance user engagement and a sense of accomplishment, and can also promote the plasticity of the brain's sensorimotor cortex, thereby accelerating the improvement of motor imagery ability. As an example, if the feedback is not effective for motor imagery, it may not be provided to the user. However, this is merely an example, and this disclosure is not limited to this; feedback corresponding to ineffective motor imagery may also be provided.

[0090] The decoding model can be updated after each trial.

[0091] For example, the EEG data and its labels collected in this trial can be used to adjust the model parameters to enhance its ability to recognize similar intentions in the future. The decoding model here can be a machine learning model, for example, it can correspond to the above formula (2). The decoding model can map the multi-channel EEG signals generated by the subject when performing the motor imagery task to specific motor intention categories.

[0092] In addition, if the task completion rate in several consecutive trials in the previous trials is lower than the preset lower limit, the task difficulty threshold of the current trial can be reset to the preset safe initial value. The preset safe initial value refers to a conservative and easily achievable benchmark task difficulty threshold set during motor imagery training to prevent subjects from continuously failing, feeling frustrated, or even giving up training due to excessive task difficulty. For example, the preset safe initial value can be between 0.4 and 0.6.

[0093] The above steps can be repeated to complete a training session that includes multiple trials.

[0094] Training is organized in units of "sessions," each containing multiple consecutive trials. In a single trial, the subject's EEG data is collected, a decoding model outputs a prediction, and this prediction is compared to the current task difficulty threshold to determine the trial's effectiveness. If effective, the decoding model is updated, and the difficulty of the next trial may be adjusted. Multiple trials together constitute a complete training session, typically lasting from several minutes to tens of minutes, representing a training unit in a relatively stable neural state.

[0095] After the session ends, higher-level model updates (such as domain adaptation) can be triggered to address cross-session non-stationarity. After each trial in the current training session, if it is determined that the corresponding trial corresponds to valid motor imagery, a trial-level update is performed on the decoding model based on the collected corresponding EEG data.

[0096] Specifically, please refer to Figure 2 The entire trial lasts approximately 19–24 seconds and is divided into four main phases: preparation phase (0–2s), motor imagery phase (2–12s), assessment phase (12–17s), and rest phase (17–24s).

[0097] During the preparation phase, a cross prompt is displayed on the screen, and the task difficulty threshold can be dynamically adjusted based on the subject's historical performance. Then, the subject enters the motor imagery phase, where they imagine completing a designated task (such as reaching for a cup) without any actual movement. EEG data is continuously collected and model predictions are made. During the evaluation phase, the effectiveness of the current trial is determined by comparing the prediction results with the threshold, and the online update of the decoding model is triggered. Finally, the subject enters the rest phase to recover.

[0098] A center-oriented training set can be constructed first. For example, previous session data can be used to generate a "center-oriented training set" with labels and logical edit values ​​through a feature extractor and a classifier. For example, the trained feature extractor can be used to encode all samples of each category in the historical session, and the mean vector of the category in the feature space can be calculated as the "category center". Then, several samples with the closest Euclidean distance to the center can be selected to form the center-oriented subset of the category.

[0099] Reference Figure 4 (a) can utilize EEG samples from previous sessions to construct center-oriented training sets for each category, which will be used for model updates after each trial. Specifically, the process for constructing a center-oriented training set for category c is as follows: calculate the mean of the feature vectors of all samples in that category as the category center; then, select the feature vector most similar to the center point in that category. A number of samples (similarity evaluated using Euclidean distance). The data, labels, and output logits of these selected samples together constitute the center-guided training set for this category, which can be represented as:

[0100] (9)

[0101] in, , and Representing the first The data, labels, and logical values ​​of each sample; This represents a set of samples from historical sessions.

[0102] Then, trial-level model updates can be performed. For example, the current trial data and the center-guided training set can be input into the model together to calculate the knowledge distillation loss and classification loss, thereby enabling rapid fine-tuning of the model parameters and improving the decoding performance of the current trial.

[0103] Reference Figure 5 The trial update of the decoding model according to the embodiments of the present disclosure may include steps S141, S142, S143, S144 and S145.

[0104] In step S141, the EEG data collected in the current trial is used to construct the current category training set, which may contain all samples in the current trial and their true category labels.

[0105] In step S142, a center-oriented training set is constructed from the historical session data for each category other than the current motion imagery category. The center-oriented training set contains multiple historical samples whose features are closest to the category center in the corresponding category.

[0106] The specific processes of steps S141 and S142 are as described above, and will not be repeated here.

[0107] For details, please refer to Figure 4 (b) can be completed by the subject in a certain trial (belonging to a category). After that, a new training set is constructed using the samples from the current trial. Used for model updates, it is defined as:

[0108] (10)

[0109] in, Represents a sample from the current session; It refers to the number of samples.

[0110] In step S143, the decoding model is trained under supervision based on the current category training set to optimize its ability to discriminate the current category. The supervised training loss can be calculated based on the difference between the true category label of each sample in the current category training set and the prediction result of the decoding model.

[0111] As an example, the cross-entropy loss function can be used, leveraging... The model is fine-tuned using samples from the dataset, and the loss function can be defined as:

[0112] (11)

[0113] Here, Indicates the first prediction made by the decoding model to be updated. The probability that a sample belongs to its true class. Furthermore, to avoid performance degradation in classifying other classes.

[0114] In step S144, the decoding model is subjected to knowledge distillation constraints based on the center-guided training set to preserve its ability to discriminate other historical categories. The knowledge distillation loss can be calculated based on the distribution difference between the softened outputs generated by the decoding model before the update for each sample in the center-guided training set and the outputs of the model to be updated for the same sample. However, this disclosure is not limited to this; for example, the center-guided training set samples can be constrained solely by the knowledge distillation loss, or supervision can be performed without using real labels.

[0115] As an example, you can use categories other than the current category. All other categories of center training set samples Perform model updates, where This represents the total number of categories. Using knowledge distillation techniques, the model is fine-tuned using these central training set samples, and its loss function can be defined as:

[0116] (12)

[0117] in, z represents the total number of samples from the central training set; i This indicates that the historical decoding model is for the first... Logical values ​​generated from each central sample; This indicates that the decoding model to be updated is the first one. Logical values ​​generated from each sample; It is a temperature parameter used to soften logic values; Represents the KL divergence used for distillation; This refers to the softmax function.

[0118] In step S145, the supervised training loss and the knowledge distillation loss are weighted and summed to serve as the overall optimization objective for the trial update, which is used to fine-tune the decoding model.

[0119] After each trial, the total loss function for model updates can be defined as:

[0120] (13)

[0121] However, the above formula (13) is merely an example, and the weighting method is not limited to this. A loss function can be used. Fine-tune the decoding model, and update the model parameters using a momentum-based method. The update formula is as follows:

[0122] (14)

[0123] in, and These represent the parameters of the updated model and the original model, respectively; These represent the model parameters obtained by fine-tuning the original model. It is the momentum coefficient used to regulate the rate of change.

[0124] Furthermore, session-level model updates can be performed. For example, multimodal distance loss can be introduced to align domains by combining data from previous and current sessions. At the same time, classification loss is used to maintain class discrimination capability, and paradigm comparison loss is used to implement class prototype consistency constraints, prevent catastrophic forgetting, and enhance cross-session stability.

[0125] After the current training session ends, the decoding model can be updated at the session level based on the EEG data of the current training session and the EEG data of the historical training sessions.

[0126] Reference Figure 6 The session-level update of the decoding model according to the embodiments of the present disclosure may include steps S210, S220, S230 and S240.

[0127] In step S210, the EEG data corresponding to the current training session is used as the target domain data, and the EEG data corresponding to the historical training sessions is used as the source domain data.

[0128] The EEG data collected in the current training session is defined as target domain data, representing the subject's latest neural response under the current physiological state and experimental conditions. In addition, all EEG data accumulated in the past training sessions can be regarded as source domain data, reflecting the subject's past stable EEG patterns.

[0129] In step S220, a representative prototype for each motion imagination category is constructed based on the source domain data.

[0130] As an example, for each category of motion imagery, its mean feature value in the source domain data is calculated, serving as a representative prototype for that category. This representative prototype can be obtained by averaging the feature vectors of all samples belonging to the same motion imagery category in the feature space of the decoding model. Furthermore, the source domain samples can be mapped to the embedding space using a feature extractor, and the feature vectors of all samples of the same category can be averaged to obtain the central point that encapsulates the typical neural representations of that category. These prototypes are used to guide the alignment of target domain features, achieving class-aware domain adaptation.

[0131] In step S230, contrastive loss, domain alignment loss, and classification loss are calculated using representative prototypes, target domain data, and source domain data. Contrastive loss is generated by calculating the degree of matching between the target domain data and each representative prototype, encouraging features of similar samples to approach their corresponding prototypes and features of dissimilar samples to move away from non-corresponding prototypes. Domain alignment loss is generated by measuring the overall distribution difference between source domain data and target domain data in the feature space. Classification loss is calculated based on the difference between the true class label of the target domain data and the prediction result of the decoding model.

[0132] As an example, the feature mean of the source domain samples can be calculated for each category as a "paradigm". By comparing the loss, the distance between the target domain's similar samples and the paradigm can be narrowed, while the dissimilar samples can be pushed away, thus achieving fine-grained alignment of class perception.

[0133] In step S240, the contrast loss, domain alignment loss, and classification loss are weighted and combined as the overall optimization objective for the session-level update.

[0134] The calculated contrast loss, domain alignment loss, and classification loss can be weighted and summed according to preset weights to form the overall optimization objective function for session-level model updates. This weighting strategy balances class discriminative ability, cross-domain distribution consistency, and class prototype alignment strength, ensuring that the model can adapt to new data in the current session while retaining historical knowledge and maintaining balanced performance across classes.

[0135] For details, please refer to Figure 4 (c) Session-level model updates can be performed as follows: the classification performance of source and target domain samples is jointly optimized by cross-entropy loss; secondly, maximum mean difference (MMD) loss is introduced to globally align the distribution of the two domains in the feature space, thereby mitigating drift caused by electrode offset, physiological state changes, etc.

[0136] Specifically, the model is further fine-tuned based on paradigm-guided domain adaptation. Samples from historical sessions are considered source domain samples, and samples from the current session are considered target domain samples. A cross-entropy loss function is used to fine-tune the decoding model using these source and target domain samples. The loss function can be defined as:

[0137] (15)

[0138] in, and These represent the predicted probabilities of samples in the source and target domains, respectively. and These are labels representing samples from the source and target domains, respectively. and These represent the number of samples in the source and target domains, respectively. Furthermore, the maximum mean difference (MMD) loss can be applied to globally align the feature distributions of the source and target domains, and it can be defined as:

[0139] (16)

[0140] in, and These represent the features of the relevant source and target domain samples, respectively. This represents mapping to the regenerating kernel Hilbert space. Feature mapping; It is space The induced norm on. Furthermore, a contrastive loss can be developed based on the prototype of each class to achieve class-aware feature alignment. Specifically, for each class... The mean of the feature vectors of that class of samples in all source domain samples can be used as an example. Based on examples of various categories, the contrast loss can be defined as:

[0141] (17)

[0142] in, and These represent the true categories of samples in the source and target domains, respectively. Represents the dot product of two vectors; It is a smoothing hyperparameter. Therefore, the total loss function for paradigm-guided domain adaptation can be defined as:

[0143] (18)

[0144] Decoding models can use loss functions Fine-tuning can be performed. As an example, gradients can be calculated through backpropagation, and the optimizer can be used to update the model parameters, allowing the model to adapt to the data distribution of the current session while maintaining its historical discriminative ability, thus completing session-level adaptive fine-tuning.

[0145] Figure 7 The accuracy comparison results of the scheme according to the example embodiment of this disclosure and the comparative scheme are shown; Figure 8 The effect of threshold adaptive adjustment in a motion imagery training method according to an example embodiment of the present disclosure is shown; Figure 9 The average accuracy of different methods varies with the number of trials; and Figure 10 The average accuracy for each category of different methods is shown as a function of trials.

[0146] As an example, the scheme disclosed herein is as follows: In a trial, a crosshair is displayed on the screen, prompting the subject to prepare for 2 seconds. During this time, based on the recorded task completion index and probability, the difficulty threshold of the current motor imagery task is adjusted by solving a Nash equilibrium and combining it with a threshold adaptive algorithm. Next, the subject performs a 10-second motor imagery task, while visual feedback is provided through a virtual scene. After the motor imagery is completed, if the subject's performance reaches the threshold, assistance is provided as feedback during the evaluation phase. This assistance lasts for 5 seconds, during which the decoding model is updated. Assistive devices used include robotic arms, soft gloves, and functional electrical stimulation (FES). If the subject's performance does not reach the threshold, no assistance is provided during the evaluation phase, but the decoding model can still be updated. After the evaluation phase, the subject is given a rest period of 2 to 7 seconds.

[0147] A motor imagery brain-computer interface (MI-BCI) system capable of deploying the motor imagery training and online decoding methods proposed in this disclosure may specifically include: an EEG acquisition system (including a 32-channel Quick-cap electrode cap, an amplifier, and a NeuroScan system based on Curry8 software), an execution device (e.g., a functional electrical stimulation device (MotionStim8), a robotic arm (Rokae CR7), and a soft glove), a feedback device (e.g., a VR scene developed based on the Unity3D engine), and a computing device (as a control center). However, this disclosure is not limited to this, and the method of providing feedback to subjects corresponding to the comparison results is not limited to VR.

[0148] To verify the effectiveness of the proposed method, an online decoding task of unilateral upper limb motor imagery can be selected as the validation scenario. Specifically, online MI-BCI experiments and online simulation experiments can be conducted. In the online MI-BCI experiment, the proposed adaptive human-machine joint learning method is compared with widely used methods that only consider subject learning and fix model parameters after calibration. If the prediction result is correct, the task is considered complete. In the online simulation experiment, different model update methods are implemented as comparison objects.

[0149] As an example, multiple healthy subjects participated in an online MI-BCI experiment, which included a human-computer co-learning experiment (i.e., the method of this disclosure) and a subject learning experiment. Each subject participated in both experiments over two days, each containing three types of unilateral limb motor imagery tasks. Before each experiment, subjects were informed of the experiment details and their consent was obtained. Subjects were instructed to avoid physical movement and remain quiet during the tasks.

[0150] The procedure for each experiment is as follows: First, data from three categories was collected for model calibration. Similar to existing studies, 40 trials were performed for each category, each lasting 5 seconds for a task without feedback. The lightweight model EEGNet was used for EEG decoding, and a sliding window method was applied for data augmentation to train the model. The window length was set to 2 seconds, and the step size to 1 second. Data was selected from the latest 10 trials for each category for validation. The learning rate and dropout rate of the model training were optimized using grid search. Second, after calibration, online decoding was performed using the appropriate methods. The online decoding phase consisted of 8 sessions, each containing 4 trials for each task. In each trial, the sliding window settings were the same as in the calibration phase. The probability values ​​were updated as new data arrived. For the rest task, subjects rested for only 10 seconds without any feedback. The classification results for each subject in each session were recorded for analysis.

[0151] In addition, for the online simulation experiment, the public MI dataset was selected, which contains data on three types of unilateral limb motor imagery tasks collected from twenty-five healthy subjects. The session and trial settings of the simulation experiment were the same as those of the online MI-BCI experiment.

[0152] For the online model update method of this disclosure, the momentum coefficient λ is set to 0.9, and the center training set size n_c for each class is set to 120. Several existing representative methods are implemented for comparison, including model retraining, model fine-tuning, real-time fine-tuning, and weighted updates. All methods use the same calibrated model.

[0153] Reference Figure 7 To compare the performance of different methods, the highest classification accuracy was selected from each subject's last two sessions for evaluation. Figure 7 As can be seen from Part (a), the average accuracy of the method disclosed herein is significantly higher than that of the subject learning method (paired t-test, p=0.006<0.01). Specifically, the average accuracy of the method disclosed herein is 49.0±7.9%, while the average accuracy of the subject learning method is 39.3±2.6%.

[0154] also, Figure 7 Part (b) shows the mean accuracy of all subjects in each session. In the last three sessions, the mean classification accuracy of the method disclosed herein significantly outperformed the subject-learned method (paired t-test, all p < 0.05), with accuracies increasing to 37.9 ± 4.4%, 41.4 ± 7.0%, and 44.9 ± 9.7%, respectively. These results demonstrate that the training and online update method of this disclosure can significantly improve the accuracy of online decoding.

[0155] Additionally, refer to Figure 8 , Figure 8 The diagram shows the curves of the average probability of trials versus the threshold, calculated based on two motor imagery tasks for all participants. The shaded area around the curve represents the standard deviation. It can be seen that when using the motor imagery training method disclosed herein, the average threshold dynamically adjusts according to the participants' actual performance, always remaining within a relatively achievable range, effectively motivating participants to generate more separable EEG signals. By generating more separable EEG signals, participants continuously attempt to approach this adjusted threshold, thereby enhancing the effectiveness of motor imagery training.

[0156] The classification results of different model update methods in online simulation experiments are as follows: Figure 9 And as shown in Table 1 below. The mean accuracy and F1 score for the first two sessions and the last two sessions were used for evaluation. Best and second-best results are indicated in bold and underlined, respectively. It can be seen that without updating the model, both accuracy and F1 score decrease. Compared to no update, all update methods achieve higher accuracy in the last two sessions (paired t-test, all p < 0.005). The method disclosed herein significantly outperforms all other update methods in accuracy in the last two sessions (paired t-test, all p < 0.05). Furthermore, Figure 9 The figure shows how the average accuracy of different methods changes with the number of trials. It can be seen that after the first three sessions, the accuracy of the method disclosed in this paper is higher than that of other methods.

[0157] Table 1. Recognition accuracy of different methods

[0158]

[0159] To further analyze the stability of model updates, such as Figure 10As shown, this disclosure further calculates the trial mean accuracy for all categories across four update methods from session 3 to session 8. For each method, after each online trial, the trial mean accuracy for each category is calculated based on the historical classification results of all subjects in that category to reflect changes in accuracy during model updates. The stability of model updates is evaluated primarily from two aspects: First, the accuracy for all categories should show a continuous improvement throughout the update process. It can be seen that for most methods, the average accuracy for each category is continuously improved. However, for the weighted update method, the accuracy for category 0 decreases in the early stages of the update, which harms online decoding performance. Second, fairness between categories needs to be considered, i.e., the updated model may have significant differences in classification performance for different categories. For example, the updated model of the real-time fine-tuning method tends to achieve high accuracy for category 0, but very low accuracy for other categories, and the improvement in accuracy for these categories is relatively limited. In contrast, the method of this disclosure enables a consistent improvement in accuracy for each category, with smaller differences in accuracy between different categories. The above results demonstrate that the method of this disclosure can effectively improve the stability of model updates.

[0160] Therefore, it can be seen that the disclosed method for motor imagery training and online decoding can effectively improve the training effect of motor imagery and the accuracy of online decoding.

[0161] The above has been referred to Figures 1 to 10 A method for motion visualization training and online decoding according to exemplary embodiments of the present disclosure is described. However, it should be understood that the devices, units, equipment, etc., shown in the figures can be configured as software, hardware, firmware, or any combination thereof to perform specific functions. For example, these units and devices may correspond to dedicated integrated circuits, pure software code, or modules combining software and hardware. Furthermore, one or more functions implemented by these systems or devices may also be uniformly executed by components in a physical entity device (e.g., a processor, client, or server).

[0162] The instructions stored in the aforementioned computer-readable storage medium can be executed in environments deployed in computer devices such as clients, hosts, agent devices, and servers. It should be noted that the instructions can also be used to perform additional steps beyond those described above, or to perform more specific processing while executing the aforementioned steps. The details of these additional steps and further processing are already provided in the reference... Figures 1 to 10 As mentioned in the description of the relevant systems and methods, they will not be repeated here to avoid repetition.

[0163] It should be noted that the process generation method according to the exemplary embodiments of this disclosure can rely entirely on the operation of computer programs or instructions to achieve the corresponding functions. That is, each device corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (e.g., a lib library) to achieve the corresponding functions.

[0164] The disclosed motor imagery training and online decoding device can be applied to brain-computer interfaces and may include a memory storing programs or instructions that, when executed by a processor, cause the processor to perform the motor imagery training and online decoding method as described above. This motor imagery training and online decoding device may be a computer-readable storage medium or a device including a computer-readable storage medium and a processor.

[0165] The motion imagery training and online decoding method according to embodiments of this disclosure can increase the number of effective trials.

[0166] The motion imagery training and online decoding method according to the embodiments of this disclosure can avoid performance fluctuations caused by blind updates and catastrophic forgetting of motion imagery training and online decoding methods.

[0167] The motor imagery training and online decoding method according to embodiments of this disclosure can be applied to the fields of rehabilitation training and motor assistance, thereby improving the efficiency of rehabilitation treatment and motor assistance.

[0168] While specific terminology has been used to describe various embodiments of this disclosure, the specification and drawings are to be regarded as illustrative rather than restrictive in order to aid in understanding this disclosure. Various modifications and changes can be made by those skilled in the art (e.g., different features in different embodiments may be combined) without departing from the broader spirit and scope of this disclosure. Therefore, the scope of this disclosure is not limited by the specific embodiments and examples, but by the claims and their equivalents.

Claims

1. A motor imagery training and online decoding method applied to a brain-computer interface, characterized in that, The method comprises the following steps: dynamically determining a task difficulty threshold of a current trial according to the task completion degree of the subject in a previous trial; obtaining a prediction result of a corresponding motor imagery category based on electroencephalogram data generated by the subject in the current trial by using a decoding model for motor imagery category classification; determining whether the current trial corresponds to effective motor imagery according to a comparison result between the prediction result and the task difficulty threshold; providing feedback corresponding to the effective motor imagery in response to determining that the current trial corresponds to the effective motor imagery; updating the decoding model after the current trial ends; repeating the above steps to complete a current training session comprising multiple trials, wherein, after each trial of the current training session ends, performing a trial-level update of the decoding model based on the corresponding electroencephalogram data collected, wherein the trial-level update of the decoding model comprises: constructing a current category training set using the electroencephalogram data collected in the current trial, the current category training set containing all samples in the current trial and their true category labels; constructing a center-oriented training set for each category except the current motor imagery category from historical session data, the center-oriented training set containing multiple historical samples in the corresponding category that are closest to the category center; supervised training the decoding model based on the current category training set to optimize its discrimination ability for the current category; knowledge distillation constraint of the decoding model based on the center-oriented training set to retain its discrimination ability for other historical categories; weighting and summing the supervised training loss and the knowledge distillation loss as the total optimization objective of the trial-level update for fine-tuning the decoding model, wherein the supervised training loss is calculated based on the difference between the true category labels of each sample in the current category training set and the prediction result of the decoding model; the knowledge distillation loss is calculated based on the distribution difference between the softened output generated by the decoding model before updating for each sample in the center-oriented training set and the output of the same sample by the model to be updated.

2. The motor imagery training and online decoding method of claim 1, wherein, After the current training session ends, performing a session-level update of the decoding model based on the electroencephalogram data of the current training session and the electroencephalogram data of historical training sessions.

3. The motor imagery training and online decoding method of claim 1, wherein, The step of dynamically determining a task difficulty threshold of a current trial according to the task completion degree of the subject in a previous trial comprises: obtaining prediction probabilities output by the decoding model in a plurality of previous trials of the subject; determining a candidate task difficulty threshold based on the prediction probabilities; using an exploration-exploitation strategy to determine whether to update the task difficulty threshold of the current trial to the candidate task difficulty threshold.

4. The motor imagery training and online decoding method of claim 3, wherein, The step of using an exploration-exploitation strategy to determine whether to update the current task difficulty threshold to the candidate task difficulty threshold comprises: calculating the average of the task completion degrees of the subject in the plurality of previous trials, and setting the average as an exploration probability; generating a random number between zero and one; in response to the random number being less than the exploration probability, updating the task difficulty threshold of the current trial to the candidate task difficulty threshold; in response to the random number being greater than or equal to the exploration probability, maintaining the task difficulty threshold unchanged.

5. The motor imagery training and online decoding method of claim 1, wherein, In response to the task completion degree in the continuous multiple trials in the previous multiple trials being lower than the preset lower limit value, the task difficulty threshold of the current trial is reset to a preset safe initial value, and the preset safe initial value is between 0.4 and 0.

6.

6. The motor imagery training and online decoding method of claim 2, wherein, The session-level updating of the decoding model comprises: The electroencephalogram data corresponding to the current training session is taken as target domain data, and the electroencephalogram data corresponding to the historical training session is taken as source domain data; A representative prototype of each motor imagery category is constructed based on the source domain data; A contrast loss, a domain alignment loss and a classification loss are calculated by using the representative prototype, the target domain data and the source domain data; The contrast loss, the domain alignment loss and the classification loss are weighted and combined as a total optimization target of the session-level updating, which is used to further fine-tune the decoding model.

7. The motor imagery training and online decoding method according to claim 6, wherein: The representative prototype is obtained by averaging the feature vectors of all samples belonging to the same motor imagery category in the source domain data in the feature space of the decoding model; The contrast loss is generated by calculating the matching degree between the target domain data and each representative prototype, and is used to encourage the features of the same samples to be close to the corresponding prototype and the features of different samples to be far away from the non-corresponding prototype; The domain alignment loss is generated by measuring the overall distribution difference between the source domain data and the target domain data in the feature space; The classification loss is calculated based on the difference between the real category label of the target domain data and the prediction result of the decoding model.

8. The motor imagery training and online decoding method of claim 1, wherein, The motor imagery training and online decoding method comprises providing feedback corresponding to the comparison result to the subject.

9. A motion imagination training and online decoding apparatus, characterized by comprising: The motor imagery training and online decoding device is applied to a brain-computer interface and comprises a memory storing programs or instructions, which, when executed by a processor, cause the processor to execute the motor imagery training and online decoding method according to any one of claims 1-8.

Images