Electroencephalogram motion decoding method based on cross-modal semantic alignment

Abstract

The invention discloses an electroencephalogram motion decoding method based on cross-modal semantic alignment, which comprises the following steps: fusing electroencephalogram (EEG) coding and video coding, mapping EEG signals and motion video features to a shared semantic space by utilizing comparative learning, and training a classifier which only depends on EEG input on the basis of the mapping, so as to obtain an electroencephalogram motion decoding result. Two innovative loss functions are adopted in the comparison learning stage and the training stage, the accuracy of the corresponding relation between the EEG and video action semantics and the EEG single-mode discrimination capability can be improved at the same time, and accurate discrimination of the corresponding action category of the EEG signal is achieved. According to the invention, high identification precision and robustness can be maintained under the cross-subject condition, and the demand for large-scale labeling and calibration of new subjects is reduced. The modular design facilitates expansion and integration, the system can be widely applied to rehabilitation training, virtual reality interaction, exoskeleton control and other scenes, and an efficient and reliable solution is provided for real-time electroencephalogram motion decoding and cross-modal retrieval.

Description

Technical Field

[0001] This invention relates to the technical field of brain-computer interface (BCI) and artificial intelligence, and in particular to a brainwave motion decoding method based on cross-modal semantic alignment. Background Technology

[0002] With the rapid development of BCI, non-invasive EEG has been widely used due to its high temporal resolution, portability, and low cost. Brain-computer interfaces (BCI) aim to translate brain activity into control commands that can drive external devices. In rehabilitation scenarios, they can form a closed loop of "intention-execution-feedback" with exoskeletons, robotic arms, etc., to promote motor function reconstruction and the recovery of neuroplasticity. Cross-subject decoding capability is a key challenge for the large-scale application of BCI. This is affected by factors such as individual brain structure and physiological differences, electrode placement, and noise environment.

[0003] However, traditional approaches based on handcrafted features and shallow learning are susceptible to individual differences and signal non-stationarity in cross-subject scenarios, resulting in insufficient generalization ability. Convolutional neural networks improve end-to-end modeling capabilities, but their limited receptive field makes it difficult to fully model long-term dependencies. Transformers have the advantage of capturing global temporal sequences and cross-channel relationships, but they still have shortcomings in cross-subject robustness. On the other hand, cross-modal alignment models (such as CLIP) have shown the potential to map different modalities to a shared semantic space and achieve zero-shot and few-shot generalization. Aligning abstract EEG embeddings with information-rich action video representations can significantly alleviate the expressive limitations of EEG unimodality and enhance cross-subject generalization and retrieval / classification capabilities. Summary of the Invention

[0004] The purpose of this invention is to address the problem of insufficient generalization ability of existing EEG decoding methods in cross-subject scenarios. It proposes an EEG motion decoding method based on cross-modal semantic alignment, which can maintain high recognition accuracy and robustness under cross-subject conditions, reduce the need for large-scale annotation and calibration of new subjects, and its modular design facilitates expansion and integration. It can be widely applied to scenarios such as rehabilitation training, virtual reality interaction, and exoskeleton control, providing an efficient and reliable solution for real-time EEG motion decoding and cross-modal retrieval.

[0005] To achieve the above objectives, the technical solution provided by this invention is: a brainwave motion decoding method based on cross-modal semantic alignment, comprising the following steps: 1) Acquire raw data and perform standardization processing. The raw data includes synchronously acquired EEG signals and motion videos. 2) Parallel encoding is performed on the standardized raw data to obtain directly comparable EEG feature vectors and video feature vectors; the standardized EEG signal is encoded using an EEG encoder to generate EEG feature vectors; the standardized motion video is encoded using a video encoder to encode the video frame sequence, extract the semantic information of the motion, and generate video feature vectors. 3) The EEG feature vectors and video feature vectors are mapped to a shared embedding space through a nonlinear projection layer, and then normalized to obtain EEG embedding vectors and video embedding vectors. Cross-modal alignment is then performed through contrastive learning. The loss function used in the contrastive learning is a weighted sum of cross-modal contrastive loss and intra-modal supervised contrastive loss. The cross-modal contrastive loss treats all EEG signal samples and motion video samples of the same category as positive sample pairs and the rest as negative sample pairs. By minimizing the distance between positive sample pairs and maximizing the distance between negative sample pairs through contrastive learning, the EEG representation and video representation are aligned. At the same time, the MIL-NCE algorithm is used for bidirectional retrieval to handle EEG signal samples and motion video samples with loose pairings. The intra-modal supervised contrastive loss treats all EEG signal samples of the same category as positive sample pairs and the rest as negative sample pairs. By contrastive learning, the discriminative ability of the EEG representation itself is enhanced. The joint optimization of the above two losses can simultaneously improve the accuracy of the correspondence between EEG signals and video motion semantics and the EEG single-modal discriminative ability. 4) Using the correspondence between EEG signals and video action semantics learned in step 3), train an independent classifier to classify new EEG signals into actions, thereby decoding the EEG signals, that is, translating the intention or state contained in the EEG signals into externally understandable information.

[0006] Furthermore, in step 1), the publicly available EEG motor imagery dataset EEGMMI is used, and standard action videos that strictly correspond to its action category are collected.

[0007] Furthermore, in step 1), the original data is standardized, that is, the EEG signal and the motion video are standardized separately. The standardization of the EEG signal includes: using bandpass and notch filters for noise reduction, and using independent component analysis to remove physiological artifacts, finally forming EEG trials of fixed duration. The standardization of the motion video includes: decoding the motion video into an image frame sequence, performing time uniform sampling and spatial size normalization.

[0008] Further, in step 2), the EEG encoder is an EEG-Conformer EEG encoder, used to extract spatiotemporal features from the standardized EEG signal. This EEG encoder adopts a Conformer model architecture that integrates convolutional networks (CNN) and Transformers, consisting of cascaded convolutional modules and self-attention modules. The convolutional modules are used to extract local spatiotemporal features, and the self-attention modules are used to capture long-range temporal dependencies. Its core formula for scaled dot product attention operation is: ; In the formula, Q, K, and V are the query, key, and value matrices, respectively; the superscript T indicates transpose. The key vector dimension of each attention head is represented by... Scale the values ​​to stabilize them; Finally, the EEG encoder outputs an EEG feature vector representing the high-level semantics of each input EEG signal. .

[0009] Furthermore, in step 2), the video encoder uses a masked self-supervised pre-trained visual Transformer-type model, VideoMAE, to extract stable, high-quality visual action semantic features from the standardized action video clips. First, the motion video clips are parsed into a series of video tubes, and learnable position embeddings are added. , forming the input sequence : ; In the formula, It is a learnable categorical lexical unit; Let n be the nth video tube block, where n = 1, 2, ..., N, and N is the number of video tube blocks; this sequence After processing by the Transformer encoder, the final layer outputs the classification terms. As a global semantic representation of the video, the probability distribution is ultimately predicted through a linear classification head. : ; In the formula, and These are the learnable weight matrix and bias vector of the linear classifier head, respectively. During the fine-tuning phase, cross-entropy loss is used. As an optimization objective, the entire video encoder is trained end-to-end: ; In the formula, y is the one-hot encoded vector of the sample's true label. It is the one-hot encoded vector of the true label of class k. It is the predicted probability of the k-th class. The total number of categories; ultimately, the video encoder assigns a value to each input motion video. Output a video feature vector representing its high-level semantics. .

[0010] Furthermore, in step 3), nonlinear projection heads are respectively installed after the EEG encoder and the video encoder. and EEG feature vectors and video feature vectors Each vector is mapped to a shared space and then normalized using L2 to distribute them onto a unit hypersphere, resulting in embedding vectors of the same dimension. and The superscripts 'e' and 'v' refer to the EEG modality and video modality, respectively. ; In the formula, This is the L2 normalization function.

[0011] Furthermore, in step 3), the loss function used for the cross-modal alignment is the total alignment loss, which is a weighted sum of a cross-modal objective and an intra-modal objective, defined as: ; In the formula, To align the total loss, For the cross-modal contrast loss corresponding to the cross-modal objective, For the intra-modal supervised contrast loss corresponding to the intra-modal target, To supervise the comparison of loss weights; Cross-modal contrast loss Using the MIL-NCE concept, for a given EEG signal sample, all motion videos of the same class are considered positive samples, while all motion videos of other classes are considered negative samples; for a given EEG embedding vector of the i-th EEG signal sample... The losses are as follows: ; In the formula, The loss is calculated by using the i-th EEG signal sample as an index to retrieve all motion videos. This indicates that the search is performed using EEG signal samples as an index to retrieve motion video samples. Let j be the video embedding vector of the j-th action video sample. A collection of motion video sample indexes of the same category. Let i be the category label for the i-th EEG signal sample. Let j be the category label of the j-th action video sample. For temperature parameters, For similarity function, Let i be the video embedding vector of the q-th action video sample, where q represents any action video sample in the batch; i, j, q = 1, 2, ..., B, where B is the batch size; Cross-modal contrast loss It is a symmetrical average value: ; In the formula, It is the loss that is retrieved from all EEG signal samples using the i-th motion video sample as an index; To enhance the discriminative power of EEG representations, supervised contrastive loss was applied within the EEG modalities. For a given EEG signal sample, all EEG embedding vectors of the same category are considered positive samples, while all EEG embedding vectors of other categories are considered negative samples; for a given EEG embedding vector of the i-th EEG signal sample... its losses for: ; In the formula, Represents the set of positive sample indices. p is the number of elements in the positive sample index set. The index in Let r be the EEG embedding vector of the r-th EEG signal sample, where r is the index of any EEG signal sample in the batch except i.

[0012] Furthermore, in step 4), after completing cross-modal alignment, a high-performance and independent classifier that relies solely on the EEG signal input is constructed. This classifier consists of a frozen EEG encoding module, a projection head, and a classification head. The classification head uses a cosine prototype head, whose initial prototype is initialized using knowledge of video modalities. First, K-Means clustering is performed on the video embedding vectors, and the final K cluster centers are taken as the first cluster of category c. prototype vectors Then, by computing the EEG embedding vector With prototype vector The cosine similarity is calculated, and the unnormalized score logit for each class is aggregated using the log-sum-exp operation, where for a given EEG embedding vector... The logit of category c is Calculate using the following formula: ; In the formula, For the number of prototypes in each class, For learnable temperature parameters, Let be the cosine similarity.

[0013] Furthermore, in step 4), the classifier performs decoding using a composite loss. : ; in, For labeled smooth cross-entropy loss, For knowledge distillation loss, For prototype regularization loss, To ensure the stability of the learning process, a weight scheduling strategy is adopted for the weights β of the knowledge distillation loss and γ of the prototype regularization loss. Specifically, an annealing mechanism is used, where the weights are set to 0 in the early stage of training and then gradually increased to the preset maximum value during the training process. This annealing mechanism ensures that the classifier focuses on the basic classification task in the early stage of training, and then gradually introduces regularization constraints and knowledge guidance from the video modality after the features are stable. Cross-entropy loss The primary objective function is classification, employing standard cross-entropy loss with label smoothing. This aims to regularize the classifier's predicted output, effectively preventing overfitting by reducing the classifier's confidence in the training labels, thereby improving its generalization ability on unseen samples. Let the batch size be B and the total number of classes be C. For the i-th sample, let the logit of the classification head output be... After the softmax function The predicted distribution of the k-th class is obtained for: ; Let the smoothing coefficient be Then the target distribution of the k-th class after smoothing of the i-th sample for: ; The labeled smooth cross-entropy loss is: ; Knowledge distillation loss A teacher-student paradigm is adopted to achieve knowledge transfer. A pre-trained video encoder with fixed parameters and an original classification head are used as the teacher network, while a classifier relying solely on EEG input is used as the student network. The implicit knowledge held by the teacher network is distilled into the student network by calculating the KL divergence between the temperature-scaled outputs of the student and teacher networks on corresponding samples. The mathematical expression is as follows: ; In the formula, and The logit values ​​for the student model and the teacher model are respectively. The distillation temperature. Let KL divergence be a metric. Prototype regularization loss As a regularization constraint, its purpose is to explicitly guide the EEG embedding vector. The mathematical expression for the category center closest to its corresponding category in the feature space is: ; In the formula, Given an EEG embedding vector The logit of category c, Given an EEG embedding vector Real category Logit, prototype regularization loss This is used to prevent the fine-tuned EEG vectors from deviating from the structured feature space with good class discrimination established during the cross-modal alignment stage, thereby ensuring the effectiveness of knowledge transfer and the consistency of the feature space.

[0014] Compared with the prior art, the present invention has the following advantages and beneficial effects: 1. The method of the present invention uses a weighted sum of cross-modal contrast loss and intra-modal supervised contrast loss for joint optimization during cross-modal alignment. The cross-modal contrast uses the MIL-NCE algorithm for bidirectional retrieval to enhance the robustness of cross-modal alignment and maintain class clustering among different subjects, thereby significantly alleviating the cross-subject domain shift problem and reducing or eliminating subject-specific calibration annotations.

[0015] 2. The method of this invention acquires prior knowledge during the training phase and relies only on EEG signal input during the deployment phase. It adopts joint training with labeled smooth cross-entropy loss, knowledge distillation loss and prototype regularization loss, and uses annealing scheduling for the weights of knowledge distillation loss and prototype regularization loss. This reduces computation and latency while ensuring performance, making it easy to deploy on the edge.

[0016] 3. The cosine prototype header of the method of the present invention uses cosine similarity and aggregates log-sum-exp to form logit for each category. It can directly output the similarity distribution and visualize the prototype, improve interpretability, facilitate the rapid adaptation of new categories and small samples, and reduce the expansion cost.

[0017] 4. The method of this invention utilizes cross-modal alignment to embed noisy and non-stationary EEG into a more robust video semantic space, and maintains good discriminability under weak labeling, slight temporal mismatch and cross-session conditions through distillation and contrastive learning. Compared with pure EEG methods, it has comprehensive advantages in cross-domain generalization, noise resistance and calibration cost.

[0018] In summary, the method of this invention can significantly improve the accuracy and robustness of EEG decoding across different subject scenarios, reduce the calibration and annotation costs for new subjects, provide stable control signals and retrieval capabilities for various downstream devices, and enhance the system's generalization and feasibility across populations and scenarios. Attached Figure Description

[0019] Figure 1 This is an architectural diagram of the method of the present invention.

[0020] Figure 2 This is a schematic diagram of the EEG encoder.

[0021] Figure 3 This is a schematic diagram of the structure of a video encoder.

[0022] Figure 4 This is a schematic diagram of the cross-modal alignment training phase. Detailed Implementation

[0023] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0024] like Figures 1 to 4 As shown in the figure, this embodiment discloses a brainwave motion decoding method based on cross-modal semantic alignment, the specific details of which are as follows: 1) Acquire raw data and perform standardization processing. The raw data includes synchronously acquired EEG signals and motion videos, as follows: We used the publicly available EEG motor imagery dataset EEGMMI and collected standard action videos that strictly corresponded to the action categories. The raw data were standardized, specifically the EEG signals and action videos. The standardization of the EEG signals included denoising with bandpass (1-40Hz) and notch (50Hz) filters and removing physiological artifacts using methods such as independent component analysis (ICA), ultimately forming EEG trials of fixed duration. The standardization of the action videos included decoding the action videos into image frame sequences, performing temporally uniform sampling, and normalizing the spatial size (224×224 pixels).

[0025] 2) Parallel encoding is performed on the standardized raw data to obtain directly comparable EEG feature vectors and video feature vectors; the standardized EEG signal is encoded using an EEG encoder to generate EEG feature vectors; the standardized motion video is encoded using a video encoder to encode the video frame sequence, extract the semantic information of the motion, and generate video feature vectors. Specifically, the EEG encoder is an EEG-Conformer brainwave encoder, used to extract spatiotemporal features from standardized EEG signals. This EEG encoder employs a Conformer model architecture that integrates convolutional networks (CNN) and Transformers, consisting of cascaded convolutional modules and self-attention modules. The convolutional modules are used to extract local spatiotemporal features, and the self-attention modules are used to capture long-range temporal dependencies. Its core formula for scaled dot product attention operations is: ; In the formula, Q, K, and V are the query, key, and value matrices, respectively; the superscript T indicates transpose. The key vector dimension of each attention head is represented by... Scale the values ​​to stabilize them; Finally, the EEG encoder outputs an EEG feature vector representing the high-level semantics of each input EEG signal. .

[0026] Specifically, the video encoder uses a masked self-supervised pre-trained visual Transformer-type model, VideoMAE, to extract stable, high-quality visual motion semantic features from standardized motion video clips. First, the motion video clips are parsed into a series of video tubes, and learnable position embeddings are added. , forming the input sequence : ; In the formula, It is a learnable categorical lexical unit; Let n be the nth video tube block, where n = 1, 2, ..., N, and N is the number of video tube blocks; this sequence After processing by the Transformer encoder, the final layer outputs the classification terms. As a global semantic representation of the video, the probability distribution is ultimately predicted through a linear classification head. : ; In the formula, and These are the learnable weight matrix and bias vector of the linear classifier head, respectively. During the fine-tuning phase, cross-entropy loss is used. As an optimization objective, the entire video encoder is trained end-to-end: ; In the formula, y is the one-hot encoded vector of the sample's true label. It is the one-hot encoded vector of the true label of class k. It is the predicted probability of the k-th class. The total number of categories; ultimately, the video encoder assigns a value to each input motion video. Output a video feature vector representing its high-level semantics. .

[0027] 3) The EEG feature vectors and video feature vectors are mapped to a shared embedding space through a nonlinear projection layer, and then normalized to obtain EEG embedding vectors and video embedding vectors. Cross-modal alignment is then performed through contrastive learning. The loss function used in the contrastive learning is a weighted sum of cross-modal contrastive loss and intra-modal supervised contrastive loss. The cross-modal contrastive loss treats all EEG signal samples and motion video samples of the same category as positive sample pairs and the rest as negative sample pairs. By minimizing the distance between positive sample pairs and maximizing the distance between negative sample pairs through contrastive learning, the EEG representation and video representation are aligned. At the same time, the MIL-NCE algorithm is used for bidirectional retrieval to handle EEG signal samples and motion video samples with loose pairings. The intra-modal supervised contrastive loss treats all EEG signal samples of the same category as positive sample pairs and the rest as negative sample pairs. By contrastive learning, the discriminative ability of the EEG representation itself is enhanced. The joint optimization of the above two losses can simultaneously improve the accuracy of the correspondence between EEG signals and video motion semantics and the EEG single-modal discriminative ability.

[0028] Specifically, a nonlinear projection head is respectively installed after the EEG encoder and the video encoder. and EEG feature vectors and video feature vectors Each vector is mapped to a shared space and then normalized using L2 to distribute them onto a unit hypersphere, resulting in embedding vectors of the same dimension. and The superscripts 'e' and 'v' refer to the EEG modality and video modality, respectively. ; In the formula, This is the L2 normalization function.

[0029] Specifically, the loss function used for the cross-modal alignment is the total alignment loss, which is a weighted sum of a cross-modal objective and an intra-modal objective, defined as: ; In the formula, To align the total loss, For the cross-modal contrast loss corresponding to the cross-modal objective, For the intra-modal supervised contrast loss corresponding to the intra-modal target, To supervise the comparison of loss weights; Cross-modal contrast loss Using the MIL-NCE concept, for a given EEG signal sample, all motion videos of the same class are considered positive samples, while all motion videos of other classes are considered negative samples; for a given EEG embedding vector of the i-th EEG signal sample... The losses are as follows: ; In the formula, The loss is calculated by using the i-th EEG signal sample as an index to retrieve all motion videos. This indicates that the search is performed using EEG signal samples as an index to retrieve motion video samples. Let j be the video embedding vector of the j-th action video sample. A collection of motion video sample indexes of the same category. Let i be the category label for the i-th EEG signal sample. Let j be the category label of the j-th action video sample. For temperature parameters, For similarity function, Let i be the video embedding vector of the q-th action video sample, where q represents any action video sample in the batch; i, j, q = 1, 2, ..., B, where B is the batch size; Cross-modal contrast loss It is a symmetrical average value: ; In the formula, It is the loss that is retrieved from all EEG signal samples using the i-th motion video sample as an index; To enhance the discriminative power of EEG representations, supervised contrastive loss was applied within the EEG modalities. For a given EEG signal sample, all EEG embedding vectors of the same category are considered positive samples, while all EEG embedding vectors of other categories are considered negative samples; for a given EEG embedding vector of the i-th EEG signal sample... its losses for: ; In the formula, Represents the set of positive sample indices. p is the number of elements in the positive sample index set. The index in Let r be the EEG embedding vector of the r-th EEG signal sample, where r is the index of any EEG signal sample in the batch except i.

[0030] 4) Using the correspondence between EEG signals and video action semantics learned in step 3), train an independent classifier to classify new EEG signals into actions, that is, decode new EEG signals and translate the intentions or states contained in the EEG signals into externally understandable information.

[0031] Specifically, after completing cross-modal alignment, a high-performance and independent classifier that relies solely on EEG signal input is constructed. This classifier consists of a frozen EEG encoding module, a projection head, and a classification head. The classification head adopts a cosine prototype head, whose initial prototype is initialized using knowledge of video modalities. First, K-Means clustering is performed on the video embedding vectors, and the final K cluster centers are taken as the k-th clusters of category c. prototype vectors Then, by computing the EEG embedding vector With prototype vector The cosine similarity is calculated, and the logit (unnormalized score) for each category is aggregated using the log-sum-exp operation, where for a given EEG embedding vector... The logit of category c is Calculate using the following formula: ; In the formula, For the number of prototypes in each class, For learnable temperature parameters, Let be the cosine similarity.

[0032] Specifically, the classifier uses a composite loss function for decoding. : ; in, For labeled smooth cross-entropy loss, For knowledge distillation loss, For prototype regularization loss, To ensure the stability of the learning process, a weight scheduling strategy is adopted for the weights β of the knowledge distillation loss and γ of the prototype regularization loss. Specifically, an annealing mechanism is used, where the weights are set to 0 in the early stage of training and then gradually increased to the preset maximum value during the training process. This annealing mechanism ensures that the classifier focuses on the basic classification task in the early stage of training, and then gradually introduces regularization constraints and knowledge guidance from the video modality after the features are stable. Cross-entropy loss The primary objective function is classification, employing standard cross-entropy loss with label smoothing. This aims to regularize the classifier's predicted output, effectively preventing overfitting by reducing the classifier's confidence in the training labels, thereby improving its generalization ability on unseen samples. Let the batch size be B and the total number of classes be C. For the i-th sample, let the logit of the classification head output be... After the softmax function The predicted distribution is as follows: ; Let the smoothing coefficient be The smoothed target distribution is: ; The labeled smooth cross-entropy loss is: ; Knowledge distillation loss A teacher-student paradigm is adopted to achieve knowledge transfer. A pre-trained video encoder with fixed parameters and the original classification head are used as a teacher network, while a classification module that relies solely on EEG input is used as a student network. The implicit knowledge held by the teacher network is distilled into the student network by calculating the KL divergence between the temperature-scaled outputs of the student and teacher networks on corresponding samples. The mathematical expression is as follows: ; In the formula, and These are the logits for the student model and the teacher model, respectively. The distillation temperature. Let KL divergence be a metric. Prototype regularization loss As a regularization constraint, its purpose is to explicitly guide the EEG embedding vector. The mathematical expression for the category center closest to its corresponding category in the feature space is: ; In the formula, Given an EEG embedding vector The logit of category c, Given an EEG embedding vector Real category Logit, prototype regularization loss This is used to prevent the fine-tuned EEG vectors from deviating from the structured feature space with good class discrimination established during the cross-modal alignment stage, thereby ensuring the effectiveness of knowledge transfer and the consistency of the feature space.

[0033] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for decoding EEG motion based on cross-modal semantic alignment, characterized in that, Includes the following steps: 1) Acquire raw data and perform standardization processing. The raw data includes synchronously acquired EEG signals and motion videos. 2) Parallel encoding is performed on the standardized raw data to obtain directly comparable EEG feature vectors and video feature vectors; the standardized EEG signal is encoded using an EEG encoder to generate EEG feature vectors; the standardized motion video is encoded using a video encoder to encode the video frame sequence, extract the semantic information of the motion, and generate video feature vectors. 3) The EEG feature vectors and video feature vectors are mapped to a shared embedding space through a nonlinear projection layer, and then normalized to obtain EEG embedding vectors and video embedding vectors. Cross-modal alignment is then performed through contrastive learning. The loss function used in the contrastive learning is a weighted sum of cross-modal contrastive loss and intra-modal supervised contrastive loss. The cross-modal contrastive loss treats all EEG signal samples and motion video samples of the same category as positive sample pairs and the rest as negative sample pairs. By minimizing the distance between positive sample pairs and maximizing the distance between negative sample pairs through contrastive learning, the EEG representation and video representation are aligned. At the same time, the MIL-NCE algorithm is used for bidirectional retrieval to handle EEG signal samples and motion video samples with loose pairings. The intra-modal supervised contrastive loss treats all EEG signal samples of the same category as positive sample pairs and the rest as negative sample pairs. By contrastive learning, the discriminative ability of the EEG representation itself is enhanced. The joint optimization of the above two losses can simultaneously improve the accuracy of the correspondence between EEG signals and video motion semantics and the EEG single-modal discriminative ability. 4) Using the correspondence between EEG signals and video action semantics learned in step 3), train an independent classifier to classify new EEG signals into actions, thereby decoding the EEG signals, that is, translating the intention or state contained in the EEG signals into externally understandable information.

2. The EEG motion decoding method based on cross-modal semantic alignment according to claim 1, characterized in that, In step 1), the publicly available EEG motor imagery dataset EEGMMI is used, and standard action videos that strictly correspond to its action category are collected.

3. The EEG motion decoding method based on cross-modal semantic alignment according to claim 1, characterized in that, In step 1), the raw data is standardized, that is, the EEG signal and motion video are standardized separately. The standardization processing of EEG signals includes: using bandpass and notch filters for noise reduction, and employing independent component analysis to remove physiological artifacts, ultimately forming EEG trials of fixed duration; the standardization processing of motion videos includes: decoding motion videos into image frame sequences, performing time-uniform sampling and spatial size normalization.

4. The EEG motion decoding method based on cross-modal semantic alignment according to claim 1, characterized in that, In step 2), the EEG encoder is an EEG-Conformer EEG encoder, used to extract spatiotemporal features from the standardized EEG signal. This EEG encoder employs a Conformer model architecture that integrates convolutional networks (CNN) and Transformers, consisting of cascaded convolutional modules and self-attention modules. The convolutional modules are used to extract local spatiotemporal features, and the self-attention modules are used to capture long-range temporal dependencies. Its core formula for scaled dot product attention operation is: ； In the formula, Q, K, and V are the query, key, and value matrices, respectively; the superscript T indicates transpose. The key vector dimension of each attention head is represented by... Scale the values ​​to stabilize them; Finally, the EEG encoder outputs an EEG feature vector representing the high-level semantics of each input EEG signal. .

5. The EEG motion decoding method based on cross-modal semantic alignment according to claim 4, characterized in that, In step 2), the video encoder uses a masked self-supervised pre-trained visual Transformer-type model, VideoMAE, to extract stable, high-quality visual motion semantic features from the standardized motion video clips. First, the motion video clips are parsed into a series of video tubes, and learnable position embeddings are added. , forming the input sequence : ； In the formula, It is a learnable categorical lexical unit; Let n be the nth video tube block, where n = 1, 2, ..., N, and N is the number of video tube blocks; this sequence After processing by the Transformer encoder, the final layer outputs the classification terms. As a global semantic representation of the video, the probability distribution is ultimately predicted through a linear classification head. : ； In the formula, and These are the learnable weight matrix and bias vector of the linear classifier head, respectively. During the fine-tuning phase, cross-entropy loss is used. As an optimization objective, the entire video encoder is trained end-to-end: ； In the formula, y is the one-hot encoded vector of the sample's true label. It is the one-hot encoded vector of the true label of class k. It is the predicted probability of the k-th class. The total number of categories; ultimately, the video encoder assigns a value to each input motion video. Output a video feature vector representing its high-level semantics. .

6. The EEG motion decoding method based on cross-modal semantic alignment according to claim 5, characterized in that, In step 3), nonlinear projection heads are respectively installed after the EEG encoder and the video encoder. and EEG feature vectors and video feature vectors Each vector is mapped to a shared space and then normalized using L2 to distribute them onto a unit hypersphere, resulting in embedding vectors of the same dimension. and The superscripts 'e' and 'v' refer to the EEG modality and video modality, respectively. ； In the formula, This is the L2 normalization function.

7. The EEG motion decoding method based on cross-modal semantic alignment according to claim 6, characterized in that, In step 3), the loss function used for cross-modal alignment is the total alignment loss, which is a weighted sum of a cross-modal objective and an intra-modal objective, defined as: ； In the formula, To align the total loss, For the cross-modal contrast loss corresponding to the cross-modal objective, For the intra-modal supervised contrast loss corresponding to the intra-modal target, To supervise the comparison of loss weights; Cross-modal contrast loss Using the MIL-NCE concept, for a given EEG signal sample, all motion videos of the same class are considered positive samples, while all motion videos of other classes are considered negative samples; for a given EEG embedding vector of the i-th EEG signal sample... The losses are as follows: ； In the formula, The loss is calculated by using the i-th EEG signal sample as an index to retrieve all motion videos. This indicates that the search is performed using EEG signal samples as an index to retrieve motion video samples. Let j be the video embedding vector of the j-th action video sample. A collection of motion video sample indexes of the same category. Let i be the category label for the i-th EEG signal sample. Let j be the category label of the j-th action video sample. For temperature parameters, For similarity function, Let be the video embedding vector of the q-th action video sample, where q represents any action video sample in the batch; i, j, q = 1, 2, ..., B, where B is the batch size; Cross-modal contrast loss It is a symmetrical average value: ； In the formula, It is the loss that is retrieved from all EEG signal samples using the i-th motion video sample as an index; To enhance the discriminative power of EEG representations, supervised contrastive loss was applied within the EEG modalities. For a given EEG signal sample, all EEG embedding vectors of the same category are considered positive samples, while all EEG embedding vectors of other categories are considered negative samples; for a given EEG embedding vector of the i-th EEG signal sample... its losses for: ； In the formula, Represents the set of positive sample indices. p is the number of elements in the positive sample index set. The index in Let r be the EEG embedding vector of the r-th EEG signal sample, where r is the index of any EEG signal sample in the batch except i.

8. The EEG motion decoding method based on cross-modal semantic alignment according to claim 7, characterized in that, In step 4), after completing cross-modal alignment, a high-performance and independent classifier that relies solely on the EEG signal input is constructed. This classifier consists of a frozen EEG encoding module, a projection head, and a classification head. The classification head uses a cosine prototype head, whose initial prototype is initialized using knowledge of video modalities. First, K-Means clustering is performed on the video embedding vectors, and the final K cluster centers are taken as the k-th clusters of category c. prototype vectors Then, by computing the EEG embedding vector With prototype vector The cosine similarity is calculated, and the unnormalized score logit for each class is aggregated using the log-sum-exp operation, where for a given EEG embedding vector... The logit of category c is Calculate using the following formula: ； In the formula, For the number of prototypes in each class, For learnable temperature parameters, Let be the cosine similarity.

9. The EEG motion decoding method based on cross-modal semantic alignment according to claim 8, characterized in that, In step 4), the classifier performs decoding using a composite loss. : ； in, For labeled smooth cross-entropy loss, For knowledge distillation loss, For prototype regularization loss, To ensure the stability of the learning process, a weight scheduling strategy is adopted for the weights β of the knowledge distillation loss and γ of the prototype regularization loss. Specifically, an annealing mechanism is used, where the weights are set to 0 in the early stage of training and then gradually increased to the preset maximum value during the training process. This annealing mechanism ensures that the classifier focuses on the basic classification task in the early stage of training, and then gradually introduces regularization constraints and knowledge guidance from the video modality after the features are stable. Cross-entropy loss The primary objective function is classification, employing standard cross-entropy loss with label smoothing. This aims to regularize the classifier's predicted output, effectively preventing overfitting by reducing the classifier's confidence in the training labels, thereby improving its generalization ability on unseen samples. Let the batch size be B and the total number of classes be C. For the i-th sample, let the logit of the classification head output be... After the softmax function The predicted distribution of the k-th class is obtained for: ； Let the smoothing coefficient be Then the target distribution of the k-th class after smoothing of the i-th sample for: ； The labeled smooth cross-entropy loss is: ； Knowledge distillation loss A teacher-student paradigm is adopted to achieve knowledge transfer. A pre-trained video encoder with fixed parameters and an original classification head are used as the teacher network, while a classifier relying solely on EEG input is used as the student network. The implicit knowledge held by the teacher network is distilled into the student network by calculating the KL divergence between the temperature-scaled outputs of the student and teacher networks on corresponding samples. The mathematical expression is as follows: ； In the formula, and The logit values ​​for the student model and the teacher model are respectively. The distillation temperature. Let KL divergence be a metric. Prototype regularization loss As a regularization constraint, its purpose is to explicitly guide the EEG embedding vector. The mathematical expression for the category center closest to its corresponding category in the feature space is: ； In the formula, Given an EEG embedding vector The logit of category c, Given an EEG embedding vector Real category Logit, prototype regularization loss This is used to prevent the fine-tuned EEG vectors from deviating from the structured feature space with good class discrimination established during the cross-modal alignment stage, thereby ensuring the effectiveness of knowledge transfer and the consistency of the feature space.

Images