Multimodal brain signal transfer learning method and system

Abstract

The embodiment of the application provides a kind of multi-modal brain signal transfer learning method and system, can solve the problem that the significant distribution difference of neurophysiological signal between different individuals exists in the cross-subject scene of multi-modal brain-computer interface, resulting in the performance of classification model trained in source domain is greatly reduced on target domain.The method comprises: obtaining source domain electroencephalogram features, target domain electroencephalogram features, source domain near-infrared brain function imaging features and target domain near-infrared brain function imaging features based on source domain multi-modal brain signal samples and target domain multi-modal brain signal samples;According to the covariance matrix difference and canonical correlation between the corresponding source domain features and target domain features, respectively determine the statistical distribution alignment loss and correlation alignment loss;In the process of training the model, the domain alignment in early training is mainly based on statistical distribution alignment loss, and the constraint corresponding to feature correlation alignment loss is gradually enhanced in later training;Multi-modal brain signals are classified using the trained model.

Description

Technical Field

[0001] This application relates to the field of brain-computer interface transfer learning technology, and in particular to a multimodal brain signal transfer learning method and system. Background Technology

[0002] Multimodal brain-computer interfaces (BCIs) improve motor imagery classification performance by simultaneously acquiring electroencephalogram (EEG) and near-infrared brain functional imaging (fNIRS) signals, leveraging the high temporal resolution of EEG and the high spatial resolution of fNIRS. However, in cross-subject scenarios, significant differences in the distribution of neurophysiological signals exist among different individuals, leading to a substantial decrease in the performance of classification models trained in the source domain on the target domain. Summary of the Invention

[0003] This application provides a multimodal brain signal transfer learning method and system, which can solve the problem that in multimodal brain-computer interface scenarios across subjects, there are significant differences in the distribution of neurophysiological signals between different individuals, which leads to a significant decrease in the performance of classification models trained in the source domain on the target domain.

[0004] The first aspect of this application provides a multimodal brain signal transfer learning method, including: Source domain multimodal brain signal samples corresponding to the source domain subjects and target domain multimodal brain signal samples corresponding to the target domain subjects are obtained to obtain source domain EEG features, target domain EEG features, source domain near-infrared brain functional imaging features, and target domain near-infrared brain functional imaging features. The source domain multimodal brain signal samples and the target domain multimodal brain signal samples both include EEG signal samples and near-infrared brain functional imaging signal samples. For EEG modalities and near-infrared brain functional imaging modalities, statistical distribution alignment loss is determined based on the difference in covariance matrix between the corresponding source domain features and target domain features, and feature correlation alignment loss is determined based on the canonical correlation between the corresponding source domain features and similar features in the target domain. During the iterative training of the multimodal brain signal transfer learning model, the first alignment strength coefficient corresponding to the statistical distribution alignment loss and the second alignment strength coefficient corresponding to the feature correlation alignment loss are adjusted according to the current training stage, so that the domain alignment in the early stage of training is mainly based on the statistical distribution alignment loss, and the constraint corresponding to the feature correlation alignment loss is gradually strengthened in the later stage of training. The trained multimodal brain signal transfer learning model was used to classify the multimodal brain signals of the subjects in the target domain.

[0005] A second aspect of this application provides a multimodal brain signal transfer learning device, comprising: The acquisition unit is used to acquire source domain multimodal brain signal samples corresponding to the source domain subjects and target domain multimodal brain signal samples corresponding to the target domain subjects, so as to obtain source domain EEG features, target domain EEG features, source domain near-infrared brain functional imaging features and target domain near-infrared brain functional imaging features. The source domain multimodal brain signal samples and the target domain multimodal brain signal samples both include EEG signal samples and near-infrared brain functional imaging signal samples. The determination unit is used to determine the statistical distribution alignment loss based on the difference in the covariance matrix between the corresponding source domain features and target domain features for EEG modal and near-infrared brain functional imaging modal, respectively, and to determine the feature correlation alignment loss based on the canonical correlation between the corresponding source domain features and similar features in the target domain. The training unit is used to adjust the first alignment strength coefficient corresponding to the statistical distribution alignment loss and the second alignment strength coefficient corresponding to the feature correlation alignment loss according to the current training stage during the iterative training of the multimodal brain signal transfer learning model. This ensures that the domain alignment in the early stage of training is mainly based on the statistical distribution alignment loss, and that the constraint corresponding to the feature correlation alignment loss is gradually strengthened in the later stage of training. The classification unit is used to classify the multimodal brain signals of the target domain subjects using the trained multimodal brain signal transfer learning model.

[0006] A third aspect of this application provides an electronic system including a memory and a processor, wherein the processor is used to execute a computer program stored in the memory to implement the steps of the above-described multimodal brain signal transfer learning method.

[0007] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the multimodal brain signal transfer learning method described above.

[0008] In summary, the multimodal brain signal transfer learning method provided in this application processes EEG signals and near-infrared brain functional imaging signals separately through two feature extraction branches with independent parameters. This preserves the physiological characteristics and temporal variation patterns of the two signals, avoiding the degradation of feature representation ability caused by forcibly processing heterogeneous signals with the same set of parameters. Within each modality, statistical distribution alignment first reduces the overall difference between the source and target domains at the covariance structure level, while feature correlation alignment further enhances the correspondence between similar features in the two domains. Therefore, this scheme addresses both the overall distribution shift among subjects and the inconsistency in the expression of category-related features among different subjects. By adjusting the strength of the two alignment losses according to the training phase, this scheme avoids imposing strong category correlation constraints prematurely when features are not yet stable. The model first forms a relatively stable cross-domain statistical structure and then learns more refined cross-domain category relationships, which helps reduce the possibility of the feature space being erroneously pulled or the category boundaries being destroyed. By determining and updating modality weights based on the baseline classification accuracy of the target domain, this scheme can adapt to individual differences among subjects in different target domains. For subjects with high-quality EEG signals, the model can utilize EEG modalities more extensively; for subjects with more discriminative near-infrared brain imaging signals, the model can correspondingly increase the role of that modality. Modality weights can also be updated as the relative effectiveness of the two modalities changes during training. Ultimately, the classification loss maintains the distinguishability of task categories, the two domain alignment losses reduce cross-subject differences, the progressive alignment strength controls the optimization focus at different training stages, and the dynamic modality weights control the relative contribution of the two modalities. This improves the applicability of source domain knowledge transfer to the target domain and enhances the adaptability of the multimodal brain signal transfer learning model to subjects with different target domains, different signal qualities, and different neurophysiological response states.

[0009] Accordingly, the multimodal brain signal transfer learning device, electronic system, and computer-readable storage medium provided in the embodiments of the present invention also have the above-mentioned technical effects. Attached Figure Description

[0010] Figure 1 A flowchart illustrating a possible multimodal brain signal transfer learning method provided in this application embodiment; Figure 2 A comparison of ablation experiments between a progressive alignment scheduling strategy and different alignment strategies in a multimodal brain signal transfer learning method provided in this application embodiment; Figure 3 A weight evolution curve of a dynamic weight fusion module in a multimodal brain signal transfer learning method provided in this application embodiment; Figure 4 A schematic structural block diagram of a possible multimodal brain signal transfer learning device provided in this application embodiment; Figure 5 A schematic diagram of the hardware structure of a possible multimodal brain signal transfer learning device provided in this application embodiment; Figure 6 A schematic structural block diagram of a possible electronic system provided for embodiments of this application; Figure 7 This is a schematic structural block diagram of a possible computer-readable storage medium provided for embodiments of this application. Detailed Implementation

[0011] This application provides a multimodal brain signal transfer learning method and system, which can solve the problem that in multimodal brain-computer interface scenarios across subjects, there are significant differences in the distribution of neurophysiological signals between different individuals, which leads to a significant decrease in the performance of classification models trained in the source domain on the target domain.

[0012] Existing domain adaptation methods have the following main shortcomings: First, single-domain alignment methods, such as using only CORAL or only MMD, can only align low-order or high-order statistics of features, resulting in a single alignment dimension and making it difficult to comprehensively eliminate distribution differences between domains. Second, when multiple alignment constraints are applied simultaneously, different alignment objectives may conflict with each other, leading to feature space distortion or training instability. Third, multimodal fusion often adopts fixed weight or equal weight strategies, failing to adaptively adjust weight allocation according to the actual classification ability of each modality in the target domain. Fourth, existing methods usually treat domain alignment as a static process, without considering the differentiated requirements of alignment strategies at different training stages.

[0013] To resolve the above issues, please refer to [link / reference]. Figure 1 The flowchart of a multimodal brain signal transfer learning method provided in this application embodiment may specifically include: S110-S150.

[0014] S110, acquire source domain multimodal brain signal samples corresponding to the source domain subjects and target domain multimodal brain signal samples corresponding to the target domain subjects, so as to obtain source domain EEG features, target domain EEG features, source domain near-infrared brain functional imaging features and target domain near-infrared brain functional imaging features. The source domain multimodal brain signal samples and the target domain multimodal brain signal samples both include EEG signal samples and near-infrared brain functional imaging signal samples.

[0015] S120 determines the statistical distribution alignment loss based on the difference in the covariance matrix between the corresponding source domain features and target domain features for EEG modalities and near-infrared brain functional imaging modalities, respectively, and determines the feature correlation alignment loss based on the canonical correlation between the corresponding source domain features and similar features in the target domain.

[0016] S130, during the iterative training of the multimodal brain signal transfer learning model, the first alignment strength coefficient corresponding to the statistical distribution alignment loss and the second alignment strength coefficient corresponding to the feature correlation alignment loss are adjusted according to the current training stage, so that the domain alignment in the early stage of training is mainly based on the statistical distribution alignment loss, and the constraint corresponding to the feature correlation alignment loss is gradually strengthened in the later stage of training.

[0017] S140 uses a trained multimodal brain signal transfer learning model to classify the multimodal brain signals of subjects in the target domain.

[0018] In one embodiment, it further includes: Based on the baseline classification accuracy of the EEG modality and the near-infrared brain functional imaging modality in the target domain, the weights of the EEG modality and the near-infrared brain functional imaging modality are determined, and the weights of the EEG modality and the near-infrared brain functional imaging modality are updated during the iterative training process. Based on the classification loss, the statistical distribution alignment loss corresponding to each modality, the feature correlation alignment loss corresponding to each modality, the first alignment strength coefficient, the second alignment strength coefficient, and the modality weights corresponding to each modality, a joint loss function is constructed. The multimodal brain signal transfer learning model is trained using the joint loss function, so as to classify the multimodal brain signals of the target domain subjects using the trained multimodal brain signal transfer learning model.

[0019] It is understandable that in addressing the issue of inconsistent brain signal distributions among different subjects leading to decreased classification performance of transfer models, even when different subjects perform the same motor imagery task, their EEG and near-infrared brain imaging signals may still exhibit significant differences due to variations in head structure, neural activity patterns, hemodynamic responses, sensor installation locations, skin and hair conditions, and the acquisition environment. If a classification model is directly trained using samples from source domain subjects and then applied to target domain subjects, the model may misinterpret individual differences between subjects as differences in task categories. This approach first processes EEG and near-infrared brain imaging signals separately, enabling each modality to extract features according to its own temporal characteristics, frequency characteristics, and physiological formation mechanisms. Subsequently, statistical distribution alignment constraints and feature correlation alignment constraints are simultaneously set within each modality. Statistical distribution alignment reduces the overall distribution differences between source and target domain features, while feature correlation alignment improves the correspondence between similar features in the source and target domains within the shared feature space. Considering that the features extracted in the early stages of model training are still unstable, applying two different alignment constraints simultaneously and forcefully may force the model to satisfy complex cross-domain correlations before a basic category structure is formed, leading to feature space disorder. This scheme adjusts the strength of the two alignment constraints according to the training phase: in the early stages of training, the statistical distribution basis between the source and target domains is primarily established; in the later stages, the correlation constraints between similar features are gradually strengthened. Simultaneously, this scheme does not pre-define that EEG modalities and near-infrared brain functional imaging modalities have the same importance. Instead, it determines the modality weights based on the actual classification performance of the two modalities in the current target domain and updates these weights during training. Thus, the model can adapt to differences in signal quality and modality effectiveness among subjects in different target domains. Finally, by using a joint loss function, the classification task, the two domain alignment tasks, and the contribution levels of the two modalities are incorporated into the same training process, enabling the model to maintain its class discrimination ability while gradually reducing domain differences among different subjects.

[0020] For example, EEG acquisition devices and near-infrared brain imaging (NIBI) acquisition devices can be set up separately for source domain subjects and target domain subjects. Both devices perform signal acquisition according to a unified task prompt time, ensuring that the EEG signal samples and NIBI signal samples correspond to the same task process. For instance, subjects can be asked to perform left-hand motor imagery, right-hand motor imagery, and resting state tasks, respectively. For each task trial, the task start time, task duration, and task category are recorded, and EEG and NIBI signals are extracted according to the corresponding time intervals, thus forming multimodal brain signal samples corresponding to the same trial. EEG signal samples may include sampling sequences from multiple EEG channels within a task time period. NIBI signal samples may include blood oxygenation change sequences corresponding to multiple light source and detector channels. Since the sampling frequencies and physiological response speeds of the two modalities are different, it is not necessary to force them to have exactly the same number of sampling points; instead, the input data can be organized according to their respective sampling frequencies and time windows. Source domain multimodal brain signal samples can come from one source domain subject or multiple source domain subjects. When using multiple source domain participants, their samples can be used together as source domain training data. Target domain multimodal brain signal samples come from the target domain participants to be adapted, with at least a portion of these samples used for target domain feature distribution estimation and modality baseline classification accuracy assessment. For example, the source domain could consist of motor imagery signals from multiple healthy participants, while the target domain could correspond to a patient with cerebral hemorrhage. The source and target domains perform the same task, but the patient's motor cortex electrical activity intensity, hemodynamic response velocity, and activation range of left and right hemispheres may differ from those of healthy participants. By simultaneously acquiring data from both modalities, a more complete signal foundation can be provided for subsequent cross-participant transfer.

[0021] For example, source domain EEG signal samples and target domain EEG signal samples are input into the same EEG feature extraction branch. This branch uses the same network parameters for both source and target domain samples to ensure they are mapped to the same EEG feature space. After feature extraction, source domain EEG features and target domain EEG features are obtained separately. Similarly, source domain near-infrared brain functional imaging (NIBMI) signal samples and target domain NIBMI signal samples are input into the same near-infrared brain functional imaging (NIBMI) feature extraction branch. This branch shares parameters between the source and target domains but not with the EEG feature extraction branch. After feature extraction, source domain NIBMI features and target domain NIBMI features are obtained separately. In specific training processes, source domain samples and target domain samples can be extracted separately in the same training batch. The features output by the EEG feature extraction branch can be organized according to the number of samples, the number of feature channels, and the number of time positions; the features output by the NIBMI feature extraction branch can also be organized according to a form adapted to its signal structure. Subsequently, the time dimension can be aggregated or flattened to obtain feature vectors for domain alignment and classification. For example, for electroencephalogram (EEG) signals, the EEG feature extraction branch can focus on extracting changes in sensorimotor rhythms and the synergistic features between related channels during motor imagery. For near-infrared (NII) brain imaging signals, the NII feature extraction branch can focus on extracting the trend of blood oxygenation changes, peak locations, and response differences between brain regions after the task begins. Furthermore, in a left-hand motor imagery task, the EEG branch might extract rhythmic changes in channels related to the right sensorimotor cortex; the NII branch might extract the features of gradual changes in blood oxygenation concentration in the right motor-related brain regions. Each branch retains the task information best suited to its expression.

[0022] For example, in each model training iteration, a batch of EEG signal samples can be selected from the source domain training set and another batch from the target domain training set. The source domain EEG feature matrix and the target domain EEG feature matrix are obtained through the EEG feature extraction branch, respectively. After centering the two feature matrices, the corresponding covariance matrices are calculated. Subsequently, the matrix distance between the source domain and target domain EEG feature covariance matrices is calculated, and this matrix distance is used as the basis for calculating the statistical distribution alignment loss of the EEG modality. For near-infrared brain functional imaging modality, the same processing method is used to obtain its statistical distribution alignment loss. For example, two feature dimensions in the source domain subjects' EEG features may always have a strong common variation relationship, while the corresponding feature dimensions in the target domain subjects have a weaker common variation relationship. By reducing the difference in covariance matrices, the model will adjust the EEG feature extraction parameters so that the dimensional relationships in the target domain features gradually approach the statistical structure in the source domain that can support classification. Furthermore, the overall fluctuation amplitude of the near-infrared brain functional imaging signal of the target domain subjects may be significantly lower than that of the source domain subjects. Statistical distribution alignment is not simply about amplifying the original signal, but rather about adjusting the feature representation in the feature space so that the discrete structure and dimensional relationship of the features in the two domains are more suitable for processing by the same classification model.

[0023] For example, source domain features and target domain features are organized according to the task category corresponding to the sample. For each category, features corresponding to that category are selected from the source domain features, and features corresponding to the same category are selected from the target domain features. Subsequently, the source domain similar features and target domain similar features are input into the corresponding nonlinear projection structures, mapping them to a shared feature subspace. In the shared feature subspace, canonical correlations between source domain similar features and target domain similar features are calculated, and feature correlation alignment loss is determined based on these canonical correlations. During training, this loss is reduced to improve the consistency of similar features in several major correlation directions. This processing is applied to the EEG modality and the near-infrared brain functional imaging modality, respectively. Feature correlation alignment in the EEG modality is used to find stable class relationships between subjects' electrophysiological characteristics; feature correlation alignment in the near-infrared brain functional imaging modality is used to find stable class relationships between subjects' hemodynamic characteristics. For example, in the left-hand motor imagery category, the EEG signal amplitudes of source domain subjects and target domain subjects may differ, but they may still be consistent in the common direction of change of corresponding features in several brain regions. Canonical correlation alignment can emphasize this stable relationship across subjects without requiring that the values ​​of each feature dimension be completely equal. For near-infrared brain functional imaging signals, the peak blood oxygen response of subjects in the source domain may appear earlier, while the peak response of subjects in the target domain may appear later, but the relative activation relationship between the two in the relevant brain regions may be consistent. Improving canonical correlation through nonlinear projection can preserve this type of class consistency that does not manifest as simple numerical equality.

[0024] For example, the total number of training epochs of the model can be predetermined, and the training process can be divided into pre-training and post-training phases based on these epochs. In each training epoch, a first alignment strength coefficient and a second alignment strength coefficient are calculated based on the current phase. At the beginning of training, the first alignment strength coefficient can be set to a low non-zero value and then gradually increased to allow statistical distribution alignment constraints to smoothly intervene in model training. The second alignment strength coefficient can remain zero or close to zero at this stage, so that the model does not temporarily bear strong constraints related to similar features. As training progresses into the post-training phase, the second alignment strength coefficient can be gradually increased, allowing feature correlation alignment to gradually participate in parameter updates. Simultaneously, the first alignment strength coefficient can be appropriately reduced, but not completely eliminated, thus maintaining the established overall statistical alignment state while strengthening the correspondence between similar features. For example, in a cross-subject transfer scenario for patients with cerebral hemorrhage, the patient's EEG signals and near-infrared brain functional imaging signals may simultaneously exhibit amplitude variations, response delays, and changes in activation regions. In the early stages of training, reducing the overall covariance variance allows patient features to gradually fall within a range suitable for the source domain classifier. Later in training, strengthening the correlation between similar features can further identify the task category structure still retained in the patient features. Conversely, if the number of samples in a particular category within the target domain is small, directly strengthening the correlation between similar features in the early stages of training may cause a small number of samples to have an excessive impact on the model parameters. Establishing a statistical distribution basis using all target domain samples first, and then gradually introducing category correlation constraints, can reduce the possibility of a small number of similar target domain samples causing erroneous biases in the early stages of training.

[0025] For example, before model training begins, baseline classification assessments can be performed using EEG modality data and near-infrared brain functional imaging (NIBMI) modality data, respectively. For the EEG modality, EEG signal samples from the target domain are input into the EEG feature extraction branch and the corresponding classification component to obtain the baseline classification results for the EEG modality. The baseline classification accuracy of the EEG modality is determined based on the consistency between the predicted category and the sample category. The baseline classification accuracy of the NIBMI modality is determined in the same way. Subsequently, the baseline classification accuracy of the two modalities is normalized so that the weights of the EEG modality and the NIBMI modality reflect the relative classification ability between the two modalities. If the baseline classification accuracy of the EEG modality is higher than that of the NIBMI modality, the weight of the EEG modality is correspondingly higher; conversely, the weight of the NIBMI modality is correspondingly higher. During model training, the feature extraction branch and domain alignment state continuously change, and the classification ability of the two modalities in the target domain may also change accordingly. Therefore, the baseline classification accuracy of the two modalities can be reassessed at preset training intervals, and the corresponding modality weights can be updated based on the new assessment results. For example, in the early stages of training, EEG modalities may exhibit strong classification ability due to better electrode contact in the target domain, thus receiving higher weights. As near-infrared brain functional imaging (NIBEM) branches complete cross-domain alignment, their blood oxygenation response characteristics gradually develop better class discrimination capabilities, and the weight of the NIBEM modal can be increased accordingly after reassessment. Furthermore, for a subject with significant EEG artifacts in the target domain, the baseline classification accuracy of the EEG modal may be low, while the NIBEM signal can still stably reflect the activation state of the relevant brain regions. In this case, a dynamic weighting mechanism can reduce the influence of EEG modalities on joint training and increase the relative role of the NIBEM modalities.

[0026] For example, in a single training iteration, the classification loss is first calculated using source domain multimodal brain signal samples and corresponding category information. If target domain category information is available, corresponding target domain training samples can also be used to calculate the classification loss. Subsequently, the statistical distribution alignment loss and feature correlation alignment loss for the EEG modality, and the statistical distribution alignment loss and feature correlation alignment loss for the near-infrared brain functional imaging modality, are obtained respectively. The first and second alignment strength coefficients are obtained based on the current training phase to control the strength of the two alignment losses. The domain alignment losses corresponding to the two modalities are weighted according to the currently evaluated EEG modality weights and near-infrared brain functional imaging modality weights. Finally, the weighted domain alignment loss and the classification loss together constitute a joint loss function. Backpropagation is performed using the joint loss function to update the model parameters of the EEG feature extraction branch, the near-infrared brain functional imaging feature extraction branch, the cross-domain correlation mapping part, and the classification part. After multiple iterations, a trained multimodal brain signal transfer learning model is obtained. In the usage phase, the EEG signals and near-infrared brain functional imaging signals of the target domain subjects are input into the corresponding feature extraction branches. The model generates classification results for target domain brain signals based on features extracted from two modalities. For example, the classification result can represent left-hand motor imagery, right-hand motor imagery, or resting state. For instance, when the EEG features of the target domain sample have formed a similar covariance structure to the source domain EEG features, and the left-hand motor imagery features in the two domains have a strong canonical correlation, the classifier can continue to use the left-hand motor imagery discrimination information learned from the source domain to process the target domain sample. For near-infrared brain functional imaging modality, even if the blood oxygen response amplitude of the target domain subject differs from that of the source domain, as long as the features after domain alignment retain the brain region response relationship related to the task category, this modality can still provide effective information for the final classification.

[0027] like Figure 2 and Figure 3 As shown, in Figure 2 In the diagram, four transfer learning strategies are set on the horizontal axis. The first strategy uses only CORAL statistical distribution alignment; the second strategy uses only DCCA feature correlation alignment; the third strategy uses both CORAL statistical distribution alignment and DCCA feature correlation alignment, but does not adjust the strength of the two alignment methods according to the training stage; the fourth strategy uses the progressive alignment strategy proposed in this invention, that is, CORAL statistical distribution alignment is mainly performed in the early stage of training, and DCCA feature correlation alignment is gradually enhanced in the later stage of training. Figure 2The ordinate represents classification accuracy as a percentage. Each transfer learning strategy corresponds to two sets of bars. The white bars represent ICH patient data (i.e., data from patients with cerebral hemorrhage); the gray bars with diagonal fills represent public datasets. For ICH patient data, the classification accuracy is 70.2% when using only CORAL statistical distribution alignment; 68.5% when using only DCCA feature correlation alignment; 71.8% when using both alignment constraints without progressive scheduling; and 74.9% when using the progressive alignment strategy of this invention. For public datasets, the classification accuracy is 76.3% when using only CORAL statistical distribution alignment; 74.8% when using only DCCA feature correlation alignment; 79.1% when using both alignment constraints without progressive scheduling; and 82.3% when using the progressive alignment strategy of this invention. On both types of data, the classification accuracy obtained by using both alignment methods simultaneously is higher than that obtained by using only one alignment method, indicating that statistical distribution alignment based on covariance matrix differences and feature correlation alignment based on canonical correlation can reduce the feature differences between the source and target domains from different aspects. Furthermore, the classification accuracy obtained by the progressive alignment strategy of this invention is higher than that of the strategy that applies two alignment constraints simultaneously, indicating that establishing the statistical distribution basis between the source and target domains in the early stage of training and gradually strengthening the correlation constraints of similar features in the later stage of training can reduce the possibility of mutual interference between the two alignment constraints before the features are stable. Figure 3 This is a graph showing the changes in EEG modal weights and near-infrared brain functional imaging modal weights with training rounds during the training process. Figure 3 The x-axis represents the number of training epochs, showing a training process that extends to approximately 120 epochs. The y-axis represents the fusion weights, ranging from approximately 0.30 to 0.70. Figure 3 It is evident that the contributions of the EEG and near-infrared brain imaging modalities can be dynamically adjusted during training, rather than maintaining equal weights or a fixed ratio. For example, in the early stages of training, when the EEG modality has relatively strong classification capabilities, it receives a higher weight. As the near-infrared brain imaging branch gradually completes domain adaptation, its weight increases and at one point approaches that of the EEG modality. In the later stages of training, the relative performance of the two modalities changes again, and the weights are redistributed accordingly. This weight variation allows the model to select a relatively reliable modality based on the actual signal state of the subject in the current target domain, mitigating the problem of fixed weights failing to adapt to changes in training states.

[0028] In one embodiment, acquiring source domain multimodal brain signal samples corresponding to the source domain subject and target domain multimodal brain signal samples corresponding to the target domain subject to obtain source domain EEG features, target domain EEG features, source domain near-infrared brain functional imaging features, and target domain near-infrared brain functional imaging features includes: The EEG signal samples and the near-infrared brain functional imaging signal samples are respectively input into the EEG feature extraction branch and the near-infrared brain functional imaging feature extraction branch with independent parameters to obtain source domain EEG features, target domain EEG features, source domain near-infrared brain functional imaging features and target domain near-infrared brain functional imaging features. The EEG feature extraction branch and the near-infrared brain functional imaging feature extraction branch have the same modular network structure and independent network parameters. Each feature extraction branch includes two one-dimensional convolutional layers and one depth-separable convolutional layer arranged sequentially. The one-dimensional convolutional kernel of the EEG feature extraction branch is configured to extract the temporal variation features in the EEG signal sample. The size of the one-dimensional convolutional kernel of the near-infrared brain functional imaging feature extraction branch is set according to the input length of the near-infrared brain functional imaging signal sample to extract the low-frequency variation features in the near-infrared brain functional imaging signal sample.

[0029] For example, the EEG feature extraction branch can receive time series corresponding to multiple EEG channels. A first-layer one-dimensional convolution extracts local temporal changes from the raw or pre-processed signal; a second-layer one-dimensional convolution further expands the sensing range and combines the local patterns obtained from the previous layer; and depthwise separable convolution processes the temporal relationships of each feature channel separately and combines information from different channels. For instance, in a motor imagery task, the first-layer convolution can identify rhythmic fluctuations within a short time range, the second-layer convolution can generate responses to longer task segments, and the depthwise separable convolution can further identify the combination relationships between channels corresponding to different brain regions. The near-infrared brain functional imaging feature extraction branch can receive blood oxygenation change sequences from multiple near-infrared measurement channels. Its first-layer convolution can extract local upward or downward trends in blood oxygenation signals, the second-layer convolution can identify the response formation, duration, and recovery processes, and the depthwise separable convolution can combine the response relationships between channels corresponding to different brain regions. The EEG branch and the near-infrared brain functional imaging branch can have the same number of layers and the same module arrangement, but the kernel size, convolution parameters, and channel combination parameters of each layer are trained separately. For example, the convolutional kernels in the EEG branch cover a relatively short time range but contain a large number of sampling points, while the convolutional kernels in the near-infrared brain functional imaging branch are set with a lower sampling frequency and a longer response time. Between the source and target domains, the same EEG branch uses the same parameters to process EEG samples from both domains, and the same near-infrared brain functional imaging branch also uses the same parameters to process corresponding samples from both domains. Therefore, source domain features and target domain features have a unified feature coordinate meaning within the same modality.

[0030] In one embodiment, determining the statistical distribution alignment loss based on the difference in the covariance matrix between the corresponding source domain features and target domain features includes: Calculate the source domain feature covariance matrix and the target domain feature covariance matrix for the same mode, respectively; Calculate the Frobenius norm distance between the source domain feature covariance matrix and the target domain feature covariance matrix; The statistical distribution alignment loss is determined based on the Frobenius norm distance, so as to align the second-order statistics between the source and target domains by reducing the statistical distribution alignment loss; Wherein, the statistical distribution alignment loss satisfies:

[0031] In the formula, This represents the statistical distribution alignment loss. This represents the feature dimension of the corresponding modality. Represents the source domain feature covariance matrix. This represents the feature covariance matrix of the target domain. This represents the Frobenius norm.

[0032] For example, in a training batch, a set of source domain EEG features is extracted from source domain EEG signal samples, and a set of target domain EEG features is extracted from target domain EEG signal samples. The features corresponding to each sample are organized into feature vectors, forming source domain feature matrices and target domain feature matrices respectively. The mean of each feature dimension in the two feature matrices is calculated, and this mean is subtracted from the corresponding feature to center the feature matrices. The source domain EEG feature covariance matrix and the target domain EEG feature covariance matrix are calculated based on the centered feature matrices. The corresponding elements of the two covariance matrices are subtracted, the Frobenius norm square of the difference matrix is ​​calculated, and normalization is performed in conjunction with the feature dimensions to obtain the EEG modality statistical distribution alignment loss. The near-infrared brain functional imaging modality is calculated using the same process to obtain its statistical distribution alignment loss. During training, the statistical distribution alignment loss is used in backpropagation. The model does not directly modify the original signal, but rather adjusts the parameters of the feature extraction branch so that its output source domain feature covariance matrix and target domain feature covariance matrix gradually approach each other. For example, two dimensions in source domain EEG features might describe rhythmic changes in the motor cortex and coordinated changes in adjacent brain regions, respectively, and these two dimensions have a stable correlation within the source domain. In the target domain, due to individual differences, the correlation between these two dimensions may be weaker. Covariance alignment can help restore the dimensional relationships between target domain features and task discrimination. As another example, near-infrared brain functional imaging features in the source domain show consistent changes in blood oxygenation response across certain brain regions, while the corresponding relationships in the target domain are not obvious due to differences in response amplitude. By aligning the covariance matrix, feature representations can be adjusted to make the statistical correlations between these brain regions closer to the effective patterns in the source domain.

[0033] In one embodiment, determining the feature relevance alignment loss based on the typical correlation between the corresponding source domain features and similar features in the target domain includes: From the source domain features and target domain features corresponding to the same modality, determine the source domain features and target domain features with the same category; The source domain features and target domain features of the same category are respectively input into the corresponding nonlinear projection network to map the source domain features and the target domain features to a shared feature subspace. The nonlinear projection network includes two fully connected layers. A cross-domain feature matrix is ​​constructed based on the source domain features and target domain features mapped to the shared feature subspace, and singular value decomposition is performed on the correlation matrix obtained based on the cross-domain feature matrix. The feature correlation alignment loss is determined based on the sum of the first preset number of singular values ​​obtained from the singular value decomposition, so as to increase the typical correlation between the source domain features and the target domain similar features by reducing the feature correlation alignment loss.

[0034] For example, source domain features and target domain features are organized according to task categories. For instance, for the left-handed motor imagery category, source domain left-handed motor imagery features and target domain left-handed motor imagery features are selected; for the right-handed motor imagery category, corresponding source domain features and target domain features are selected. Similar source domain features are input into a source domain-side nonlinear projection network, and similar target domain features are input into a target domain-side nonlinear projection network. Both projection networks can include two fully connected layers, and the hidden layer dimension can be larger than the input feature dimension to improve the network's ability to express nonlinear relationships. The output dimensions after projection are kept consistent to compute correlations in a shared feature subspace. The projected source and target domain features are centered and subjected to necessary numerical stabilization processing, and then a cross-domain feature matrix reflecting the relationship between the two domain features is constructed. A correlation matrix is ​​obtained from this matrix, and singular value decomposition is performed on the correlation matrix. A predetermined number of large singular values ​​are selected, and the feature correlation alignment loss is determined based on the sum of these singular values. During training, reducing this loss encourages the selected singular values ​​to increase, thereby enhancing the typical correlation between similar features in the source and target domains. The above processing was performed on the EEG modality and the near-infrared brain functional imaging modality, respectively. The two modalities did not share the same set of cross-domain feature matrices to avoid interference between the feature scales and physiological meanings of different signals. For example, the EEG features of left-hand motor imagery in the source and target domains may differ significantly in specific values, but after nonlinear projection, they may both exhibit patterns related to right-side sensorimotor cortex activity in several shared directions. Singular value decomposition can extract these main correlation directions. For the near-infrared brain functional imaging modality, the peak blood oxygenation response of subjects in the source domain may appear earlier, while that of subjects in the target domain may appear later, but the relative response relationships of different brain regions may still be consistent. Nonlinear projection can convert different response patterns into comparable feature representations, and then enhance the class consistency between the two through canonical correlation constraints.

[0035] In one embodiment, adjusting the first alignment strength coefficient corresponding to the statistical distribution alignment loss and the second alignment strength coefficient corresponding to the feature correlation alignment loss according to the current training stage includes: The total training rounds of the multimodal brain signal transfer learning model are divided into an early training phase and a late training phase, with the early training phase accounting for 30% to 50% of the total training rounds and the late training phase accounting for 50% to 70% of the total training rounds. During the initial training phase, the first alignment strength coefficient is gradually increased from a first initial value to a first target value, while the second alignment strength coefficient is kept at zero or less than a preset strength threshold. In the later training phase, the first alignment strength coefficient is gradually decreased from the first target value, and the second alignment strength coefficient is gradually increased from zero or less than the preset strength threshold to the second target value.

[0036] For example, the initial training phase can be set at 30% to 50% of the total training rounds, with the remaining rounds designated as the later training phase. The specific proportion can be determined based on the sample size, model convergence speed, and the degree of difference between the target and source domains. In the initial training phase, the first alignment strength coefficient gradually increases from an initial value to a target value. This change can be implemented evenly across training rounds or gradually increased after several training rounds. Simultaneously, the second alignment strength coefficient remains zero or at a low level. In the later training phase, the second alignment strength coefficient is gradually increased, allowing feature correlation alignment to begin to have a significant effect. The first alignment strength coefficient can be gradually decreased to a non-zero value, ensuring the model maintains the established covariance alignment while strengthening similar correlations. For instance, when the source and target domains differ significantly, the initial training phase can be appropriately extended, giving the model more time to adjust the overall feature distribution. When the source and target domains have similar signal acquisition conditions, the initial training phase can be appropriately shortened, allowing similar correlation constraints to participate in training earlier. Furthermore, the target domain features of brain-injured patients may differ significantly from the source domain features of healthy subjects. In the early stages of training, enhancing covariance alignment can reduce the differences in overall feature scale caused by pathological conditions. After the model forms a basic class structure, increasing canonical correlation constraints can prevent prematurely mapping of the patient's unstable features to the source domain class center.

[0037] In some examples, determining the EEG modality weights and near-infrared brain functional imaging modality weights based on their respective baseline classification accuracy in the target domain includes: The baseline classification models corresponding to the EEG modal and the near-infrared brain functional imaging modal were used respectively to classify and evaluate the target domain EEG signal samples and the target domain near-infrared brain functional imaging signal samples, and the baseline classification accuracy of the EEG modal and the near-infrared brain functional imaging modal were obtained. The EEG modal weights and the near-infrared brain functional imaging modal weights are determined according to the following formula:

[0038]

[0039] In the formula, This represents the weights of the EEG modalities. This represents the modal weights of the near-infrared brain functional imaging. This indicates the baseline classification accuracy of the stated EEG modality. This indicates the baseline classification accuracy of the near-infrared brain functional imaging modality; During the training process of the multimodal brain signal transfer learning model, the baseline classification accuracy of the EEG modality and the baseline classification accuracy of the near-infrared brain functional imaging modality are obtained again every 5 to 15 training rounds, and the EEG modality weights and the near-infrared brain functional imaging modality weights are updated.

[0040] For example, before formally performing joint training of the model, baseline classification assessments are performed on the EEG modality and the near-infrared brain functional imaging modality, respectively. For the EEG modality, EEG signal samples from the target domain are input into the EEG feature extraction branch and the corresponding classification part. The baseline classification accuracy of the EEG modality is determined based on the consistency between the output category and the actual category. The baseline classification accuracy of the near-infrared brain functional imaging modality is determined in the same way. The baseline classification accuracy of the EEG modality is divided by the sum of the baseline classification accuracies of the two modalities to obtain the EEG modality weight. The baseline classification accuracy of the near-infrared brain functional imaging modality is also divided by the sum of the baseline classification accuracies of the two modalities to obtain the near-infrared brain functional imaging modality weight. During joint training, the baseline classification assessment is re-performed every 5 to 15 training epochs. The updated weights are used for joint loss calculation in subsequent training epochs. The update interval should not be too short. If re-evaluation is performed for every training batch, fluctuations in the prediction of a small number of samples may cause frequent changes in weights, affecting training stability. The update interval should also not be too long, otherwise the weights may not be able to reflect the performance changes of a certain modality after domain alignment in a timely manner. Setting update intervals across several training rounds can strike a balance between response speed and weight stability. For example, if a subject's EEG electrode contact is poor in a certain target domain, the initial classification ability of the EEG modality is weak, while the near-infrared brain functional imaging modality can more stably identify the blood oxygenation response of the relevant brain regions. In the initial stage, the near-infrared brain functional imaging modality can be assigned a higher weight. As the EEG branches complete several rounds of statistical distribution alignment, individual differences in EEG features gradually decrease, and its baseline classification accuracy may improve. After re-evaluation, the weight of the EEG modality will also increase accordingly, allowing the model to reuse the gradually recovering EEG category information. As another example, if a subject in a certain target domain has a significant hemodynamic response delay, the near-infrared brain functional imaging modality has weak classification ability in the early stages of training. Through feature extraction and cross-domain alignment, the model may gradually identify category-related brain region relationships. Periodic updates allow this modality to gain appropriate weight as its classification ability improves, rather than remaining in a state of low contribution indefinitely.

[0041] In some examples, the joint loss function satisfies:

[0042] In the formula, Denotes the joint loss function, This represents the classification loss. This represents the weights of the EEG modalities. This represents the modal weights of the near-infrared brain functional imaging. This represents the first alignment strength coefficient. This represents the second alignment strength coefficient. and These represent the statistical distribution alignment losses corresponding to the EEG modality and the near-infrared brain functional imaging modality, respectively. and These represent the feature correlation alignment loss corresponding to the EEG modality and the near-infrared brain functional imaging modality, respectively. The model parameters of the multimodal brain signal transfer learning model are updated using the Adam optimizer based on the joint loss function, and the learning rate of the Adam optimizer is 0.0005 to 0.002.

[0043] For example, in a training batch, two modalities are obtained through EEG feature extraction and near-infrared brain imaging feature extraction branches, respectively, and the classification loss is calculated based on the task category probabilities output by the model. Subsequently, the EEG modality statistical distribution alignment loss, EEG modality feature correlation alignment loss, near-infrared brain imaging modality statistical distribution alignment loss, and near-infrared brain imaging modality feature correlation alignment loss are calculated. A first alignment strength coefficient and a second alignment strength coefficient are obtained based on the current training epoch. The statistical distribution alignment loss and feature correlation alignment loss for each modality are adjusted using the two strength coefficients, and the overall effect of the corresponding losses for the two modalities is adjusted using the EEG modality weights and near-infrared brain imaging modality weights. The domain alignment loss after strength adjustment and modality weight adjustment is combined with the classification loss to form a joint loss function. Backpropagation is performed using the joint loss function to calculate the gradients corresponding to each model parameter, and the model parameters are updated using the Adam optimizer. The Adam optimizer can maintain the first-moment and second-moment estimates of the model parameter gradients separately and adjust the update magnitude of different parameters accordingly. For models that simultaneously include multiple feature extraction branches and multiple loss terms, the gradient scales of different parameters may vary significantly. The Adam optimizer can reduce the instability caused by a uniform fixed update amplitude. The learning rate can be set in the range of 0.0005 to 0.002. When the number of training samples is small or the target domain features fluctuate greatly, a relatively low learning rate can be used to ensure smooth update of model parameters. When the training data is sufficient and the joint loss decreases slowly, a relatively high learning rate can be selected within this range. For example, in the early stages of training, the first alignment strength coefficient is high, and the second alignment strength coefficient is low. The joint loss mainly promotes the classifier to form basic class discrimination ability and reduce the difference in covariance matrix. In the later stages of training, the second alignment strength coefficient gradually increases, and the joint loss further promotes the formation of stable correlations between similar features in the source domain and similar features in the target domain. As another example, when the weight of the EEG modality in the current target domain is high, the domain alignment loss corresponding to the EEG modality has a significant impact on parameter updates; when subsequent evaluations show that the near-infrared brain functional imaging modality classification ability improves, its weight increases, and the corresponding loss's impact on model training also increases.

[0044] In some cases, existing protocols assume a relatively stable correspondence between electroencephalography (EEG) activity and the hemodynamic response reflected by near-infrared brain functional imaging (NIBEM). However, for subjects in the target domain who have suffered from cerebral hemorrhage, cerebral infarction, cerebral vascular stenosis, or are receiving vasoactive drug treatment, neurovascular decoupling may occur between neuroelectric activity and blood oxygenation response. For example, when a target domain subject performs left-hand motor imagery, the EEG signal may show clear sensorimotor rhythm changes, but changes in oxygenated hemoglobin in the brain regions surrounding the lesion may be significantly delayed, abnormal in amplitude, or even show the opposite direction of change to those in healthy subjects in the source domain. In such cases, the NIBEM modality may still exhibit relatively stable repeatability, thus achieving a baseline classification accuracy that appears not low on the surface. Existing dynamic weighting mechanisms may continue to assign it a high weight. Canonical correlation alignment may also force abnormal blood flow responses in the target domain to approximate normal responses in the source domain, thereby erasing pathologically significant differences. Based on this, in one embodiment, before performing cross-domain alignment, the timing of neural activity changes is determined from the EEG features of the target domain, and the onset time, direction, and duration of the hemodynamic response are determined from the near-infrared brain functional imaging features of the target domain. The neurovascular coupling state corresponding to the subject in the target domain is determined based on the response time interval, the degree of consistency in response direction, and the stability across trials between the two modalities. When the neurovascular coupling state meets the normal coupling conditions, statistical distribution alignment and feature correlation alignment of the two modalities are performed according to the original scheme. When the neurovascular coupling state indicates a stable time delay, the task-correspondence interval of the near-infrared brain functional imaging features is first corrected according to the estimated hemodynamic delay, and then the correlation alignment of similar features is performed. When the neurovascular coupling state indicates an abnormal response direction or no stable correspondence between neural activity and hemodynamic response, the use of common category correspondence constraints between the two modalities is stopped, the EEG modality and the near-infrared brain functional imaging modality are migrated separately, and the influence of abnormal near-infrared brain functional imaging features on EEG branch parameters is limited.

[0045] In some cases, considering that EEG signals typically recover quickly after a task, while the hemodynamic response of near-infrared brain functional imaging (NIBEM) signals exhibits significant delay and persistence, when the interval between adjacent task trials is short, the NIBEM signal captured in the current trial may simultaneously contain blood oxygenation responses from the previous trial that have not yet subsided. For example, the previous trial involved left-hand motor imagery, while the current trial involves right-hand motor imagery. The first half of the current NIBEM sample may primarily reflect the left-hand motor imagery from the previous trial, while the sample label is overall marked as right-hand motor imagery. This contamination may not necessarily appear as obvious noise in the original signal and may even exhibit a smooth and stable trend. Therefore, it is difficult to detect using conventional signal quality assessment. If canonical correlation alignment is directly performed, the model may learn the residual response of the previous category as a feature of the current category. Based on this, in one embodiment, for each target domain NIBEM sample, the current task category, the previous task category, and the blood oxygenation slope before the start of the current task are obtained. Based on the slope of blood oxygenation changes before the task begins, the inflection point of the response after the task begins, and the reference blood flow response morphology corresponding to the previous task category, the residual response interval caused by the previous task is determined in the current sample. When the residual response interval and the current task response interval can be distinguished temporally, the residual response component is separated from the current sample, and near-infrared brain functional imaging features for feature correlation alignment are extracted only based on the current task response component. When the two response components cannot be reliably separated, the near-infrared brain functional imaging sample is not used for feature correlation alignment of the same type, but its uncontaminated latter interval can still be used for statistical distribution estimation, or only the corresponding EEG sample can be used for classification training of the current task. Different processing methods can also be set according to whether the previous task category and the current task category are the same. If the two adjacent task categories are the same, the residual response and the current response have consistent category semantics, and the degree of rejection can be reduced; if the categories are different, the strength of residual response identification and isolation is increased.

[0046] In some cases, considering that subjects performing different tasks in the target domain may simultaneously experience changes in respiratory rate, heart rate, blood pressure, or scalp blood flow, these systemic physiological changes can also affect near-infrared brain functional imaging (NIBEM) signals. If a task cue itself causes tension, breath-holding, or physical exertion, the NIBEM model may achieve higher classification accuracy based on scalp blood flow changes rather than cortical activity. Existing dynamic modality weighting mechanisms interpret this spurious classification ability as reliable NIBEM modality, thereby increasing its weight. Based on this, in one embodiment, the NIBEM channel can be divided into a reference channel that mainly reflects superficial tissue changes and a task channel that mainly covers the target brain region. Physiological change features are extracted from the reference channel and the task channel, and common change components in the two channels are identified. If a classification feature appears simultaneously in both the reference channel and the task channel and does not have target brain region specificity, it is marked as a systemic physiological interference feature. The baseline classification accuracy of the original NIBEM features and the baseline classification accuracy after weakening the systemic physiological interference features are calculated separately. When the difference between the two classification accuracies exceeds a preset range, the near-infrared brain functional imaging modality weights are adjusted for confidence based on the degree of difference, instead of directly using the original baseline classification accuracy to calculate the weights. This also reduces the impact of feature dimensions labeled as whole-body physiological interference features on loss calculation during covariance matrix alignment and canonical correlation alignment.

[0047] In some cases, considering that EEG acquisition devices and near-infrared brain imaging (NIBI) acquisition devices may have different internal clocks, even if synchronization is achieved at the start of acquisition, the time axes of the two devices may gradually shift as acquisition time increases. This shift may be smaller than a single task time window and therefore not manifest as significant data loss, but in long-term training tasks, the EEG and NIBI responses in later trials may gradually correspond to different task stages. If the two modalities are directly organized according to a unified task label, the model may misinterpret the actual cross-modal time misalignment as individual differences in the target domain. Based on this, in one embodiment, time reference information related to task cues is extracted from the EEG and NIBI signals, respectively. The time reference information can come from synchronization trigger markers or from recurring stable signal changes after the task starts. The time correspondence between the two devices is calculated at multiple time points during the acquisition process, and a clock drift function is determined based on the offset at different time points, rather than calculating only a fixed time offset. The task time boundary of the NIBI samples is segmented and corrected according to the clock drift function to ensure that the NIBI response interval of each task trial is consistent with the corresponding EEG task interval. During periods when reliable correction is not possible, the constraint of the cross-modal joint classification loss on the samples in that period is paused, but the two modalities are still allowed to learn intra-domain features separately.

[0048] In some cases, the task category ratios in the source and target domains may differ. For example, in the target domain, patients may not complete some right-hand motor imagery trials due to fatigue, resulting in a significantly larger number of left-hand motor imagery samples than right-hand motor imagery samples. The global covariance matrix is ​​affected by both intra-category differences and category ratios. If the global covariance matrices of the source and target domains are directly aligned, the model may compensate for the category ratio differences by shifting features of a minority of categories, thus disrupting the discrimination space of the minority categories. Therefore, in one embodiment, the source domain feature covariance matrix and the target domain feature covariance matrix can be calculated separately for each task category, and the category reliability can be determined based on the number of valid samples for each category in both domains. For categories with sufficient samples, covariance alignment under category conditions is implemented. For categories with insufficient samples in the target domain, the unstable target domain covariance matrix is ​​not used directly; instead, a restricted alignment direction is determined based on the stable feature subspace of the corresponding category in the source domain and the reliability dimension of the small number of samples in the target domain. When calculating the overall statistical distribution alignment loss, category quality normalization is applied to the loss of each category to prevent the majority of categories from dominating model parameter updates simply because of their larger sample size.

[0049] In some cases, it is considered that all samples from a subject in a target domain are typically treated as a single target domain. However, during long-term acquisition, the EEG cap may move slightly, and the near-infrared probe may change position or contact pressure due to head movement. This change does not necessarily lead to complete channel failure, but may cause the same channel to correspond to different spatial positions or have different signal scales before and after acquisition. Therefore, a sample from a subject in a target domain may actually contain multiple temporally distributed subdomains. If the same covariance matrix is ​​calculated for all target domain samples, the model will mix the differences before and after sensor displacement, resulting in an average distribution that does not correspond to any real acquisition state. Based on this, in one embodiment, the spatial covariance characteristics of the EEG channels, the response relationship of the near-infrared channels, and the channel quality index are continuously calculated according to the acquisition time sequence. When the spatial covariance structure or channel quality relationship between adjacent time intervals undergoes a continuous abrupt change, it is determined that the target domain acquisition state has changed, and the target domain samples are divided into multiple target subdomains with the change time as the boundary. Statistical distribution alignment loss, feature correlation alignment loss, and modal weights are calculated for each target subdomain. For subdomains before and after sensor displacement, the same target domain covariance matrix is ​​not directly shared, but the classification category semantics can be shared. When the model processes new target domain samples, it selects the appropriate domain alignment parameters or modal weights based on the similarity of the sample to the channel structure of each target subdomain.

[0050] In some cases, while progressive scheduling can stagger the two types of alignment constraints in time, it cannot guarantee that the two losses will always be coordinated in the later stages of training. In some batches, the parameter update direction required to reduce covariance differences may be opposite to the update direction required to improve canonical correlations. If fixed coefficients are still set only according to the training rounds, the two losses may cancel each other out in local training phases, or one loss may destroy the alignment result already formed by the other loss. Based on this, in one embodiment, for the same modality, the gradient directions of the statistical distribution alignment loss and the feature correlation alignment loss relative to the shared feature extraction parameters are calculated separately. The alignment consistency state in the current training batch is determined according to the angle or inner product relationship between the two gradient directions. When the two gradient directions are basically consistent, the parameters are updated normally according to the intensity coefficient set in the current training phase. When the two gradient directions are significantly opposite, the feature dimension causing the conflict is identified, and one of the losses is restricted to update parameters along the conflict dimension; or the alignment loss that has reached a stable state is temporarily maintained, and only the other loss is used to update along the non-conflict direction. This processing is implemented separately for the EEG modality and the near-infrared brain functional imaging modality, rather than using a global conflict judgment result.

[0051] In some cases, when healthy subjects in the source domain and brain-damaged patients in the target domain perform the same motor imagery task, the task category may be the same, but the brain regions responsible for the task may differ. Patients may compensate using areas surrounding the lesion, the contralateral hemisphere, or supplementary motor areas. If similar features are directly aligned according to channel location, the model may force the patient's compensatory brain region features to closely resemble the original functional areas of the healthy subject. This could not only cause negative transfer but also potentially eliminate the patient's genuinely effective functional reorganization features as domain shifts. Therefore, in one embodiment, a brain region functional relationship structure can be constructed based on the task-conditional correlations between brain region features in the source and target domains. This functional relationship structure describes the relative synergistic relationships between different brain regions under the same task category, without requiring the source and target domains to use brain regions in the same spatial location to perform the same function. Based on the source and target domain brain region functional relationship structures, functional substitution regions in the target domain are determined. For example, if the relationship between a region in the contralateral hemisphere of the target domain and other task-related brain regions is similar to the relationship between the lesion-side motor area of ​​the source domain and healthy subjects, then this region is identified as a candidate functional substitution region. When performing feature relevance alignment, instead of forcibly establishing correspondences based on fixed channel numbers, the correspondences between source domain brain regions and candidate functional substitute regions of the target domain are established according to the functional relationship structure. For compensatory features that are unique to the target domain and have stable class discrimination capabilities, their target domain-specific components are preserved, and alignment is performed only on the parts that are semantically consistent with the category of the source domain.

[0052] In some cases, near-infrared brain functional imaging signals are affected by hair, skin, probe pressure, and optical path length. Changes in probe contact state may cause changes in the signal amplitude and covariance structure of certain channels, but the signal remains continuous and may not be recognized by conventional bad channel detection. If such changes are directly used for domain alignment, the model may learn the probe contact state instead of the brain region's blood oxygenation response. Based on this, in one embodiment, for each near-infrared channel, the response relationship between the two wavelength signals, the spatial support relationship between adjacent channels, and the stability of the channel's class response across multiple task trials are extracted. When the two wavelength relationship of a channel is abnormal and its change is not supported by adjacent channels, the channel is identified as an optical path contact abnormality channel, rather than a brain region functional abnormality channel. When calculating the near-infrared brain functional imaging covariance matrix, the contribution of the channel and its related feature dimensions is reduced. When implementing canonical correlation alignment, only feature dimensions that have adjacent spatial support and cross-trial stability are used. If the channel subsequently recovers, its lost contribution is gradually restored based on its stability across multiple consecutive task trials, rather than immediately restoring all weights.

[0053] In some cases, existing dynamic modality weights are determined based on the baseline accuracy of each modality's individual classification. However, modalities with low individual classification accuracy are not necessarily without value. A modality may fail to independently distinguish all categories but can provide crucial auxiliary information on samples where another modality is most prone to error. If weights are assigned solely based on the overall classification accuracy of a single modality, such modalities with remedial value may be incorrectly downweighted. For example, EEG modality can distinguish motor imagery from resting states well but easily confuses left-hand motor imagery with right-hand motor imagery. Near-infrared brain functional imaging modality may have low overall classification accuracy but can provide information on spatial differences in brain regions when EEG modality confuses left-hand and right-hand categories. Based on this, in one embodiment, in addition to calculating the individual baseline classification accuracy of each modality, it is also determined whether the second modality can provide correct category information when the first modality is in a low-confidence state or has a classification error. The conditional remedial capability of a modality is determined based on the correction ratio of a modality for difficult samples of another modality, the stability of the correction results, and the degree of overlap of error samples between the two modalities. Modality weights are determined not only based on the overall classification accuracy of a single modality but also adjusted according to conditional recovery capabilities. For modalities with lower overall classification accuracy but capable of consistently correcting errors in another modality, their corresponding weights are retained. During specific classification, the influence of auxiliary modalities can be adjusted based on the classification confidence of the primary modality in the current sample. When the confidence of the primary modality is high, auxiliary modalities maintain a normal contribution; when the confidence of the primary modality is low, the influence of modalities with conditional recovery capabilities on the current sample is increased.

[0054] Please see Figure 4One embodiment of the multimodal brain signal transfer learning device in this application may include: The acquisition unit 21 is used to acquire source domain multimodal brain signal samples corresponding to the source domain subject and target domain multimodal brain signal samples corresponding to the target domain subject, so as to obtain source domain EEG features, target domain EEG features, source domain near-infrared brain functional imaging features and target domain near-infrared brain functional imaging features. The source domain multimodal brain signal samples and the target domain multimodal brain signal samples both include EEG signal samples and near-infrared brain functional imaging signal samples. The determination unit 22 is used to determine the statistical distribution alignment loss based on the difference in the covariance matrix between the corresponding source domain features and the target domain features for the EEG modality and the near-infrared brain functional imaging modality, respectively, and to determine the feature correlation alignment loss based on the canonical correlation between the corresponding source domain features and the same type of features in the target domain. Training unit 23 is used to adjust the first alignment strength coefficient corresponding to the statistical distribution alignment loss and the second alignment strength coefficient corresponding to the feature correlation alignment loss according to the current training stage during the iterative training of the multimodal brain signal transfer learning model, so that the domain alignment in the early stage of training is mainly based on the statistical distribution alignment loss, and the constraint corresponding to the feature correlation alignment loss is gradually strengthened in the later stage of training. Classification unit 24 is used to classify multimodal brain signals of subjects in the target domain using a trained multimodal brain signal transfer learning model.

[0055] above Figure 4 The multimodal brain signal transfer learning device in this application embodiment has been described from the perspective of modular functional entities. The following is a detailed description of the multimodal brain signal transfer learning device in this application embodiment from the perspective of hardware processing. Please refer to... Figure 5 One embodiment of the multimodal brain signal transfer learning device 300 in this application includes: The system includes an input device 301, an output device 302, a processor 303, and a memory 304, wherein the number of processors 303 can be one or more. Figure 5 Taking a processor 303 as an example. In some embodiments of this application, the input device 301, output device 302, processor 303, and memory 304 can be connected via a bus or other means, wherein... Figure 5 Taking the example of a connection between China and Israel via a bus.

[0056] Among them, by calling the operation instructions stored in memory 304, processor 303 is also used to execute Figure 1 Any of the methods in the corresponding embodiments.

[0057] Please see Figure 6 , Figure 6A schematic diagram of an embodiment of the electronic system provided in this application.

[0058] like Figure 6 As shown, this application provides an electronic system including a memory 410, a processor 420, and a computer program 411 stored in the memory 410 and executable on the processor 420. When the processor 420 executes the computer program 411, it can achieve... Figure 1 Any of the corresponding implementation methods in the embodiments.

[0059] Since the electronic system described in this embodiment is the device used to implement a multimodal brain signal transfer learning device in the embodiments of this application, those skilled in the art can understand the specific implementation method and various variations of the electronic system in this embodiment based on the method described in the embodiments of this application. Therefore, how the electronic system implements the method in the embodiments of this application will not be described in detail here. Any device used by those skilled in the art to implement the method in the embodiments of this application is within the scope of protection of this application.

[0060] Please see Figure 7 , Figure 7 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided in this application.

[0061] like Figure 7 As shown, this embodiment provides a computer-readable storage medium 500 on which a computer program 511 is stored. When the computer program 511 is executed by a processor, it can perform... Figure 1 Any of the corresponding implementation methods in the embodiments.

[0062] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0063] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0064] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0065] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0066] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0067] This application also provides a computer program product, which includes computer software instructions that, when executed on a processing device, cause the processing device to perform actions such as... Figure 1 The flowchart of the multimodal brain signal transfer learning method in the corresponding embodiment.

Claims

1. A multimodal brain signal transfer learning method, characterized in that, include: Source domain multimodal brain signal samples corresponding to the source domain subjects and target domain multimodal brain signal samples corresponding to the target domain subjects are obtained to obtain source domain EEG features, target domain EEG features, source domain near-infrared brain functional imaging features, and target domain near-infrared brain functional imaging features. The source domain multimodal brain signal samples and the target domain multimodal brain signal samples both include EEG signal samples and near-infrared brain functional imaging signal samples. For EEG modalities and near-infrared brain functional imaging modalities, statistical distribution alignment loss is determined based on the difference in covariance matrix between the corresponding source domain features and target domain features, and feature correlation alignment loss is determined based on the canonical correlation between the corresponding source domain features and similar features in the target domain. During the iterative training of the multimodal brain signal transfer learning model, the first alignment strength coefficient corresponding to the statistical distribution alignment loss and the second alignment strength coefficient corresponding to the feature correlation alignment loss are adjusted according to the current training stage, so that the domain alignment in the early stage of training is mainly based on the statistical distribution alignment loss, and the constraint corresponding to the feature correlation alignment loss is gradually strengthened in the later stage of training. The trained multimodal brain signal transfer learning model was used to classify the multimodal brain signals of the subjects in the target domain.

2. The method according to claim 1, further comprising: Based on the baseline classification accuracy of the EEG modality and the near-infrared brain functional imaging modality in the target domain, the weights of the EEG modality and the near-infrared brain functional imaging modality are determined, and the weights of the EEG modality and the near-infrared brain functional imaging modality are updated during the iterative training process. Based on the classification loss, the statistical distribution alignment loss corresponding to each modality, the feature correlation alignment loss corresponding to each modality, the first alignment strength coefficient, the second alignment strength coefficient, and the modality weights corresponding to each modality, a joint loss function is constructed. The multimodal brain signal transfer learning model is trained using the joint loss function, so as to classify the multimodal brain signals of the target domain subjects using the trained multimodal brain signal transfer learning model.

3. The method according to claim 1, characterized in that, The acquisition of source domain multimodal brain signal samples corresponding to the source domain subjects and target domain multimodal brain signal samples corresponding to the target domain subjects, to obtain source domain EEG features, target domain EEG features, source domain near-infrared brain functional imaging features, and target domain near-infrared brain functional imaging features, includes: The EEG signal samples and the near-infrared brain functional imaging signal samples are respectively input into the EEG feature extraction branch and the near-infrared brain functional imaging feature extraction branch with independent parameters to obtain source domain EEG features, target domain EEG features, source domain near-infrared brain functional imaging features and target domain near-infrared brain functional imaging features. The EEG feature extraction branch and the near-infrared brain functional imaging feature extraction branch have the same modular network structure and independent network parameters. Each feature extraction branch includes two one-dimensional convolutional layers and one depth-separable convolutional layer arranged sequentially. The one-dimensional convolutional kernel of the EEG feature extraction branch is configured to extract the temporal variation features in the EEG signal sample. The size of the one-dimensional convolutional kernel of the near-infrared brain functional imaging feature extraction branch is set according to the input length of the near-infrared brain functional imaging signal sample to extract the low-frequency variation features in the near-infrared brain functional imaging signal sample.

4. The method according to claim 1, characterized in that, The step of determining the statistical distribution alignment loss based on the difference in the covariance matrix between the corresponding source domain features and target domain features includes: Calculate the source domain feature covariance matrix and the target domain feature covariance matrix for the same mode, respectively; Calculate the Frobenius norm distance between the source domain feature covariance matrix and the target domain feature covariance matrix; The statistical distribution alignment loss is determined based on the Frobenius norm distance, so as to align the second-order statistics between the source and target domains by reducing the statistical distribution alignment loss; Wherein, the statistical distribution alignment loss satisfies: In the formula, This represents the statistical distribution alignment loss. This represents the feature dimension of the corresponding modality. Represents the source domain feature covariance matrix. This represents the feature covariance matrix of the target domain. This represents the Frobenius norm.

5. The method according to claim 2, characterized in that, The step of determining the feature relevance alignment loss based on the typical correlation between the corresponding source domain features and similar features in the target domain includes: From the source domain features and target domain features corresponding to the same modality, determine the source domain features and target domain features with the same category; The source domain features and target domain features of the same category are respectively input into the corresponding nonlinear projection network to map the source domain features and the target domain features to a shared feature subspace. The nonlinear projection network includes two fully connected layers. A cross-domain feature matrix is ​​constructed based on the source domain features and target domain features mapped to the shared feature subspace, and singular value decomposition is performed on the correlation matrix obtained based on the cross-domain feature matrix. The feature correlation alignment loss is determined based on the sum of the first preset number of singular values ​​obtained from the singular value decomposition, so as to increase the typical correlation between the source domain features and the target domain similar features by reducing the feature correlation alignment loss.

6. The method according to claim 1, characterized in that, Adjusting the first alignment strength coefficient corresponding to the statistical distribution alignment loss and the second alignment strength coefficient corresponding to the feature correlation alignment loss according to the current training phase includes: The total training rounds of the multimodal brain signal transfer learning model are divided into an early training phase and a late training phase, with the early training phase accounting for 30% to 50% of the total training rounds and the late training phase accounting for 50% to 70% of the total training rounds. During the initial training phase, the first alignment strength coefficient is gradually increased from a first initial value to a first target value, while the second alignment strength coefficient is kept at zero or less than a preset strength threshold. In the later training phase, the first alignment strength coefficient is gradually decreased from the first target value, and the second alignment strength coefficient is gradually increased from zero or less than the preset strength threshold to the second target value.

7. The method according to claim 2, characterized in that, The determination of EEG modality weights and near-infrared brain functional imaging modality weights based on their respective baseline classification accuracy in the target domain includes: The baseline classification models corresponding to the EEG modal and the near-infrared brain functional imaging modal were used respectively to classify and evaluate the target domain EEG signal samples and the target domain near-infrared brain functional imaging signal samples, and the baseline classification accuracy of the EEG modal and the near-infrared brain functional imaging modal were obtained. The EEG modal weights and the near-infrared brain functional imaging modal weights are determined according to the following formula: In the formula, This represents the weights of the EEG modalities. This represents the modal weights of the near-infrared brain functional imaging. This indicates the baseline classification accuracy of the stated EEG modality. This indicates the baseline classification accuracy of the near-infrared brain functional imaging modality; During the training process of the multimodal brain signal transfer learning model, the baseline classification accuracy of the EEG modality and the baseline classification accuracy of the near-infrared brain functional imaging modality are obtained again every 5 to 15 training rounds, and the EEG modality weights and the near-infrared brain functional imaging modality weights are updated.

8. A multimodal brain signal transfer learning device, characterized in that, include: The acquisition unit is used to acquire source domain multimodal brain signal samples corresponding to the source domain subjects and target domain multimodal brain signal samples corresponding to the target domain subjects, so as to obtain source domain EEG features, target domain EEG features, source domain near-infrared brain functional imaging features and target domain near-infrared brain functional imaging features. The source domain multimodal brain signal samples and the target domain multimodal brain signal samples both include EEG signal samples and near-infrared brain functional imaging signal samples. The determination unit is used to determine the statistical distribution alignment loss based on the difference in the covariance matrix between the corresponding source domain features and target domain features for EEG modal and near-infrared brain functional imaging modal, respectively, and to determine the feature correlation alignment loss based on the canonical correlation between the corresponding source domain features and similar features in the target domain. The training unit is used to adjust the first alignment strength coefficient corresponding to the statistical distribution alignment loss and the second alignment strength coefficient corresponding to the feature correlation alignment loss according to the current training stage during the iterative training of the multimodal brain signal transfer learning model. This ensures that the domain alignment in the early stage of training is mainly based on the statistical distribution alignment loss, and that the constraint corresponding to the feature correlation alignment loss is gradually strengthened in the later stage of training. The classification unit is used to classify the multimodal brain signals of the target domain subjects using the trained multimodal brain signal transfer learning model.

9. An electronic system comprising a memory and a processor, characterized in that, When the processor executes a computer program stored in the memory, it implements the steps of the multimodal brain signal transfer learning method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the multimodal brain signal transfer learning method as described in any one of claims 1 to 7.

Images