Double-domain double-scale cross-subject SSVEP decoding method and device based on transfer learning

Abstract

The invention discloses a double-domain double-scale cross-subject SSVEP (Steady-State Visual Evoked Potential) decoding method and device based on transfer learning, and belongs to the technical field of brain-computer interfaces. The method comprises the following steps: acquiring SSVEP electroencephalogram data of a source domain and SSVEP electroencephalogram data to be decoded of a target domain; sub-band filtering and sine and cosine reference matrix template generation; aiming at source domain data of each sub-band in the source domain multi-sub-band data, constructing a source domain dual-scale template pair of each sub-band for each candidate stimulation frequency, and extracting a single-cycle cross-tested spatial filter; applying time local constraint to target domain data of each sub-band, and extracting an adaptive spatial filter; and calculating multi-path evidence scores, carrying out weighted fusion based on the sub-band weights, and judging and outputting a decoding result. Target subject training data is not needed, high-precision and strong-robustness cross-subject decoding is achieved through double-domain double-scale feature fusion and self-adaptive adaptation, the calculation amount is low, real-time operation is supported, and the practicability and generalization ability of a brain-computer interface are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of brain-computer interface, and particularly relates to a dual-domain and dual-scale cross-subject SSVEP (Steady-State Visually Evoked Potential) decoding method and device based on transfer learning. BACKGROUND

[0002] The brain-computer interface (BCI) based on steady-state visually evoked potential (SSVEP) has attracted extensive attention in the field of brain-computer interface due to high information transmission rate, multiple control targets and other advantages. When a user gazes at a stimulus source that flickers at a stable frequency, the visual cortex of the occipital lobe of the brain will produce an electroencephalogram (EEG) response corresponding to the stimulation frequency and its harmonics. By analyzing the steady-state response in the scalp EEG signal, the user's gaze target can be identified, thereby realizing instruction decoding. The brain-computer interface based on SSVEP has shown very high performance in brain-computer interface applications such as spelling, device control and games.

[0003] It is a core problem of SSVEP-BCI to design an efficient SSVEP feature decoding method. The classic SSVEP decoding methods such as canonical correlation analysis (CCA), filter bank canonical correlation analysis (FBCCA) have limited recognition accuracy due to the existence of certain noise in the signal; task-related component analysis (TRCA) and periodically repeated component analysis (PRCA) and other training recognition algorithms perform well within the subject, but generally rely on a large amount of training data of the target subject. In real-world applications, the cumbersome data collection process will increase the mental fatigue of the user, seriously affecting the practicability of the BCI system; at the same time, the individual differences between subjects generally exist, which exacerbates the instability of cross-subject decoding, and direct filter or template migration will limit the generalization ability and affect the decoding performance and recognition accuracy. In addition, transfer template-based canonical correlation analysis (TTCCA) and cross-subject spatial filter transfer (CSSFT) and other algorithms mostly use the spatial filter obtained by CCA for transfer, only considering the overall steady-state response, and the feature extraction capability is weak, and the template scale is single during transfer, which limits the final recognition effect of the algorithm.

[0004] Therefore, there is an urgent need for a decoding method that can directly utilize the group prior formed by the historical data of other subjects for transfer learning without any calibration or training for the target subject, and still maintain high recognition accuracy and stability under different subject conditions, to reduce deployment costs, shorten deployment cycle and improve the universality and robustness of practical application. SUMMARY

[0005] The technical purpose of the present application is to solve the problems of existing SSVEP brain-computer interface decoding methods, such as dependence on individual training, insufficient cross-subject generalization ability, and single template scale during transfer, and proposes a dual-domain dual-scale cross-subject SSVEP decoding method and device based on transfer learning for SSVEP frequency recognition.

[0006] To achieve the above technical purpose, the embodiments of the present application adopt the following technical solutions.

[0007] In a first aspect, the embodiments provide a dual-domain and dual-scale cross-subject SSVEP decoding method based on transfer learning, comprising: obtaining source domain SSVEP electroencephalogram data and target domain SSVEP electroencephalogram data to be decoded;

[0008] Based on the source domain and target domain SSVEP electroencephalogram data, multi-subband filtering is respectively performed to obtain source domain multi-subband data and target domain multi-subband data; for all candidate stimulation frequencies of SSVEP decoding, corresponding sine and cosine reference matrix templates are generated;

[0009] For source domain data of each subband in the source domain multi-subband data, a source domain overall average template and a source domain single-cycle template are respectively constructed for each candidate stimulation frequency, forming a source domain dual-scale template pair specific to each subband;

[0010] For the source domain single-cycle template of each subband, a single-cycle cross-subject spatial filter of the corresponding subband is extracted;

[0011] For target domain data of each subband in the target domain multi-subband data, a time local constraint is applied to obtain target domain data after time constraint specific to each subband;

[0012] For each subband and the corresponding candidate stimulation frequency, the target domain data of the subband and the sine and cosine reference matrix templates are combined to extract a target domain adaptive filter specific to each subband;

[0013] For each subband, based on the corresponding target domain data after time constraint, the single-cycle cross-subject spatial filter, the target domain adaptive filter, and the source domain dual-scale template pair, three pieces of evidence scores are calculated, and a total evidence score is calculated; by combining preset weight coefficients of each subband, the total evidence scores of all subbands are weighted and fused to obtain a discrimination score of each candidate stimulation frequency, and the candidate stimulation frequency with the maximum discrimination score is selected as the decoding result output.

[0014] In another aspect, the embodiments also provide a dual-domain and dual-scale cross-subject SSVEP decoding device based on transfer learning, comprising:

[0015] A data acquisition module is configured to obtain source domain SSVEP electroencephalogram data and target domain SSVEP electroencephalogram data to be decoded;

[0016] A subband data determination module is configured to perform multi-subband filtering based on source domain and target domain SSVEP electroencephalogram data to obtain source domain multi-subband data and target domain multi-subband data;

[0017] A reference matrix template generation module is configured to generate corresponding sine and cosine reference matrix templates for all candidate stimulation frequencies of SSVEP decoding;

[0018] The template pair generation module is configured to, for source domain data of each subband in the source domain multi-subband data, construct a source domain overall average template and a source domain single-cycle template for each candidate stimulation frequency, and form a source domain double-scale template pair specific to each subband;

[0019] The spatial filter generation module is configured to extract a single-cycle cross-subject spatial filter of a corresponding subband for the source domain single-cycle template of each subband;

[0020] The time-constrained target domain data generation module is configured to apply a time-local constraint to target domain data of each subband in the target domain multi-subband data, and obtain time-constrained target domain data specific to each subband;

[0021] The adaptive filter extraction module is configured to, for each subband and a corresponding candidate stimulation frequency, combine target domain data of the subband and the sine-cosine reference matrix template to extract a target domain adaptive filter specific to each subband;

[0022] The decoding module is configured to, for each subband, calculate three-way evidence scores based on corresponding target domain data or time-constrained target domain data, a single-cycle cross-subject spatial filter, a target domain adaptive filter, and the source domain double-scale template pair, and calculate a total evidence score; combine preset weight coefficients of each subband to perform weighted fusion on the total evidence scores of all subbands, obtain a discrimination score of each candidate stimulation frequency, and select a candidate stimulation frequency with the maximum discrimination score as a decoding result output.

[0023] Compared with the prior art, the beneficial technical effects of the double-domain double-scale cross-subject SSVEP decoding method and device based on transfer learning provided by the embodiments of the present application include:

[0024] 1. The present application does not require individual training data of a target subject, and does not require calibration time, thereby greatly improving the practicability of a BCI system;

[0025] 2. In the transfer feature construction process, both the overall average template and the single-cycle template of the source domain data (training data) of each subband in the source domain multi-subband data are used: the overall average template reflects the overall steady-state law of SSVEP, and the single-cycle template captures the detailed changes of a single stimulation cycle. By fusing these two types of scale features in the transfer process, the present application can maintain overall robustness while taking into account the subtle differences at the cycle level, thereby improving the transfer ability;

[0026] 3. By extracting a stable single-cycle cross-subject spatial filter in the source domain and transferring it to the target domain, and simultaneously combining the specific spatial distribution characteristics of the target domain EEG, the collaborative use of cross-subject knowledge and individual characteristics is realized, and the precision and robustness of cross-subject training-free decoding are significantly improved.

[0027] It is to be understood that the Summary is not intended to identify key or essential features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the disclosure will be readily apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0028] The drawings described herein are for purposes of illustration only and are not intended to limit the scope of the present disclosure in any way.

[0029] In the drawings:

[0030] Figure 1 A flowchart of a dual-domain and dual-scale cross-subject SSVEP decoding method based on transfer learning provided for an embodiment;

[0031] Figure 2 A flowchart of a pre-processing procedure for electroencephalogram signals in an embodiment;

[0032] Figure 3 A flowchart of a procedure for obtaining normalized sub-band signals and a sine-cosine reference matrix template based on pre-processed normalized signals in an embodiment;

[0033] Figure 4 A flowchart of a procedure for obtaining a pair of dual-scale templates based on normalized sub-band signals in an embodiment;

[0034] Figure 5 A flowchart of a procedure for obtaining a single-cycle cross-subject spatial filter based on a single-cycle stacked tensor in an embodiment;

[0035] Figure 6 A flowchart of a procedure for obtaining a time-local constraint result based on target domain data and a sine-cosine reference matrix template in an embodiment;

[0036] Figure 7 A flowchart of a procedure for an or target domain adaptive filter in an embodiment;

[0037] Figure 8 A flowchart of a procedure for performing three-way correlation calculation to output a recognition result and applying the same in an embodiment;

[0038] Figure 9 A technical roadmap diagram of a dual-domain and dual-scale cross-subject SSVEP decoding method based on transfer learning provided for an embodiment. DETAILED DESCRIPTION

[0039] In order for those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor should fall within the scope of protection of the present application.

[0040] A dual-domain and dual-scale cross-subject SSVEP decoding method based on transfer learning, as shown in Figure 1 and Figure 9 , comprises the following steps:

[0041] Step 1: Obtain source domain SSVEP electroencephalogram data and target domain SSVEP electroencephalogram data to be decoded.

[0042] In some embodiments, as shown in Figure 2 , the public Benchmark dataset is used in step 1, and the collected electroencephalogram signals are recorded at the occipital lobe, parietal lobe related electrode sites Pz, POz, PO3, PO4, PO5, PO6, Oz, O1, O2 according to the international 10-10 system, and the sampling frequency f s = 250 Hz.

[0043] In some embodiments, step 1 further comprises standardizing and preprocessing the source domain SSVEP electroencephalogram data and the target domain SSVEP electroencephalogram data to be decoded, thereby obtaining standardized signal input. Taking the public Benchmark dataset as an example, the standardization preprocessing includes: to suppress power frequency interference, first use a 50 Hz comb notch filter for denoising; all trials are uniformly aligned at the stimulation start time.

[0044] Some embodiments take a specified time node (such as 0.5s) after the stimulation start time as the starting point of data analysis, and simultaneously use a uniform latency fixed compensation time (such as 140 ms) for early transient, for example, 0.5s + 0.14s (0.64s) as the final starting point.

[0045] Step 2: As shown in Figure 3 , based on the source domain and target domain SSVEP electroencephalogram data (after standardization preprocessing), multi-subband filtering is respectively performed to obtain source domain multi-subband data and target domain multi-subband data; for all candidate stimulation frequencies of SSVEP decoding, corresponding sine and cosine reference matrix templates (also known as standard sine and cosine templates) are generated.

[0046] To improve the frequency band resolution and anti-interference performance, after the pre-processing is completed, each segment of EEG of the source domain and the target domain is synchronously sent into a sub-band filter bank composed of K (K is a positive integer, and K is 5 in some embodiments) band-pass filters for decomposition.

[0047] As an example, the multi-sub-band filtering is implemented by using a Chebyshev type I band-pass filter bank, and the data is divided into K sub-bands, and the Chebyshev type I band-pass filter bank includes K Chebyshev type I band-pass filters, where K is a positive integer.

[0048] The passband and the stopband of the Chebyshev type I band-pass filter of the kth sub-band are respectively set to [8k, 90] Hz and [8k-2, 100] Hz, the passband ripple is not greater than 0.5 dB, and the stopband attenuation is not less than 40 dB.

[0049] In the source domain, let the frequency be f, the sub-band serial number be k ∈ {1, …, K}, the number of channels be C, and the sampling frequency be f s , the signal length of the standard analysis window is T (the number of sample points), and N tr is the number of trials of the source domain data ， is a real set. Then, for the same candidate stimulation frequency and sub-band, let the mth candidate stimulation frequency f m after sub-band filtering be f s . .

[0050] In the target domain, let the sub-band signal of the target domain trial be , and the target domain trial signal in the kth sub-band after sub-band filtering be: .

[0051] In some embodiments, a sine-cosine reference matrix template is generated for each candidate stimulation frequency of SSVEP decoding, which is the most commonly used reference signal in SSVEP decoding research. s and f m respectively represent the sampling frequency and the mth SSVEP candidate stimulation frequency, N h represents the harmonic number (the harmonic number N h may be set to 5, and may be set):

[0052]

[0053] is the sine-cosine reference matrix template generated for the mth SSVEP candidate stimulation frequency f m .

[0054] Step 3: For each sub-band of the source domain multi-sub-band data, a source domain overall average template and a source domain single-cycle template are constructed for each candidate stimulation frequency, forming a source domain double-scale template pair specific to each sub-band.

[0055] Step 3: The source domain data is superimposed and averaged to obtain an "overall scale" template at each frequency. After upsampling, it is sliced according to the cycle length and superimposed and averaged again to obtain a "single-cycle scale" template, forming a "double-scale" representation of the source domain template.

[0056] In some embodiments, the source domain overall average template and the source domain single-cycle template are constructed as shown in Figure 4 , which includes: for each sub-band, the source domain data of the same candidate stimulation frequency within the sub-band is averaged trial by trial to obtain the source domain overall average template of the candidate stimulation frequency corresponding to the sub-band, expressed as:

[0057] ,

[0058] is the source domain overall average template of the candidate stimulation frequency corresponding to the kth sub-band.

[0059] The source domain overall average template is upsampled along the time axis using two-dimensional interpolation (the upsampling factor can be up=10), and the upsampled source domain overall average template is divided into several cycle segments according to the cycle length (i.e. window length) of the candidate stimulation frequency, obtaining a cycle stacking tensor. In embodiments, the test signal of each source domain training trial is sliced along the time axis according to the window , obtaining a cycle stacking tensor: , is the single-cycle sampling point number (i.e. cycle length) of the mth SSVEP candidate stimulation frequency f m , p m is the cycle segment index, and there are P m cycle segments).

[0060] In embodiments, the single-cycle sampling point number of the mth SSVEP candidate stimulation frequency f m and the divisible cycle P m are as follows:

[0061] ;

[0062] where round(·) represents rounding to an integer, represents rounding down, and up is the upsampling multiple (set to 10 in experiments).

[0063] The single-cycle components (PRC) are obtained by averaging the cross-cycle stacked tensors of all cycles:

[0064] ;

[0065] All single-cycle components are repeated along the time dimension and truncated to the length of the analysis window of the source domain SSVEP electroencephalogram data, obtaining the source domain single-cycle templates corresponding to the candidate stimulation frequencies of the sub-band:

[0066] .

[0067] Thus, a pair of "double-scale" templates is obtained: The former describes the overall steady-state morphology, and the latter emphasizes the cycle-level migratable details, retaining the phase-locked SSVEP components and suppressing non-coherent noise in a statistical sense, making the resulting templates more stable in morphology and more usable in cross-subject migration.

[0068] Step 4: Extract the single-cycle cross-subject spatial filter corresponding to the sub-band for the source domain single-cycle template of each sub-band.

[0069] In some embodiments, as shown in Figure 5 Step 4 includes single-cycle cross-subject spatial filter extraction, and the extraction process includes:

[0070] For the mthSSVEP candidate stimulation frequency f m and the sub-band k, take the single-cycle stacked tensor obtained in step 3 , construct the task-related covariance matrix and the global covariance matrix ;

[0071] Where the task-related covariance matrix is:

[0072] ;

[0073] The global covariance matrix is:

[0074] .

[0075] Solve the generalized eigenvalue problem using the two constructed covariance matrices, and take the eigenvector corresponding to the maximum eigenvalue as the initial single-cycle cross-subject spatial filter:

[0076] , where w represents the spatial filter to be solved in the generalized eigenvalue problem, To solve the eigenvector corresponding to the largest eigenvalue, i.e. the single-cycle cross-subject spatial filter used for subsequent migration.

[0077] The brain spatial responses between different SSVEP stimuli have similarities, so the spatial filters of all candidate stimulus frequencies in the same sub-band are cascaded to further enhance the feature extraction capability, and a single-cycle cross-subject spatial filter is obtained, and the expression is as follows: .

[0078] The single-cycle cross-subject spatial filter is completely learned from the source domain template and is irrelevant to the target subject data, and is used as a fixed projection in the subsequent step, which maps the target trial and the cycle template to the same single-cycle subspace, and is used for correlation evaluation of single-cycle detail similarity.

[0079] The present application is mainly composed of small-scale generalized eigenvalue decomposition and correlation calculation, has low calculation complexity and small memory occupation, and adopts the construction form of the average template, so that a large amount of source domain data is compressed into an average template of one category, greatly reducing the calculation amount and storage requirement, and can complete frequency discrimination in milliseconds on a general-purpose CPU without the need for special acceleration hardware, facilitating embedded or low-power implementation, and having high real-time performance and calculation efficiency.

[0080] Step 5: Apply time-local constraints to the target domain data in each sub-band of the target domain multi-sub-band data to obtain time-constrained target domain data specific to each sub-band.

[0081] Step 5 constructs a time-local Laplacian matrix for each sub-band of the target domain, applies time-local constraints to the target trial signal, enhances the consistency of adjacent time points and suppresses transient noise, thereby reflecting the time structure characteristics of the target domain, and applies the same time-local constraints to the sine and cosine reference signals to maintain consistent constraints.

[0082] To enhance the time consistency in a short time window and suppress non-stationary disturbances, in some embodiments, as shown in Figure 6 , step 5 specifically includes: setting the time scale τ k (unit: sampling points) for the kth sub-band, i.e. embedding the Laplacian matrix L in the covariance matrix to extract the time-local information of the data and improve the recognition performance of the SSVEP. Assuming that the window length used for correlation calculation is T, the construction method of the time-local transformation matrix, i.e. the Laplacian matrix L, is as follows:

[0083] First, based on the time scale τ k , the adjacency matrix A is defined using the Tukey weighting function as follows:

[0084] ;

[0085] wherein:​ ;

[0086] Where i and j are time point indices; τ represents the connection weight between time points i and j; k This represents the time scale parameter set for the k-th sub-band, which determines the temporal neighborhood range of the local constraints; r is the polynomial order, typically taken as r = 3. That is, the closer two time points are in time, the greater their weights.

[0087] Based on the time scale τ k Determine the diagonal matrix :

[0088] ;

[0089] diagonal matrix The diagonal element represents the sum of the connection weights between the i-th time point and all other time points.

[0090] This leads to the Graph Laplace matrix. : .

[0091] The graph Laplacian matrix is ​​used as a local time matrix and simultaneously right-multiplied by the target domain trial signal in the k-th subband before correlation calculation. With respect to the frequency f of the m-th SSVEP candidate stimulus m Generated sine and cosine reference matrix template Transform it in the time dimension to obtain the signal after time local constraint correction. With reference matrix :

[0092] ;

[0093] ;

[0094] This strengthens coherent components and weakens long-distance correlations and slow drift interference within the local temporal neighborhood, thereby enhancing the dynamic information in time and reflecting the temporal structure characteristics of the target domain.

[0095] Step 6: For each subband and its corresponding candidate stimulus frequency, combine the target domain data of that subband with the sine and cosine reference matrix template to extract the target domain adaptive filter for each subband.

[0096] like Figure 7 As shown, step 6 combines the current trial data of the target subject and uses CCA to obtain its own target domain adaptive spatial filter. The target trial is then projected in the same direction as the overall template of the source domain to calculate the consistency of the "overall scale" in order to improve individual fit.

[0097] In an embodiment, the step 6 performs target domain adaptive projection to extract adaptive filter.

[0098] As an example, the step of extracting adaptive filter includes: for the mthSSVEP candidate stimulation frequency f m With the sub-band k, the target domain trial signal on the kthsub-band is With the mthSSVEP candidate stimulation frequency f m The generated sine and cosine reference matrix template (contains N h Perform canonical correlation analysis (CCA) on the target domain:

[0099] ;

[0100] Take the EEG side solution of the first canonical vector As the target domain adaptive filter :

[0101] ;

[0102] The adaptive filter The corresponding canonical correlation coefficient is denoted as .

[0103] Vector As the target domain adaptive spatial projection, it is used later to project the target trial and the source domain overall average template to the same EEG side one-dimensional subspace for consistency comparison, so as to compensate for individual differences and improve the adaptability of cross-subject decoding.

[0104] Step 7: For each sub-band, based on the corresponding target domain data or time-constrained target domain data, single-cycle cross-subject spatial filter, target domain adaptive filter and source domain double-scale template pair, calculate three evidence scores, and calculate the total evidence score; combine the weight coefficients of each sub-band, and weight and fuse the total evidence scores of all sub-bands to obtain the discrimination score of each candidate stimulation frequency, and select the candidate stimulation frequency with the maximum discrimination score as the decoding result output.

[0105] In an embodiment, as shown in Figure 8 Step 7, the calculation of three evidence scores includes FBTCCA path score, single-cycle migration path score and target domain adaptive path score, which specifically includes:

[0106] (a) FBTCCA (Filter Bank Time-weighted Canonical Correlation Analysis) path: FBCCA correlation with time local constraint (individual's own SSVEP response feature).

[0107] The time local weighted canonical correlation analysis is performed on the target domain data of each subband after time constraint and the corresponding sine and cosine reference matrix template of the subband, the first canonical correlation coefficient is taken, and the band signed square of the first canonical correlation coefficient is taken as the FBTCCA path score of the subband.

[0108] In the embodiment, the time local constraint result obtained in step 5 is used The CCA calculation is performed, the first canonical correlation coefficient is taken , and the band signed square is taken as the reference matching FBTCCA path score . .

[0109] (b) Single-cycle migration path: "single-cycle scale" consistency (target domain detail change feature). The target domain data of each subband is projected into the single-cycle feature space with the single-cycle cross-subject spatial filter of the subband respectively, and the single-cycle correlation coefficient of the projected data is calculated, and the band signed square of the single-cycle correlation coefficient is taken as the single-cycle migration path score of the subband.

[0110] In the embodiment, the single-cycle cross-subject spatial filter learned in step 4 is used , the target trial and the source domain single-cycle template are projected into the same single-cycle feature space at the same time, the single-cycle correlation coefficient under the single-cycle migration path at the frequency f m is calculated .

[0111] ;

[0112] Wherein corr(·) is the Pearson correlation coefficient.

[0113] The consistency single-cycle migration path score is obtained :

[0114] .

[0115] (c) Target domain adaptation path: "overall scale" consistency (overall change feature of target domain). Using the target domain adaptation filter of each subband, project the target domain data of the subband and the source domain overall average template into the same feature subspace respectively, calculate the adaptive correlation coefficient of the projected data, and take the band signed square of the adaptive correlation coefficient as the target domain adaptation path score of the subband.

[0116] In the embodiment, the target domain adaptation filter obtained by step 6 , project the target trial and the source domain overall average template into the same EEG side one-dimensional subspace, calculate the adaptive correlation coefficient of the projected data in frequency f m The adaptive correlation coefficient under the target domain adaptation path

[0117]

[0118] Obtain the target domain adaptation path score of "target domain-overall scale" consistency

[0119]

[0120] The above three path scores then enter the multi-path fusion and subband weighting process for final frequency decision.

[0121] In the embodiment, the multi-path fusion and subband weighting process includes: for the mthSSVEP candidate stimulation frequency f m In each subband k=1, …, K, obtain the three correlation scores in step 7 To realize the joint use of "double domain x double scale" consistency in the subband dimension, the embodiment linearly synthesizes the above three evidence scores to obtain the total evidence score corresponding to each subband:

[0122]

[0123] Where w k is the weight coefficient of the kthsubband.

[0124] In some embodiments, the weight coefficients of each subband are configured in advance for subsequent subband weighting fusion.

[0125] As an example, when the output of each subband is fused, the corresponding subband number k is weighted and fused based on the following power law weight:

[0126] Then sum the total evidence scores of all subbands in the frequency dimension to form the discrimination score of the candidate stimulation frequency:

[0127] ​​​​​​​​ ;

[0128] The fusion structure takes the time-local constraint FBTCCA matching as the basis, and introduces two pieces of migration evidence of “source domain overall average template” and “source domain single cycle template”, which not only captures the overall steady state rule of SSVEP, but also strengthens the detail changes of a single stimulation cycle, and combines the specific spatial distribution characteristics of the target domain EEG, to realize the collaborative use of cross-subject knowledge and individual characteristics;

[0129] The system selects the frequency corresponding to the maximum discriminant score among the discriminant scores calculated at all candidate stimulation frequencies as the recognition output: ;

[0130] In the embodiment, the result can be used for brain control device control or man-machine interaction interface display, so as to complete the cross-subject robust decoding under the condition of no training.

[0131] In the embodiment, the classification accuracies of different models are evaluated, as shown in Tables 1 and 2.

[0132] Table 1 shows the classification accuracy results of different models

[0133] Table 2 shows the full-time window average ablation experiment results of 0.6-1.5s in the embodiment

[0134] In the embodiment, the accuracy rate refers to the ratio of the number of correctly recognized trials (T correct ) to the total number of trials (T total ), which is mathematically expressed as:

[0135] .

[0136] Table 1 shows the classification accuracy of each algorithm under different time window lengths (0.6s-1.5s), where the horizontal axis is the time window (s) and the vertical axis is the accuracy rate (%).

[0137] Where Trans-eCCA is a statistical learning method deeply integrated with transfer learning (Transfer Learning) and extended canonical correlation analysis (eCCA).

[0138] CIRCST is a cross-stimulus transfer method using common impulse response (Cross-Stimulus Transfer Method Using Common Impulse Response).

[0139] According to Table 1, it can be analyzed that the classification accuracy of all algorithms increases with the increase of the time window length, which shows that the signal of longer time can provide more stable steady-state response, and thus improve the recognition stability. The dual-domain and dual-scale cross-subject SSVEP decoding method (3DSCST) based on transfer learning provided in the application performs best under all time windows, and the short window is more obvious. The results prove that the method in this paper still maintains high precision under short time window, which shows that it has strong time robustness and generalization; and fully verifies the superiority of 3DSCST in multi-domain feature fusion.

[0140] Table 2 is the ablation experiment result of the full time window of 0.6-1.5s, that is, the influence on the final classification performance after gradually removing each feature module (single cycle detail feature, target domain overall feature, time local constraint, individual feature), and the average accuracy and significance test result (p value) under each condition are given.

[0141] According to Table 2, it can be analyzed that Table 2 shows the average classification accuracy and significance results after removing each feature module independently within the time window of 0.6-1.5s, which is used to verify the effectiveness of different functional modules in the method. In the experiment, the single cycle detail change feature, the target domain overall change feature, the time local constraint and the individual feature are removed respectively, and compared with the complete model. The results show that the complete method (Cond5) achieves the highest average accuracy of 77.94% under all conditions, which is significantly better than the rest of the feature missing conditions (p<0.001). When the individual feature is removed (Cond4), the accuracy decreases most obviously, which is only 64.89%, indicating that the individual SSVEP response feature has the greatest influence on the model performance; when the time local constraint is removed (Cond3), the accuracy decreases to 76.59%, indicating that the local consistency has a positive effect on suppressing transient noise; when the single cycle detail change feature (Cond1) and the target domain overall change feature (Cond2) are removed, the accuracy is 75.97% and 74.34% respectively, which are also significantly lower than the complete model. Therefore, the multi-scale and multi-domain fusion structure proposed in the application plays a key role in the time, space and individual levels, and the missing of any module will weaken the overall recognition performance, which verifies the effectiveness of the proposed method in maintaining high robustness and adaptability under complex cross-subject conditions.

[0142] The above "step 1", "step 2" and the like are only for description purposes, and cannot be understood as indicating or implying relative importance or limiting the order of steps or implicitly indicating the indicated technical features.

[0143] Based on the same inventive concept as the dual-domain and dual-scale cross-subject SSVEP decoding method based on transfer learning provided in the above embodiments, the embodiments of the present application also provide a dual-domain and dual-scale cross-subject SSVEP decoding device based on transfer learning, which comprises a data acquisition module, a sub-band data determination module, a reference matrix template generation module, a template pair generation module, a spatial filter generation module, a target domain data generation module after time constraint, an adaptive filter extraction module and a decoding module.

[0144] The data acquisition module is configured to acquire source domain SSVEP electroencephalogram data and target domain SSVEP electroencephalogram data to be decoded.

[0145] The sub-band data determination module is configured to perform multi-sub-band filtering based on the SSVEP electroencephalogram data of the source domain and the target domain, respectively, to obtain source domain multi-sub-band data and target domain multi-sub-band data.

[0146] The reference matrix template generation module is configured to generate corresponding sine and cosine reference matrix templates for all candidate stimulation frequencies of SSVEP decoding.

[0147] The template pair generation module is configured to, for the source domain data of each sub-band in the source domain multi-sub-band data, construct a source domain overall average template and a source domain single-cycle template for each candidate stimulation frequency, to form a source domain dual-scale template pair exclusive to each sub-band.

[0148] The spatial filter generation module is configured to extract a single-cycle cross-subject spatial filter corresponding to each sub-band for the source domain single-cycle template of each sub-band.

[0149] The target domain data generation module after time constraint is configured to apply a time-local constraint to the target domain data of each sub-band in the target domain multi-sub-band data, to obtain target domain data after time constraint exclusive to each sub-band.

[0150] The adaptive filter extraction module is configured to, for each sub-band and corresponding candidate stimulation frequency, combine the target domain data of the sub-band and the sine and cosine reference matrix templates to extract a target domain adaptive filter exclusive to each sub-band.

[0151] The decoding module is configured to, for each sub-band, calculate three-way evidence scores based on the corresponding target domain data or target domain data after time constraint, the single-cycle cross-subject spatial filter, the target domain adaptive filter and the source domain dual-scale template pair, and calculate a total evidence score; combine preset weight coefficients of each sub-band to weight and fuse the total evidence scores of all sub-bands to obtain a discrimination score of each candidate stimulation frequency, and select the candidate stimulation frequency with the maximum discrimination score as the decoding result output.

[0152] The apparatuses or modules illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a tablet computer, or a combination of any of these devices.

[0153] The dual-domain dual-scale cross-subject SSVEP decoding method and device based on transfer learning provided by the present application are described in detail above, and the principles and implementation manners of the present application are described by using specific examples. The above embodiment descriptions are only used to help understand the concept of the present application and should not be construed as limiting the protection scope of the present application.

Claims

1. A dual-domain dual-scale cross-subject SSVEP decoding method based on transfer learning, characterized in that, The method comprises the following steps: obtaining source domain SSVEP electroencephalogram data and target domain to-be-decoded SSVEP electroencephalogram data; based on the source domain and target domain SSVEP electroencephalogram data, performing multi-subband filtering respectively to obtain source domain multi-subband data and target domain multi-subband data; generating corresponding sine and cosine reference matrix templates for all candidate stimulation frequencies of SSVEP decoding; for the source domain data of each subband in the source domain multi-subband data, a source domain overall average template and a source domain single-cycle template are constructed for each candidate stimulation frequency to form a source domain double-scale template pair specific to each subband; extracting a single-cycle cross-subject spatial filter corresponding to each subband for the source domain single-cycle template of each subband; applying a time-local constraint to the target domain data of each subband in the target domain multi-subband data to obtain target domain data after time constraint specific to each subband; for each subband and the corresponding candidate stimulation frequency, combining the target domain data of the subband and the sine and cosine reference matrix templates to extract a target domain adaptive filter specific to each subband; for each subband, based on the corresponding target domain data or target domain data after time constraint, single-cycle cross-subject spatial filter, target domain adaptive filter and the source domain double-scale template pair, three evidence scores are calculated, and a total evidence score is calculated; combining the preset weight coefficients of each subband, the total evidence scores of all subbands are weighted and fused to obtain a discrimination score of each candidate stimulation frequency, and the candidate stimulation frequency with the maximum discrimination score is selected as the decoding result output.

2. The dual-domain dual-scale cross-subject SSVEP decoding method based on transfer learning according to claim 1, wherein, The method further comprises standardizing and preprocessing the source domain SSVEP electroencephalogram data and the target domain to-be-decoded SSVEP electroencephalogram data, and the standardizing and preprocessing comprises the following steps: using a 50Hz comb notch filter to realize power interference suppression; aligning the time axes of all trials based on the SSVEP stimulation start time; taking a specified time node after the SSVEP stimulation start time as the analysis starting point of the data, and uniformly compensating the latency of all data for a preset time length.

3. The dual-domain dual-scale cross-subject SSVEP decoding method based on transfer learning according to claim 1, wherein, The multi-subband filtering is realized by using a Chebyshev type I bandpass filter set, and the data is divided into K subbands, and the Chebyshev type I bandpass filter set comprises K Chebyshev type I bandpass filters, wherein K is a positive integer.

4. The dual-domain dual-scale cross-subject SSVEP decoding method based on transfer learning according to claim 3, characterized in that, The weight coefficient of each subband is a power-law weight determined according to the corresponding subband number.

5. The dual-domain dual-scale cross-subject SSVEP decoding method based on transfer learning according to claim 1, wherein, The source domain overall average template and the source domain single-cycle template are constructed by the following steps: for each subband, the source domain data of the same candidate stimulation frequency in the subband is averaged trial by trial to obtain a source domain overall average template corresponding to the candidate stimulation frequency of the subband; the source domain overall average template is up-sampled along the time axis by two-dimensional interpolation, and the up-sampled source domain overall average template is divided into several period segments according to the period length of the candidate stimulation frequency to obtain period stacking tensors; the average values of all period stacking tensors are obtained to obtain single-cycle components, and the single-cycle components are repeated along the time dimension and cut off to the same length as the analysis window length of the source domain SSVEP electroencephalogram data to obtain a source domain single-cycle template corresponding to the candidate stimulation frequency of the subband.

6. The dual-domain dual-scale cross-subject SSVEP decoding method based on transfer learning according to claim 5, characterized in that, The specific extraction process of the single-cycle cross-subject spatial filter comprises the following steps: Solving a generalized eigenvalue problem composed of the task-related covariance matrix and the global covariance matrix, obtaining an eigenvector corresponding to a maximum eigenvalue as an initial single-cycle cross-subject spatial filter of a candidate stimulation frequency corresponding to the subband; cascading initial single-cycle cross-subject spatial filters of all candidate stimulation frequencies in the subband to obtain a single-cycle cross-subject spatial filter of the subband. The process of applying time-local constraints includes: for each subband, defining an adjacency matrix A using a Tukey weighting function; 7. The dual-domain dual-scale cross-subject SSVEP decoding method based on transfer learning according to claim 1, characterized in that, Constructing a diagonal matrix D based on the adjacency matrix A; Calculating a graph Laplacian matrix L through the formula L=D-A; and multiplying the graph Laplacian matrix L on the right of target domain data of the subband to obtain time-constrained target domain data of the subband. The three-way evidence score includes an FBTCCA path score, a single-cycle migration path score, and a target domain adaptive path score; 8. The dual-domain dual-scale cross-subject SSVEP decoding method based on transfer learning according to claim 1, wherein, Wherein, performing time-local weighted canonical correlation analysis on the time-constrained target domain data of each subband and a corresponding sine-cosine reference matrix template of the subband, taking a first canonical correlation coefficient, and taking a band sign square of the first canonical correlation coefficient as the FBTCCA path score of the subband; Using the single-cycle cross-subject spatial filter of each subband, respectively projecting the target domain data and the source domain single-cycle template of the subband into a single-cycle feature space to calculate a single-cycle correlation coefficient of the projected data, and taking a band sign square of the single-cycle correlation coefficient as the single-cycle migration path score of the subband; Using the target domain adaptive filter of each subband, respectively projecting the target domain data and the source domain overall average template of the subband into the same feature subspace to calculate an adaptive correlation coefficient of the projected data, and taking a band sign square of the adaptive correlation coefficient as the target domain adaptive path score of the subband. The decoding result is used for brain control device control or man-machine interaction interface display, realizing cross-subject SSVEP robust decoding under no training condition.

9. The dual-domain dual-scale cross-subject SSVEP decoding method based on transfer learning according to claim 1, wherein, It comprises:

10. A dual-domain dual-scale cross-subject SSVEP decoding device based on transfer learning, characterized in that, A data acquisition module for acquiring source domain SSVEP electroencephalogram data and target domain SSVEP electroencephalogram data to be decoded; A subband data determination module for performing multi-subband filtering on the source domain and target domain SSVEP electroencephalogram data respectively to obtain source domain multi-subband data and target domain multi-subband data; A reference matrix template generation module for generating corresponding sine-cosine reference matrix templates for all candidate stimulation frequencies of SSVEP decoding; A template pair generation module for constructing a source domain overall average template and a source domain single-cycle template for each candidate stimulation frequency based on source domain data of each subband in the source domain multi-subband data, forming a source domain double-scale template pair exclusive for each subband; A spatial filter generation module for extracting a single-cycle cross-subject spatial filter of a corresponding subband for a source domain single-cycle template of each subband; A time-constrained target domain data generation module for applying time-local constraints to target domain data of each subband in the target domain multi-subband data to obtain time-constrained target domain data exclusive for each subband; A time-constrained target domain data generation module for applying time-local constraints to target domain data of each subband in the target domain multi-subband data to obtain time-constrained target domain data exclusive for each subband; an adaptive filter extraction module, configured to extract, for each subband and corresponding candidate stimulating frequency, a target domain adaptive filter specific to the subband by combining target domain data of the subband and the sine and cosine reference matrix template; a decoding module, configured to, for each subband, calculate three-way evidence scores based on corresponding target domain data or time-constrained target domain data, the single-cycle across-subject spatial filter, the target domain adaptive filter, and the source domain double-scale template pair, and calculate a total evidence score; and combine preset weight coefficients of each subband to perform weighted fusion on the total evidence scores of all subbands to obtain a discrimination score of each candidate stimulating frequency, and select a candidate stimulating frequency with the largest discrimination score as a decoding result output.

Images