DTWICA-based thought typewriting character signal template construction method

Abstract

The invention discloses a DTWICA-based thought typewriting character signal template construction method, and relates to construction of a thought typewriting character signal template, which comprises the following steps of: firstly, constructing preprocessing, DTWICA and template construction on equipment, and visualizing and classifying aligned data; the electroencephalogram signals are sent into a DTWICA model, the model constructs a distortion function for the EEG signals of each trial through FastDTW, and signal decomposition is carried out based on FastICA to obtain time factors and neuron factors; carrying out mapping deformation on the decomposed time factor through a distortion function, and combining the time factor with a neuron factor to form a new EEG signal; then, a sliding time window is used for carrying out self-adaptive processing on the new EEG signal, and a deformed EEG signal is obtained; finally, averaging the deformed EEG signals of different test times to obtain each type of character signal template; according to the method, the performance of the template is verified in the imaginary sentence data set, and technical support is provided for real-time imaginary handwriting typing.

Description

Technical Field

[0001] The present invention relates to the field of electroencephalogram (EEG) signal research, and particularly to a method for constructing an ideographic typing character signal template based on DTWICA. Background Art

[0002] It is known that any activity of all living organisms is accompanied by a discharge process. When a human being has a heartbeat, blinks the eyelids, or thinks, a voltage can be generated. Taking the example of thinking, a voltage of 0.2 to 1 millivolt can be generated. If electrodes are implanted in the brain, the voltage fluctuations generated during thinking can be captured. These electrical signals can also be detected by electrodes placed near the brain. By analyzing these electrical signals with equipment, scientists can read brain waves; that is, these brain waves constitute the physiological basis for the control of matter by the mind. The control of matter by the mind is to analyze these brain waves through professional equipment to manipulate matter. The discovery and research of brain waves by humans have a history of nearly a hundred years, and scientists have also proven that there is a certain correlation between brain waves and consciousness states. The brain wave frequencies of a person in different consciousness states are different. Scientists interpret this EEG information and convert it into specific action instructions; in 1998, scientists from the University of Emmerich and the University of Tübingen in Germany implanted a thin glass electrode into the brain of a stroke patient who was paralyzed, and then connected it to a computer and interpreted his brain signals. This patient could "see" the cursor on the computer screen and control its movement through consciousness. This marked the possibility of direct interaction between the human brain and the computer. Subsequently, neuroscientist John Donoghue of Brown University improved this system and named it "brain-computer interface". This technology aims to create a "thought dictionary" for humans. With the continuous development of brain-computer interface technology, ideographic typing, as an important application technology, has become an important direction in EEG signal research; however, EEG signals often have the characteristics of high complexity and low signal-to-noise ratio. In the ideographic handwriting task, the time difference between trials may lead to difficulties in signal alignment, thus affecting the classification performance. Especially in the multi-trial handwriting task, the time dynamic differences between different trials are significant. How to effectively eliminate these differences and improve the stability and accuracy of the signals has become a major challenge in EEG signal research. Summary of the Invention

[0003] The purpose of the present invention is to disclose a method for constructing an ideographic typing character signal template based on DTWICA based on the existing deficiencies. The classification accuracy of the single-character template constructed by the present invention on the imagined handwriting single-character dataset is relatively high.

[0004] In order to achieve the invention purpose of the present invention, the following technical solutions are disclosed in this application:

[0005] A method for constructing an ideographic typing character signal template based on DTWICA, the method for constructing the ideographic typing character signal template comprising:

[0006] Step S1, first construct preprocessing, DTWICA, template construction, visualization of aligned data, and classification on the device;

[0007] Step S2, collect the electroencephalogram of the user through an electrode array arranged on the brain;

[0008] Step S3, in view of the fact that the baseline discharge rates of different data blocks will drift over time, normalize the discharge rate of each electrode, and then perform Gaussian filtering on the neural activity data in the time dimension. The standard deviation of the Gaussian filtering is set to 3, and the degree of smoothing is controlled;

[0009] Step S4, DTWICA stage: DTWICA processes the data of multiple trials through time warping (FastDTW) and independent component analysis (FastICA), which not only considers the spatial structure of the data, but also considers the time differences between different trials; capture the differences caused by time delays, different experimental stages, or other time variations between trials through DTWICA processing;

[0010] Step S5, template construction stage: after obtaining the time warped data corresponding to the character, first perform mean processing on all trials along the trial dimension to obtain a more regular electroencephalogram signal;

[0011] Step S6, after the time alignment processing of the overall DTWICA framework, perform t-SNE (t-Distributed Stochastic Neighbor Embedding) dimensionality reduction processing on the neural activity data to map the high-dimensional data to a two-dimensional space;

[0012] Step S7, template evaluation stage: classify the aligned single-character data using ten-fold cross-validation based on a KNN (K-Nearest Neighbors) classifier, and evaluate the classification accuracy.

[0013] In the method for constructing an ideographic typing character signal template based on DTWICA, in the process of step S3, in the formula, x represents the current data in each sample, represents the mean of the current block of the sample; in step S4, DTWICA not only learns the time factor (time*components) and the neuron factor (neuron*components), but also learns a time warping function W for each trial k; Among them, each column of the time factor represents the activity pattern of an independent component at each time point, and the neuron factor represents the weight of each neuron on each independent component; Although there may be differences in the time axis among different trials, DTWICA aligns these trials into a shared time space through time warping, so that the EEG signals between different trials can be compared and analyzed in the same time frame.

[0014] In the method for constructing an ideographic typing character signal template based on DTWICA, in step S5, the average value of each time point and each neuron is calculated, and NaN values are ignored to eliminate individual differences between trials; Then, for the time dimension, one-dimensional Gaussian filtering is performed to smooth the time series and reduce noise; Subsequently, according to the time length reconstructed by each character handwriting, the effective time window of the character is intercepted; For the obtained time window, it is sent to PCA for training to perform dimensionality reduction on neural activities; The data after dimensionality reduction will be restored to the original space, but only the first 10 principal components are retained, so as to achieve noise reduction and remove components with low contributions; Finally, through these processing steps, a spatio-temporal template is obtained, providing high-quality template data for subsequent signal analysis and classification tasks.

[0015] In the method for constructing an ideographic typing character signal template based on DTWICA, in step S6, the perplexity in the t-SNE algorithm is used to control the balance of local data structures, and its value is set to 20; Subsequently, ten-fold cross-validation based on a KNN classifier is used to classify the aligned single-character data to evaluate the classification accuracy; This step aims to verify the visualization effect and classification performance of the data after dimensionality reduction after time alignment, so as to evaluate the effectiveness of the DTWICA alignment method in practical tasks.

[0016] In the method for constructing an ideographic typing character signal template based on DTWICA, the overall framework of DTWICA in step S6 is divided into the following three stages, namely signal alignment, iterative optimization, and sliding time window processing;

[0017] Among them, FastDTW is used in signal alignment to initialize the warping function W k , and perform preliminary time alignment on the data; Specifically, first calculate the mean value of each trial of the single-character data to obtain an initial template . Then, calculate the path relationship between the neural activity of each trial and this initial template through FastDTW to obtain the relative time offset, and construct the warping function W of each trial k . Finally, the warping function W kApplied to the raw data to obtain preliminarily aligned data; the core idea of this process is to calculate the similarity between each trial and the initialized template through the dynamic programming algorithm of FastDTW, and construct a warping function accordingly; FastDTW efficiently aligns two time series through multi-scale approximation and local search window; specifically, its core operation is: align the time series at a rough resolution to obtain a general matching path; then refine the path at a higher resolution to narrow the alignment range; finally, recursively perform this process layer by layer until the alignment is completed at the original resolution; the process is as follows:

[0018] Coarse-graining: Perform standard DTW on each trial and the initial template according to the following formula (1) to obtain a general matching path; where the input data X k ∈R t×n represents the activities of t time points and n neurons in the k-th trial, (i,j) is the index of the time point;

[0019] D(i, j) = Dist(i, j) + min[D(i - 1, j), D(i, j - 1), D(i - 1, j - 1)] (1)

[0020] Projection: Project the coarse-grained path points onto the high-resolution sequences X k and to expand a local window around the path points to limit the search range for precise alignment;

[0021] Fine-graining: Perform standard DTW within the local window, calculating the distance only within the window to reduce the time complexity;

[0022] Then perform FastICA matrix decomposition on the preliminarily aligned data to obtain the initial time factor T and neuron factor N, as shown in the following formula (1); FastICA is a commonly used blind source separation algorithm for separating statistically independent components from multi-dimensional signals; its core idea is to use non-Gaussianity as a measure of independence and extract independent components by maximizing negentropy: FastICA is used to extract potential independent components from EEG activity data and decompose them into T and N. The specific steps are as follows:

[0023]

[0024] To ensure the stability and performance of the algorithm, first centralize the data according to the following formula (1) so that the signal mean of each neuron is zero;

[0025]

[0026] Next, to remove the correlation in the data and standardize the variance of the data, the whitening method in formula (1) below is used to process the centered data; first, calculate the covariance matrix of the data and perform eigenvalue decomposition to obtain the eigenvectors and eigenvalues; then, standardize the data by using the reciprocal of the square root of the eigenvalues to obtain the whitened data X w . Among them, E is the eigenvector matrix and D is the eigenvalue matrix;

[0027]

[0028] Initialize the weight vector: Randomly initialize a weight vector ω, which is used to gradually extract independent components;

[0029] Iterative optimization: Through non-Gaussianity metrics (such as using nonlinear activation functions like tanh or u 3 etc.), optimize each extracted independent component; the process of updating the weight vector is as shown in formula (1), where y = Xw × ω is the signal obtained by projecting through the weight vector;

[0030]

[0031] Orthogonalization and normalization: To ensure that the extracted components are linearly independent, after each update, remove the linear components of the already extracted components through orthogonalization and normalize to obtain a weight vector of unit length;

[0032] Convergence check: If the change between the new and old weight vectors is less than the set threshold, it is considered that the algorithm has converged and the iteration stops;

[0033] The time factor is calculated through T = X w W, where W is the finally extracted mixing weight matrix; the neuron factor is obtained through N = W -1 ; these factors can reveal the potential independent sources in the signal, corresponding to the time dynamic pattern and the neuron spatial pattern respectively; to make the time factor and the neuron factor have the same scale, they are scaled according to formulas (1) and (2) below respectively;

[0034]

[0035] Here, T scaled and N scaled are the scaled time factor and neuron factor respectively, and "||*||2" represents calculating the Euclidean norm of each column of the factor matrix;

[0036] Using the current time warping function W k , time factor T, the warped time factor W k (T) is obtained and the reconstructed data predicted by the model can be expressed as formula (1) below, The signal representing the estimated template;

[0037]

[0038] The iterative optimization calculates the reconstruction error L of the neural signal according to the following formula (1) recon , and calculates the errors of the time factor, neuron factor, and warping function in the same way;

[0039] A. Construct the total loss function L total , as shown in the following formula (2);

[0040]

[0041] L total = L recon + L T + L N + L W (2)

[0042] B. Calculate the gradients of the loss function with respect to T, N, and W k ;

[0043] C. Update the time factor T, neuron factor N, and time warping function W k ;

[0044] D. Repeat the above steps until the loss function converges or reaches the maximum number of iterations;

[0045] Sliding time window processing

[0046] Perform sliding time window processing on the predicted data to further extract time-domain features and obtain the final time warping function W k , time factor T, neuron factor N, model prediction data

[0047] The method for constructing an ideographic typing character signal template based on DTWICA, the system of the method for constructing an ideographic typing character signal template includes:

[0048] An electrode array acquisition module in the brain is used to acquire electroencephalograms for identifying and decoding specific brain features or intentions; a data analysis module is used to analyze raw data, neurons, trials, and time data analysis; a data preprocessing unit is used to perform data preprocessing on electroencephalograms; a DTWICA model construction module is used to introduce a time warping factor to model the time series of single spike events and estimate the uncertainty of time warping through Bayesian inference, significantly improving the detection ability for neural sequences of different durations; a data secondary analysis module is used to analyze the secondary analysis of raw data, neurons, trials, and time data; a DTWICA model module is used for dynamic time warping independent component analysis; t-SNE: t-distributed stochastic neighbor embedding; PCA denoising is performed on the data averaged over trials, and an electroencephalogram feature extraction module is used for character spatio-temporal templates, and ten-fold cross-validation based on KNN is used to evaluate the performance of the constructed templates.

[0049] The method for constructing an ideographic typing character signal template based on DTWICA described above uses a device including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described in any one of the foregoing.

[0050] Through the above disclosure, the beneficial effects of the present invention are:

[0051] For the method for constructing an ideographic typing character signal template based on DTWICA of the present invention, the classification accuracy of the single character template constructed by the present invention on the imagined handwritten single character data set can reach 94.39%. In addition, the present invention verifies the performance of the model in template construction in a larger imagined sentence data set; the results show that the model of the present invention can effectively eliminate the time differences between different trials, thereby extracting stable and representative features, providing strong technical support for the development of subsequent electroencephalogram signal analysis methods. Description of the Drawings

[0052] Figure 1 is the overall framework diagram of the present invention;

[0053] Figure 2 is the 2D visualization diagram of t-SNE neural activities of different alignment models of the present invention;

[0054] Figure 3 is the ten-fold cross-validation comparison diagram based on KNN of the present invention;

[0055] Figure 4 The confusion matrix diagram of different alignment models under the KNN classifier;

[0056] where the horizontal axis represents the predicted labels of different models, and the vertical axis represents the true labels. The values in the confusion matrix represent the recall rate of each character, ranging from 0 to 1;

[0057] Figure 5 are the alignment effect diagrams of the twisted function solved by different methods of the present invention;

[0058] Taking the character 'a' as an example, the neural activities in 5 independent components (ICs) are selected;

[0059] Figure 6 is the schematic diagram of the influence of the matrix decomposition method of the present invention on the template accuracy; Detailed implementation manners

[0060] The following will combine the accompanying drawings to elaborate on the preferred embodiments of the present invention, so as to more clearly understand the invention purpose, features and advantages of the present invention. It should be understood that the embodiments shown in the drawings are not limitations on the scope of the present invention, but only to explain the implementation manners of the technical solution of the present invention that are not limited to one kind.

[0061] Combined with the Figures 1 - 3 The method for constructing an idea typing character signal template based on DTWICA described above, the method for constructing an idea typing character signal template includes:

[0062] Step S1, referring to the Figure 1 First, construct preprocessing, DTWICA, template construction, visualization of aligned data and classification on the device;

[0063] Step S2, collect the electroencephalogram of the user through the electrode array set in the brain;

[0064] Step S3, in view of the fact that the baseline discharge rate of different data blocks will drift over time, normalize the discharge rate of each electrode, as shown in formula (1) below. Wherein, x represents the current data in each sample, represents the mean value of the current block of this sample; then perform Gaussian filtering on the neural activity data in the time dimension, and set the standard deviation of Gaussian filtering to 3 and control the degree of smoothing;

[0065] Step S4, DTWICA stage: DTWICA processes the data of multiple trials through time warping (FastDTW) and independent component analysis (FastICA), which not only considers the spatial structure of the data, but also considers the time differences between different trials; through time warping, DTWICA can capture the differences caused by time delays, different experimental stages or other time variations between trials; specifically, DTWICA not only learns the time factor T (time * components) and the neuron factor N (neuron * components), but also learns a time warping function W for each trial kAmong them, each column of the time factor represents the activity pattern of an independent component at each time point, and the neuron factor represents the weight of each neuron on each independent component. Although there may be differences in the time axis among different trials, DTWICA aligns these trials into a shared time space through time warping, so that the EEG signals between different trials can be compared and analyzed within the same time frame.

[0066] Step S5, template construction stage: After obtaining the time warping data corresponding to the characters, first perform mean processing on all trials along the trial dimension to obtain more regular EEG signals. Specifically, the present invention calculates the average value of each time point and each neuron, and ignores NaN values, thereby eliminating individual differences between trials. Then, for the time dimension, the present invention performs one-dimensional Gaussian filtering to smooth the time series and reduce noise. Subsequently, according to the time length reconstructed by each character's handwriting, the effective time window of the character is intercepted. For the obtained time window, the present invention sends it into PCA for training to perform dimensionality reduction on neural activities. The data after dimensionality reduction will be restored to the original space, but only the first 10 principal components are retained, thereby achieving noise reduction and removing low-contribution components. Finally, after these processing steps, the present invention obtains a spatio-temporal template, providing high-quality template data for subsequent signal analysis and classification tasks.

[0067] Step S6, after the time alignment processing of the DTWICA framework, perform t-SNE (t-Distributed Stochastic Neighbor Embedding) dimensionality reduction processing on the neural activity data to map the high-dimensional data to a two-dimensional space;

[0068] Step S7, template evaluation stage: Use ten-fold cross-validation based on the KNN (K-Nearest Neighbors) classifier to classify the aligned single-character data and evaluate the classification accuracy.

[0069] The overall framework of DTWICA combines the attached Figure 1 As shown, it can be roughly divided into the following three stages, namely signal alignment, iterative optimization, and sliding time window processing;

[0070] Among them, FastDTW is used in signal alignment to initialize the warping function W k , and perform preliminary time alignment on the data. Specifically, first calculate the mean value of each trial of the single-character data to obtain the initial template . Then, calculate the path relationship between the neural activity of each trial and the initial template through FastDTW, so as to obtain the relative time offset, and construct the warping function W of each trial k . Finally, the warping function Wk Applied to the original data, preliminary aligned data is obtained. The core idea of this process is to calculate the similarity between each trial and the initialized template through the dynamic programming algorithm of FastDTW, and construct a warping function accordingly. FastDTW efficiently aligns two time series through multi-scale approximation and local search window. Specifically, its core operations are as follows: Align the time series at a rough resolution to obtain a general matching path. Then refine this path at a higher resolution to narrow the alignment range. Finally, recursively perform this process layer by layer until the alignment is completed at the original resolution. The process is as follows:

[0071] Coarsening, perform standard DTW on each trial and the initial template according to formula (2) to obtain a general matching path. Among them, the input data X k ∈R t×n represents the activities of t time points and n neurons in the k-th trial, (i,j) is the index of the time point.

[0072] D(i, j) = Dist(i, j) + min[D((i - 1, j), D(i, j - 1), D(i - 1, j - 1)] (2)

[0073] Projection, project the coarse-grained path points onto the high-resolution sequences X k and extend a local window around the path points to limit the search range of the exact alignment.

[0074] Fine-graining. Perform standard DTW within the local window, calculate the distance only within the window, and reduce the time complexity.

[0075] Then perform FastICA matrix decomposition on the preliminarily aligned data to obtain the initial time factor T and neuron factor N, as shown in formula (3). FastICA is a commonly used blind source separation algorithm for separating statistically independent components from multi-dimensional signals. Its core idea is to use non-Gaussianity as a measure of independence and extract independent components by maximizing negative entropy. In this study, FastICA is used to extract potential independent components from EEG activity data and decompose them into T and N. The specific steps are as follows:

[0076]

[0077] To ensure the stability and performance of the algorithm, first centralize the data through formula (4) so that the signal mean of each neuron is zero.

[0078]

[0079] Next, to remove the correlation in the data and standardize the variance of the data, the whitening method in formula (5) is used to process the centered data. First, calculate the covariance matrix of the data and perform eigenvalue decomposition to obtain the eigenvectors and eigenvalues. Then, standardize the data by using the reciprocal of the square root of the eigenvalues to obtain the whitened data X w . Among them, E is the eigenvector matrix and D is the eigenvalue matrix.

[0080]

[0081] Initialize the weight vector: Randomly initialize a weight vector ω, which is used to gradually extract independent components.

[0082] Iterative optimization: Through non-Gaussianity measures (such as using non-linear activation functions such as tanh or u 3 etc.), optimize each extracted independent component. The process of updating the weight vector is as shown in formula (6), where y = Xw × ω is the signal obtained by projecting the weight vector.

[0083]

[0084] Orthogonalization and normalization: To ensure that the extracted components are linearly independent, after each update, remove the linear components of the extracted components through orthogonalization and normalize to obtain a unit-length weight vector. Convergence check: If the change between the new and old weight vectors is less than the set threshold, it is considered that the algorithm converges and the iteration stops.

[0085] The time factor is calculated through T = X w W, where W is the finally extracted mixing weight matrix. The neuron factor is obtained through N = W -1 . These factors can reveal the potential independent sources in the signal, corresponding to the time dynamic pattern and the neuron spatial pattern respectively. To make the time factor and the neuron factor have the same scale, they are scaled according to formulas (7) and (8) respectively.

[0086]

[0087] Here, T scaled and N scaled are the scaled time factor and neuron factor, and "||*||2" represents calculating the Euclidean norm of each column of the factor matrix.

[0088] Using the current time warping function W k and the time factor T, the warped time factor W k (T) is obtained and the reconstructed data predicted by the model can be expressed as formula (9), represents the signal estimating the template.

[0089] The iterative optimization calculates the reconstruction error L of the neural signal according to the following formula (1) recon , and calculates the errors of the time factor, neuron factor, and warping function in the same way to form the total loss function L total , as shown in the following formula (2);

[0090]

[0091] L total = L recon + L T + L N + L W (11)

[0092] Calculate the gradients of the loss function with respect to T, N, and W k .

[0093] Update the time factor T, neuron factor N, and time warping function W k

[0094] Repeat the above steps until the loss function converges or the maximum number of iterations is reached.

[0095] Sliding time window processing

[0096] Perform sliding time window processing on the predicted data to further extract time-domain features and obtain the final time warping function W k , time factor T, neuron factor N, model prediction data

[0097] Evaluation metrics

[0098] To evaluate the performance of the proposed model, this application uses a KNN classifier to perform ten-fold cross-validation on the aligned signals and uses four evaluation metrics: accuracy, recall, precision, and F1-score. Accuracy measures the proportion of correctly predicted samples in the total number of samples; recall represents the proportion of samples that are actually positive and are correctly identified as positive by the model; precision measures the proportion of samples predicted as positive that are actually positive; the F1-score is the harmonic mean of precision and recall, comprehensively reflecting the overall performance of the model. The calculation formulas for the above metrics are shown in formulas (12), (13), (14), and (15). During the ten-fold cross-validation process, the dataset is divided into a training set and a test set in a ratio of 8:2. Each time cross-validation is performed, two subsamples are reserved from the dataset for model validation, and the remaining eight subsamples are used to train the KNN model. The cross-validation is repeated 10 times, and each subset is alternately validated and the evaluation metrics are executed, and the results of the ten times are averaged to obtain a reliable estimate of the model performance.

[0099]

[0100] Where TP is true positive, FN is false negative, TN is true negative, and FP is false positive.

[0101] The template optimization result of this application

[0102] The t-SNE visualization of the character template signal is shown in the appendix Figure 2 as follows. The character signals processed by the DTWICA model show significantly better separability, especially between easily confused character pairs, such as the characters "x" and "y", "z" and "e", and "d" and "o". In addition, the classification accuracy of 31 characters is evaluated using the K-Nearest Neighbor classifier (KNN). The results show that the average classification accuracy after ten-fold cross-validation has increased from 92.73% to 94.39%. The specific classification results are shown in Figure 3 the appendix. Except for the results of the third and ninth iterations being slightly lower than the report of Willett et al., the DTWICA method performs better than TWPCA in the other eight iterations.

[0103] To evaluate and analyze the robustness and accuracy of the model in constructing each type of imagined character template, the appendix Figure 4 shows the confusion matrix diagram of the neural activities after being processed by different models. It can be seen from Figure 6 that the model proposed in this paper has different degrees of improvement in the recall rates of multiple characters, such as the characters "h", "j", "k", "m", "n", "o", "x", ">", and ",". Different degrees of improvement in the recall rate are shown. Although the performance of individual characters has decreased, the decrease is small. For example, the recall rate of the character "b" has decreased from 1 to 0.88. Generally speaking, the recall rates of most characters have increased, reflecting the effectiveness and robustness of the model in most cases.

[0104] The appendix Figure 4 Confusion matrices of different alignment models under the KNN classifier. The horizontal axis represents the predicted labels of different models, and the vertical axis represents the true labels. The values in the confusion matrix represent the recall rate of each character, ranging from 0 to 1.

[0105] To discuss the robustness of the model in more detail, the present invention randomly selects 5 characters (i.e., "h", "k", "m", "o", and ">") for further analysis, and evaluates its performance in detail from three aspects: recall rate, accuracy rate, and F1 score. As shown in Table 1 below, the model proposed in this paper shows the highest F1 score for these characters, demonstrating its good classification ability.

[0106] Table 1 below: Experimental performance comparison of recall rate, accuracy rate, and F1 score of 5 characters

[0107]

[0108] The character templates constructed in this paper are applied to the classification task of the imagined handwritten sentence dataset. The alignment between single-character templates and the sentence dataset is achieved through the Hidden Markov Model (HMM), and the character signals in the sentence data are converted into a one-hot probability matrix, which is finally fed into a Recurrent Neural Network (RNN) for training and classification. The character error rate and word error rate of the present invention are statistically analyzed, and the results are shown in Table 2 below.

[0109] Table 2 Influence of single-character templates on the sentence dataset

[0110] Character Error Rate (%) Word Error Rate (%) Willett et al. 2.78 12.88 Our 2.51 12.13

[0111] Ablation experiment of this application

[0112] Since the warping function plays a key role in template construction, the present invention conducts a more detailed discussion on the method for solving the warping function, and the relevant results are shown in Table 3. After constructing the warping function using FastDTW, the average accuracy of ten-fold cross-validation is increased to 93.45% ± 1.71. The Shift method is a method for solving the warping function proposed by Willett et al. As a comparison, the present invention also evaluates the Piecewise method. Although Piecewise is a relatively new model for solving the warping function, because the overall template it constructs has a high-dimensional and noisy neural signal, the Piecewise method is not ideal in the research of this paper.

[0113] Table 3 below Influence of the warping function on the template accuracy

[0114] Shift Piecewise FastDtw Accuracy Rate (%) 92.73±3.75 87.79±3.59 93.45±1.71

[0115] The present invention conducts a visual analysis on the neural activity data processed by warping functions constructed by different methods, and the results are as shown in the appendix Figure 5 For the original data without being processed by the DTWICA model, on the first two independent components, the neural activities of 27 trials of the character 'a' are generally stable within a certain time range, while on independent components 3 to 5, the neural activities show a relatively chaotic pattern, which increases the complexity of subsequent decoding operations. From the appendix Figure 5It can be seen that compared with the original data, the Piecewise method only slightly improved the signal and successfully eliminated most of the inter-trial differences. Both Shift and FastDTW showed an obvious trend of spike alignment on the five independent components. However, as the number of independent components increased, FastDTW was able to better capture complex neural signals, thus achieving a more refined trial alignment. In independent component 3 (IC 3), the energy distribution of trials 4, 5, 11, 13, 23, and 24 was more concentrated than that of the Shift method. In independent components 4 (IC 4) and 5 (IC 5), the performance between trials was also more regular.

[0116] Given the characteristics of EEG signals such as high dimensionality and large noise, when performing time warping alignment, the present invention uses FastICA to reduce the dimension and decompose the data, thereby obtaining both time factors and neuron factors simultaneously to achieve a more refined data calibration. In addition, the present invention also statistically analyzed ten dimensionality reduction algorithms based on matrix decomposition and evaluated their performance through ten-fold cross-validation. Figure 6 Shows the results of the average accuracy and standard deviation of ten-fold cross-validation when using these ten matrix decomposition algorithms respectively when constructing the warping coefficient by the Shift method. The results show that Factor Analysis obtained the highest average accuracy (94.0%), followed by SVD (92.73%) and FastICA (92.81%).

[0117] Based on the matrix decomposition dimensionality reduction algorithm, the present invention further verified the influence of FastDTW on template construction when constructing the warping function. As shown in Table 4 below, at this time, the average accuracy of ten-fold cross-validation of TruncatedSVD reached 93.45%, while the accuracy of FastICA was the highest, reaching 94.31%. The results show that the accuracies of seven decomposition dimensionality reduction algorithms, such as TruncatedSVD, FastICA, PCA, SparsePCA, DictionaryLearning, MiniBatchDictionaryLearning, and TCA, have all increased, while the accuracies of MiniBatchSparsePCA, Incremental PCA, and FactorAnalysis have decreased, with the maximum decrease of 0.43%. Therefore, although the performance of some algorithms has decreased slightly, the FastDTW method still shows higher robustness and effectiveness compared with the Shift method.

[0118] Table 4 below Influence of FastDtw on template accuracy with different matrix decomposition dimensionality reduction algorithms

[0119]

[0120] After data calibration, the present invention performs a sliding time window process on the data to select the most suitable window to remove noise data irrelevant to the imagination task. The influence of different window sizes on the template accuracy is shown. Under three models, it can be observed that the optimal window size for FastDTW and FastDTW+FastICA is 181 time points, while the optimal window size for the Willett etal. model is 191 time points. The model proposed by the present invention performs best in terms of classification accuracy, and the average accuracy of ten-fold cross-validation can reach 94.39%. In addition, the present invention also finds that as the time window slides backward, the accuracy gradually decreases (see Table 5). This is related to the task schedule, because the first 60 time points are used for preparing the prompt, and the 61st to 201st time points are for the time of performing the imagination task. Therefore, the EEG signals generated during the task prompt are helpful for subsequent analysis.

[0121] Table 5 below Influence of the sliding time window on the template accuracy

[0122]

[0123] The device used in the present invention includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

[0124] The preferred embodiments of the present invention have been described in detail above. However, it should be understood that after reading the above teachings of the present invention, those skilled in the art can make various changes or modifications within the protection scope of the present invention. These equivalent forms also fall within the scope defined by the appended claims of this application.

[0125] The parts not detailed in the present invention are prior art.

Claims

1. A method for constructing a character signal template for mind typing based on DTWICA, characterized by: The method for constructing a thought typing character signal template comprises: Step S1, first build preprocessing, DTWICA, template construction, visualization and classification of aligned data on the device; Step S2, collecting the user's electroencephalogram (EEG) through an electrode array disposed on the brain; Step S3: considering that the baseline discharge rate of different data blocks will drift over time, the discharge rate of each electrode is normalized, and then the neural activity data is Gaussian filtered in the time dimension. The standard deviation of the Gaussian filter is set to 3, and the degree of smoothing is controlled; Step S4, DTWICA stage: DTWICA processes the data of multiple trials by performing time warping (FastDTW) and independent component analysis (FastICA), which takes into account both the spatial structure of the data and the time differences between different trials. DTWICA captures the differences between trials caused by time delays, different experimental stages, or other time changes. Step S5, template construction stage: after obtaining the time-warped data of the corresponding character, firstly perform mean processing on all trials along the trial dimension to obtain a more regular EEG signal; Step S6: After the time alignment processing of the DTWICA overall framework, the neural activity data is subjected to t-SNE (t-Distributed Stochastic Neighbor Embedding) dimensionality reduction processing to map the high-dimensional data into a two-dimensional space; Step S7, template evaluation stage: Use a ten-fold cross validation based on a KNN (K-Nearest Neighbors) classifier to classify the aligned single-character data and evaluate the classification accuracy.

2. The method for constructing a character signal template for mind typing based on DTWICA according to claim 1, characterized in that: The process in step S3 is as follows: In the formula, x represents the current data in each sample, and x represents the mean of the current block of the sample; in step S4, DTWICA not only learns the time factor (time*components) and the neuron factor (neuron*components), but also learns a time warp function (W) for each trial. k ), where each column of the time factor represents the activity pattern of an independent component at each time point, and the neuron factor represents the weight of each neuron on each independent component; although different trials may have differences in the time axis, DTWICA aligns these trials into a shared time space through time warping, so that the EEG signals between different trials can be compared and analyzed in the same time frame.

3. The method for constructing a DTWICA-based mind-typing character signal template according to claim 1, characterized in that: In step S5, the average value of each time point and each neuron is calculated, and NaN values ​​are ignored to eliminate individual differences between trials; then, a one-dimensional Gaussian filter is performed on the time dimension to smooth the time series and reduce noise; Subsequently, the effective time window of each character is intercepted according to the time length of the handwriting reconstruction of each character; the obtained time window is sent to PCA for training to reduce the dimensionality of neural activity; the data after dimensionality reduction will be restored back to the original space, but only the first 10 principal components will be retained to achieve noise reduction and remove low-contribution components; finally, after these processing steps, a spatiotemporal template is obtained to provide high-quality template data for subsequent signal analysis and classification tasks.

4. The method for constructing a character signal template for mind typing based on DTWICA according to claim 1, characterized in that: The perplexity in the t-SNE algorithm in step S6 is used to control the balance of the local data structure, and its value is set to 20; then, the aligned single-character data is classified using a ten-fold cross validation based on the KNN classifier to evaluate the classification accuracy; This step aims to verify the visualization effect and classification performance of the time-aligned data after dimensionality reduction, so as to evaluate the effectiveness of the DTWICA alignment method in practical tasks.

5. The method for constructing a character signal template for mind typing based on DTWICA according to claim 1, characterized in that: The overall framework of DTWICA in step S6 is divided into the following three stages, namely, signal alignment, iterative optimization, and sliding time window processing; The signal alignment by FastDTW is used to initialize the warping function W k , and perform preliminary time alignment on the data; specifically, first calculate the mean of each trial of the single-character data to obtain the initial template After that, FastDTW is used to calculate the path relationship between the neural activity of each trial and the initial template, thereby obtaining the relative time offset and constructing the distortion function W for each trial. k Finally, the distortion function W k Applied to the original data to obtain preliminary aligned data; The core idea of ​​this process is to calculate the similarity between each trial and the initialization template through FastDTW's dynamic programming algorithm, and construct the distortion function based on it; FastDTW uses multi-scale approximation and local search windows to efficiently align two time series; Specifically, its core operation is: align the time series at a coarse resolution to obtain a rough matching path; then refine the path at a higher resolution to narrow the alignment range; finally, recursively perform this process layer by layer until the alignment is completed at the original resolution; the process is as follows: Coarse-graining: Perform standard DTW on each trial and initial template according to the following formula (1) to obtain a rough matching path; Among them, the input data X k ∈R t×n represents the activities of n neurons at t time points in the kth trial, (i,j) is the index of the time point; Projection: Project the coarse-grained path points to the high-resolution sequence X k and On the other hand, a local window is expanded around the waypoints to limit the search range for exact alignment; Fine-grained: Standard DTW is performed in a local window, and the distance is calculated only within the window to reduce time complexity; Then, the data after preliminary alignment is subjected to FastICA matrix decomposition to obtain the initial time factor T and neuron factor N, as shown in the following formula (1); FastICA is a commonly used blind source separation algorithm used to separate statistically independent components from multidimensional signals; its core idea is to use non-Gaussianity as a measure of independence and extract independent components by maximizing negative entropy: FastICA is used to extract potential independent components from EEG activity data and decompose them into T and N. The specific steps are as follows: To ensure the stability and performance of the algorithm, the data is first centered using the following formula (1) so that the mean value of the signal of each neuron is zero; Next, in order to remove the correlation in the data and standardize the variance of the data, the whitening method in the lower part of the formula (1) is used to process the centered data; first, the covariance matrix of the data is calculated, and the eigenvalue decomposition is performed to obtain the eigenvector and eigenvalue; then, the data is standardized by using the inverse square root of the eigenvalue to obtain the whitened data Xw. Among them, E is the eigenvector matrix, and D is the eigenvalue matrix; Initialize the weight vector: Randomly initialize a weight vector ω, which is used to gradually extract independent components; Iterative optimization: By using non-Gaussian metrics (such as tanh or u 3 Nonlinear activation functions, etc.) are used to optimize each extracted independent component; the process of updating the weight vector is as shown in formula (1), where y = Xw × ω is the signal obtained by projecting the weight vector; Orthogonalization and normalization: To ensure that the extracted components are linearly independent, the linear components of the extracted components are removed by orthogonalization after each update, and normalized to obtain a weight vector of unit length; Convergence check: If the change between the new and old weight vectors is less than the set threshold, the algorithm is considered to have converged and the iteration stops; By T=X w W is calculated to obtain the time factor, where W is the final extracted mixing weight matrix; the neuron factor is calculated by N = W -1 It is obtained that these factors can reveal the potential independent sources in the signal, corresponding to the temporal dynamic pattern and the neuronal spatial pattern respectively; in order to make the temporal factor and the neuronal factor have the same scale, they are scaled according to the following formulas (1) and (2) respectively; Here, T scaled and N scaled are the scaled time factors and neuron factors, "||*||2" represents the Euclidean norm of the factor matrix on each column; Use the current time warp function W k , time factor T, get the distortion time factor W k (T) and calculate the reconstructed data predicted by the model, which can be expressed as the following formula (1), A signal representing an estimated template; The iterative optimization calculates the reconstruction error L of the neural signal according to the following formula (1): recon , the same way is used to calculate the error of the time factor, neuron factor and distortion function; A. Constructing the total loss function L total , as shown in the following formula (2); L total =L recon +L T +L N +L W (2) B. Calculate the loss function for T, N and W k The gradient of C. Update time factor T, neuron factor N and time warp function W k ; D. Repeat the above steps until the loss function converges or the maximum number of iterations is reached; Sliding time window processing The predicted data is processed by sliding time window to further extract the time domain features and obtain the final time warp function W k , time factor T, neuron factor N, model prediction data 6. The method for constructing a character signal template for mind typing based on DTWICA according to claim 1, characterized in that: The system of the method for constructing the idea typing character signal template comprises: The electrode array acquisition module of the brain is used to collect EEGs that identify and decode specific brain features or intentions; the data analysis module is used to analyze raw data, neurons, trials and time data analysis; the data preprocessing unit is used to preprocess EEG data; the DTWICA model construction module is used to introduce a time warp factor to model the time series of single spike events, and estimate the uncertainty of time warp through Bayesian inference, which significantly improves the detection ability of neural sequences of different durations; the data secondary analysis module is used to analyze the secondary analysis of raw data, neurons, trials and time data; the DTWICA model module is used for dynamic time warping independent component analysis; t-SNE: t-distributed random neighbor embedding; PCA denoising is performed on the data after averaging the trials, and the EEG feature extraction module is used for character spatiotemporal templates, and ten-fold cross validation based on KNN: used to evaluate the performance of the constructed templates.

7. The method for constructing a character signal template for mind typing based on DTWICA according to claim 1, characterized in that: The device used comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method according to any one of claims 1 to 6 when executing the computer program.

Images