A multi-scale time-frequency analysis method for decoding imagined speech brain electrical signals

Abstract

The application discloses a multi-scale time-frequency analysis method for public speech and imagined speech electroencephalogram (EEG) collaborative decoding, which comprises the following steps: making a subject perform a public speech task and an imagined speech task simultaneously; collecting and pre-processing speech EEG data including public speech data and imagined speech data; marking corresponding labels according to categories for the public speech task and the imagined speech data; constructing public spatial mode features of the public speech data and the imagined speech data cross-band coupling; constructing an EEG multi-scale time-frequency analysis network including a multi-scale time convolution module and a multi-scale band attention module; inputting the public spatial mode features into the EEG multi-scale time-frequency analysis network for speech category prediction; training the EEG multi-scale time-frequency analysis network; and inputting test samples into the trained EEG multi-scale time-frequency analysis network to generate a classification result for judging the speech category of the subject. The application can extract public speech and imagined speech features and pay attention to the importance of the features in the time domain and the frequency domain at the same time.

Description

Technical Field

[0001] This invention relates to the field of electroencephalogram (EEG) signal processing technology, and in particular to a cross-band coupled EEG multi-scale time-frequency analysis method for speech imagery decoding. Background Technology

[0002] Implicit verbal imagery refers to the process by which an individual generates verbal representations in their brain or "speaks in their mind" without external vocalization. In recent years, the connection between implicit verbal imagery and its clinical / engineering applications (such as non-invasive brain-computer interfaces and aphasia rehabilitation) has increasingly become a research hotspot in neuroscience and engineering. Current research generally considers both explicit and implicit verbal imagery to be outputs of different stages in the language production process within cognitive theory. Explicit verbal imagery includes both phonological planning and motor execution, while implicit verbal imagery primarily reflects the internal representation of the phonological planning stage. However, compared to explicit verbal EEG signals, implicit verbal imagery EEG signals still have a series of inherent limitations in actual decoding. First, due to the lack of actual vocal tract and muscle motor execution, the neural activity energy of implicit verbal imagery is usually significantly weaker than the corresponding signals evoked by explicit verbal imagery, resulting in a low signal-to-noise ratio, weak amplitude, and poor detectability. Second, the activation patterns associated with implicit verbal imagery are more dispersed, and cortical responses are more susceptible to fluctuations in cognitive state, changes in attention levels, and cross-trial variability, leading to insufficient feature stability and cross-temporal reproducibility. Third, compared to open speech, imaginary speech often lacks clear temporal boundary markers. For example, event-related trigger points such as the sound origin are difficult to obtain accurately, making the implementation of time-locked features more challenging. At the same time, existing neurolinguistics and brain-computer interface research has shown that although open speech and imaginary speech lack the same external behavioral manifestations at the output level, they have significant coupling and consistency in cortical processing mechanisms, especially sharing overlapping neural networks at the levels of speech planning, semantic integration, and auditory feedback prediction.

[0003] Current methods for using EEG signals to visualize speech have the following problems:

[0004] 1. Relying solely on single-band analysis and ignoring cross-band coupling features. Most existing methods extract features based on single-band power spectra, single-scale convolution, or uniform filter banks, which can only capture single oscillatory activities and lack modeling of the interactions and coupling structures between different neural oscillation frequency bands.

[0005] 2. Ignoring the physiological basis of multi-scale language generation. Language production is a hierarchical, cross-timescale dynamic process—from semantic planning and phoneme conversion to the execution of articulation movements, each stage corresponds to the coordination of neural oscillations at different frequency bands.

[0006] 3. Lack of time-frequency attention analysis for both spoken and imagined speech. Existing methods cannot capture the key time segments and frequency bands of common attention during language production, thus limiting cross-task information transfer and robust decoding performance improvement. Summary of the Invention

[0007] This invention addresses the shortcomings of existing technologies by proposing a multi-scale time-frequency analysis method for collaborative decoding of public and imagined speech EEG. This method constructs data features coupled across frequency bands of public and imagined speech, and employs a multi-scale feature extraction mechanism combined with a dual attention mechanism. This approach not only extracts features at different granularities of public and imagined speech but also considers their importance in both the time and frequency domains. Furthermore, public speech can be used to further enhance the classification performance of imagined speech.

[0008] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0009] A multi-scale time-frequency analysis method for co-decoding public and imagined speech EEG includes the following steps:

[0010] S1. Have the subjects perform both a public speech task and an imagined speech task simultaneously; collect and preprocess speech EEG data including public speech data and imagined speech data; label the public speech task and imagined speech data according to their categories.

[0011] S2. Construct public space pattern features that couple the cross-frequency bands of public speech data and imagined speech data;

[0012] S3. Construct an EEG multi-scale time-frequency analysis network including a multi-scale temporal convolution module and a multi-scale frequency band attention module; input public space pattern features into the EEG multi-scale time-frequency analysis network for speech category prediction;

[0013] S4. Train the EEG multi-scale time-frequency analysis network; input the test samples into the trained EEG multi-scale time-frequency analysis network to generate classification results that determine the subject's speech category.

[0014] As a preferred embodiment, it also includes: S5, constructing a time attention sub-model and a frequency band attention sub-model based on the trained cross-frequency coupling driven multi-scale EEG speech analysis model, using training set or validation set samples to input the time attention sub-model and the frequency band attention sub-model for inference, and obtaining the time attention weight and the frequency band attention weight.

[0015] Preferably, S2 includes:

[0016] S21. Divide the speech EEG data into a training set and a test set; divide the public speech data and imagined speech data in the training set into the following 5 frequency bands: delta: [1,4], theta: [4,8], alpha: [8,13], beta: [13,30], gamma: [30,45]; generate 25 frequency band combinations by pairwise permutation and combination; merge and classify the speech EEG data in the training set according to the category to which the label belongs;

[0017] S22. A one-to-many public space pattern feature extraction strategy is adopted to generate public space pattern features.

[0018] Preferably, S3 includes:

[0019] S31. Construct a multi-scale temporal convolution module to extract temporal convolutional feature maps of common spatial pattern features;

[0020] S32. Perform average pooling on the temporal convolutional feature map to obtain temporal features from a global perspective; perform convolution on the temporal features to capture the dependencies between adjacent time points and generate a temporal weight vector.

[0021] S33. Construct a multi-scale frequency band attention module, obtain the frequency band weight matrix, apply frequency band dimension attention weighting to the temporal convolutional feature map, and generate a frequency band feature map.

[0022] S34. Perform deep spatial filtering on the frequency band feature map;

[0023] S35. Use separable convolution to extract deep temporal features;

[0024] S36. Perform residual connection on the data before and after the extraction of deep temporal features;

[0025] S37. The features after the residual connection are flattened by using a flattening layer, then passed through a fully connected layer, and finally the class probability is output through the softmax function.

[0026] Preferably, in S31, the multi-scale temporal convolution module includes three two-dimensional convolutional layers of different scales.

[0027] Preferably, S33 includes:

[0028] The multi-scale frequency band attention module receives temporal convolutional feature maps; it performs three different average pooling operations on the temporal convolutional feature maps to generate feature descriptors of three different scales for each of the 25 frequency band combinations.

[0029] The three feature descriptors at different scales are expanded in dimension, and the three expanded feature descriptors are concatenated in the last dimension to generate a concatenated feature matrix.

[0030] The concatenated feature matrix is ​​passed through a multilayer perceptron to generate a frequency band weight matrix. The frequency band weight matrix is ​​then expanded in dimension and a time dimension is added. It is then multiplied with the temporal convolutional feature map at corresponding elements to generate a frequency band feature map.

[0031] Preferably, three different average pooling operations are included:

[0032] Global average pooling: Performs global average pooling on the temporal convolutional feature map in both the temporal and filter dimensions;

[0033] Temporal window average pooling: Using a pooling kernel of size [1,3], average pooling is performed on the temporal dimension of the temporal convolutional feature map, and then global average pooling is performed on the pooling result on both the temporal and filter dimensions.

[0034] Temporal window average pooling: Using a pooling kernel of size [1,5], average pooling is performed on the temporal dimension of the temporal convolutional feature map, and then global average pooling is performed on the pooling result on both the temporal and filter dimensions.

[0035] Preferably, the multilayer perceptron includes:

[0036] The hidden layer is a fully connected neural network with an input dimension of 3 and an output dimension of 8, and uses ReLU as the activation function; the input is processed using a time-step independent layer wrapper.

[0037] The output layer is another fully connected neural network with an input dimension of 8 and an output dimension of 1, and it uses the Sigmoid activation function. The input is processed using a time-step independent layer wrapper, and the final output is a frequency band weight matrix with a dimension of [batch size, number of frequency bands, 1].

[0038] As a preferred option, constructing a time attention sub-model includes: inheriting the weights from the trained multi-scale EEG speech analysis model, forcibly terminating the data flow at the time attention layer, and outputting the time attention weights; constructing a frequency band attention sub-model includes: inheriting the weights from the trained multi-scale EEG speech analysis model, forcibly terminating the data flow at the multi-scale frequency band attention layer, and outputting the frequency band attention weights.

[0039] A multi-scale time-frequency analysis system for co-decoding public and imagined speech EEG includes:

[0040] The data acquisition module is used to collect and preprocess speech EEG data, including public speech data and imagined speech data, and to label the public speech task and imagined speech data according to categories.

[0041] The coupling feature construction module is used to construct common spatial pattern features that are coupled across frequency bands of public speech data and imagined speech data;

[0042] A multi-scale time-frequency EEG analysis network, including a multi-scale temporal convolution module and a multi-scale frequency band attention module, is used to predict speech categories based on public space pattern features and generate classification results that determine the speech category of the test subject.

[0043] The aforementioned multi-scale time-frequency analysis system for co-decoding public and imagined speech EEG is used to implement the multi-scale time-frequency analysis method and steps for co-decoding public and imagined speech EEG as described in the first aspect.

[0044] Compared with the prior art, the beneficial effects of the present invention are reflected in:

[0045] 1. Unlike traditional speech imagination decoding methods that rely excessively on single frequency band (such as low frequency) features or have difficulty capturing high frequency features on non-invasive devices, this invention adopts a technical solution that discloses the common space pattern features of the cross-frequency bands of speech and imagined speech, which significantly improves the richness of available information and the reliability of features in non-invasive EEG.

[0046] 2. Unlike traditional EEG classification models that primarily focus on classification results and lack detailed analysis and dynamic adaptation of feature contribution, this invention employs a multi-scale convolutional approach to extract neurodynamic features at different temporal resolutions. This combined temporal and frequency attention mechanisms enables dynamic and adaptive optimization and recognition of the contribution of key time segments and cross-frequency band combinations in spoken and imagined speech. This suppresses noise interference and significantly improves recognition accuracy and generalization performance.

[0047] 3. Unlike traditional speech imagination decoding, which faces the fundamental challenge of scarce imagined speech samples, leading to insufficient model training and difficulties in practical application, this invention adopts a technical solution that establishes a set of stable cross-frequency coupling identifiers between public speech and imagined speech, and uses these identifiers to incorporate a large amount of public speech data into model training. This alleviates the problem of scarce imagined speech samples, enhances model training with easily accessible public speech data, and lays the foundation for practical non-invasive brain-computer interfaces (such as input methods based on imagined speech). Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of the method framework for Embodiment 1. Detailed Implementation

[0050] To make the technical means, inventive features, objectives, and effects of the invention readily understandable, the invention is further described below with reference to specific illustrations. However, the invention is not limited to the embodiments described below.

[0051] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0052] Example 1:

[0053] like Figure 1 The multi-scale time-frequency analysis method for co-decoding public and imagined speech EEG includes the following steps:

[0054] S1. Have the subjects perform both a public speech task and an imagined speech task simultaneously; collect and preprocess speech EEG data including public speech data and imagined speech data; label the public speech task and imagined speech data according to their categories.

[0055] S11. Collect verbal EEG data and label the public verbal task and imagined verbal data according to the category.

[0056] Participants, wearing EEG caps, were instructed to perform public and imagined speech tasks with relevant cues, such as audio prompts. The resulting EEG signals were collected as raw data. The EEG signal from the imagined speech segment was tagged accordingly. Because EEG data is multi-rhythmic and multi-channel, this embodiment used speech EEG data with 64 channels, where the Cz electrode was selected as the reference electrode. All signals were acquired according to the international 10-20 system.

[0057] S12. Preprocess the speech EEG data;

[0058] In this embodiment, a fifth-order Butterworth bandpass filter with a frequency range of 1Hz to 70Hz is applied to the acquired speech EEG data to eliminate noise and eye movement artifacts. After filtering, a downsampling operation is performed, adjusting the sampling rate to 256Hz.

[0059] S2. Construct public space pattern features that couple the cross-frequency bands of public speech data and imagined speech data;

[0060] S21. Divide the verbal EEG data into training and testing sets according to a certain ratio; divide the public verbal data and imagined verbal data into 5 frequency bands each, and then combine them in pairs through permutation and combination to generate 25 frequency band combinations; and merge and classify the data in the training set according to their respective categories. Among them, the merged data are all public verbal data and the training set of imagined verbal data.

[0061] S22. Next, a one-vs-rest common space pattern feature extraction strategy is adopted. For each category, the current category trial is taken as the positive class; all other category trials are merged as the negative class; k sets of binary classification tasks are formed, thus constructing K sets of common space pattern spatial filters. Only one set of the most discriminative spatial projection is retained for each category, thereby obtaining the common space pattern features.

[0062] In the "one-to-many common spatial pattern feature extraction" process, for the covariance decomposition results corresponding to each category, only the spatial projection vector with the strongest discriminative power is selected as the effective common spatial pattern component; in other words, the number of common spatial pattern components is set to 1. The core advantage of this strategy is that, based on the characteristics of the verbal imagination EEG task, the common spatial pattern projection with the highest or lowest feature value can often concentrate on carrying most of the category discrimination information, thus achieving effective compression of feature dimensions without significantly reducing classification performance. This not only reduces the input size and total number of parameters of the subsequent neural network, reducing computational complexity, but also avoids the interference of redundant information caused by multiple projection vectors, thereby improving feature utilization, network training convergence speed, and model stability.

[0063] S3. Construct an EEG multi-scale time-frequency analysis network including a multi-scale temporal convolution module and a multi-scale frequency band attention module; input public space pattern features into the EEG multi-scale time-frequency analysis network for speech category prediction;

[0064] S31. Use the multi-scale temporal convolution module to extract temporal convolutional feature maps of common spatial pattern features;

[0065] Three Conv2D(1,k) convolutions (k=8 / 16 / 32, convolution only in the time dimension) are used for breadth scanning to capture features of the entire EEG signal across different time spans, enriching the feature information.

[0066] S32. Temporal Attention: The temporal convolutional feature map output from step S31 is subjected to average pooling in the first dimension (number of frequency bands) and the third dimension (number of filters), ignoring frequency band differences and focusing on temporal changes to obtain temporal features from a global perspective. Weight learning is then performed, using a 1*3 convolution kernel to convolve the aggregated temporal features, capturing the dependencies between adjacent time points. Finally, a sigmoid function is used to activate and generate a temporal weight vector, highlighting features from key time segments.

[0067] S33. Construct a multi-scale frequency band attention module to perform frequency band-dimensional attention weighting on the temporal convolutional feature map, generating a frequency band feature map "out". The frequency band feature map "out" is more discriminative for classification tasks, while suppressing interference from noise or irrelevant frequency bands. The specific implementation steps are as follows:

[0068] S331. Input Feature Map: Receives a temporal convolutional feature map x from the temporal convolutional layer, whose dimensions are (batch size, number of frequency bands, time point, number of filters), i.e. (batch size, number of frequency bands, number of samples, number of filters).

[0069] S332. Multi-scale Feature Descriptor Extraction: To capture information for each frequency band at different time scales, the following three different average pooling operations are performed on the temporal convolutional feature map x to generate three feature descriptors for each of the 25 frequency band combinations:

[0070] ① Global average pooling: Global average pooling is performed on the temporal convolutional feature map x in both the temporal dimension (number of samples) and the filter dimension (number of filters).

[0071] ② Temporal Window Average Pooling: First, an average pooling operation is performed on the temporal dimension of the temporal convolutional feature map x using a pooling kernel of size (1,3) (i.e., averaging over every 3 consecutive time points). The pooling operation has a stride of 1, and the padding method is 'same' to maintain the length of the temporal dimension. Then, the pooled result is subjected to global average pooling on both the temporal and filter dimensions.

[0072] ③ Temporal Window Average Pooling: Similarly, an average pooling operation is performed on the temporal dimension of the temporal convolutional feature map x using a pooling kernel of size (1,5). Then, the pooled result is subjected to global average pooling on both the temporal and filter dimensions.

[0073] S333. Feature Descriptor Concatenation: The three feature descriptors desc1, desc3, and desc5 obtained at different scales are expanded in dimension, changing from (batch size, number of frequency bands) to (batch size, number of frequency bands, 1). Then, these three expanded feature descriptors are concatenated in the last dimension to form a concatenated feature matrix concat, with dimensions (batch size, number of frequency bands, 3). Each row of this matrix contains the feature information of the corresponding frequency band at three different time scales.

[0074] S334. Frequency Band Weight Learning: To adaptively learn the importance weights of each frequency band, the concatenated feature matrix is ​​input into a Multi-Layer Perceptron (MLP). This MLP consists of the following two layers:

[0075] i) Hidden Layer: A fully connected neural network with an input dimension of 3 and an output dimension of 8, using ReLU (Rectified Linear Unit) as the activation function. Since this operation needs to be performed independently for each frequency band, a time-step independent layer wrapper is used to enable it to handle inputs in the format (batch size, number of frequency bands, 3) and outputs in the format (batch size, number of frequency bands, 8).

[0076] ii) Output layer: Another fully connected neural network with an input dimension of 8 and an output dimension of 1, using the Sigmoid activation function. The Sigmoid function ensures that the output weight values ​​fall within the range of 0 to 1. Similarly, this layer is also wrapped by time-step independent layers, and the final output is a band weight matrix td2 with a dimension of (batch size, number of bands, 1).

[0077] S335, Weight Broadcasting and Feature Weighting: The band weight matrix td2 (dimension (batch size, number of bands, 1)) is expanded by adding a time dimension, making it a scale matrix of (batch size, number of bands, 1, 1). This scale matrix is ​​then multiplied element-wise with the temporal convolutional feature map x (dimension (batch size, number of bands, number of samples, number of filters)). In this way, the entire time series of each band and all filter responses are scaled by their corresponding learned weights.

[0078] S336. Output result: Output the weighted frequency band feature map out, whose dimension is the same as the temporal convolution feature map x, i.e. (batch size, number of frequency bands, number of samples, number of filters).

[0079] S34. Perform deep spatial filtering on the frequency band feature map (out): Perform depth convolution only in the frequency band dimension, combined with maximization regularization constraints to prevent overfitting, followed by pooling (downsampling by 4 times in the time dimension) and dropout regularization. This is equivalent to performing spatial filtering in the frequency domain feature dimension, further compressing redundant frequency bands.

[0080] S35, Temporal Convolution. After reducing the data dimensionality through separable convolutional pooling (8x temporal downsampling) and dropout regularization, a single fixed temporal convolutional layer is used to slide along the time axis. Separable convolution is used to extract deep temporal features, perform depth focusing, and optimize feature quality.

[0081] S36. Perform residual connections on the data from step S35 before and after deep temporal feature extraction. Effective features of EEG are easily lost during deep processing. Residual connections can directly preserve these core features, preventing the model from learning incorrectly, reducing the training difficulty of deep networks, allowing the model to converge faster, and preventing overfitting to noisy EEG features. This improves generalization ability.

[0082] S37. Flatten the features after residual connection in step S36 using the flattening layer, then pass them through the fully connected layer, and finally output the class probability through the softmax function.

[0083] In this embodiment, the multi-scale temporal module adopts multi-scale temporal characteristics of speech EEG signals (such as the fast temporal response (millisecond level) of speech perception and the slow integration (hundreds of milliseconds level) of semantic understanding). It captures "short-term fine features" (such as the instantaneous response of speech acoustic features), "mid-term transition features" (such as the neural coding of syllable switching) and "long-term global features" (such as the EEG pattern of sentence-level semantic integration) through three sets of 1D temporal convolutions with different kernel sizes (8, 16, 32).

[0084] Multi-scale frequency band attention also employs average pooling at three different scales. This combination of sizes can effectively cover the temporal scale range of speech-related features in EEG signals, balancing the detail of feature extraction with global relevance.

[0085] The temporal attention mechanism can be applied first to the multi-scale temporal convolution module, and then added after the fine-grained temporal convolution with pooling. This allows us to obtain both the importance of fine-grained time points and the attention trend within that time period.

[0086] S4. Train the EEG multi-scale time-frequency analysis network; input the test samples into the trained EEG multi-scale time-frequency analysis network to generate classification results that determine the subject's speech category;

[0087] Based on a multi-scale time-frequency EEG analysis network, the Adam optimizer was configured, and classification cross-entropy was used as the loss function and accuracy as the model evaluation metric to complete model compilation. Next, appropriate training parameters, such as learning rate, batch size, and number of training epochs, were set according to different datasets. The preprocessed training and test sets were input into the model for iterative training. During training, the validation set accuracy was monitored through the ModelCheckpoint callback function, and the model with the best generalization performance was automatically saved. Finally, the saved best model was used to classify and predict new EEG signal samples, outputting the classification results to determine the speech category of the subject during EEG data collection.

[0088] The optimizer is set to the Adam optimizer. Compared to the traditional SGD optimizer, the Adam optimizer combines momentum term and adaptive learning rate characteristics. In EEG signal classification tasks, it not only improves convergence speed but also effectively avoids local optima. The optimal learning rate is 0.005, which results in faster model convergence during training. In contrast, 0.001 (too small) may still converge, but requires more training epochs, leading to slower convergence and wasting time and computational resources. Compared to 0.01, it can cause oscillations and non-convergence. The optimal batch size is 8. Considering that EEG datasets typically have a small sample size (hundreds to thousands of samples per subject), a batch size of 8 ensures the stability of gradient estimation while reducing memory usage.

[0089] S5. Based on the trained multi-scale EEG speech analysis model driven by cross-frequency coupling, create a sub-model with consistent input and outputs of temporal attention layer output and frequency band attention layer output. Use training or validation set samples as input to this sub-model for inference, obtaining the temporal attention weights and frequency band attention weights as outputs. Temporal attention sub-model: Inherits the weights from the entire trained model. After obtaining the temporal weight vector in step S32, the data flow is forcibly terminated at the temporal attention layer. Frequency band attention sub-model: Inherits the weights from the entire trained model. After obtaining the frequency band weight matrix in step S33, the data flow is forcibly terminated at the multi-scale frequency band attention layer.

[0090] In this embodiment, by constructing common spatial pattern features of the cross-frequency bands of public speech and imagined speech, and proposing a multi-scale network, the final model can effectively identify the labels of test set samples. In this embodiment, on the one hand, a temporal attention mechanism is used to analyze EEG signals under public and imagined speech tasks. This mechanism can automatically identify and focus on the most critical time segments for classification decisions in both speech patterns, effectively extracting common temporal dynamic features, thereby enhancing the model's ability to capture task-related EEG responses. On the other hand, with the help of a frequency band attention mechanism, the model can adaptively learn the importance weights of different frequency band EEG signals, uncovering the cross-frequency band correlation features between public and imagined speech. This helps reveal the common patterns of the two speech patterns at the neural activity frequency band level. Through the synergistic effect of the above-mentioned temporal attention and frequency band attention, the model's feature extraction efficiency and classification performance for public and imagined speech EEG signals can be significantly improved, providing more accurate and robust technical support for speech task recognition in brain-computer interface systems.

[0091] Example 2:

[0092] A multi-scale time-frequency analysis system for co-decoding public and imagined speech EEG includes:

[0093] The data acquisition module is used to collect and preprocess speech EEG data, including public speech data and imagined speech data, and to label the public speech task and imagined speech data according to categories.

[0094] The coupling feature construction module is used to construct common spatial pattern features that are coupled across frequency bands of public speech data and imagined speech data;

[0095] A multi-scale time-frequency EEG analysis network, including a multi-scale temporal convolution module and a multi-scale frequency band attention module, is used to predict speech categories based on public space pattern features and generate classification results that determine the speech category of the test subject.

[0096] Example 3:

[0097] This embodiment discloses a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The memory stores the computer program; the processor executes the cross-band coupled EEG multi-scale time-frequency analysis method for speech imagery decoding described in Embodiment 1.

[0098] Example 4:

[0099] This embodiment discloses a readable storage medium storing a computer program; when the computer program is executed by a processor, it is used to implement the cross-band coupled EEG multi-scale time-frequency analysis method for speech imagery decoding described in Embodiment 1.

[0100] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A multi-scale time-frequency analysis method for co-decoding public and imagined speech EEG, characterized in that, Includes the following steps: S1. Have the subjects perform both a public speech task and an imagined speech task simultaneously; collect and preprocess speech EEG data including public speech data and imagined speech data; label the public speech task and imagined speech data according to their categories. S2. Construct public space pattern features that couple the cross-frequency bands of public speech data and imagined speech data; S3. Construct an EEG multi-scale time-frequency analysis network including a multi-scale temporal convolution module and a multi-scale frequency band attention module; input public space pattern features into the EEG multi-scale time-frequency analysis network for speech category prediction; S4. Training the EEG multi-scale time-frequency analysis network; The test samples are input into a trained EEG multi-scale time-frequency analysis network to generate classification results that determine the subject's speech category.

2. The multi-scale time-frequency analysis method for co-decoding public and imagined speech EEG according to claim 1, characterized in that, Also includes: S5. Based on the trained multi-scale EEG speech analysis model driven by cross-frequency coupling, construct a time attention sub-model and a frequency band attention sub-model. Use training set or validation set samples to input the time attention sub-model and the frequency band attention sub-model for inference, and obtain the time attention weight and the frequency band attention weight.

3. The multi-scale time-frequency analysis method for co-decoding public and imagined speech EEG according to claim 1, characterized in that, S2 include: S21. Divide the speech EEG data into a training set and a test set; divide the public speech data and imagined speech data in the training set into the following 5 frequency bands: delta: [1,4], theta: [4,8], alpha: [8,13], beta: [13,30], gamma: [30,45]; combine them by permutation and combination to generate 25 frequency band combinations; merge and classify the speech EEG data in the training set according to the category to which the label belongs; S22. A one-to-many public space pattern feature extraction strategy is adopted to generate public space pattern features.

4. The multi-scale time-frequency analysis method for co-decoding public and imagined speech EEG according to claim 1, characterized in that, S3 include: S31. Construct a multi-scale temporal convolution module to extract temporal convolutional feature maps of common spatial pattern features; S32. Perform average pooling on the temporal convolutional feature map to obtain temporal features from a global perspective; perform convolution on the temporal features to capture the dependencies between adjacent time points and generate a temporal weight vector. S33. Construct a multi-scale frequency band attention module, obtain the frequency band weight matrix, apply frequency band dimension attention weighting to the temporal convolutional feature map, and generate a frequency band feature map. S34. Perform deep spatial filtering on the frequency band feature map; S35. Use separable convolution to extract deep temporal features; S36. Perform residual connection on the data before and after the extraction of deep temporal features; S37. The features after the residual connection are flattened by using a flattening layer, then passed through a fully connected layer, and finally the class probability is output through the softmax function.

5. The multi-scale time-frequency analysis method for co-decoding public and imagined speech EEG according to claim 4, characterized in that, In S31, the multi-scale temporal convolution module includes three two-dimensional convolutional layers of different scales.

6. The multi-scale time-frequency analysis method for co-decoding public and imagined speech EEG according to claim 4, characterized in that, S33 includes the following steps: The multi-scale frequency band attention module receives temporal convolutional feature maps; it performs three different average pooling operations on the temporal convolutional feature maps to generate feature descriptors of three different scales for each of the 25 frequency band combinations. The three feature descriptors at different scales are expanded in dimension, and the three expanded feature descriptors are concatenated in the last dimension to generate a concatenated feature matrix. The concatenated feature matrix is ​​passed through a multilayer perceptron to generate a frequency band weight matrix. The frequency band weight matrix is ​​then expanded in dimension and a time dimension is added. It is then multiplied with the temporal convolutional feature map at corresponding elements to generate a frequency band feature map.

7. The multi-scale time-frequency analysis method for co-decoding public and imagined speech EEG according to claim 6, characterized in that, The three different average pooling operations include: Global average pooling: Performs global average pooling on the temporal convolutional feature map in both the temporal and filter dimensions; Temporal window average pooling: Using a pooling kernel of size [1,3], average pooling is performed on the temporal dimension of the temporal convolutional feature map, and then global average pooling is performed on the pooling result on both the temporal and filter dimensions. Temporal window average pooling: Using a pooling kernel of size [1,5], average pooling is performed on the temporal dimension of the temporal convolutional feature map, and then global average pooling is performed on the pooling result on both the temporal and filter dimensions.

8. The multi-scale time-frequency analysis method for co-decoding public and imagined speech EEG according to claim 6, characterized in that, Multilayer perceptrons include: The hidden layer is a fully connected neural network with an input dimension of 3 and an output dimension of 8, and uses ReLU as the activation function; the input is processed using a time-step independent layer wrapper. The output layer is another fully connected neural network with an input dimension of 8 and an output dimension of 1, and it uses the Sigmoid activation function. The input is processed using a time-step independent layer wrapper, and the final output is a frequency band weight matrix with a dimension of [batch size, number of bands, 1].

9. A multi-scale time-frequency analysis method for co-decoding public and imagined speech EEG according to claim 2, characterized in that, The construction of the time attention sub-model includes: inheriting the weights from the multi-scale EEG speech analysis model after training, forcibly terminating the data flow at the time attention layer, and outputting the time attention weights; the construction of the frequency band attention sub-model includes: inheriting the weights from the multi-scale EEG speech analysis model after training, forcibly terminating the data flow at the multi-scale frequency band attention layer, and outputting the frequency band attention weights.

10. A multi-scale time-frequency analysis system for co-decoding public and imagined speech EEG, characterized in that, include: The data acquisition module is used to collect and preprocess speech EEG data, including public speech data and imagined speech data, and to label the public speech task and imagined speech data according to categories. The coupling feature construction module is used to construct common spatial pattern features that are coupled across frequency bands of public speech data and imagined speech data; A multi-scale time-frequency EEG analysis network, including a multi-scale temporal convolution module and a multi-scale frequency band attention module, is used to predict speech categories based on public space pattern features and generate classification results that determine the speech category of the test subject. The aforementioned multi-scale time-frequency analysis system for co-decoding of public and imagined speech EEG is used to implement the multi-scale time-frequency analysis method and steps of co-decoding of public and imagined speech EEG as described in claim 1.

Images