Transfer learning-based imaginary voice classification method and system

Abstract

The invention relates to the technical field of brain-computer interfaces, in particular to an imaginary voice classification method and system based on transfer learning, and the method comprises the steps: training a full-module model through health feature electroencephalogram signals, extracting spatial attention features through a perceptual coding module, capturing cross-brain region correlation features in combination with a sequence correlation module, and classifying the cross-brain region correlation features; semantic conversion from motion to language is realized; multi-modal noise is introduced through the injury simulation module, electroencephalogram signal characteristics in an injury scene are simulated, and the model in the first stage has higher robustness; by freezing the perceptual coding layer parameters of the first-stage model and only updating the subsequent module parameters, the obtained second-stage model improves the classification accuracy and generalization ability in the damage feature space; according to the method, the accuracy of the language intention recognition task is improved, and the method has higher individual adaptability and anti-noise capability.

Description

Technical Field

[0001] This invention relates to the fields of brain-computer interface and artificial intelligence technology, specifically to an imaginary speech classification method and system based on transfer learning. Background Technology

[0002] In the field of brain-computer interface technology, accurately interpreting brain signals and converting them into understandable instructions or information has always been a research challenge. This is especially true for patients with damage to the language center due to stroke, traumatic brain injury, or ALS, who, due to the loss of language expression ability, cannot express their thoughts and needs through normal language channels, but may still retain motor imagination abilities in their brains.

[0003] In existing technologies, brain-computer interface (BCI) systems based on traditional machine learning mainly rely on signal features from single brain regions, such as classifying EEG signals collected through motor imagery tasks (e.g., imagining hand movements). These methods have significant limitations in language intent recognition scenarios: First, the neural mechanisms of motor imagery signals and language generation signals differ considerably, making direct transfer difficult due to feature decoupling; second, differences in EEG patterns among individual patients necessitate individual model training for each patient, limiting generalization ability; finally, traditional transfer learning frameworks (such as parameter transfer and feature transfer) struggle to handle nonlinear mapping relationships between signals across brain regions, especially in scenarios where damage to the language center leads to signal quality degradation, resulting in low classification accuracy. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides an imaginary speech classification method and system based on transfer learning, aiming to solve the problem that patients with damaged language centers cannot communicate through traditional brain-computer interfaces. By constructing a cross-brain region transfer learning model of motor imagination and language intention, the accuracy of brain signal decoding is improved.

[0005] To achieve the above objectives, the present invention provides the following technical solution: On one hand, embodiments of the present invention provide an imaginary speech classification method based on transfer learning, the method comprising the following steps: Acquire health characteristic EEG signals, input these health characteristic EEG signals into the full-module model, and extract the spatial attention features of these health characteristic EEG signals through the perception coding module; By using the sequence association module, spatial attention features are captured through multi-scale temporal convolution and cross-brain region temporal, frequency band and spatial association features to obtain deep temporal features; The linear mapping layer of the damage simulation module transforms the deep temporal features into the damage feature space, and then the multimodal noise injection layer dynamically adds noise to obtain noisy damage features. The noisy damage features are classified by combining the flattening layer and the fully connected layer of the decision classification module, and the category probability distribution is output by the normalized exponential function. The cross-entropy loss function is used to optimize the multi-classification task. The parameters of the full-module model are adjusted by minimizing the cross-entropy loss value between the true label and the predicted probability distribution of the health feature EEG signal, and a one-stage model is obtained. The injury-feature EEG signals are acquired, and a one-stage model is trained using these signals. During the training process, the parameters of the perception coding module in the one-stage model are frozen, and the trainable parameters of the sequence association module, injury simulation module, and decision classification module in the one-stage model are updated to obtain the two-stage model. The newly acquired EEG signals of injury characteristics are input into the two-stage model to achieve semantic conversion from motor intention to linguistic intention.

[0006] Optionally, the step of extracting the spatial attention features of the EEG signal of this health characteristic through the perception coding module includes: The health characteristic EEG signal is input into the perception coding module. The frequency domain feature extraction layer in the perception coding module performs convolution operation on the EEG signal through the frequency domain weight matrix. The multi-scale convolution kernel is used to extract the EEG rhythm features that reflect the brain's activity state in the set frequency band. Then, the signal is linearly activated by the batch normalization layer to obtain the frequency domain features.

[0007] The spatial attention convolutional layer obtains spatial attention features by performing deep convolution on the spatial attention weight matrix and frequency domain features. Each channel of the spatial attention feature corresponds to the joint response of the set frequency band and spatial location in the original EEG signal.

[0008] Optionally, the step of using the sequence association module to perform multi-scale temporal convolution on spatial attention features and capture temporal, frequency band, and spatial association features across brain regions to obtain deep temporal features, including: Spatial attention features are input into the multi-scale temporal convolutional layer of the sequence association module. Sliding convolution is performed in the temporal dimension through convolution kernels of different scales. For each scale convolution kernel, the convolution result with the spatial attention features is calculated, and the corresponding temporal bias term is added to obtain the temporal features at that scale. The temporal features of different scales are combined to form multi-scale temporal features. The multi-scale temporal features are linearly transformed by the attention weight matrix and then processed by the Sigmoid function to output neural oscillation attention features. The cross-domain association Transformer layer generates query matrix, key matrix, and value matrix by performing a linear transformation on neural oscillatory attention features, and uses multi-head attention mechanism to capture temporal, frequency band, and spatial association features across brain regions; The residual fusion layer adds the associated features and multi-scale temporal features element by element to output the fused features; The fused features are compressed in time dimension by a temporal deep dimensionality reduction layer to output deep temporal features.

[0009] Optionally, the transformation of depth-time features into a damage feature space via a linear mapping layer of the damage simulation module, followed by dynamic noise addition via a multimodal noise injection layer to obtain noisy damage features, includes: The deep temporal features are input into the linear mapping layer, and the deep temporal features are linearly transformed by the transformation weight matrix to generate intermediate features that match the damage scene. By using a transformation bias term to shift and adjust the intermediate features, damage features that conform to the set damage mode are obtained; Laplace distributed noise is generated based on a preset noise intensity coefficient. The Laplace noise is then added to the damage features element by element to obtain the noisy damage features.

[0010] Optionally, the method of classifying noisy damage features by combining the flattening layer and the fully connected layer of the decision classification module, and outputting a predicted probability distribution through a normalized exponential function, includes: The decision classification module converts noisy damage features from multimodal to one-dimensional feature vectors through a flattening layer; The fully connected layer maps the one-dimensional feature vector to the classification space through the weight matrix, generating the raw scores for each category; The original scores are converted into a probability distribution by a normalized exponential function, and a predicted probability distribution is output, which contains the predicted probability for each category.

[0011] Optionally, the optimization of the multi-classification task using the cross-entropy loss function involves adjusting the parameters of the full-module model by minimizing the cross-entropy loss value between the true label and the predicted probability distribution of the health feature EEG signal, resulting in a first-stage model, including: Determine the true labels of health characteristic EEG signals and convert the true labels into one-hot encoded forms; the true labels represent the actual category affiliation of the samples. Calculate the cross-entropy loss between the true label distribution and the predicted probability distribution; The cross-entropy loss value is minimized by the backpropagation algorithm, and the parameters of the whole module model are updated according to the cross-entropy loss value until the cross-entropy loss value converges to below the preset threshold, thus obtaining the first-stage model.

[0012] Optionally, the step of acquiring injury-feature EEG signals, training a one-stage model using these signals, freezing the parameters of the perception-encoding module in the one-stage model during training, and updating the trainable parameters of the sequence association module, injury simulation module, and decision classification module in the one-stage model to obtain a two-stage model includes: The damage feature EEG signal is acquired and input into the trained one-stage model, keeping the parameters of the frequency domain feature extraction layer and the spatial attention convolution layer of the perception coding module unchanged. The parameters of the multi-scale temporal convolutional layer, neural oscillation attention layer, and cross-domain association Transformer layer of the sequence association module, the parameters of the linear mapping layer and multimodal noise injection layer of the damage simulation module, and the parameters of the fully connected layer of the decision classification module are updated by backpropagation algorithm. Continue iterative training until the cross-entropy loss value of the first-stage model converges to below a preset threshold, thus obtaining a second-stage model adapted to the damage feature space.

[0013] On the other hand, embodiments of the present invention provide an imaginary speech classification system based on transfer learning, comprising: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0014] On the other hand, embodiments of the present invention provide a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the above-described method.

[0015] The beneficial effects of this invention are as follows: This invention discloses a method and system for classifying imaginary speech based on transfer learning. By constructing a cross-brain region transfer learning model of motor imagination and language intention, this invention effectively solves the problem of language intention recognition in patients with damaged language centers due to insufficient signal specificity, significant individual differences, and severe noise interference. The method first trains a one-stage model using healthy EEG signals. Spatial attention features are extracted through a perceptual coding module, and cross-brain region association features are captured by a sequence association module, achieving semantic conversion from motor to language. Subsequently, multimodal noise is introduced through an injury simulation module to simulate EEG signal features under injury scenarios, enhancing the model's robustness. Finally, the parameters of the perceptual coding layer are frozen in a two-stage model, and only the parameters of subsequent modules are updated, significantly improving the model's classification accuracy and generalization ability in the injury feature space. Compared with traditional methods, this invention improves the accuracy of language intention recognition tasks and has stronger individual adaptability and noise resistance. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating an imaginary speech classification method based on transfer learning according to an embodiment of the present invention.

[0018] Figure 2 This is a structural diagram of the full-module model in an embodiment of the present invention;

[0019] Figure 3 yes Figure 2 Structure diagram of the sensing and coding module;

[0020] Figure 4 yes Figure 2 Structure diagram of the sequence association module;

[0021] Figure 5 yes Figure 2 Structural diagram of the damage simulation module;

[0022] Figure 6 yes Figure 2 Structure diagram of the decision classification module;

[0023] Figure 7 This is a schematic diagram of the structure of an imaginary speech classification system based on transfer learning in an embodiment of the present invention. Detailed Implementation

[0024] The following will provide a clear and complete description of the concept, specific structure, and technical effects of the present invention in conjunction with embodiments and accompanying drawings, so as to fully understand the purpose, solution, and effects of the present invention. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0025] Among related technologies, brain signals for motor imagery have the following limitations:

[0026] While brain signals associated with motor imagery (such as μ / β rhythms in EEG) can be detected, their use in language simulation presents the following problems:

[0027] Insufficient signal specificity: There is a large difference between the neural representations of motor imagery and language intention, resulting in low accuracy of direct mapping (approximately 40%-60%).

[0028] Significant individual differences: The motor imagery patterns of different patients lack a unified feature space, resulting in poor model generalization ability.

[0029] Severe noise interference: Muscle artifacts and environmental noise can easily mask weak speech-related signals.

[0030] Preliminary application of transfer learning in BCI (Brain-Computer Interface):

[0031] Current transfer learning is mostly applied to signals from the same brain region (such as the transfer of motor imagery signals between different individuals), while cross-brain region transfer learning (motor → language) faces the following challenges:

[0032] Differences in neural mechanisms: The activation patterns and signal frequency characteristics of the motor cortex and the language center are different.

[0033] Data scarcity: Patients with damage to the language center are unable to provide a large amount of paired motor-language imagery data.

[0034] Insufficient model adaptability: Traditional transfer algorithms struggle to handle cross-modal and nonlinear brain signal mapping relationships.

[0035] To address the technical problems in related technologies, this invention provides a method and system for imaginary speech classification based on transfer learning. This method overcomes the limitations of traditional brain-computer interface technology in language intent recognition through an innovative cross-brain region transfer learning framework. Specifically, this invention employs a two-stage training strategy: In the first stage, a basic model is trained using EEG signals from healthy subjects. A perceptual coding module extracts spatially selective frequency domain features, and a sequence association module constructs a cross-brain region time-frequency-spatial joint representation, achieving a preliminary mapping from motor imagination to language intent. In the second stage, a damage simulation module introduces multimodal noise (such as Laplace noise, frequency band attenuation, etc.) to simulate the degradation pattern of EEG signals in patients with damage to the language center. Simultaneously, the parameters of the perceptual coding module are frozen to maintain the stability of feature extraction, and only the parameters of subsequent modules are updated to adapt to the damage feature space. Experiments show that this method significantly improves accuracy compared to traditional methods in language intent classification tasks and significantly enhances robustness to individual differences.

[0036] refer to Figure 1 ,like Figure 1 The image shows an imaginary speech classification method based on transfer learning provided by an embodiment of the present invention. The method includes the following steps: S100: Acquire health feature EEG signals, input the health feature EEG signals into the full module model, and extract the spatial attention features of the health feature EEG signals through the perception coding module; S200 utilizes a sequence association module to perform multi-scale temporal convolution on spatial attention features and capture temporal, frequency, and spatial association features across brain regions to obtain deep temporal features; it achieves semantic transformation mapping from motor intention to linguistic intention through association features; S300 transforms the deep temporal features into the damage feature space through the linear mapping layer of the damage simulation module, and then dynamically adds noise through the multimodal noise injection layer to obtain noisy damage features; S400 uses a combination of flattened and fully connected layers in the decision classification module to classify noisy damage features and outputs the category probability distribution through a normalized exponential function. S500 uses the cross-entropy loss function to optimize multi-classification tasks. It adjusts the parameters of the full-module model by minimizing the cross-entropy loss value between the true label and the predicted probability distribution of the health feature EEG signal, thus obtaining a one-stage model. S600: Acquire injury feature EEG signals, train a one-stage model using injury feature EEG signals, freeze the parameters of the perception coding module in the one-stage model during training, update the trainable parameters of the sequence association module, injury simulation module and decision classification module in the one-stage model, and obtain a two-stage model. The S700 inputs newly acquired injury feature EEG signals into a two-stage model to achieve semantic conversion from motor intention to linguistic intention.

[0037] The core logic of the two-layer cross-task transfer learning framework proposed in this invention is as follows: Figure 2As shown, the method is divided into a health feature pre-training stage and an injury scenario adaptation stage. The health feature pre-training stage builds a basic mapping capability across brain regions. The injury scenario adaptation stage freezes the perceptual encoding module and inputs injury feature EEG signals into the first-stage model trained based on health feature EEG signals for parameter tuning. The resulting second-stage model can more accurately adapt to the EEG signal features of patients with language center damage, achieving efficient semantic conversion from motor intent to language intent. In practical applications, this method demonstrates significant advantages: Firstly, the basic cross-brain region mapping capability built through the health feature pre-training stage provides a stable feature extraction framework for subsequent injury scenario adaptation, avoiding the complexity and instability of training from scratch. Secondly, the injury scenario adaptation stage freezes the parameters of the perceptual encoding module, updating parameters only for subsequent modules. This preserves the model's effective feature extraction capability in the health feature space while quickly adapting to the specificity of the injury feature space, thus significantly improving the model's accuracy and generalization ability in language intent recognition tasks. Furthermore, this method also possesses strong individual adaptability and noise resistance. Because motor imagery patterns vary significantly among patients, and EEG signals are easily interfered with by muscle artifacts and environmental noise, traditional methods often struggle to accurately identify verbal intent. This invention, however, introduces multimodal noise to simulate EEG signal degradation patterns under injury scenarios. This allows the model to be exposed to noise interference similar to real-world scenarios during training, thereby enhancing its robustness and noise resistance. Furthermore, since the model has already established basic cross-brain region mapping capabilities during the pre-training phase focusing on healthy features, it can adjust and adapt more quickly to different patients' motor imagery patterns, achieving more accurate verbal intent recognition.

[0038] The following is a specific embodiment provided by the present invention: Overall architecture overview: Phase 1: Training the full-module model using healthy EEG data (healthy characteristic EEG signals); The second stage involves freezing the parameters of the sensory layer, injecting damage data (damage characteristic EEG signals), fine-tuning the sequence association and classification modules, and simulating robust classification under the scenario of nerve injury.

[0039] Specifically, the full-module model includes: a perception encoding module, a sequence association module, a damage simulation module, and a decision classification module; the module hierarchy is as follows:

[0040] In some embodiments, the extraction of spatial attention features of the health characteristic EEG signal through the perception coding module includes: The health characteristic EEG signal is input into the perception coding module. The frequency domain feature extraction layer in the perception coding module performs convolution operation on the EEG signal through the frequency domain weight matrix. The multi-scale convolution kernel is used to extract the EEG rhythm features that reflect the brain's activity state in the set frequency band. Then, the signal is linearly activated by the batch normalization layer to obtain the frequency domain features. The spatial attention convolutional layer obtains spatial attention features by performing deep convolution on the spatial attention weight matrix and frequency domain features. Each channel of the spatial attention feature corresponds to the joint response of the set frequency band and spatial location in the original EEG signal.

[0041] refer to Figure 3 The perceptual coding module (trainable for Task I, frozen for Task II) includes a frequency domain feature extraction layer, a batch normalization layer, a spatial attention convolutional layer, a normalized activation layer, a temporal downsampling pooling layer, and a Dropout layer. Frequency domain feature extraction layer: use One 1D convolutional kernel (kernel size) Extract EEG rhythm features and output dimensions. The frequency domain characteristics. Corresponding formula: ; in, For the input EEG signal, This represents the convolution operation. Linear activation; The center frequency can be adjusted to cover different frequency bands according to actual needs. These refer to the number of convolutional kernels, spatial dimension, and temporal dimension, respectively. Frequency domain features extracted by the frequency domain feature extraction layer The frequency domain weight matrix, This is the frequency domain bias term.

[0042] Spatial attention convolutional layer: through depthwise convolution (kernel size) Interconnection between learning channels, output dimensions The calculation formula is: ; in, Represents depthwise convolution. For the attention dimension, ; For spatial dimensions, For the attention dimension, Here is the spatial attention weight matrix. Spatial attention features This is the spatial attention weight matrix.

[0043] Spatial attention weight matrix By learning the importance of EEG signals at different spatial locations, the spatial representation of key frequency band features is enhanced; the output spatial attention features are improved. Each channel corresponds to the combined response of a specific frequency band and spatial location in the original EEG signal.

[0044] In practice, frequency domain features are extracted from the input EEG signal. The frequency domain feature extraction layer uses multiple 1D convolution kernels to perform convolution operations to obtain the frequency domain features. Then, the frequency domain features The input spatial attention convolutional layer learns the inter-channel relationships through deep convolution, and the spatial attention weight matrix... Learn the importance of EEG signals at different spatial locations, enhance the spatial representation of key frequency band features, and output spatial attention features. Each channel corresponds to the combined response of a specific frequency band and spatial location in the original EEG signal.

[0045] Spatial attention features are dimensionality reduced and regularized using temporal downsampling pooling layers and Dropout layers. The temporal downsampling pooling layer employs max pooling (1×2 kernel size, 1×2 stride) to compress feature length along the time dimension, reducing computation while preserving key temporal information. The Dropout layer randomly deactivates some neurons with a preset probability (e.g., 0.3) to prevent overfitting. Batch normalization layers and normalized activation layers (using the ReLU function) further standardize the feature distribution and enhance non-linear expressive power, resulting in a final output dimension of [missing value]. Spatial attention features are used for subsequent sequence association modules.

[0046] In some embodiments, the use of the sequence association module to perform multi-scale temporal convolution on spatial attention features and capture temporal, frequency band, and spatial association features across brain regions to obtain deep temporal features includes:

[0047] Spatial attention features are input into the multi-scale temporal convolutional layer of the sequence association module. Sliding convolution is performed in the temporal dimension through convolution kernels of different scales. For each scale convolution kernel, the convolution result with the spatial attention features is calculated, and the corresponding temporal bias term is added to obtain the temporal features at that scale. The temporal features of different scales are combined to form multi-scale temporal features.

[0048] The multi-scale temporal features are linearly transformed by the attention weight matrix and then processed by the Sigmoid function to output neural oscillation attention features.

[0049] The cross-domain association Transformer layer generates query matrix, key matrix and value matrix by linearly transforming neural oscillatory attention features, and captures the temporal, frequency band and spatial association features across brain regions by using multi-head attention mechanism;

[0050] The captured correlation features are fused with multi-scale temporal features through residual connections to form correlation features;

[0051] The residual fusion layer adds the associated features and multi-scale temporal features element by element to output the fused features;

[0052] The fused features are compressed in time dimension by a temporal deep dimensionality reduction layer to output deep temporal features.

[0053] refer to Figure 4 The sequence association module includes: multi-scale temporal convolutional layers, batch normalized activation layers, neural oscillation attention layers, dynamic position encoding layers, cross-domain association Transformer layers, residual fusion layers, and temporal depth dimensionality reduction layers;

[0054] Multi-scale temporal convolutional layers: Employ hybrid kernels (primarily 1×17) to extract long- and short-term features, outputting dimensions... The calculation formula is: ; in, This is the kernel index, which can be either 1 or 2. Let be the temporal weight matrix of the i-th convolutional kernel. This is the temporal bias term for the i-th convolutional kernel; For multi-scale time-domain features, It is an integer division operator.

[0055] It should be noted that the time-domain weight matrix Convolutional kernel parameters of different scales (such as combinations of 1×17 and smaller scale kernels) are used to capture multi-scale features in the temporal domain. Multi-scale convolutional layers slide convolutions in the time dimension through convolutional kernels of different scales to extract short-term (such as within 17 time points) and long-term (across multiple 17-point windows) temporal features of EEG signals, preserving both local details and extracting global trends, thus forming multi-scale temporal features. .

[0056] In practical implementation, spatial attention features As input, the data is fed into a multi-scale temporal convolutional layer. This layer performs sliding convolutions along the temporal dimension using two different scale kernels (with a 1×17 kernel as the main kernel, supplemented by smaller kernels). For each kernel, the convolution result with the input feature is calculated, and the corresponding bias term is added to obtain the temporal feature at that scale. By integrating the convolution results at different scales, multi-scale temporal features are formed. Its dimensions are , where T / / 4 represents downsampling the time dimension by a factor of 4 to reduce computation and preserve key time-domain information.

[0057] Neural oscillatory attention layer: Channel compression is achieved by focusing the mu / beta rhythm (8-30Hz) through Sigmoid activation. ; in, This is the attention weight matrix. It is the Sigmoid activation function. This is a characteristic of neural oscillation attention.

[0058] In practical implementation, the attention weight matrix The dimensional design is for multi-scale temporal features Channel number matching ensures that the feature vector at each time point can independently calculate the attention weights, using the attention weight matrix. Multi-scale temporal features Perform a linear transformation by multiplying each channel. The attention weight map is fused with multi-scale temporal features, and then compressed to the [0,1] interval using the Sigmoid function, so that the final output dimension remains unchanged. The neural oscillatory attention layer enhances the response intensity of mu / beta rhythm-related channels and suppresses irrelevant frequency noise by dynamically adjusting the weights of each channel, dynamically focusing on neural oscillatory patterns related to language intent, improving the feature alignment efficiency of cross-brain region transfer learning, and enhancing the expression of language-related neural oscillatory signals.

[0059] Cross-domain association Transformer layer: 8-head self-attention mechanism simulates brain region interaction, the formula is:

[0060] Where Q, K, and V are the query, key, and value matrices, For key dimensions. This is the result of self-attention calculation; The correlation features of different subspaces are processed in parallel using a multi-head mechanism, and the output dimension is... The layer utilizes an 8-head self-attention mechanism to simulate the nonlinear interaction between the motor cortex and the language center. Each attention head independently learns a specific frequency band-spatial location association pattern, and then the multi-head results are fused through splicing and linear transformation. The Q, K, and V matrices in the formula are derived from neural oscillatory attention features. Linear transformation generation, key dimension To control the sparsity of attention distribution, the softmax function ensures that the weights sum to 1. By capturing the temporal-frequency-spatial trimodal association features across brain regions, these features simultaneously contain temporal dynamics, frequency-specific information, and spatial interaction information between brain regions, providing multi-dimensional feature support for subsequent semantic transformation. Specifically, the 8-head self-attention mechanism decomposes attention computation into multiple subspaces, enabling the model to simultaneously focus on association features of different frequency bands (such as mu rhythms and beta rhythms) and spatial locations (such as frontal lobes and parietal lobes). The concatenation and linear transformation of multi-head results further integrate these heterogeneous information, ultimately outputting association features containing dynamic interactions across brain regions. The cross-domain association Transformer layer solves the problem of feature space misalignment in traditional transfer learning, providing a more discriminative intermediate representation for subsequent damage simulation and classification.

[0061] The associated features and multi-scale temporal features are added element-wise through a residual fusion layer to output fused features. A temporal depth dimensionality reduction layer then compresses the temporal dimension of the fused features to output deep temporal features. These deep temporal features will serve as input to the injury simulation module, participating in the feature transfer process under simulated neural injury scenarios.

[0062] During the injury scenario adaptation phase, after freezing the perception encoding module, the cross-domain association Transformer layer dynamically focuses on the associated features of remaining healthy brain regions or compensatory activation areas when simulating neural injury scenarios by adjusting attention weights. Specifically, when the input is an injury-featured EEG signal, the cross-domain association Transformer layer first generates a new query matrix Q', key matrix K', and value matrix V' through linear transformation. The generation process of Q' and K' incorporates the brain region activity mask output by the injury simulation module, which is dynamically generated based on the injury simulation module's prediction of the brain region functional connectivity strength. Through the masking mechanism, the 8-head self-attention mechanism assigns lower weights to key-value pairs corresponding to the damaged brain region when calculating attention scores, while enhancing the extraction of associated features from healthy brain regions or compensatory areas. For example, when the left side of the language center is damaged, the model automatically reduces the attention weight of key-value pairs in the left frontal lobe and instead focuses on symmetrical regions in the right frontal or parietal lobe, achieving cross-brain region feature compensation. The dynamically adjusted attention weights are renormalized through a softmax function to ensure that the total weight sum is 1 while highlighting the contribution of healthy brain regions. The final output of the associated features maintains a dimension of Kp,1,T / 4, but the feature distribution is significantly biased towards healthy brain regions, and its channel response patterns reflect functional reorganization features under injury conditions. This mechanism solves the problem of feature space misalignment in injury scenarios in traditional transfer learning through dynamic redistribution of attention weights, enabling the model to adapt to injury scenarios simply by adjusting cross-brain region association patterns without updating the parameters of the perceptual encoding module. When the adjusted associated features are added to multi-scale temporal features through the residual fusion layer, the brain region activity mask generated by the injury simulation module also applies dynamic gains to the residual connection paths, further enhancing the feature transfer of healthy brain regions. The final output deep temporal features retain the original temporal dynamics while highlighting the functional compensation patterns under injury conditions, providing more discriminative input features for the decision classification module.

[0063] In some embodiments, the transformation of depth temporal features to a damage feature space via a linear mapping layer of the damage simulation module, followed by dynamic addition of noise via a multimodal noise injection layer to obtain noisy damage features, includes:

[0064] The deep temporal features are input into the linear mapping layer, and the deep temporal features are linearly transformed by the transformation weight matrix to generate intermediate features that match the damage scene.

[0065] By using a transformation bias term to shift and adjust the intermediate features, damage features that conform to the set damage mode are obtained;

[0066] Laplace distributed noise is generated based on a preset noise intensity coefficient. The Laplace noise is then added to the damage features element by element to obtain the noisy damage features.

[0067] refer to Figure 5 The damage simulation module includes a feature space transformation layer and a multimodal noise injection layer;

[0068] Feature space transformation layer: The linear mapping layer transforms health features Transformed into a damage feature space; ; in, To transform the weight matrix, , For damage feature dimensions; As a health characteristic, For transforming the bias term; This represents the noisy impairment features output by the linear mapping layer.

[0069] By transforming the weight matrix With transformation bias term Perform a linear transformation to generate a feature representation that matches the damage scenario. This layer uses a learnable transformation weight matrix. With transformation bias term This method maps the frequency-spatial features of healthy EEG to the damage feature space, simulating signal distortion patterns caused by nerve damage. The dimensions of the weight matrix are designed as follows: ,in The dimensions are dynamically adjusted based on the injury type (e.g., stroke, traumatic brain injury) to ensure flexibility in feature space transformation. Transformation weight matrix. Each row corresponds to a feature transformation rule for a specific damage mode, which is optimized through gradient descent to simulate damage features. The distribution is close to real patient data.

[0070] Multimodal noise injection layer: dynamically adding Laplace noise to simulate nerve injury (such as stroke); ; in, Noise intensity coefficient, It follows a Laplace distribution. These are the location and scale parameters of the Laplace distribution, respectively. This represents the noisy damage characteristics output by the multimodal noise injection layer.

[0071] The multimodal noise injection layer simulates neurological injuries of varying severity (such as mild concussion and severe stroke) by dynamically adjusting the noise intensity coefficient λ. The Laplace distribution parameters μ and b are set according to the range of distortion of the injury signal based on clinical statistics. By adding noise channel by channel, the structural information of the original injury features is preserved while introducing non-stationary interference from real brain injury scenarios, thereby improving the robustness of the model in noisy environments.

[0072] In some embodiments, the classification of noisy damage features using a combination of flattening and fully connected layers in the decision classification module, and the output of a predicted probability distribution through a normalized exponential function, includes:

[0073] The decision classification module converts noisy damage features from multimodal to one-dimensional feature vectors through a flattening layer;

[0074] The fully connected layer maps the one-dimensional feature vector to the classification space through the weight matrix, generating the raw scores for each category;

[0075] The original scores are converted into a probability distribution by a normalized exponential function, and a predicted probability distribution is output, which contains the predicted probability for each category.

[0076] refer to Figure 6 The decision classification module includes a combination of a flattening layer and a fully connected layer: ; in, This represents the weight matrix of the fully connected layer. , This represents the number of categories. Indicates the characteristics of noise-infused damage A flattening operation is performed to convert multidimensional features into one-dimensional feature vectors for processing by fully connected layers. y represents the bias term of the fully connected layer, and y represents the class probability distribution output by the classifier.

[0077] The decision classification module transforms noisy damage features from a multidimensional structure into a one-dimensional vector through a flattening layer, eliminating redundant information in the spatial and temporal dimensions and providing standardized input to the fully connected layer. The fully connected layer uses a fully connected layer weight matrix... Achieve a linear mapping from the feature space to the class space, combined with a bias term. The decision boundary is adjusted, and the output is finally normalized to a probability distribution y using the Softmax function. This module utilizes a learnable weight matrix. (dimension is) (where N is the number of classification categories) captures the non-linear relationship between features and categories, and the bias term... This is used to fine-tune the classification threshold. By optimizing the weight parameters through gradient descent, the model automatically learns the association pattern between EEG signals and language intent in injury scenarios during training, and finally outputs the probability value corresponding to each language category, achieving an efficient mapping from EEG features to language classification.

[0078] In some embodiments, the optimization of the multi-classification task using the cross-entropy loss function involves adjusting the parameters of the full-module model by minimizing the cross-entropy loss value between the true label and the predicted probability distribution of the health feature EEG signal, resulting in a one-stage model, including:

[0079] Determine the true labels of health characteristic EEG signals and convert the true labels into one-hot encoded forms; the true labels represent the actual category affiliation of the samples.

[0080] Calculate the cross-entropy loss between the true label distribution and the predicted probability distribution;

[0081] The cross-entropy loss value is minimized by the backpropagation algorithm, and the parameters of the whole module model are updated according to the cross-entropy loss value until the cross-entropy loss value converges to below the preset threshold, thus obtaining the first-stage model.

[0082] Specifically, the multi-class classification task is optimized using cross-entropy loss, with the loss function being: ; in, For real labels, To predict probabilities. is the cross-entropy loss value, where i is the category index and N is the total number of categories.

[0083] The predicted probability represents the model's confidence in predicting each category, while the true label represents the actual category to which the sample belongs. This loss function quantifies the classification error and guides parameter updates by calculating the cross-entropy between the true label distribution and the model's predicted probability distribution. Specifically, the true labels are one-hot encoded, and the predicted probabilities are normalized using a softmax function to ensure that the probability value for each category is in the [0,1] interval and sums to 1. The sensitivity of the cross-entropy loss to the predicted probability increases with the position of the true label; the loss approaches 0 when the predicted probability is close to the true label, and conversely, it generates a large gradient to drive model optimization. By minimizing the cross-entropy loss, the model can gradually improve the accuracy of classifying imagined speech intentions, especially in scenarios with class imbalance or noise interference, where the gradient characteristics of the cross-entropy loss help stabilize the training process.

[0084] By minimizing this loss function, the model can continuously adjust the weight parameters, making the predicted probability distribution y gradually approximate the true label distribution p, thereby optimizing multi-class classification tasks. During training, cross-entropy loss can effectively measure the uncertainty of the model's predictions and guide the model to update towards the correct classification.

[0085] For an N-class classification task, the loss function predicts the probability of each class. With real labels The logarithmic difference is calculated point-by-point, and the total loss is obtained by summing the negative logarithms. This allows the model to focus on misclassified samples during training, especially when the predicted probability corresponding to the true label is low, resulting in a larger loss gradient and accelerating the optimization of model parameters. Furthermore, the combination of cross-entropy loss and the softmax function ensures the normalization of output probabilities, enabling the model to generate an effective class probability distribution in multi-class tasks, providing a reliable basis for subsequent classification decisions. During implementation, the importance of different classes or samples can be balanced by adjusting the learning rate or introducing loss weights, further improving the model's classification performance in complex scenarios.

[0086] In some embodiments, the acquisition of injury-feature EEG signals, training a one-stage model using these signals, freezing the parameters of the perception encoding module in the one-stage model during training, and updating the trainable parameters of the sequence association module, injury simulation module, and decision classification module in the one-stage model to obtain a two-stage model includes:

[0087] The damage feature EEG signal is acquired and input into the trained one-stage model, keeping the parameters of the frequency domain feature extraction layer and the spatial attention convolution layer of the perception coding module unchanged.

[0088] The parameters of the multi-scale temporal convolutional layer, neural oscillation attention layer, and cross-domain association Transformer layer of the sequence association module, as well as the parameters of the linear mapping layer and multimodal noise injection layer of the injury simulation module, and the parameters of the fully connected layer of the decision classification module are updated by backpropagation algorithm.

[0089] Continue iterative training until the cross-entropy loss value of the first-stage model converges to below a preset threshold, thus obtaining a second-stage model adapted to the damage feature space.

[0090] The parameters for the multi-scale temporal convolutional layer include the kernel weights and temporal bias terms. The parameters for the neural oscillatory attention layer include the attention weight matrix. The parameters for the cross-domain association Transformer layer include the query matrix weights, key matrix weights, value matrix weights, and linear transformation parameters in the multi-head attention mechanism. The parameters for the linear mapping layer include the transformation weight matrix and transformation bias terms. The parameters for the multimodal noise injection layer include the noise intensity coefficients. The parameters for the fully connected layer include the weight matrix, all of which are the parameters that need to be updated. This update process optimizes the parameters using a gradient descent algorithm, ensuring that the two-stage model has stronger feature extraction and classification capabilities in the damage feature space. The final two-stage model retains the general feature extraction capabilities of the healthy state by freezing the perceptual coding module, while simultaneously achieving accurate adaptation to damage features by updating the parameters of other modules, significantly improving the model's classification performance and robustness in scenarios involving damage to the language center.

[0091] Specifically, the cross-task migration strategy employs a parameter freezing mechanism, freezing all parameters of the perception encoding module (approximately) in Task II. (where C is the number of channels), only the sequence association layer, damage simulation layer, and classification layer are updated, and the formula is expressed as follows: ; ; in, This is the set of frozen parameters for the perception encoding module. For fixed parameter identification; These are the sets of trainable parameters for the sequence association module, the damage simulation module, and the decision classification module, respectively. This is an identifier for trainable parameters.

[0092] Fixed parameter identifier This indicates that the parameters of the perceptual encoding module remain unchanged during the training of Task II. Trainable parameter identifier. The parameters of the sequence association module, damage simulation module, and decision classification module are continuously updated during the training of Task II. In the specific implementation, the parameter freezing of the perceptual coding module is achieved by disabling gradient backpropagation, while the parameter updates of other modules are completed through regular backpropagation.

[0093] During cross-task transfer, the parameter freezing mechanism avoids overfitting of the feature extraction layer due to task differences by fixing the parameters of the perceptual coding module (such as frequency domain convolutional kernels and spatial attention weights). Specifically, the perceptual coding module includes a frequency domain feature extraction layer and a spatial attention convolutional layer, with a parameter scale of approximately (Frequency domain convolution kernel parameters) and (Spatial attention weight matrix parameters), freezing these parameters preserves the general frequency-spatial feature representation capabilities learned in the source task, while allowing the sequence association layer, damage simulation layer, and classification layer to adapt across tasks based on the frozen parameters. By updating the multi-scale temporal convolution kernel, attention weight matrix, and classifier parameters, transfer learning from healthy to damaged states is achieved. The set of trainable parameters for the sequence association module, damage simulation module, and decision classification module ( , , It can be dynamically adjusted through backpropagation to adapt to the specific needs of the target task (such as imagined speech classification).

[0094] For example, in Task II, when transitioning from healthy EEG signals to the imagined speech impairment classification task, the sequence association module adjusts the temporal feature extraction scale by updating the multi-scale temporal convolution kernel parameters (e.g., a combination of 1×17 and smaller-scale kernels), enabling the model to capture more complex temporal fluctuation patterns under impairment conditions; the neural oscillation attention layer optimizes the attention weight matrix. The dimensions and values ​​are dynamically focused on the damage-related mu / beta rhythm (8-30Hz), suppressing abnormal frequency noise caused by neural damage and enhancing the expression of neural oscillation patterns related to language intent. The cross-domain association Transformer layer simulates the changes in nonlinear interaction between the motor cortex and the language center under damage conditions by adjusting the query-key-value matrix (Q,K,V) generation rules of the 8-head self-attention mechanism, capturing the time-frequency-spatial trimodal association features across brain regions. With bias term Adjust the linear mapping rule from health features to damage features to simulate damage features. The distribution of noise is closer to real patient data; the multimodal noise injection layer simulates different degrees of neurological injury (such as mild concussion and severe stroke) by dynamically adjusting the noise intensity coefficient λ and the Laplace distribution parameter (μ, b), introducing non-stationary disturbances from real injury scenarios and improving the model's robustness in noisy environments. The decision classification module updates the weight matrix of the fully connected layer. Bias terms with fully connected layers The model optimizes the classification boundary, enabling it to distinguish between different categories such as imagined speech and resting state. Gradient descent is used to adjust parameters, improving classification stability in small sample scenarios. The combination of parameter freezing and dynamic adjustment of trainable parameters preserves the general feature extraction capabilities learned from the source task while achieving cross-task adaptation through local parameter updates. This effectively solves the feature space misalignment problem caused by task differences in traditional transfer learning, improving the model's performance in the imagined speech impairment classification task.

[0095] Compared with the prior art, the present invention has the following improvements:

[0096] I. Breakthrough in Signal Source Utilization Methods

[0097] Existing technologies rely solely on EEG signals (such as P300, N400, and other event-related potentials) or functional magnetic resonance imaging (BOLD) signals generated by the language center. When the language center loses its function due to damage (such as stroke or traumatic brain injury), these technologies become completely ineffective and cannot obtain effective signals.

[0098] This invention innovatively enhances the accuracy of brain signal extraction by constructing a model through transfer learning between motor imagery brain signals and language centers. Even with damage to the language center, patients can still generate detectable EEG activity by imagining limb movements (such as hand movements), providing a new pathway for signal acquisition.

[0099] II. Cross-domain upgrade of transfer learning strategies

[0100] Existing technology: Transfer learning is mostly applied to cross-individual scenarios within the same brain region (e.g., transferring the motor imagery model of healthy person A to healthy person B). Essentially, it is parameter optimization between similar tasks and does not involve cross-domain mapping of brain region functions.

[0101] This invention proposes for the first time a cross-brain region transfer strategy from the motor cortex to the language center. By forcibly learning the shared feature space of signals from the two brain regions, it achieves semantic conversion from motor intention to language intention, solving the problem of dependence on the integrity of the language center in traditional techniques.

[0102] V. Decoding Performance and Application Scenarios Expansion

[0103] Existing technology: The decoding accuracy in patients with impaired language center is generally less than 50%, and it can only achieve simple "yes / no" judgments. It cannot support complex semantic communication, and its application scenarios are strictly limited to people with intact language center function (such as healthy subjects or people with mild language disorders).

[0104] This invention enables semantic recognition of common words (such as "drink water" and "help") in scenarios where the simulated language center is damaged.

[0105] refer to Figure 7 This invention also provides an imaginary speech classification system based on transfer learning, comprising: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0106] The content of the above method embodiments is applicable to this embodiment. The specific functions implemented in this embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments. Therefore, they will not be repeated here.

[0107] Furthermore, one embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions for performing the above-described method.

[0108] It is worth noting that, since the computer-readable storage medium of the present invention is capable of executing the methods of any of the above embodiments, the specific implementation methods and technical effects of the computer-readable storage medium of the present invention can be referred to the specific implementation methods and technical effects of the methods of any of the above embodiments.

[0109] Although the description of this disclosure has been quite detailed and particularly focused on several of the described embodiments, it is not intended to limit itself to any of these details or embodiments or any particular embodiment, but should be considered as effectively covering the intended scope of this disclosure by referring to the appended claims and taking into account the broad possible interpretations of these claims provided by the prior art. Furthermore, the foregoing description of this disclosure with respect to embodiments foreseeable by the inventors is intended to provide a useful description, and non-substantial modifications to this disclosure that have not yet been foreseen may still represent equivalent modifications.

Claims

1. A method for classifying imaginary speech based on transfer learning, characterized in that, The method includes the following steps: Acquire health characteristic EEG signals, input these health characteristic EEG signals into the full-module model, and extract the spatial attention features of these health characteristic EEG signals through the perception coding module; By using the sequence association module, spatial attention features are captured through multi-scale temporal convolution and cross-brain region temporal, frequency band and spatial association features to obtain deep temporal features; The linear mapping layer of the damage simulation module transforms the deep temporal features into the damage feature space, and then the multimodal noise injection layer dynamically adds noise to obtain noisy damage features. The noisy damage features are classified by combining the flattening layer and the fully connected layer of the decision classification module, and the category probability distribution is output by the normalized exponential function. The cross-entropy loss function is used to optimize the multi-classification task. The parameters of the full-module model are adjusted by minimizing the cross-entropy loss value between the true label and the predicted probability distribution of the health feature EEG signal, and a one-stage model is obtained. The injury-feature EEG signals are acquired, and a one-stage model is trained using these signals. During the training process, the parameters of the perception coding module in the one-stage model are frozen, and the trainable parameters of the sequence association module, injury simulation module, and decision classification module in the one-stage model are updated to obtain the two-stage model. The newly acquired EEG signals of injury characteristics are input into the two-stage model to achieve semantic conversion from motor intention to linguistic intention.

2. The method according to claim 1, characterized in that, The extraction of spatial attention features from the EEG signal of this health characteristic via the perception coding module includes: The health characteristic EEG signal is input into the perception coding module. The frequency domain feature extraction layer in the perception coding module performs convolution operation on the EEG signal through the frequency domain weight matrix. The multi-scale convolution kernel is used to extract the EEG rhythm features that reflect the brain's activity state in the set frequency band. Then, the signal is linearly activated by the batch normalization layer to obtain the frequency domain features. The spatial attention convolutional layer obtains spatial attention features by performing deep convolution on the spatial attention weight matrix and frequency domain features. Each channel of the spatial attention feature corresponds to the joint response of the set frequency band and spatial location in the original EEG signal.

3. The method according to claim 1, characterized in that, The sequence association module is used to perform multi-scale temporal convolution on spatial attention features and capture temporal, frequency, and spatial association features across brain regions to obtain deep temporal features, including: Spatial attention features are input into the multi-scale temporal convolutional layer of the sequence association module. Sliding convolution is performed in the temporal dimension through convolution kernels of different scales. For each scale convolution kernel, the convolution result with the spatial attention features is calculated, and the corresponding temporal bias term is added to obtain the temporal features at that scale. The temporal features of different scales are combined to form multi-scale temporal features. The multi-scale temporal features are linearly transformed by the attention weight matrix and then processed by the Sigmoid function to output neural oscillation attention features. The cross-domain association Transformer layer generates query matrix, key matrix, and value matrix by performing a linear transformation on neural oscillatory attention features, and uses multi-head attention mechanism to capture temporal, frequency band, and spatial association features across brain regions; The residual fusion layer adds the associated features and multi-scale temporal features element by element to output the fused features; The fused features are compressed in time dimension by a temporal deep dimensionality reduction layer to output deep temporal features.

4. The method according to claim 1, characterized in that, The process involves transforming the depth-time features into the damage feature space through a linear mapping layer of the damage simulation module, followed by dynamically adding noise through a multimodal noise injection layer to obtain noisy damage features, including: The deep temporal features are input into the linear mapping layer, and the deep temporal features are linearly transformed by the transformation weight matrix to generate intermediate features that match the damage scene. By using a transformation bias term to shift and adjust the intermediate features, damage features that conform to the set damage mode are obtained; Laplace distributed noise is generated based on a preset noise intensity coefficient. The Laplace noise is then added to the damage features element by element to obtain the noisy damage features.

5. The method according to claim 1, characterized in that, The method employs a combination of flattening and fully connected layers in the decision classification module to classify noisy damage features, and outputs a predicted probability distribution through a normalized exponential function, including: The decision classification module converts noisy damage features from multimodal to one-dimensional feature vectors through a flattening layer; The fully connected layer maps the one-dimensional feature vector to the classification space through the weight matrix, generating the raw scores for each category; The original scores are converted into a probability distribution by a normalized exponential function, and a predicted probability distribution is output, which contains the predicted probability for each category.

6. The method according to claim 1, characterized in that, The method of optimizing a multi-class classification task using the cross-entropy loss function involves adjusting the parameters of the full-module model by minimizing the cross-entropy loss value between the true label and the predicted probability distribution of the health feature EEG signal, resulting in a first-stage model, including: Determine the true labels of health characteristic EEG signals and convert the true labels into one-hot encoded forms; the true labels represent the actual category affiliation of the samples. Calculate the cross-entropy loss between the true label distribution and the predicted probability distribution; The cross-entropy loss value is minimized by the backpropagation algorithm, and the parameters of the whole module model are updated according to the cross-entropy loss value until the cross-entropy loss value converges to below the preset threshold, thus obtaining the first-stage model.

7. The method according to claim 1, characterized in that, The process involves acquiring injury-feature EEG signals, training a one-stage model using these signals, freezing the parameters of the perception-encoding module in the one-stage model during training, and updating the trainable parameters of the sequence association module, injury simulation module, and decision classification module in the one-stage model to obtain a two-stage model, including: The damage feature EEG signal is acquired and input into the trained one-stage model, keeping the parameters of the frequency domain feature extraction layer and the spatial attention convolution layer of the perception coding module unchanged. The parameters of the multi-scale temporal convolutional layer, neural oscillation attention layer, and cross-domain association Transformer layer of the sequence association module, the parameters of the linear mapping layer and multimodal noise injection layer of the damage simulation module, and the parameters of the fully connected layer of the decision classification module are updated by backpropagation algorithm. Continue iterative training until the cross-entropy loss value of the first-stage model converges to below a preset threshold, thus obtaining a second-stage model adapted to the damage feature space.

8. An imaginary speech classification system based on transfer learning, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images