Electroencephalogram signal recognition method and device, computer equipment, storage medium and computer program product

Abstract

The invention provides an electroencephalogram signal recognition method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a multi-channel electroencephalogram signal when a user completes a motor imagery task; inputting the multi-channel electroencephalogram signals into a time-frequency feature extraction network of a motor imagery task recognition model to obtain time-domain features and frequency-domain features; inputting the time domain features and the frequency domain features into a time-frequency feature fusion network of a motor imagery task recognition model to obtain fusion features; inputting the fusion features into a time sequence modeling network of a motor imagery task recognition model to obtain time sequence features; and inputting the time sequence characteristics into a classification network of a motor imagery task identification model to obtain the task type of the motor imagery task. By adopting the method, the task type of the motor imagery task can be accurately identified.

Description

Technical Field

[0001] This application relates to the field of brain-computer interface technology, and in particular to a method, device, computer equipment, storage medium and computer program product for recognizing electroencephalogram (EEG) signals. Background Technology

[0002] Motor imagery brain-computer interface technology identifies the EEG signal patterns generated when a user imagines specific limb movements (such as left and right hand movements) and converts them into control commands, providing a pathway for rehabilitation medicine, neuroprosthetic control, and advanced human-computer interaction.

[0003] Currently, classification methods based on motor imagery EEG signals mainly rely on analyzing EEG signals from a single dimension. However, these classification methods cannot comprehensively characterize the neural representations of motor imagery tasks, leading to insufficient information utilization and thus inaccurate classification of EEG signals for motor imagery tasks.

[0004] Therefore, traditional technologies have the problem of being unable to accurately classify EEG signals. Summary of the Invention

[0005] Based on this, the purpose of this application is to at least solve one of the above-mentioned technical defects, especially the technical defect that the prior art cannot accurately classify EEG signals. This application provides an EEG signal recognition method, device, computer equipment, computer-readable storage medium and computer program product that can accurately classify EEG signals.

[0006] In a first aspect, this application provides a method for recognizing electroencephalogram (EEG) signals, the method comprising: Acquire multi-channel EEG signals from users while they are completing a motor imagery task; Multi-channel EEG signals are input into the time-frequency feature extraction network of the motor imagery task recognition model to obtain time-domain and frequency-domain features; The time-domain features and frequency-domain features are input into the time-frequency feature fusion network of the motion imagery task recognition model to obtain the fused features; The fused features are input into the temporal modeling network of the motion imagery task recognition model to obtain temporal features; The temporal features are input into the classification network of the motion imagery task recognition model to obtain the task type of the motion imagery task.

[0007] In an exemplary embodiment, the time-frequency feature fusion network includes a temporal convolutional unit, a frequency convolutional unit, and an attention weight generation unit; the temporal and frequency features are input into the time-frequency feature fusion network of the motion imagery task recognition model to obtain fused features, including: Temporal features are input into temporal convolution units to obtain temporal convolutional features, and frequency features are input into frequency convolution units to obtain frequency convolutional features; The frequency domain convolutional features are input into the attention weight generation unit to generate the attention weight matrix; The temporal convolutional features are weighted using an attention weight matrix to obtain fused features.

[0008] In an exemplary embodiment, the temporal convolutional unit includes multiple temporal convolutional layers of various scales; the frequency convolutional unit includes multiple frequency convolutional layers of various scales; inputting temporal features into the temporal convolutional unit to obtain temporal convolutional features includes: Temporal features are input into temporal convolutional layers of multiple scales to obtain output features of temporal convolutional layers of multiple scales; The output features of multiple temporal convolutional layers at different scales are concatenated to obtain temporal convolutional features; Frequency domain features are input into frequency domain convolution units to obtain frequency domain convolution features, including: Frequency domain features are input into frequency domain convolutional layers of multiple scales to obtain frequency domain convolutional layer output features of multiple scales; The output features of multiple frequency domain convolutional layers at different scales are concatenated to obtain frequency domain convolutional features.

[0009] In an exemplary embodiment, the attention weight generation unit includes a convolutional layer, a batch normalization layer, and an activation layer; inputting frequency domain convolutional features into the attention weight generation unit to generate an attention weight matrix includes: The frequency domain convolutional features are input into the convolutional layer to obtain the initial weight matrix; The initial weight matrix is ​​input into the normalization layer to obtain the normalized weight matrix; The normalized weight matrix is ​​input into the activation layer to obtain the attention weight matrix.

[0010] In an exemplary embodiment, the temporal modeling network includes a feature reshaping unit, a forward temporal modeling unit, a backward temporal modeling unit, and a bidirectional feature fusion unit; the fused features are input into the temporal modeling network of the motion imagery task recognition model to obtain temporal features, including: The fused features are input into the feature reshaping unit to obtain the feature reshaping result; The feature reshaping results are input into the forward temporal modeling unit to obtain forward temporal features, and the feature reshaping results are input into the backward temporal modeling unit to obtain reverse temporal features; The forward and reverse time series features are input into the bidirectional feature fusion unit to obtain the time series features.

[0011] In an exemplary embodiment, the time-frequency feature extraction network includes a time-domain feature extraction unit and a frequency-domain feature extraction unit. Multi-channel EEG signals are input into the time-frequency feature extraction network of the motor imagery task recognition model to obtain time-domain features and frequency-domain features, including: Feature extraction was performed on multi-channel EEG signals to obtain time-domain and frequency-domain representations; Align the time-domain representation and the frequency-domain representation along the time dimension to obtain the aligned time-domain representation and the aligned frequency-domain representation; The aligned time-domain representation is input into the time-domain feature extraction unit to obtain time-domain features, and the aligned frequency-domain representation is input into the frequency-domain feature extraction unit to obtain frequency-domain features.

[0012] Secondly, this application provides a brainwave signal recognition device, the device comprising: The signal acquisition module is used to acquire multi-channel EEG signals from the user while performing a motor imagery task; The feature extraction module is used to input multi-channel EEG signals into the time-frequency feature extraction network of the motor imagery task recognition model to obtain time-domain features and frequency-domain features; The feature fusion module is used to input time-domain features and frequency-domain features into the time-frequency feature fusion network of the motion imagery task recognition model to obtain fused features; The temporal modeling module is used to input the fused features into the temporal modeling network of the motion imagery task recognition model to obtain temporal features; The task classification module is used to input temporal features into the classification network of the motion imagery task recognition model to obtain the task type of the motion imagery task.

[0013] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0014] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0015] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0016] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: The EEG signal recognition method, device, computer equipment, storage medium, and computer program product provided in this application acquire multi-channel EEG signals from a user while performing a motor imagery task; input the multi-channel EEG signals into the time-frequency feature extraction network of a motor imagery task recognition model to obtain time-domain and frequency-domain features; input the time-domain and frequency-domain features into the time-frequency feature fusion network of the motor imagery task recognition model to obtain fused features; input the fused features into the temporal modeling network of the motor imagery task recognition model to obtain temporal features; and input the temporal features into the classification network of the motor imagery task recognition model to obtain the task type of the motor imagery task. Thus, through an end-to-end recognition model that includes time-frequency feature extraction, time-frequency feature fusion, temporal modeling, and classification, it can automatically and layer-by-layer extract and fuse discriminative information in the time and frequency domains from the original multi-channel EEG signals, and model their dynamic temporal patterns. This comprehensively utilizes the multi-dimensional characteristics of EEG signals, effectively solving the problem of inaccurate classification of EEG signals due to incomplete feature expression and insufficient utilization of temporal information, and significantly improving the accuracy of motor imagery task recognition. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating an electroencephalogram (EEG) signal recognition method provided in this application embodiment; Figure 2 A schematic diagram of the motion imagery task recognition model provided in an embodiment of this application; Figure 3 A flowchart illustrating another EEG signal recognition method provided in this application embodiment; Figure 4 A structural block diagram of an electroencephalogram (EEG) signal recognition device provided in an embodiment of this application; Figure 5 This is an internal structural diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] In one exemplary embodiment, such as Figure 1 As shown, a method for recognizing electroencephalogram (EEG) signals is provided. Taking the application of this method to a server as an example, the method includes the following steps S102 to S110. Wherein: Step S102: Acquire multi-channel EEG signals from the user while completing the motor imagery task.

[0021] Among them, multichannel EEG signals refer to time-series signals collected through multiple electrodes that reflect changes in brain potentials during specific thought activities.

[0022] Among them, motor imagery tasks refer to cognitive tasks in which users consciously imagine performing actions of specific body parts (such as left hand, right hand, and foot).

[0023] Optionally, the server can acquire, in real-time or non-real-time, the raw multi-channel EEG signals generated by the user during motor imagery based on prompts using an EEG acquisition device.

[0024] Step S104: Input the multi-channel EEG signal into the time-frequency feature extraction network of the motor imagery task recognition model to obtain time-domain features and frequency-domain features.

[0025] The motion imagery task recognition model is a deep learning model for automatically identifying the types of user motion imagery. It sequentially comprises a time-frequency feature extraction network, a time-frequency feature fusion network, a time-series modeling network, and a classification network. (See reference...) Figure 2 The diagram shows a schematic of the model. The motion imagery task recognition model can be built based on the EnBiTE (Enhanced Bidirectional Time-Freq Pyramid Network) model, which is an improvement on the BiTE (Bidirectional Time-Freq Pyramid Network) model.

[0026] Among them, the time-frequency feature extraction network is a neural network module used to extract time-dimensional change patterns (time-domain features) and frequency-dimensional energy distribution patterns (frequency-domain features) from multi-channel EEG signals.

[0027] Among them, temporal features are feature vectors or feature maps that characterize the changes in amplitude, morphology, etc. of EEG signals over time.

[0028] Among them, frequency domain features are feature vectors or feature maps that characterize the energy or power distribution of EEG signals in different frequency bands (such as μ rhythm and β rhythm).

[0029] Optionally, the server inputs multi-channel EEG signals into a pre-trained motor imagery task recognition model, in which a time-frequency feature extraction network extracts time-domain and frequency-domain features.

[0030] Step S106: Input the time-domain features and frequency-domain features into the time-frequency feature fusion network of the motion imagery task recognition model to obtain fused features.

[0031] Among them, the time-frequency feature fusion network is a neural network module used to adaptively integrate time-domain features and frequency-domain features to generate fused features containing cross-domain information.

[0032] Among them, fusion features refer to composite feature representations that simultaneously contain key time-domain patterns and frequency-domain context information after being processed by the fusion network.

[0033] Optionally, the server inputs the time-domain features and frequency-domain features extracted by the time-frequency feature extraction network into the time-frequency feature fusion network of the motion imagery task recognition model, thereby fusing the time-domain features and frequency-domain features to obtain fused features.

[0034] Step S108: Input the fused features into the temporal modeling network of the motion imagery task recognition model to obtain temporal features.

[0035] Among them, the temporal modeling network is a neural network module used to capture the dynamic evolution and contextual dependencies of fused features in time series, thereby outputting temporal features.

[0036] Among them, temporal features refer to dynamic feature representations that further model the sequential dependencies of feature sequences on the time axis.

[0037] Optionally, the server inputs the fused features output by the time-frequency feature fusion network into the temporal modeling network of the motion imagery task recognition model, and performs temporal modeling on the fused features through the temporal modeling network to obtain temporal features.

[0038] Step S110: Input the temporal features into the classification network of the motion imagery task recognition model to obtain the task type of the motion imagery task.

[0039] Among them, the classification network is a neural network module used to ultimately determine the specific category of the motion imagery task based on temporal features.

[0040] Optionally, the server inputs the temporal features output by the temporal modeling network into the classification network of the motion imagery task recognition model, and the classification network finally outputs the judgment result of the user's current motion imagery category.

[0041] In practical applications, for each multi-channel EEG signal, the imagination task recognition model outputs which type of movement it belongs to. For example, it determines whether a multi-channel EEG signal is neural activity data under a motor imagination task such as left hand, right hand, both feet, or tongue.

[0042] The aforementioned EEG signal recognition method acquires multi-channel EEG signals from a user while performing a motor imagery task. These signals are then input into the time-frequency feature extraction network of the motor imagery task recognition model to obtain time-domain and frequency-domain features. The time-domain and frequency-domain features are then input into the time-frequency feature fusion network of the same model to obtain fused features. These fused features are then input into the temporal modeling network of the same model to obtain temporal features. Finally, the temporal features are input into the classification network of the same model to determine the task type. Thus, through an end-to-end recognition model encompassing time-frequency feature extraction, fusion, temporal modeling, and classification, the method automatically and layer-by-layer extracts and fuses discriminative information in the time and frequency domains from the original multi-channel EEG signals, models their dynamic temporal patterns, and comprehensively utilizes the multi-dimensional characteristics of EEG signals. This effectively solves the problem of inaccurate classification of EEG signals due to incomplete feature representation and insufficient utilization of temporal information, significantly improving the accuracy of motor imagery task recognition.

[0043] In an exemplary embodiment, the time-frequency feature fusion network includes a time-domain convolutional unit, a frequency-domain convolutional unit, and an attention weight generation unit. Inputting time-domain and frequency-domain features into the time-frequency feature fusion network of the motion imagery task recognition model to obtain fused features includes: inputting time-domain features into the time-domain convolutional unit to obtain time-domain convolutional features, and inputting frequency-domain features into the frequency-domain convolutional unit to obtain frequency-domain convolutional features; inputting the frequency-domain convolutional features into the attention weight generation unit to generate an attention weight matrix; and weighting the time-domain convolutional features using the attention weight matrix to obtain the fused features.

[0044] The attention weight matrix is ​​a numerical matrix in which each element represents an assessment of the importance of the corresponding part of the temporal convolutional feature. The larger the value, the higher the contribution of that part of the feature to the final classification.

[0045] In this application, the numerical distribution of the attention weight matrix reflects the importance of frequency domain features in the time-frequency space. Regions with high values ​​represent key information (such as salient features), while regions with low values ​​represent redundant information (such as noise).

[0046] Weighted processing refers to the operation of multiplying the attention weight matrix element-wise with the temporal convolutional features to enhance key features and suppress non-key features.

[0047] Optionally, the server feeds the temporal and frequency domain features into separate convolutional units for deep feature transformation. Then, based on the frequency domain convolutional features obtained from the frequency domain features, an attention map is generated. This attention map is used to recalibrate the saliency of the temporal convolutional features, so that the fused features can focus on the information segments that are most important to the current frequency domain context in the temporal domain.

[0048] In this embodiment, by introducing an attention mechanism based on frequency domain context information, adaptive interactive fusion between time domain features and frequency domain features is achieved, thereby enhancing features and effectively solving the problem that traditional simple splicing or addition fusion methods cannot dynamically adjust the feature contribution based on the signal content. This provides a more discriminative feature representation for subsequent classification and directly improves the accuracy of classification.

[0049] In an exemplary embodiment, the temporal convolutional unit includes multiple temporal convolutional layers of various scales; the frequency convolutional unit includes multiple frequency convolutional layers of various scales; inputting temporal features into the temporal convolutional unit to obtain temporal convolutional features includes: inputting temporal features into multiple temporal convolutional layers of various scales to obtain output features of multiple temporal convolutional layers; concatenating the output features of multiple temporal convolutional layers of various scales to obtain temporal convolutional features; inputting frequency domain features into the frequency convolutional unit to obtain frequency convolutional features includes: inputting frequency domain features into multiple frequency convolutional layers of various scales to obtain output features of multiple frequency convolutional layers of various scales; concatenating the output features of multiple frequency convolutional layers of various scales to obtain frequency convolutional features.

[0050] Multiple scales refer to the fact that convolutional kernels have different sizes to capture feature patterns of different ranges or granularities.

[0051] Optionally, the server uses convolutional layers with different kernel sizes in parallel on both the time and frequency domain branches. For time domain features, convolutional layers of different scales can simultaneously capture short-term transient patterns and long-term dependent patterns. For frequency domain features, convolutional layers of different scales can extract spectral details of different bandwidth ranges. The server integrates the output features of all scales in the time and frequency domain branches to form time-domain convolutional features and frequency-domain convolutional features with rich multi-scale information.

[0052] In this embodiment, by employing multi-scale convolution in both the time and frequency domains, local and global patterns of EEG signals at different time and frequency resolutions can be extracted in parallel. This overcomes the shortcomings of single-scale convolution, which has a fixed receptive field and may miss important scale information, ensuring the comprehensiveness of feature extraction. It provides a rich feature foundation for subsequent fusion and classification steps, thereby improving the motor imagery task recognition model's ability to represent complex and varied EEG patterns and its classification accuracy from the source.

[0053] In an exemplary embodiment, the attention weight generation unit includes a convolutional layer, a batch normalization layer, and an activation layer; inputting frequency domain convolutional features into the attention weight generation unit to generate an attention weight matrix includes: inputting frequency domain convolutional features into the convolutional layer to obtain an initial weight matrix; inputting the initial weight matrix into the normalization layer to obtain a normalized weight matrix; and inputting the normalized weight matrix into the activation layer to obtain an attention weight matrix.

[0054] Among them, the batch normalization layer is a technique layer used to stabilize network training and accelerate convergence. It standardizes the input data to keep its mean and variance stable.

[0055] The activation layer refers to the Sigmoid function, which maps weight values ​​to the (0, 1) interval and directly represents the importance weight.

[0056] Optionally, the server compresses and transforms the input frequency domain convolutional features through a convolutional layer to generate initial attention scores. Then, it stabilizes the distribution of these scores through a batch normalization layer and finally normalizes them to probability values ​​between 0 and 1 through a Sigmoid activation function, which serve as the final attention weights.

[0057] In this embodiment, attention weights are generated by a lightweight network consisting of convolutional layers, batch normalization layers, and activation layers. This enables the attention mechanism to efficiently learn the modulation signal of temporal features from frequency domain features in a data-driven manner, ensuring the rationality and effectiveness of the generated attention weights. This makes the feature fusion process more reliable and accurate, thereby stabilizing and improving the final classification performance.

[0058] In an exemplary embodiment, the temporal modeling network includes a feature reshaping unit, a forward temporal modeling unit, a backward temporal modeling unit, and a bidirectional feature fusion unit. Inputting the fused features into the temporal modeling network of the motion imagery task recognition model to obtain temporal features includes: inputting the fused features into the feature reshaping unit to obtain a feature reshaping result; inputting the feature reshaping result into the forward temporal modeling unit to obtain a forward temporal feature; and inputting the feature reshaping result into the backward temporal modeling unit to obtain a reverse temporal feature; and inputting the forward and reverse temporal features into the bidirectional feature fusion unit to obtain temporal features.

[0059] The feature reshaping unit is used to adjust the dimensions of features to fit the input format of the time series modeling unit.

[0060] The forward temporal modeling unit and the backward temporal modeling unit are components of the bidirectional temporal convolutional network (TCN), which are used to model sequence dependencies from front to back and from back to front, respectively.

[0061] The bidirectional feature fusion unit is used to integrate temporal context information from both the forward and reverse directions.

[0062] Optionally, the server first reshapes the fused features in the spatial or channel dimension to form a time step sequence. Then, it uses forward TCN and backward TCN to encode the sequence bidirectionally. Forward TCN captures the dependencies from the past to the future, and backward TCN captures the dependencies from the future to the past. Finally, through a feature weighted fusion strategy, the final hidden states in both directions or the outputs of each time step are combined to obtain a temporal feature containing complete bidirectional contextual information.

[0063] In practical applications, the importance of forward and backward features can be balanced using a learnable parameter α.

[0064] In this embodiment, by employing bidirectional temporal modeling, the causal relationships of EEG signals in the temporal dimension can be captured simultaneously, i.e., how the current EEG pattern is influenced by previous states and how it influences subsequent states. This modeling of the complete temporal context solves the problem that unidirectional modeling can only obtain local historical information and may lose key future context, enabling the extracted temporal features to more comprehensively and accurately reflect the dynamic neural response process induced in the cerebral cortex by the motor imagery task, thus improving the accuracy of classification.

[0065] In an exemplary embodiment, the time-frequency feature extraction network includes a time-domain feature extraction unit and a frequency-domain feature extraction unit. Multi-channel EEG signals are input into the time-frequency feature extraction network of the motor imagery task recognition model to obtain time-domain features and frequency-domain features. This includes: extracting features from the multi-channel EEG signals to obtain time-domain representations and frequency-domain representations; aligning the time-domain representations and frequency-domain representations along the time dimension to obtain aligned time-domain representations and aligned frequency-domain representations; inputting the aligned time-domain representations into the time-domain feature extraction unit to obtain time-domain features; and inputting the aligned frequency-domain representations into the frequency-domain feature extraction unit to obtain frequency-domain features.

[0066] Among them, the time domain representation is the original or pre-processed signal form (such as the filtered signal) that directly reflects the change of signal amplitude over time. Unlike the time domain features mentioned above, the time domain representation is obtained after preliminary information extraction from multi-channel EEG signals and is information that has not been processed by neural network units.

[0067] Among them, frequency domain representation is a representation of the distribution of signal energy at different frequencies. Unlike the frequency domain features mentioned above, frequency domain representation is obtained after preliminary information extraction of multi-channel EEG signals and is information that has not been processed by neural network units.

[0068] In this context, alignment in the time dimension means ensuring that each point in time in the time domain representation corresponds to the same period of brain activity in the corresponding time window (or time slice) in the frequency domain representation.

[0069] Optionally, the server first processes the original multi-channel EEG signal in parallel, obtains a clean time-domain waveform through bandpass filtering and other methods, obtains a spectrum through time-frequency transformation (such as STFT), and then timestamps the two to ensure that the time-domain sample point at each moment corresponds strictly in time with the corresponding frequency-domain spectral line. The aligned data are then sent to the time-frequency feature extraction unit to further extract deeper time-domain and frequency-domain features.

[0070] In this embodiment, the extraction path is extracted through parallel time-domain and frequency-domain representations and is strictly aligned in the time dimension. This provides physically meaningful and time-synchronized dual-stream features for subsequent processing, ensuring a precise correspondence between dynamic changes in the time domain and energy distribution in the frequency domain. This lays the foundation for subsequent cross-domain feature fusion, ensuring the accuracy and consistency of feature extraction from the source. It solves the feature confusion problem caused by misalignment or insufficient representation of time-frequency information, which is an important prerequisite for achieving accurate classification.

[0071] To facilitate understanding by those skilled in the art, the following description is provided for the EnBiTE model.

[0072] The EnBiTE model includes a time-frequency feature extraction network (including time-domain processing branches and frequency-domain processing branches), a time-frequency feature fusion network (PTFA, Progressive Time-Frequency Attention), a time-series modeling network (bidirectional TCN), and a classification network.

[0073] The inference process of the EnBiTE model is as follows: Step 1: Preprocess the multi-channel EEG signals. The model inputs two tensors (X and ffqs). X represents the multi-channel EEG signals, and ffqs represents the frequency domain representation of the multi-channel EEG signals. Channel dimensions are added to X to achieve dimensionality change, and X and ffqs are aligned in the time dimension. Step 2: Extract time-frequency features to obtain time-domain features and frequency-domain features. Step 3: Feature fusion. An attention weight matrix is ​​generated based on the frequency-domain features. The attention weight matrix is ​​used to weight the time-domain features to obtain fused features. Step 4: Feature reshaping of the fused features. Based on the feature reshaping results, bidirectional temporal feature modeling is performed. The forward and reverse temporal features are fused to obtain temporal features. Step 5: Input the temporal features into the classification network to obtain the classification results.

[0074] The EnBiTE model creates convolutional layers for all S scales, capturing local differences in frequency bands and time steps through convolutions at different scales, thus improving feature representation capabilities. Furthermore, the EnBiTE model's time-frequency feature fusion network is a three-stage attention generator (convolution + batch normalization + sigmoid), which improves the quality of attention weights. In this application, `self.dropout = nn.Dropout(dropout_rate)` can be added and applied after the attention weight matrix is ​​generated, effectively preventing overfitting.

[0075] Compared to the original BiTE model, the EnBiTE model exhibits stronger generalization ability. It reduces overfitting to training data through Dropout regularization and achieves more stable multi-scale feature extraction. The EnBiTE model also boasts stronger feature representation capabilities, ensuring appropriate transformation of features across all scales through unified convolution processing and better capturing correlations between time-frequency features through an improved attention mechanism. Furthermore, the EnBiTE model demonstrates greater training stability, with batch normalization stabilizing the training process and clearer forward propagation logic facilitating debugging and maintenance. Finally, the EnBiTE model offers better feature selectivity, more accurately identifying important time-frequency regions through an enhanced attention module and ensuring consistent feature extraction through the uniformity of multi-scale processing. Overall, the EnBiTE model demonstrates stronger expressive power and better generalization performance when processing time-frequency features of EEG signals, while maintaining computational efficiency.

[0076] This application uses the BCICIV_2b database as an example for testing. The EnBiTE model achieves a higher average classification accuracy than the BiTE model, indicating that the EnBiTE model can improve classification performance. BCICIV_2b (BCIcompetition IV 2b) is an electroencephalogram (EEG) dataset based on visually evoked motor imagery (MI), specifically designed for brain-computer interface (BCI) research. This dataset contains EEG signals from nine subjects with normal or corrected vision in both hands, collecting neural activity data during left- and right-handed motor imagery tasks.

[0077] In another embodiment, such as Figure 3 As shown, a method for recognizing electroencephalogram (EEG) signals is provided. Taking the application of this method to a server as an example, the method includes the following steps: Step S302: Acquire multi-channel EEG signals from the user while completing the motor imagery task.

[0078] Step S304: Input the multi-channel EEG signal into the time-frequency feature extraction network of the motor imagery task recognition model to obtain time-domain features and frequency-domain features.

[0079] Step S306: Input the temporal features into the temporal convolution unit to obtain temporal convolution features, and input the frequency domain features into the frequency convolution unit to obtain frequency convolution features.

[0080] Step S308: Input the frequency domain convolutional features into the attention weight generation unit to generate the attention weight matrix.

[0081] Step S310: The temporal convolutional features are weighted using an attention weight matrix to obtain fused features.

[0082] Step S312: Input the fused features into the temporal modeling network of the motion imagery task recognition model to obtain temporal features.

[0083] Step S314: Input the temporal features into the classification network of the motion imagery task recognition model to obtain the task type of the motion imagery task.

[0084] It should be noted that the specific limitations of the above steps can be found in the specific limitations of a brainwave signal recognition method described above.

[0085] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0086] The following describes the EEG signal recognition device provided in the embodiments of this application. The EEG signal recognition device has the same inventive concept as the EEG signal recognition method described above. The solution to the problem provided by the device is similar to the solution described in the above method. Therefore, the specific limitations of one or more EEG signal recognition device embodiments provided below can be referred to the limitations of the EEG signal recognition method above. The EEG signal recognition device described below and the EEG signal recognition method described above can be referred to each other, and will not be repeated here.

[0087] In one exemplary embodiment, Figure 4 This is a schematic diagram of the structure of an electroencephalogram (EEG) signal recognition device provided in an embodiment of this application, as shown below. Figure 4 As shown, the EEG signal recognition device includes: a signal acquisition module 402, a feature extraction module 404, a feature fusion module 406, a temporal modeling module 408, and a task classification module 410, wherein: The signal acquisition module 402 is used to acquire multi-channel EEG signals of the user when completing a motor imagery task; Feature extraction module 404 is used to input multi-channel EEG signals into the time-frequency feature extraction network of the motor imagery task recognition model to obtain time-domain features and frequency-domain features; The feature fusion module 406 is used to input time-domain features and frequency-domain features into the time-frequency feature fusion network of the motion imagery task recognition model to obtain fused features; The temporal modeling module 408 is used to input the fused features into the temporal modeling network of the motion imagery task recognition model to obtain temporal features; The task classification module 410 is used to input temporal features into the classification network of the motion imagination task recognition model to obtain the task type of the motion imagination task.

[0088] In an exemplary embodiment, the time-frequency feature fusion network includes a time-domain convolutional unit, a frequency-domain convolutional unit, and an attention weight generation unit; the feature fusion module 406 is specifically used to input time-domain features into the time-domain convolutional unit to obtain time-domain convolutional features, and input frequency-domain features into the frequency-domain convolutional unit to obtain frequency-domain convolutional features; input the frequency-domain convolutional features into the attention weight generation unit to generate an attention weight matrix; and use the attention weight matrix to weight the time-domain convolutional features to obtain fused features.

[0089] In an exemplary embodiment, the temporal convolutional unit includes multiple temporal convolutional layers at multiple scales; the frequency convolutional unit includes multiple frequency convolutional layers at multiple scales; the feature fusion module 406 is specifically used to input temporal features into the multiple-scale temporal convolutional layers to obtain multiple-scale temporal convolutional layer output features; concatenate the multiple-scale temporal convolutional layer output features to obtain temporal convolutional features; input frequency domain features into the multiple-scale frequency convolutional layers to obtain multiple-scale frequency convolutional layer output features; and concatenate the multiple-scale frequency convolutional layer output features to obtain frequency convolutional features.

[0090] In an exemplary embodiment, the attention weight generation unit includes a convolutional layer, a batch normalization layer, and an activation layer; the feature fusion module 406 is specifically used to input frequency domain convolutional features into the convolutional layer to obtain an initial weight matrix; input the initial weight matrix into the normalization layer to obtain a normalized weight matrix; and input the normalized weight matrix into the activation layer to obtain an attention weight matrix.

[0091] In an exemplary embodiment, the temporal modeling network includes a feature reshaping unit, a forward temporal modeling unit, a backward temporal modeling unit, and a bidirectional feature fusion unit. The temporal modeling module 408 is specifically used to input the fused features into the feature reshaping unit to obtain the feature reshaping result; input the feature reshaping result into the forward temporal modeling unit to obtain the forward temporal feature; and input the feature reshaping result into the backward temporal modeling unit to obtain the reverse temporal feature; and input the forward temporal feature and the reverse temporal feature into the bidirectional feature fusion unit to obtain the temporal feature.

[0092] In an exemplary embodiment, the time-frequency feature extraction network includes a time-domain feature extraction unit and a frequency-domain feature extraction unit. The feature extraction module 404 is specifically used to extract features from multi-channel EEG signals to obtain time-domain representations and frequency-domain representations; align the time-domain representations and frequency-domain representations in the time dimension to obtain aligned time-domain representations and aligned frequency-domain representations; input the aligned time-domain representations to the time-domain feature extraction unit to obtain time-domain features; and input the aligned frequency-domain representations to the frequency-domain feature extraction unit to obtain frequency-domain features.

[0093] In one exemplary embodiment, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the EEG signal recognition methods described above.

[0094] In one exemplary embodiment, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the EEG signal recognition methods described above.

[0095] In one exemplary embodiment, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the EEG signal recognition methods described in the above embodiments.

[0096] Indicatively, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the internal structure of a computer device 500 provided in an embodiment of this application. The computer device 500 can be provided as a server. (Refer to...) Figure 5 The computer device 500 includes a processing component 502, which further includes one or more processors, and memory resources represented by memory 501 for storing instructions, such as application programs, that can be executed by the processing component 502. The application programs stored in memory 501 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 502 is configured to execute instructions to perform the EEG signal recognition method of any of the above embodiments.

[0097] The computer device 500 may also include a power supply component 503 configured to perform power management of the computer device 500, a wired or wireless network interface 504 configured to connect the computer device 500 to a network, and an input / output (I / O) interface 505. The computer device 500 may operate on an operating system stored in memory 501, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0098] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0099] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0100] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0101] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0102] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for recognizing electroencephalogram (EEG) signals, characterized in that, The method includes: Acquire multi-channel EEG signals from users while they are completing a motor imagery task; The multi-channel EEG signals are input into the time-frequency feature extraction network of the motor imagery task recognition model to obtain time-domain features and frequency-domain features; The time-domain features and the frequency-domain features are input into the time-frequency feature fusion network of the motion imagery task recognition model to obtain fused features; The fused features are input into the temporal modeling network of the motion imagery task recognition model to obtain temporal features; The temporal features are input into the classification network of the motion imagery task recognition model to obtain the task type of the motion imagery task.

2. The method according to claim 1, characterized in that, The time-frequency feature fusion network includes a time-domain convolutional unit, a frequency-domain convolutional unit, and an attention weight generation unit; the process of inputting the time-domain features and the frequency-domain features into the time-frequency feature fusion network of the motion imagery task recognition model to obtain fused features includes: The temporal features are input into the temporal convolution unit to obtain temporal convolutional features, and the frequency domain features are input into the frequency domain convolution unit to obtain frequency domain convolutional features; The frequency domain convolutional features are input into the attention weight generation unit to generate an attention weight matrix; The attention weight matrix is ​​used to weight the temporal convolutional features to obtain the fused features.

3. The method according to claim 2, characterized in that, The temporal convolutional unit includes multiple temporal convolutional layers of various scales; the frequency convolutional unit includes multiple frequency convolutional layers of various scales; the step of inputting the temporal features into the temporal convolutional unit to obtain temporal convolutional features includes: The temporal features are input into the temporal convolutional layers of the multiple scales to obtain the output features of the temporal convolutional layers of the multiple scales. The output features of the multiple temporal convolutional layers are concatenated to obtain the temporal convolutional features; The step of inputting the frequency domain features into the frequency domain convolution unit to obtain frequency domain convolution features includes: The frequency domain features are input into the frequency domain convolutional layers of the multiple scales to obtain the output features of the frequency domain convolutional layers of the multiple scales. The frequency domain convolutional features are obtained by concatenating the output features of the multiple frequency domain convolutional layers.

4. The method according to claim 2, characterized in that, The attention weight generation unit includes a convolutional layer, a batch normalization layer, and an activation layer; the step of inputting the frequency domain convolutional features into the attention weight generation unit to generate an attention weight matrix includes: The frequency domain convolutional features are input into the convolutional layer to obtain the initial weight matrix; The initial weight matrix is ​​input into the normalization layer to obtain the normalized weight matrix; The normalized weight matrix is ​​input into the activation layer to obtain the attention weight matrix.

5. The method according to claim 1, characterized in that, The temporal modeling network includes a feature reshaping unit, a forward temporal modeling unit, a backward temporal modeling unit, and a bidirectional feature fusion unit; the process of inputting the fused features into the temporal modeling network of the motion imagery task recognition model to obtain temporal features includes: The fused features are input into the feature reshaping unit to obtain the feature reshaping result; The feature reshaping result is input into the forward temporal modeling unit to obtain forward temporal features, and the feature reshaping result is input into the backward temporal modeling unit to obtain reverse temporal features; The forward and reverse time series features are input into the bidirectional feature fusion unit to obtain the time series features.

6. The method according to claim 1, characterized in that, The time-frequency feature extraction network includes a time-domain feature extraction unit and a frequency-domain feature extraction unit. The process of inputting the multi-channel EEG signal into the time-frequency feature extraction network of the motor imagery task recognition model to obtain time-domain and frequency-domain features includes: Feature extraction is performed on the multi-channel EEG signals to obtain time-domain and frequency-domain representations; Align the time-domain representation and the frequency-domain representation along the time dimension to obtain the aligned time-domain representation and the aligned frequency-domain representation; The aligned time-domain representation is input to the time-domain feature extraction unit to obtain the time-domain features, and the aligned frequency-domain representation is input to the frequency-domain feature extraction unit to obtain the frequency-domain features.

7. A brainwave signal recognition device, characterized in that, The device includes: The signal acquisition module is used to acquire multi-channel EEG signals from the user while performing a motor imagery task; The feature extraction module is used to input the multi-channel EEG signals into the time-frequency feature extraction network of the motor imagery task recognition model to obtain time-domain features and frequency-domain features; The feature fusion module is used to input the time-domain features and the frequency-domain features into the time-frequency feature fusion network of the motion imagery task recognition model to obtain fused features; The temporal modeling module is used to input the fused features into the temporal modeling network of the motion imagery task recognition model to obtain temporal features; The task classification module is used to input the temporal features into the classification network of the motion imagery task recognition model to obtain the task type of the motion imagery task.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images