User intention classification method based on multi-task learning time-frequency double-branch network

Abstract

The invention provides a user intention classification method based on a multi-task learning time-frequency double-branch network, and the method comprises the steps: data preprocessing: employing LaBrM to extract motor imagery and N-back task features from an original EEG signal, and generating a final EEG feature embedding vector; based on an adaptive spectrum feature fusion attention module and a multi-scale expansion factor convolution time feature extraction module, features of EEG feature embedding vectors are extracted respectively; fusing the extracted time features and spectrum features; and classifying the fused features through a multi-task classifier. According to the method, the distribution characteristics of the electroencephalogram signals in the time domain and the frequency domain are fully utilized, and accurate recognition of intention information in multiple cognitive tasks of the user is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of biological signal processing technology, and in particular to a user intention classification method based on a multi-task learning time-frequency dual-branch network. Background Art

[0002] Brain-Computer Interface (BCI) is a technology that enables direct interaction between the human brain and external devices based on neural signals such as EEG. It is widely used in neurorehabilitation, intelligent control, virtual reality, fatigue detection, emotion recognition and other fields. Figure 1 As shown in Figure 1, a BCI system consists of five stages: brain activity measurement, preprocessing, feature extraction, classification, and conversion into commands. In BCI systems, EEG is one of the most commonly used signal acquisition methods due to its non-invasiveness, low cost, portability, and high temporal resolution. However, the inherent complexity of EEG signals poses significant challenges for subsequent processing. EEG data is highly nonlinear, nonstationary, and susceptible to noise. Furthermore, EEG patterns vary significantly between subjects, making efficient and stable feature extraction and classification difficult during model training.

[0003] In the existing technology, traditional EEG decoding methods mainly rely on manually designed features, such as the commonly used time domain mean, frequency band energy, power spectral density, co-spatial pattern, etc. Although these methods are effective in specific tasks, they generally have the defects of overfitting to specific tasks and subjects and poor generalization ability, especially in multi-task recognition scenarios. With the development of deep learning, researchers have gradually introduced structures such as convolutional neural networks (CNN) and recurrent neural networks, trying to improve recognition accuracy through automatic feature extraction. However, most studies still focus on single-task scenarios, and only optimize for a certain type of task such as motor imagery or cognitive load, and lack a general architecture that can uniformly handle multiple types of user intentions.

[0004] In addition, in the process of user intent recognition, temporal information and spectral information usually contain key discriminative features at the same time. Existing methods often focus on time domain modeling and ignore the interaction between multiple frequency bands in the frequency domain, resulting in insufficient perception of task-related frequency features by the model. At the same time, the importance of spectral information varies in different tasks, and the lack of interpretability and adaptability of spectral modeling methods also limits the versatility and generalizability of the model. In terms of time modeling, although some studies have introduced techniques such as causal convolution and dilated convolution, a complete framework for collaborative modeling with frequency domain features has not yet been formed. In addition, the application of multi-task learning in EEG is still in its early stages, and existing methods generally lack the effective separation and integration of shared features and task-specific features, which affects the classification ability and generalization performance of the model.

[0005] In summary, in order to overcome the problems of poor multi-task adaptability, weak time domain modeling and frequency domain feature modeling capabilities of existing EEG signal user intention recognition methods, insufficient fusion of time-frequency features, and single-task learning, it is very meaningful to propose a user intention classification method based on multi-task learning time-frequency dual-branch network. Summary of the Invention

[0006] In view of this, the present invention discloses a user intention classification method based on a multi-task learning time-frequency dual-branch network, which identifies the user intention of EEG signals by constructing an EEG signal multi-task user intention recognition network and system with a time-frequency dual-branch structure based on EEG feature embedding.

[0007] The technical solution provided by the present invention is a user intent classification method based on a multi-task learning time-frequency dual-branch network, comprising:

[0008] S1: Data preprocessing: LaBrM is used to extract motor imagery and N-back task features from the original EEG signal. The dual-branch module performs refinement on each task to generate the final EEG feature embedding vector.

[0009] S2: A temporal feature extraction module based on an adaptive spectral feature fusion attention module and a multi-scale dilation factor convolution to extract the features of the EEG feature embedding vector respectively;

[0010] S3: Fusing the extracted temporal features and spectral features;

[0011] S4: Classify the fused features through a multi-task classifier.

[0012] Preferably, in S2, the adaptive spectrum feature fusion attention module includes: multi-scale spectrum feature extraction and adaptive spectrum attention mechanism;

[0013] The multi-scale spectral feature extraction is to use multiple convolution kernels with different receptive fields to capture spectral features of different scales;

[0014] The adaptive spectral attention mechanism calculates the correlation between multi-scale features and assigns different weights.

[0015] The attention matrix is ​​calculated by scaling the dot product attention, and its expression is as follows:

[0016]

[0017] Among them, F multi-scale Represents the multi-frequency information learned through multi-scale convolution, d k is the feature dimension.

[0018] Preferably, the temporal feature extraction module of the multi-scale dilation factor convolution adopts causal convolution to ensure the calculation of each time step, and its mathematical expression is as follows:

[0019]

[0020] Among them, y t represents the output at time step t, x t-i represents the input of the past i time steps, w i is the convolution kernel weight, k is the convolution kernel size;

[0021] The convolution based on the expansion factor is introduced. By applying interval sampling on the input data, the receptive field grows exponentially with the increase of the number of network layers. Its mathematical expression is as follows:

[0022]

[0023] Where d is the dilation rate.

[0024] The present invention also provides a user intention classification system based on a multi-task learning time-frequency dual-branch network, comprising a preprocessing unit, a feature extraction unit and a classification unit;

[0025] The preprocessing unit uses LaBrM to extract the shared representation of motor imagery and N-back task features from the original EEG signal, and uses a dual-branch module to refine the feature embedding vector and learn time-frequency features;

[0026] The feature extraction unit captures spectral features through an adaptive spectral feature fusion attention module, models temporal information using a causal convolutional network, and further refines feature information with the help of an attention mechanism;

[0027] The classification unit classifies the fused temporal features and spectral features by gradually reducing the dimensions through a multi-layer perceptron.

[0028] This paper builds a unified neural network framework, fully utilizing the distribution characteristics of EEG signals in the time and frequency domains to accurately identify user intent in various cognitive tasks (such as motor imagery and working memory tasks). By introducing a pre-trained embedding module based on LaBraM, the raw EEG signals are deeply encoded to extract global spatiotemporal shared features. Furthermore, a spectral convolution-self-attention mechanism and a temporal causal convolutional network are combined to construct feature extraction branches in the frequency and time domains, respectively, to achieve multi-scale and multi-dimensional task-related information modeling, thereby improving the model's classification accuracy and generalization ability in complex tasks.

[0029] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0032] Figure 1 This is a schematic diagram of the processing flow of the brain-computer interface system disclosed in the present invention;

[0033] Figure 2 A schematic diagram of a time-frequency dual-branch network architecture based on multi-task learning provided by an embodiment of the present invention;

[0034] Figure 3 A schematic diagram of a multi-task learning architecture for EEG feature embedding based on LaBraM provided in an embodiment of the present invention;

[0035] Figure 4 A schematic diagram of the architecture of a spectrum feature extraction module based on adaptive fusion attention provided in an embodiment of the present invention;

[0036] Figure 5 A schematic diagram of the architecture of a temporal feature extraction module based on multi-scale dilation factor convolution provided by an embodiment of the present invention;

[0037] Figure 6 Schematic diagram of the before and after results of the ablation experiment model of each module of TFDB-EENet in the motor imagery task provided by the embodiment of the present invention;

[0038] Figure 7 Schematic diagram of the before-and-after comparison results of the ablation experimental model of each module of TFDB-EENet in the N-back task provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0039] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of systems consistent with certain aspects of the present invention, as detailed in the appended claims.

[0040] In order to overcome the problems of existing EEG signal user intention recognition methods in terms of poor multi-task adaptability, weak time domain modeling and frequency domain feature modeling capabilities, insufficient time-frequency feature fusion, and single-task learning, the present invention provides a user intention classification method based on a multi-task learning time-frequency dual-branch network;

[0041] The terms used in the present invention have the following meanings:

[0042] Epoch: The number of iterations of the neural network

[0043] Batch size: The number of samples fed into the model for forward propagation and backpropagation at one time when training a neural network.

[0044] Learning rate: A hyperparameter that controls the size of the step size used by the neural network during each parameter update.

[0045] EEGMMIDB: This dataset is based on electroencephalogram (EEG) data collected from subjects while they performed or imagined specific motor tasks. Using a high-density EEG system, the dataset records EEG activity from multiple subjects under different experimental conditions, covering a variety of motor and imaginary tasks. During data collection, subjects were asked to perform or imagine movements of their hands, feet, and tongue, capturing movement-related EEG signals. The data is preprocessed and labeled to ensure quality and usability.

[0046] BCI: A technology that establishes a direct communication channel between the brain and external devices, which can convert neural signals such as EEG into instructions that can be used for control. It is widely used in medical rehabilitation, human-computer interaction and other fields.

[0047] N-back: This dataset includes 26 healthy participants who used the N-BACK task to examine brain activation across links with different load levels (0-back, 2-back, and 3-back).

[0048] EEG: It is a medical technique used to measure the electrical activity of the brain. It is assessed by placing several electrodes (usually adhesive tapes on the scalp) and recording the electrical signals generated by the brain.

[0049] BCI2000: A modular brain-computer interface software platform for acquiring and processing EEG signals and implementing a variety of real-time BCI applications.

[0050] CNN: A deep learning model that excels at processing grid-structured data such as images, automatically extracting features and performing efficient classification and recognition.

[0051] Multi-Wavelet CNN: A network that uses multiple parallel CNNs for multi-band feature extraction.

[0052] Concatenate: An operation that concatenates feature vectors.

[0053] Multiply & Normal: A mechanism that combines multiplication operations with normalization strategies in neural networks to enhance feature expression capabilities and stabilize the training process.

[0054] Softmax: An activation function that converts the input vector into a probability distribution, often used in multi-classification problems, so that the sum of the output values ​​is 1.

[0055] Transformer: A deep learning model architecture based on the self-attention mechanism, widely used in fields such as natural language processing.

[0056] LaBraM: The full name is Large Brain Model, a large model pre-training, based on the Transformer EEG pre-training model.

[0057] Dilation rate: A parameter in the convolution operation that controls the spacing between convolution kernels, thereby expanding the receptive field and helping to capture a wider range of feature information.

[0058] Adaptive Local Temporary Attention: is a neural network module that uses a linear layer to learn attention scores.

[0059] MLP: It is a neural network model composed of multiple fully connected layers, which transmits information through nonlinear activation functions and is widely used in classification and regression tasks.

[0060] Python 3.8: A high-level programming language used for writing computer programs. It is developed as an open source project led by Guido van Rossum and has broad community support.

[0061] NVIDIA GeForce GTX 4090 GPU: A high-end graphics processing unit (GPU) designed for demanding games and computer graphics applications.

[0062] Pytorch: An open-source software library for data flow and differentiable programming, covering a wide range of tasks. It is primarily used for machine learning and deep learning, allowing researchers and developers to easily build and deploy models.

[0063] This paper is based on the overall framework of multi-task learning time-frequency dual-branch network TFDB-EENet, and connects the time-frequency dual-branch feature learning network downstream to learn the characteristics of user intention (motor imagery task and N-back task) in both time domain and frequency domain. Figure 2 The TFDB-EENet shown in the figure consists of three key components: a preprocessing unit, a feature extraction unit, and a classification unit. First, the preprocessing module uses LaBrM to extract shared representations of the two task features from the raw EEG signal, effectively learning the common feature space between the motor imagery and N-back tasks. Next, a downstream dual-branch module refines these feature embeddings and learns time and frequency features to meet the specific learning requirements of the tasks. The feature extraction unit captures spectral features through an adaptive spectral feature fusion attention module, models temporal information using a causal convolutional network, and further refines the feature information using an attention mechanism.

[0064] Specifically, the user intent classification method based on the multi-task learning time-frequency dual-branch network includes:

[0065] S1: Data preprocessing: LaBrM is used to extract motor imagery and N-back task features from the original EEG signal. The dual-branch module performs refinement on each task to generate the final EEG feature embedding vector.

[0066] LaBraM is used for feature embedding of EEG signals. LaBraM is a representative method that applies the Transformer self-attention architecture combined with large-scale pre-training to the EEG signal field. It first pre-trains on a relatively large and extensive set of EEG data to learn a feature vector representation of the EEG signal with universal spatiotemporal characteristics. It then fine-tunes the downstream task on the target task data, allowing it to inherit knowledge from the pre-training phase while also adapting to the specific characteristics of the target task.

[0067] The present invention utilizes the LaBraM model in feature extraction of EEG signals, and with its powerful spatiotemporal information modeling capability, extracts a universal EEG feature embedding vector.

[0068] like Figure 2 As shown in the pre-training in , the original EEG features of the motor imagery and N-back tasks are embedded in the feature construction, and then each task is sent to the dual-branch network to extract features, and finally sent to the multi-task classifier for classification.

[0069] Specifically include: Figure 3 As shown; S2: Temporal feature extraction module based on adaptive spectral feature fusion attention module and multi-scale dilation factor convolution, which extracts the features of EEG feature embedding vector respectively;

[0070] The above-mentioned adaptive spectral feature fusion attention module: In the time-frequency dual-branch network, the extraction and fusion of spectral features are key to improving the model's discriminative ability. Traditional methods mostly rely on fixed-scale spectral analysis (such as wavelet transform), which makes it difficult to adaptively capture multi-scale frequency domain information. To this end, the present invention designs an adaptive spectral feature fusion attention module (ASFA), which dynamically learns task-related spectral feature representations through the combination of multi-scale convolution and self-attention mechanism.

[0071] The adaptive spectral feature fusion attention module mainly consists of two core parts, namely multi-scale spectral feature extraction and adaptive spectral attention mechanism.

[0072] like Figure 4 As shown in the figure, first, during the multi-scale spectral feature extraction process, the adaptive spectral feature fusion attention module uses multiple convolution kernels with different receptive fields to capture spectral features at different scales. Smaller convolution kernels can capture local high-frequency information, while larger convolution kernels can extract low-frequency features, allowing the model to simultaneously focus on features in different frequency ranges. These convolution operations can extract spectral information of different ranges from local to global, making the feature representation richer. This multi-scale convolution strategy ensures that more fine-grained frequency features are extracted in the spectral dimension, providing a stronger feature foundation for the subsequent learning of the attention mechanism.

[0073] Secondly, in the adaptive spectral attention mechanism, by calculating the correlation between multi-scale features, the adaptive spectral feature fusion attention module assigns different weights to further enhance the expressiveness of important features. The attention matrix is ​​calculated by scaled dot product attention, and its expression is as follows:

[0074]

[0075] Among them, F multi-scale Represents the multi-frequency information learned through multi-scale convolution, d k is the feature dimension.

[0076] like Figure 5 As shown, a temporal feature extraction module based on multi-scale dilation factor convolution: In EEG signal analysis tasks, modeling of temporal information is crucial for accurately identifying user intentions. Since EEG signals are highly non-stationary time series data, they have complex dynamic characteristics at different time scales. Therefore, simple traditional convolution or recurrent neural networks often cannot fully capture these temporal dependencies. To solve this problem, the present invention adopts a temporal feature extraction module based on causal convolution with dilation factors to maintain the integrity of temporal information and enhance the modeling capability of long-term dependencies.

[0077] In traditional standard convolution operations, the output of the current time step may be affected by future time steps, which is unreasonable in time series modeling, especially in EEG signal processing, to ensure the causality of the data flow and the integrity of the time information. Therefore, the present invention uses causal convolution to ensure that the calculation of each time step depends only on the current and past inputs, thereby strictly following the natural order of the time series. Its mathematical expression is as follows:

[0078]

[0079] Among them, y t represents the output at time step t, x t-i represents the input of the past i time steps, w i is the convolution kernel weight, and k is the convolution kernel size. The advantage of causal convolution is that it can capture local temporal features through convolution while ensuring the temporal consistency of the information flow. However, when using causal convolution alone, the fixed size of the convolution kernel limits the network's receptive field and makes it difficult to capture long-term dependencies.

[0080] To enhance the model's ability to model long-term dependencies, the Temporal CNN based on Causal CNN (TC-CC) module further introduces dilation-factor-based convolution. Dilation-factor-based convolution applies interval sampling to the input data, allowing the receptive field to grow exponentially with the number of network layers without increasing computational complexity. Its mathematical expression is as follows:

[0081]

[0082] Here, d is the dilation rate. As the network depth increases, d takes on values ​​of {1, 2, 4}. This allows for efficient capture of long-term dependencies while minimizing computational overhead. The main advantage of this method is that, through exponentially growing receptive fields, deeper network layers can perceive features spanning longer timescales, effectively modeling EEG dynamics. This is crucial for EEG signal processing, as the temporal features of different tasks exhibit long-term dependencies.

[0083] S3: Fusing the extracted temporal features and spectral features;

[0084] S4: Classify the fused features through a multi-task classifier.

[0085] Table 1 provides a set of important hyperparameters of the model of the present invention, including basic hyperparameters: number of iterations, batch size, and learning rate.

[0086] Table 1: TFDB-EENet main hyperparameters

[0087] category Parameter size Epoch 200 Batchsize 16 Learning rate 0.0001

[0088] The motor imagery experimental results are shown in Table 2. The proposed model outperforms existing methods across all evaluation metrics. In terms of accuracy, the proposed model achieved 98.61%, outperforming existing mainstream methods, including the traditional Li et al. (98.09%), DSCNN+GRU (97.76%), and Zhang et al. (98.31%) approaches. Furthermore, the proposed model also demonstrated superior performance across other evaluation metrics, with both precision and recall reaching 98.61%, and the F1 score reaching 98.60%.

[0089] The experimental results of N-back are shown in Table 3. This method achieved an accuracy of 79.23%, outperforming several previous studies, including the models of Parveen et al. (75.90%), Khanam et al. (77.20%), Chakladar et al. (78.90%), and DBC (71.20%). Table 4 also details the performance of various evaluation metrics, including precision of 78.41%, recall of 76.12%, and F1 score of 76.95%.

[0090] Table 2: Performance comparison of different methods on the EEGMMIDB dataset

[0091] method years Accuracy Accuracy Recall F1 value TD-LSTM 2024 95.86 95.87 95.86 95.86 Lietal. 2024 98.09 97.76 97.98 97.86 DSCNN+GRU 2022 97.76 - - - Zhangetal 2018 98.31 - - 97.91 Zhangetal. 2019 96.32 96.32 96.32 96.32 Method of the present invention (TFDB-EENet) 98.61 98.61 98.61 98.60

[0092] The dual-branch network structure significantly improves model performance. On the one hand, the multi-scale convolution and adaptive multidimensional spatial attention mechanism in the spectral branch effectively capture the multidimensional spectral characteristics of EEG signals. On the other hand, the local adaptive temporal attention mechanism in the time domain branch significantly enhances the model's ability to learn the temporal characteristics of EEG signals. Furthermore, the proposed multi-task learning framework significantly improves the decoding capability and robustness of EEG signals by integrating shared and task-specific feature representations.

[0093] Table 3: Accuracy comparison of different methods on the N-back dataset

[0094] Method years Accuracy Parveenetal. 2024 75.9 Khanametal. 2023 77.2 Chakladaretal. 2024 78.9 Method of the present invention (TFDB-EENet) 79.23

[0095] Table 4: Performance indicators on the N-back dataset

[0096] method Accuracy Accuracy Recall F1 value Method of the present invention (TFDB-EENet) 79.23 78.41 76.12 76.95

[0097] To evaluate the impact of individual model components, we perform an ablation study by systematically eliminating key modules: LaBraM (EEG feature embedding), adaptive spectral feature fusion attention module, attention mechanism within the adaptive spectral feature fusion attention module, and causal convolution-based temporal feature extraction module. Figure 6 and Figure 7 As shown in Figure 3. The results show that removing LaBraM leads to the most significant performance drop, indicating the importance of effective EEG feature embedding. The lack of the adaptive spectral feature fusion attention module also leads to a significant drop, emphasizing the importance of frequency domain information. Furthermore, eliminating the attention mechanism within the adaptive spectral feature fusion attention module reduces performance, confirming that spectral attention facilitates better feature learning. Finally, removing the causal convolution-based temporal feature extraction module affects the ability to model temporal dependencies, thereby reducing overall accuracy.

[0098] The present invention is described in detail below in conjunction with specific embodiments. Taking the EEG signal classification task as an example, the implementation process of the LaBraM-based time-frequency dual-branch EEG embedding network (TFDB-EENet) is explained.

[0099] Hardware and software environment configuration

[0100] 1) Hardware Platform:

[0101] Computing equipment: NVIDIA RTX 4090 GPU (24GB video memory), Intel Core i9-13900KS (6.0GHz, 24 cores).

[0102] Data acquisition equipment: A 64-channel EEG acquisition system (BCI2000) was used for the MI task, and a 30-channel EEG acquisition system with a 10-5 system was used for the N-back task, with a sampling rate of 160 Hz (MI task) or 1000 Hz (N-back task).

[0103] 2) Software Framework:

[0104] Deep learning framework: PyTorch 1.7.1, CUDA 11.7.

[0105] Data processing tools: Python 3.8, NumPy 1.22.4, MNE-Python 1.6.1.

[0106] 1. Data preprocessing and LaBraM module implementation:

[0107] Step 1: EEG signal input and segmentation: input multi-channel EEG raw signal;

[0108] Step 2: Block parameters, divide the original signal into different patches;

[0109] Step 3: Perform spatiotemporal encoding. Each patch is further divided into different time segments to generate local spatiotemporal blocks. Spatial encoding is based on the electrode coordinates of the international 10-20 system to generate position vectors. Temporal encoding uses a sine function to generate segment-level temporal position vectors.

[0110] Step 4: Use the Transformer encoder to generate the final EEG feature embedding vector.

[0111] 2. Feature extraction of EEG feature embedding vector:

[0112] Step 1: Input LaBraM’s EEG feature embedding vector;

[0113] Step 2: Use multi-scale wavelet convolution to extract multi-frequency. The convolution kernel sizes are: 1×3, 1×5, 1×7, 1×9, the step size is 1, and the number of output channels of each convolution group is 32.

[0114] Step 3: Multi-scale feature splicing,

[0115] Step 4: Calculate cross-channel spectral self-attention, where the query and key are multi-scale concatenated vectors and the value is a small kernel convolution map. Then calculate the attention weight.

[0116] Step 5: Generate spectrum feature vector after Flatten

[0117] Step 6: Next is the time domain branch, which also takes as input the LaBraM EEG feature embedding vector;

[0118] Step 7: Then use causal convolution to learn temporal information, with 3 layers, dilation sequence d = {1, 2, 4}, convolution kernel size 3, and stride 1.

[0119] Step 8: Splicing the outputs of each layer Then the temporal features and spectral features are fused;

[0120] Step 9: Finally, use a multi-layer perceptron to gradually reduce the dimension for classification.

[0121] In order to verify the performance of the present invention on multiple tasks, experiments were conducted on the EEGMMIDB dataset and the N-back dataset. The experimental results are shown in Table 5 below.

[0122] Table 5: Performance comparison with other methods on EEGMMIDB and N-back

[0123] method EEGMMIDB accuracy (%) N-back accuracy (%) Brain Model 96.02 76.79 Temporal Causal Convolution 97.41 77.86 Multi-scale spectral network 98.13 78.01 Adaptive spectral fusion attention 98.41 78.16 TFDB-EENet (this invention) 98.61 79.23

[0124] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These changes and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A user intention classification method based on a multi-task learning time-frequency dual-branch network, characterized by: include: S1: Data preprocessing: LaBrM is used to extract motor imagery and N-back task features from the original EEG signal. The dual-branch module performs refinement on each task to generate the final EEG feature embedding vector. S2: A temporal feature extraction module based on an adaptive spectral feature fusion attention module and a multi-scale dilation factor convolution to extract the features of the EEG feature embedding vector respectively; S3: Fusing the extracted temporal features and spectral features; S4: Classify the fused features through a multi-task classifier.

2. The user intention classification method based on multi-task learning time-frequency dual-branch network according to claim 1 is characterized in that In S2, the adaptive spectrum feature fusion attention module includes: multi-scale spectrum feature extraction and adaptive spectrum attention mechanism; The multi-scale spectral feature extraction is to use multiple convolution kernels with different receptive fields to capture spectral features of different scales; The adaptive spectral attention mechanism calculates the correlation between multi-scale features and assigns different weights. The attention matrix is ​​calculated by scaled dot product attention, and its expression is as follows: Among them, F multi-scale Represents the multi-frequency information learned through multi-scale convolution, d k is the feature dimension.

3. The user intention classification method based on multi-task learning time-frequency dual-branch network according to claim 1 is characterized in that The temporal feature extraction module of the multi-scale dilation factor convolution adopts causal convolution to ensure the calculation of each time step, and its mathematical expression is as follows: Among them, y t represents the output at time step t, x t-i represents the input of the past i time steps, w i is the convolution kernel weight, k is the convolution kernel size; The convolution based on the expansion factor is introduced. By applying interval sampling on the input data, the receptive field grows exponentially with the increase of the number of network layers. Its mathematical expression is as follows: Where d is the dilation rate.

4. A user intention classification system based on a multi-task learning time-frequency dual-branch network, characterized by: It includes a pre-processing unit, a feature extraction unit and a classification unit; The preprocessing unit uses LaBrM to extract the shared representation of motor imagery and N-back task features from the original EEG signal, and uses a dual-branch module to refine the feature embedding vector and learn time-frequency features; The feature extraction unit captures spectral features through an adaptive spectral feature fusion attention module, models temporal information using a causal convolutional network, and further refines feature information with the help of an attention mechanism; The classification unit classifies the fused temporal features and spectral features by gradually reducing the dimensions through a multi-layer perceptron.

Images