Data enhancement-based electroencephalogram signal decoding method and device, medium and equipment

Abstract

The invention relates to an electroencephalogram signal decoding method and device based on data enhancement, a medium and equipment. The method comprises the following steps: pre-training an RM-GAN generation network by adopting preprocessed electroencephalogram characteristics of a source domain and a target domain to obtain a generator; adopting the preprocessed motor imagery electroencephalogram data of the source domain to carry out pre-training on a BAdaptNet network to obtain a BAdaptNet model; the method comprises the following steps: generating motor imagery electroencephalogram features of a target subject by using feature data obtained by fusing resting electroencephalogram data of the current target subject and source domain motor imagery electroencephalogram data, and carrying out transfer learning and retraining on a BAdaptNet model to obtain a personalized electroencephalogram signal decoding model matched with the target subject; and the personalized electroencephalogram signal decoding model is adopted to decode the motion state electroencephalogram data, collected in real time, of the target subject to obtain the current motion intention of the target subject. By applying the method, the personalized electroencephalogram signal decoding model of the target subject can be quickly constructed, and the electroencephalogram data acquisition time of the subject is shortened.

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface technology, and in particular to a method, device, medium and equipment for decoding electroencephalogram (EEG) signals based on data enhancement. Background Art

[0002] With the rapid development of brain-computer interface (BCI) technology, motor imagery (MI)-based BCIs have become a core research area in rehabilitation medicine and neuroengineering. By decoding motor intent from EEG signals, motor imagery enables real-time control of external devices such as exoskeletons and prosthetics, providing a novel approach for restoring motor function in patients. Current BCI systems face numerous challenges in practical application, particularly the time-consuming and inefficient calibration process due to individual variability, which significantly limits the clinical application of this technology.

[0003] Traditional BCI systems typically require the collection of large amounts of motor imagery EEG data from each patient to perform personalized training for specific classifiers. This process not only requires patients to maintain sustained focus but also easily induces fatigue, reducing the effectiveness of rehabilitation training. Furthermore, EEG signals inherently have low signal-to-noise ratios and significant inter-individual variability, further complicating the development of BCI systems.

[0004] Therefore, how to effectively utilize existing data resources when data is limited and quickly adapt to the individual needs of target patients has become a technical problem that needs to be urgently solved in the current field of brain-computer interface technology. Summary of the Invention

[0005] The purpose of the present invention is to provide a method, device, medium and equipment for decoding EEG signals based on data enhancement, which are used to realize real-time decoding and control of EEG signals for motor imagery.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] According to one aspect of the present invention, a method for decoding an EEG signal based on data enhancement is provided, comprising the following steps:

[0008] Preprocessing the resting data of the source domain, the motor imagery data of the source domain, and the resting data of the target domain to obtain preprocessed EEG features;

[0009] The pre-processed EEG features are used to pre-train the RM-GAN network model including the generator and the discriminator to obtain the pre-trained generator;

[0010] The BAdaptNet network model is trained using the pre-processed source domain motor imagery features to obtain a pre-trained BAdaptNet network model.

[0011] The source domain resting data, source domain motor imagery data and target subject resting data are pre-processed and fused, and the pre-trained generator is used to generate the target subject motor imagery features.

[0012] The generated target subject's motor imagery features are used to retrain the pre-trained BAdaptNet network model through transfer learning. Through transfer learning fine-tuning, a personalized EEG signal decoding model adapted to the target subject is obtained.

[0013] A personalized EEG signal decoding model is used to decode the target subject's current motor imagery EEG data collected in real time to obtain the target subject's current movement intention.

[0014] According to one embodiment of the present invention, the preprocessing of the source domain resting data, the source domain motor imagery data, and the target domain resting data includes:

[0015] Channel screening is performed on the EEG signals including the resting data of the source domain, the motor imagery data of the source domain, and the resting data of the target domain to obtain channels related to the motor sensory cortex;

[0016] Band-pass filtering was performed to extract rhythmic activity related to the motor imagery task;

[0017] Perform multiple average downsampling to reduce the amount of data;

[0018] Frequency domain features are extracted through wavelet functions to capture key time-frequency information in EEG signals.

[0019] According to one embodiment of the present invention, in the step of using the preprocessed EEG features to pre-train the RM-GAN network model including the generator and the discriminator, the generator generates a target domain motor imagination feature that is highly similar to the motor imagination feature of the target domain based on the preprocessed EEG features, which is used for the discriminator to perform authenticity judgment and adversarial training of the generator; the discriminator performs authenticity judgment on the target domain motor imagination feature generated by the generator.

[0020] According to one embodiment of the present invention, the RM-GAN generator adopts an encoder-decoder structure, wherein the encoder includes multiple two-dimensional convolutional layers as time-frequency filters, a regularization layer is added after each convolutional layer, and LeakyReLU is used as the activation function; the decoder includes multiple deconvolutional layers, a regularization layer and a ReLU activation function are added after the deconvolution layer before the last deconvolution layer, and a Tanh activation function is added after the last deconvolution layer.

[0021] According to one embodiment of the present invention, the feature data obtained by preprocessing and data fusion of the source domain resting data, the source domain motor imagery data and the target subject resting data is obtained by respectively performing channel screening, bandpass filtering, downsampling, extracting frequency domain features and performing feature fusion on the source domain resting data, the source domain motor imagery data and the target subject resting data.

[0022] According to one embodiment of the present invention, the BAdaptNet network model includes a deep attention feature extraction layer, a dynamic convolution layer, a multiple convolution feature fusion layer and an output layer; wherein, the deep attention feature extraction layer uses the SE attention mechanism to screen channels, so that the transfer learning network can perform dynamic channel feature recalibration to improve the network's representation ability; the dynamic convolution layer uses dynamic convolution to adapt to the non-stationary characteristics of EEG signals; the multiple convolution feature fusion layer improves the model's information capture capability by fusing features through multiple convolution layers.

[0023] According to one embodiment of the present invention, the method of using the generated target subject's motor imagery features to perform transfer learning and retraining on the pre-trained BAdaptNet network model, and obtaining a personalized EEG signal decoding model adapted to the target subject through transfer learning fine-tuning, includes:

[0024] Freeze the weight parameters of the deep attention feature extraction layer and dynamic convolution layer in the pre-trained BAdaptNet network model, and only adjust the weight parameters of the multiple convolution feature fusion layer and output layer in the pre-trained BAdaptNet network model.

[0025] According to one embodiment of the present invention,

[0026] On the other hand, the present invention also provides an EEG signal decoding device based on data enhancement, comprising:

[0027] a preprocessing unit configured to preprocess the resting data of the source domain, the motor imagery data of the source domain, and the resting data of the target domain to obtain preprocessed EEG features;

[0028] A generator pre-training unit is configured to pre-train an RM-GAN network model including a generator and a discriminator using the pre-processed EEG features to obtain a pre-trained generator;

[0029] The transfer learning model pre-training unit is configured to train the BAdaptNet network model using the pre-processed motor imagery features of the source domain to obtain a pre-trained BAdaptNet network model;

[0030] a data generation unit configured to generate target subject motor imagery features using a pre-trained generator using feature data of the source domain resting data, the source domain motor imagery data, and the target subject resting data after preprocessing and data fusion;

[0031] The transfer learning model retraining unit is configured to use the generated target subject's motor imagery features to perform transfer learning retraining on the pre-trained BAdaptNet network model, and obtain a personalized EEG signal decoding model adapted to the target subject through transfer learning fine-tuning;

[0032] The real-time decoding unit is configured to use a personalized EEG signal decoding model to decode the current motor imagery EEG data of the target subject collected in real time to obtain the current motor intention of the target subject.

[0033] On the other hand, the present invention also provides a computer storage medium, in which instructions are stored. When the instructions are executed, the EEG signal decoding method based on data enhancement is implemented.

[0034] On the other hand, the present invention also provides a computing device, including a processor and a communication interface coupled to the processor; the processor is used to run a computer program or instruction to implement the data enhancement-based EEG signal decoding method.

[0035] Beneficial effects:

[0036] This invention addresses the challenges of high target subject fatigue and difficulty in calibrating the decoding model in BCI applications, often caused by long calibration data acquisition times. This significantly improves decoding efficiency and accuracy, reduces the physical and psychological stress on users, and significantly enhances the effectiveness of BCI systems. This method has been validated in rehabilitation training scenarios for stroke patients, accurately decoding the left and right leg stepping intentions and controlling the movement of lower limb exoskeleton devices in real time. This improves rehabilitation training effectiveness, reduces the burden on patients, and promotes the practical application of BCI technology in rehabilitation medicine.

[0037] This paper proposes an EEG decoding method based on RM-GAN data generation and transfer learning, which has the following beneficial effects compared with the existing technology:

[0038] 1. The method consists of an RM-GAN model and a BAdaptNet model. The target subject data generated by the RM-GAN model is used instead of the previously collected data of the target subject application, reducing calibration time. Transfer learning technology is then used to pre-train and retrain the BAdaptNet model, effectively solving the problem of personalized calibration of the brain-computer interface system.

[0039] 2. This method, through the generation of target subject motor imagery data features and transfer learning technology, addresses the key challenge of clinical applicability of brain-computer interface technology in real-time applications, promoting its widespread application in the field of rehabilitation medicine.

[0040] 3. While improving the accuracy of EEG signal decoding, the method reduces the initial calibration time by more than 90% compared with current traditional methods, significantly reducing training time and patient burden, and providing a new solution for the widespread application of brain-computer interface technology in the field of rehabilitation medicine.

[0041] 4. This method solves the problem of insufficient data for personalized adaptation of brain-computer interface systems by generating target subject motor imagery EEG data through RM-GAN.

[0042] 5. The method significantly improves decoding efficiency and accuracy and reduces calibration time by combining the BAdaptNet network and the transfer learning method.

[0043] 6. The method is applicable to various scenarios in rehabilitation medicine and provides a new technical path for the widespread application of brain-computer interface technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0045] Figure 1 4 is a flow chart of an EEG signal decoding method based on data enhancement according to an exemplary embodiment of the present invention.

[0046] Figure 2 is a schematic diagram of an RM-GAN generation network according to an exemplary embodiment of the present invention.

[0047] Figure 3 Schematic diagram of a generator G of an RM-GAN generation network according to an exemplary embodiment of the present invention.

[0048] Figure 4 Schematic diagram of a discriminator D of an RM-GAN generation network according to an exemplary embodiment of the present invention.

[0049] Figure 5 FIG. 4 is a schematic diagram of a BAdaptNet network model according to an exemplary embodiment of the present invention.

[0050] Figure 6 2 is a schematic diagram of a BAdaptNet network model transfer learning training process according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0051] To facilitate a clear description of the technical solutions of the embodiments of the present invention, the words "first" and "second" are used in the embodiments of the present invention to distinguish between identical or similar items with substantially the same functions and effects. For example, the first threshold and the second threshold are merely used to distinguish between different thresholds and do not limit their order. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or execution order, and the words "first" and "second" do not necessarily mean different.

[0052] It should be noted that, in the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present invention should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0053] In the present invention, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. The following at least one item (item) or similar expressions thereof refer to any combination of these items, including any combination of single items (items) or plural items (items). For example, at least one item (item) of a, b or c can represent: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b and c, where a, b, c can be single or multiple.

[0054] Generative Adversarial Networks (GANs) have attracted widespread attention due to their powerful ability to generate high-quality data. In the field of brain-computer interface technology, the introduction of GAN technology provides a new solution to the problem of insufficient data for personalized model calibration. Transfer learning, as a technology that effectively utilizes existing knowledge to fine-tune models, can transfer data from multiple patients to target patients, thereby improving the system's robustness to individual differences. However, the current generative networks and transfer learning methods for EEG signal characteristics are still in the exploratory stage, and their shortcomings in real-time and generalization capabilities limit practical applications. Therefore, it is necessary to enhance the coding noise resistance and ultimately improve the performance of neural network models.

[0055] To address the challenges of high target subject fatigue and difficulty in personalized decoding model calibration in application scenarios due to the long target subject calibration data collection time, the present invention proposes an EEG signal decoding method based on RM-GAN network data generation and transfer learning. This method is based on the target subject's resting EEG and source domain resting and motor imagery data. The RM-GAN network model is used to generate enhanced motor imagery data for the target subject. The BAdaptNet network model is pre-trained on the source domain motor imagery data to obtain a pre-trained BAdaptNet network model. The pre-trained BAdaptNet network model is then retrained using transfer learning technology to fine-tune parameters based on the target subject's enhanced motor imagery data, resulting in a personalized decoding model adapted for the target subject. The EEG signal decoding method based on RM-GAN network data generation and transfer learning can achieve rapid calibration and high-precision decoding of the target subject's personalized motor intention decoding model.

[0056] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0057] like Figure 1 As shown in FIG, a flow chart of an EEG signal decoding method based on data enhancement is given, and the method includes the following steps:

[0058] EEG data preprocessing: Preprocess the resting data of the source domain, the motor imagery data of the source domain, and the resting data of the target domain to obtain the preprocessed EEG features;

[0059] Pre-training the RM-GAN network model: Using the pre-processed EEG features, the RM-GAN network model including the generator and the discriminator is pre-trained to obtain the pre-trained RM-GAN generator.

[0060] Pre-trained BAdaptNet network model: The pre-processed source domain motor imagery data is used to train the transfer learning network to obtain the pre-trained BAdaptNet network model;

[0061] The pre-trained RM-GAN generator is used to generate the target subject's motor imagery EEG features: the target resting data, the source domain resting data, and the source domain's motor imagery data are pre-processed and fused, and the pre-trained RM-GAN generator is used to generate the target subject's motor imagery features;

[0062] The generated target subject motor imagery features are used to retrain the pre-trained BAdaptNet network model through transfer learning. Through transfer learning fine-tuning, a personalized EEG signal decoding model adapted to the target subject is obtained.

[0063] The current motor imagery data of the target subject collected in real time is used to decode the current motor intention of the target subject using a personalized EEG signal decoding model.

[0064] The resting data in the source domain refers to the resting-state data in the public motor imagery EEG database and the motor imagery EEG dataset previously collected from the target subject. The motor imagery data in the source domain refers to the motor imagery data in the public motor imagery EEG database and the motor imagery EEG dataset previously collected from the target subject. The resting data in the target domain refers to the resting-state data collected when the target subject uses the method. The motor imagery data in the target domain refers to the motor imagery data collected when the target subject uses the method. The target subject usually refers to the user or patient currently using the method.

[0065] The RM-GAN network, known as the Rest and Motor Imagery Generative Adversarial Network, is a network that can be translated as "Rest and Motor Imagery Generative Adversarial Network." The RM-GAN network consists of a discriminator D and a generator G. The discriminator D employs a multi-layer convolutional network to extract the complex distribution of motor imagery features, thereby determining the authenticity of the input data. The generator G employs an encoder-decoder structure, combining convolutional and deconvolutional layers to achieve nonlinear mapping of the input feature matrix, gradually generating data that approximates the target motor imagery features. The generator's encoding module consists of four two-dimensional convolutional layers, which effectively extract the time-frequency features of the EEG signal. A regularization layer is added after each convolutional layer to stabilize training, and LeakyReLU is used as the activation function to enhance nonlinear expression capabilities. The encoded features are gradually restored through four deconvolutional layers in the decoding module, matching the target motor imagery features. A regularization layer and ReLU activation function are added after the first three deconvolutional layers to ensure decoding stability and nonlinear mapping capabilities. The Tanh activation function is used after the final deconvolutional layer to map the output features to the target feature space.

[0066] The BAdaptNet network stands for Brain Adapt Net, which translates to brain adaptation network. The BAdaptNet network first extracts preliminary features through a standard convolutional layer, and uses the SE attention mechanism to adaptively adjust the importance of the channel. Subsequently, dimensionality reduction and regularization are performed through pooling and Dropout to reduce redundant information and prevent overfitting. The subsequent dynamic convolution layer can adaptively extract multi-scale features and improve the model's adaptability to different data patterns. In the deep feature extraction stage, the network gradually extracts high-dimensional features through multi-layer convolution, and uses batch normalization and ReLU activation functions to ensure training stability. Finally, features are integrated through a fully connected layer, and the category probability is output through Softmax to achieve the binary classification task.

[0067] The preprocessing includes: performing channel screening on the EEG signals including the resting data of the source domain, the motor imagery data of the source domain, and the resting data of the target domain, and screening out channels related to the motor sensory cortex;

[0068] Band-pass filtering was performed to extract rhythmic activity related to the motor imagery task;

[0069] Perform multiple average downsampling to reduce the amount of data;

[0070] Frequency domain features are extracted through wavelet functions to capture key time-frequency information in EEG signals.

[0071] The generator generates motor imagery features that are highly similar to the motor imagery features of the target domain based on the preprocessed EEG features, which are used for authenticity judgment by the discriminator and adversarial training of the generator; the discriminator performs authenticity judgment on the target domain motor imagery features generated by the generator.

[0072] The generator adopts an encoder-decoder structure. The encoder includes multiple two-dimensional convolutional layers as time-frequency filters. A regularization layer is added after each convolutional layer, and LeakyReLU is used as the activation function. The decoder includes multiple deconvolutional layers. A regularization layer and a ReLU activation function are added after the deconvolution layer before the last deconvolution layer, and a Tanh activation function is added after the last deconvolution layer.

[0073] The preprocessed EEG signal is obtained by performing channel screening, bandpass filtering, downsampling, extracting frequency domain features, and performing feature fusion on the resting data of the source domain, the motor imagery data of the source domain, and the resting data of the target domain.

[0074] The BAdaptNet network model includes a deep attention feature extraction layer, a dynamic convolution layer, a multi-convolution feature fusion layer and an output layer;

[0075] The deep attention feature extraction layer uses the SE attention mechanism to filter channels, enabling the transfer learning network to perform dynamic channel feature recalibration to improve the network's representation ability;

[0076] The dynamic convolution layer adopts dynamic convolution to adapt to the non-stationary characteristics of the EEG signal;

[0077] The multiple convolutional feature fusion layer improves the information capture capability of the model by fusing features through multiple convolutional layers.

[0078] The generated target subject motor imagery features are used to perform transfer learning and retraining on the pre-trained BAdaptNet network model, and a personalized EEG signal decoding model adapted to the target subject is obtained through transfer learning fine-tuning, including:

[0079] Freeze the weight parameters of the deep attention feature extraction layer and the dynamic convolution layer, and only adjust the weight parameters of the multi-convolution feature fusion layer and the output layer.

[0080] Based on the above method, an EEG signal decoding device based on data enhancement includes:

[0081] a preprocessing unit configured to preprocess the resting data of the source domain, the motor imagery data of the source domain, and the resting data of the target domain to obtain preprocessed EEG features;

[0082] A generator pre-training unit is configured to pre-train an RM-GAN network model including a generator and a discriminator using the pre-processed EEG features to obtain a pre-trained generator;

[0083] The transfer learning model pre-training unit is configured to train the BAdaptNet network model using the pre-processed motor imagery features of the source domain to obtain a pre-trained BAdaptNet network model;

[0084] a data generation unit configured to generate target subject motor imagery features using a pre-trained generator using feature data of the source domain resting data, the source domain motor imagery data, and the target subject resting data after preprocessing and data fusion;

[0085] The transfer learning model retraining unit is configured to use the generated target subject's motor imagery features to perform transfer learning retraining on the pre-trained BAdaptNet network model, and obtain a personalized EEG signal decoding model adapted to the target subject through transfer learning fine-tuning;

[0086] The real-time decoding unit is configured to use a personalized EEG signal decoding model to decode the current motor imagery EEG data of the target subject collected in real time to obtain the current motor intention of the target subject.

[0087] In the actual application of the brain-computer interface system, the application of the method and device described in the present invention can solve the problem of long target subject calibration data acquisition time, high target subject fatigue and difficulty in personalized calibration of the decoding model. In the specific implementation, the GAN network is an RM-GAN network, and the transfer learning network is a BAdaptNet network. Based on the resting data of the source domain, the resting data of the target domain and the motor imagery data of the source domain, the motor imagery data of the target domain is generated by RM-GAN, and the BAdaptNet network is trained based on the resting data of the source domain and the motor imagery data of the source domain. The generated data is then used for transfer learning and retraining to achieve rapid calibration and high-precision decoding of the personalized decoding model.

[0088] The following combination Figure 1 Provide detailed explanation.

[0089] Step 1: Perform EEG signal preprocessing and feature fusion on the resting data of the source domain, the resting data of the target domain, and the motor imagery data of the source domain, respectively, to obtain EEG features after preprocessing and feature fusion; the frequency domain feature extraction is used to ensure the quality and feature representation capability of the input data;

[0090] Step 2: Use the pre-processed feature data as the input of the RM-GAN network to train the generator of the RM-GAN network. Figure 2 As shown in the figure, the RM-GAN network consists of two neural networks: the generator (G) and the discriminator (D), which are mutually optimized through adversarial training.

[0091] like Figure 3 As shown, the architecture of the generator G adopts an encoder-decoder structure, combining convolutional layers and deconvolutional layers to perform nonlinear mapping on the input comprehensive feature matrix, and gradually generate data close to the target domain motion imagery features.

[0092] The generator's encoder consists of four two-dimensional convolutional (Conv2D) layers, each of which acts as a time-frequency filter, effectively extracting time-frequency features from EEG data. A regularization layer is added after each convolutional layer to stabilize the training process, and LeakyReLU is used as the activation function to enhance the model's nonlinear expression capabilities.

[0093] like Figure 4 As shown in the figure, the encoded features are further fed into the generator's decoder, which comprises four deconvolutional layers that progressively decode the features and generate a feature matrix consistent with the motor imagery features of the target domain. A regularization layer and a ReLU activation function are added after the first three deconvolutional layers to ensure stability and nonlinear mapping during decoding. A Tanh activation function is added after the final deconvolutional layer to map the output features into the target feature space.

[0094] pass Figure 2 In the encoder-decoder structure shown, the generator can gradually transform the input comprehensive feature matrix, generating features that are highly similar to the motor imagery features in the target domain. This is used by the discriminator D for authenticity determination and adversarial training of the entire framework. The input data of the RM-GAN network is the EEG signal after preprocessing and feature fusion. The preprocessed EEG signal is obtained by channel screening, bandpass filtering, downsampling, extracting frequency domain features, and then fusion of the resting data of the source domain, the resting data of the target domain, and the motor imagery data of the source domain.

[0095] Step 3: Pre-train the BAdaptNet network based on the pre-processed source domain motor imagery feature data.

[0096] The pre-processed source domain motor imagery feature data is obtained by subjecting the source domain motor imagery signal to the pre-processing and wavelet transformation described in step 1. Figure 5 As shown in the figure, the BAdaptNet network includes a deep attention feature extraction layer Block1, a dynamic convolution layer Block2, a multiple convolution feature fusion layer Block3 and an output layer.

[0097] The deep attention feature extraction layer Block1 uses the SE attention mechanism to filter channels, enabling the network to perform dynamic channel feature recalibration to improve the network's representation ability and enhance the spatial representation information extraction of EEG signals; the dynamic convolution layer Block2 uses dynamic convolution to adapt to the non-stationary characteristics of EEG signals and improve the flexibility of feature extraction; the multiple convolution feature fusion layer Block3 enhances the information capture capability of the model by fusing features through multiple convolution layers.

[0098] Step 4: Generate target motor imagery features using the pre-processed and fused feature data of the target resting data, the source resting data, and the motor imagery data, using the generator of the trained RM-GAN network. The feature data is obtained by pre-processing and fusion of the target resting data, the source resting data, and the motor imagery data as described in step 1. The generator of the trained RM-GAN network is the generator obtained by pre-training the RM-GAN network described in step 2.

[0099] Step 5: Use the target motion imagery features generated in step 4 to perform transfer learning and retrain the pre-trained BAdaptNet network model;

[0100] like Figure 6 As shown, the transfer learning retraining specifically involves fine-tuning the pre-trained BAdaptNet network based on the target motor imagery EEG features generated by RM-GAN to achieve personalized adaptation for the target subject. The pre-trained BAdaptNet network is obtained by pre-training the BAdaptNet network in step 3.

[0101] The transfer learning fine-tuning is specifically as follows: by freezing the weight parameters of Block1 and Block2, only adjusting the weight parameters of Block3 and the output layer, it quickly adapts to the EEG characteristics of the target subjects and effectively improves the decoding accuracy and real-time performance of the model in individualized applications.

[0102] Example 1: EEG data preprocessing

[0103] Data preprocessing was performed on the resting data from the source and target domains, as well as the motor imagery data from the source domain. This preprocessing step included channel filtering to identify channels related to the motor sensory cortex, 8-30 Hz bandpass filtering to extract rhythmic activity related to the motor imagery task, 10x average downsampling to reduce the data volume, and wavelet function extraction of frequency domain features to capture key time-frequency information in the EEG signals.

[0104] In the data preprocessing stage, to ensure that the EEG data input to the model has a higher signal-to-noise ratio and includes features closely related to motor intention, the resting data of the source and target domains, as well as the motor imagery data of the source domain, are first filtered to select EEG channels related to the motor sensory area to focus on key feature areas.

[0105] Assume that the original EEG data is Where C is the number of channels and T is the number of time sampling points. The channel filtering operation extracts the selected motion-related channels (such as C3, C4 and their surrounding channels) into a new data matrix:

[0106]

[0107] Among them, S(·) is the channel selection function, C s is the number of channels after filtering.

[0108] To extract the frequency band signals related to movement, the EEG signal is band-pass filtered. The frequency band of the band-pass filter is set to 8Hz≤f≤30Hz, corresponding to the α band (8-14Hz) and β band (14-30Hz) related to movement intention. The filtered signal can be expressed as:

[0109] X band =H BP (X sel ,f l ,f h )

[0110] Among them, H BP (·,f l ,f h ) is the bandpass filter function, f l and f h are the low and high cutoff frequencies of the filter, respectively.

[0111] In order to reduce the data dimension while retaining the key characteristic information of the signal, the filtered data is subjected to mean downsampling operation to reduce the number of sampling points T on the time axis. Assuming the downsampling ratio is r, the signal matrix after downsampling is:

[0112]

[0113] in, is the number of time points after downsampling,

[0114] During the feature fusion stage, resting and motor imagery data are fused using a wavelet transform (WT) to extract time-frequency features, thereby constructing a comprehensive feature matrix suitable for input into a generative adversarial network. The wavelet transform (WMT) maps signals into the time-frequency domain, extracting spectral information at different time points. Here, a continuous wavelet transform (CWT) is performed using the Morlet wavelet function as the mother wavelet.

[0115] Assume that the input resting data of the source domain and target domain and the motor imagery data of the source domain are and Its shape is Where C is the number of channels and T is the number of time points. The wavelet transform is defined as:

[0116]

[0117] Among them, ψ a,b (t) is the daughter wavelet obtained by translating and scaling the mother wavelet ψ(t):

[0118]

[0119] Among them, a>0 is the scaling factor, which determines the frequency resolution; b is the translation factor, which determines the time resolution; * represents the complex conjugate. The expression of the Morlet wavelet function is:

[0120]

[0121] in, is the Gaussian envelope, j is the imaginary unit, and f0 is the center frequency.

[0122] The characteristic of Morlet wavelet is that it has good localization characteristics in the frequency domain and time domain, and is particularly suitable for processing EEG signals.

[0123] For each channel c and each scale a, the time-frequency feature matrix can be obtained after wavelet transform After wavelet transform of all channels, the characteristic matrix obtained is:

[0124]

[0125] Where A and B are the number of sampling points for scale and time translation, respectively.

[0126] In the feature fusion stage, the source domain resting data is first Target domain resting data and source domain motor imagery data Perform wavelet transform respectively to extract its time-frequency characteristics and

[0127] To effectively fuse the source and target domain data and ensure that the resting data features match the length of the motor imagery data features, the resting data features are cropped and averaged. Specifically, based on the temporal length L of the source domain motor imagery data features, the source and target domain resting data features are cropped to the same temporal length L, and the cropped resting data features are averaged over the temporal dimension.

[0128] Subsequently, in order to enhance the correspondence between the source domain and the target domain resting data, and strengthen the dynamic correlation between the motor imagery features and the resting features, the following feature fusion strategy is adopted: First, the source domain motor imagery features are Subtract the cropped and averaged source domain resting features To extract the feature difference of the motion feature relative to the resting state; then, this difference feature is added to the cropped target domain resting feature Finally, the source domain and target domain features are fully integrated to generate a comprehensive feature matrix W fusion , which is calculated as follows:

[0129]

[0130] in, and They represent the averaging results of the source domain and target domain resting features in the time dimension, respectively. The calculation formula is:

[0131]

[0132] Where t represents the index of the time point, and L is the length of time after clipping.

[0133] Through the above fusion strategy, not only the consistency of the source domain and target domain data in time length is guaranteed, but also the low noise characteristics of the resting data and the difference information of the motor imagery data are fully utilized. This feature matrix is ​​used as the input of the generative adversarial network.

[0134] Example 2: RM-GAN network generation and pre-training

[0135] like Figure 2 As shown in Figure 2, the RM-GAN generation network module includes two neural networks: the generator (G) and the discriminator (D), which are mutually optimized through adversarial training. The task of the generator G is to generate the fusion feature W of the input.fusion Generate fake features G(W) that are close to the real target domain motion imagery features fusion ); the discriminator D is used to distinguish whether the input feature is the real target domain motion image feature Or is it a fake feature G(W generated by the generator? fusion ).

[0136] During the training phase, the input data of the RM-GAN generation network is the comprehensive EEG feature matrix Its specific dimensions and structures are as follows: Figure 2 shown.

[0137] like Figure 3 As shown, the architecture of the generator G adopts an encoder-decoder structure, combining convolutional layers and deconvolutional layers to input the comprehensive feature matrix W fusion Nonlinear mapping is performed to gradually generate data that approximates the motor imagery characteristics of the target domain. Specifically, the generator's encoding module consists of four two-dimensional convolutional layers, each with a kernel size of 4×4 and a stride of 2×2. Each convolutional layer acts as a time-frequency filter, effectively extracting the time-frequency characteristics of the EEG data. A regularization layer is added after each convolutional layer to stabilize the training process, and LeakyReLU is used as the activation function to enhance the model's nonlinear representation capabilities. The four convolutional layers in the encoding module use kernels of 40, 80, 160, and 320, respectively, to gradually encode the comprehensive feature matrix into a high-level feature representation. The encoded features are further input into the decoding module, which consists of four deconvolutional layers to gradually decode the features and generate a feature matrix consistent with the motor imagery characteristics of the target domain. The kernel size and stride of the deconvolutional layers are both 4×4 and 2×2. A regularization layer and ReLU activation function are added after the first three deconvolutional layers, respectively, to ensure stability and nonlinear mapping during the decoding process. The Tanh activation function is added after the last deconvolution layer to map the output features into the target feature space. The number of convolution kernels in the four deconvolution layers in the decoding module is 160, 80, 40, and 20, respectively. The resulting feature matrix is ​​consistent with the structure of the target domain motion imagery features. Through this encoder-decoder structure, the generator can gradually transform the input comprehensive feature matrix W. fusion Perform feature conversion to generate motion imagery features similar to the target domain Highly similar forged features are used by the discriminator to determine authenticity and conduct adversarial training of the entire framework.

[0138] like Figure 4 As shown in Figure 2, the architecture of the discriminator D is designed as a multi-layer convolutional network, which aims to effectively extract the complex distribution information of motion imagery features, thereby realizing the discrimination of the authenticity of the input data.

[0139] During RM-GAN training, the goal of the discriminator D is to maximize its accuracy in classifying real and fake features, while the goal of the generator G is to minimize the discriminator's ability to distinguish fake features. To achieve this goal, the training of the entire RM-GAN module is based on the binary cross-entropy loss function. Specifically, the discriminator loss function is defined as follows:

[0140]

[0141] Among them, N is the number of training samples, y i Indicates the target category (real feature The label 1 generates the feature G(W fusion ) is labeled as 0), D(·) represents the probability that the feature output by the discriminator is judged to be true. Generator loss function:

[0142]

[0143] The goal of the generator is to maximize the discriminator's ability to generate features G(W fusion ) is to make its output as close to the real feature as possible. The generator and discriminator are trained by alternating optimization. Specifically, in each training step, the parameters of the generator G are fixed and the parameters of the discriminator D are optimized to minimize Then, the parameters of the discriminator D are fixed and the parameters of the generator G are optimized to minimize The final optimization goal can be expressed as:

[0144]

[0145] Among them, W fusion The fused feature matrix is ​​obtained by fusing the resting data features of the source and target domains, as well as the source domain motor imagery data features, according to the method used in the feature fusion stage. We trained the model for 500 iterations and saved the model weights and hyperparameters that minimized the loss on the test data.

[0146] Through the above adversarial training, the generator gradually learns to fusion Generate high-quality forged features that are close to the real target domain motion imagery features, while the discriminator continuously improves its ability to distinguish between real features and forged features. Ultimately, the trained generator can generate high-quality motion imagery features G(W fusion ), providing input support for the transfer learning stage. This process significantly alleviates the problem of insufficient target domain data and provides sufficient feature expression capabilities for subsequent decoding tasks.

[0147] Example 3: BAdaptNet network and pre-training

[0148] like Figure 5 Figure 1 shows a schematic diagram of the BAdaptNet transfer learning network. The BAdaptNet network is pre-trained using source domain motor imagery data features. The BAdaptNet model consists of four core modules: Block1, Block2, Block3, and the output layer.

[0149] Block 1 is a deep attention feature extraction layer that uses the SE attention mechanism to filter channels, enabling the network to perform dynamic channel feature recalibration to improve the network's representation ability and enhance the extraction of spatial representation information of EEG signals.

[0150] Block2 is a dynamic convolution feature extraction layer that uses dynamic convolution to adapt to the non-stationary characteristics of EEG signals and improve the flexibility of feature extraction.

[0151] Block3 is a multi-convolution feature integration layer, which improves the model's information capture capability by fusing features through multiple convolution layers;

[0152] The output layer uses a fully connected layer to predict the features and output the prediction results.

[0153] The pre-training process is as follows.

[0154] First, the input layer converts the source domain motor imagery data into EEG features after preprocessing and feature extraction. The network is sent to the deep attention feature extraction layer (Block 1), which extracts local features through the depthwise separable convolution operation (DepthwiseConv2D) and combines the SE attention mechanism (Squeeze-and-ExcitationAttention Layer) to weight the importance of different channels to achieve adaptive redistribution of global features. After global average pooling, the channel vector is obtained.

[0155]

[0156] Where c represents the channel index, H and W represent the height and width of the feature matrix respectively. Then, the channel vector z is processed by two fully connected layers and nonlinear activation functions to generate channel weights

[0157] s=σ(W2·ReLU(W1·z))

[0158] Where W1 and W2 are trainable weight matrices, and σ is the Sigmoid activation function. The final channel weight s is used to reweight the original feature matrix X to obtain the enhanced feature matrix X att .

[0159] X att,c (h,w)=s c ·X c (h,w)

[0160] Among them, c is the channel index, h and w are the spatial position indexes of the feature matrix respectively.

[0161] Next, feature X att Enter the dynamic convolution module (Block 2), which uses two layers of dynamic convolution layers to dynamically generate convolution kernels to perform adaptive convolution operations on different input features. The dynamic convolution module first generates K weight coefficients π through two fully connected layers and a nonlinear activation function. k .

[0162]

[0163] Among them, W1 and W2 are trainable weight matrices, is the Softmax activation function, which ensures that all π k Satisfy non-negativity and normalize. Define K convolution kernels By adding the attention weight π k With the convolution kernel W k Weighted summation to generate dynamic convolution kernel W dynamic , the dynamic convolution kernel can adaptively adjust the convolution operation according to the content of the input feature matrix.

[0164]

[0165] Using dynamically generated convolution kernel W dynamic For the input feature matrix X att Perform a two-dimensional convolution operation to obtain the output feature matrix X dyn .

[0166] X dyn =Conv2D(X att ,W dynamic )

[0167] Among them, Conv2D(·) is a two-dimensional convolution function. Through the dynamic convolution module, the network can flexibly adjust the parameters of the convolution kernel to adapt to the changes in input features, thereby improving its adaptability to data non-stationarity.

[0168] The features are then fed into the multi-convolution feature fusion layer (Block 3), which processes the features by cascading multiple convolution operations with different kernel sizes. The input features pass through two consecutive 2D convolution layers, each followed by a batch normalization layer and a ReLU activation function. The compressed features are then fed into the fully connected layer and classified using the Softmax activation function. Assuming the classification target is y, the network output is The cross entropy loss function is used for optimization during the training phase.

[0169]

[0170] Where N represents the total number of categories, y i is the true label, For the category probability predicted by the network, BAdaptNet realizes deep extraction, adaptive enhancement, multi-feature fusion and classification prediction of input features, providing an effective pre-training model for transfer learning applications in the subsequent testing phase.

[0171] Example 4: BAdaptNet network model transfer learning training

[0172] like Figure 6 As shown in Figure 2, after completing data preprocessing and BAdaptNet network pretraining, the RM-GAN network is used to generate the target subject's motor imagery EEG features. The specific implementation process of generating data is shown below.

[0173] Utilize the resting data collected from the target Resting data of the source domain and source domain motor imagery data Extract its time-frequency features, which are expressed as and

[0174] In order to generate the target domain MI feature matrix, the source domain MI features are fused with the resting data, and the features are fused using the following formula:

[0175]

[0176] Among them, MEan(·) is the mean function of the feature matrix along the time axis, and the feature matrix W fusion Generate target domain motion imagery features through the trained RM-GAN generator

[0177]

[0178] Here, G(·) is the data generation function of the trained generator. The generator adjusts the source domain motor imagery features to be consistent with the target domain resting feature distribution through the encoder-decoder structure, thereby generating motor imagery data with target domain characteristics.

[0179] like Figure 6 As shown in Figure 2, during transfer learning retraining of the BAdaptNet network, a transfer learning approach was combined with freezing some pre-trained layers. The model then fine-tuned using the target subject's personalized data and the target subject-specific data generated by RM-GAN. By retaining the parameters of the common feature layers during the transfer learning phase, the model training computational complexity and time were reduced while also enhancing its adaptability to the target subject's data. The resulting model was able to quickly adapt to the target subject's personalized EEG signal characteristics, improving decoding accuracy and efficiency.

[0180] The transfer learning retraining process is as follows:

[0181] The generated target domain motor imagery feature data Used to retrain the pre-trained BAdaptNet model to achieve cross-domain adaptation. BAdaptNet consists of four modules, of which Block 1 and Block 2 are deep attention feature extraction modules and dynamic feature enhancement modules for extracting key time-frequency features. Since they have captured effective feature expressions in source domain training, the parameters of these two modules remain frozen during the transfer learning phase to ensure that the knowledge of the source domain is transferred to the target domain. Block 3 and Block 4 serve as heavy feature fusion and classification modules, and their parameters are adjusted during the transfer learning phase based on the generated target domain motion imagery data.

[0182]

[0183] in, is the generated target domain motion imagery feature matrix, is the target domain resting feature matrix, are the parameters of frozen Block 1 and Block 2, θ Block3,Block4 are the parameters of Block 3 and Block 4 that need to be trained, The optimized parameters of Block 3 and Block 4 are updated after retraining.

[0184] The BAdaptNet model, retrained through transfer learning, can be deployed in rehabilitation training systems for real-time EEG decoding, driving the exoskeleton device to achieve left and right leg stepping movements for the target subject. By decoding the target subject's movement intentions in real time and dynamically adjusting rehabilitation training tasks and plans, the effectiveness of brain-computer interface systems in rehabilitation scenarios is enhanced.

[0185] In addition, according to an exemplary embodiment of the present invention, a computer-readable storage medium having a computer program stored therein may also be provided. The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform a data-enhanced EEG signal decoding method according to an exemplary embodiment of the present invention. The computer-readable recording medium is any data storage device that can store data read out by a computer system. Examples of computer-readable recording media include: read-only memory, random access memory, read-only optical disc, magnetic tape, floppy disk, optical data storage device, and carrier wave (such as data storage via the Internet via a wired or wireless transmission path).

[0186] Furthermore, according to an exemplary embodiment of the present invention, a computing device may be provided. The computing device includes a processor and a memory. The memory is configured to store a computer program. The computer program is executed by the processor so that the processor performs the data-enhanced EEG signal decoding method according to an exemplary embodiment of the present invention.

[0187] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the drawings, etc. In the specification, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the specification. Certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0188] Although the present invention has been described with reference to specific features and embodiments thereof, it will be apparent that various modifications and combinations thereof may be made without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely illustrative of the present invention and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the invention. It will be apparent that various modifications and variations of the present invention may be made by those skilled in the art without departing from the spirit and scope of the invention. Thus, the present invention is intended to include such modifications and variations as fall within the scope of the invention and its equivalents.

Claims

1. A method for decoding EEG signals based on data enhancement, characterized in that: include: Preprocessing the resting data of the source domain, the motor imagery data of the source domain, and the resting data of the target domain to obtain preprocessed EEG features; The pre-processed EEG features are used to pre-train the RM-GAN network model including the generator and the discriminator to obtain the pre-trained generator; The BAdaptNet network model is trained using the pre-processed source domain motor imagery features to obtain a pre-trained BAdaptNet network model. The source domain resting data, source domain motor imagery data and target subject resting data are pre-processed and fused, and the pre-trained generator is used to generate the target subject motor imagery features. The generated target subject's motor imagery features are used to retrain the pre-trained BAdaptNet network model through transfer learning. Through transfer learning fine-tuning, a personalized EEG signal decoding model adapted to the target subject is obtained. A personalized EEG signal decoding model is used to decode the target subject's current motor imagery EEG data collected in real time to obtain the target subject's current movement intention.

2. The method for decoding EEG signals based on data enhancement according to claim 1, characterized in that: The preprocessing of the resting data of the source domain, the motor imagery data of the source domain, and the resting data of the target domain includes: Channel screening is performed on the EEG signals including the resting data of the source domain, the motor imagery data of the source domain, and the resting data of the target domain to obtain channels related to the motor sensory cortex; Band-pass filtering was performed to extract rhythmic activity related to the motor imagery task; Perform multiple average downsampling to reduce the amount of data; Frequency domain features are extracted through wavelet functions to capture key time-frequency information in EEG signals.

3. The method for decoding EEG signals based on data enhancement according to claim 1, wherein: In the step of pre-training the RM-GAN network model including the generator and the discriminator using the pre-processed EEG features, the generator generates target domain motor imagery features that are highly similar to the motor imagery features of the target domain based on the pre-processed EEG features, which are used for the discriminator to perform authenticity judgment and adversarial training of the generator; the discriminator performs authenticity judgment on the target domain motor imagery features generated by the generator.

4. The method for decoding EEG signals based on data enhancement according to claim 3, characterized in that: The RM-GAN generator adopts an encoder-decoder structure, wherein the encoder includes multiple two-dimensional convolutional layers as time-frequency filters, each convolutional layer is followed by a regularization layer, and LeakyReLU is used as the activation function; the decoder includes multiple deconvolutional layers, and the deconvolution layers before the last deconvolution layer are followed by a regularization layer and a ReLU activation function, and the last deconvolution layer is followed by a Tanh activation function.

5. The method for decoding EEG signals based on data enhancement according to claim 1, wherein: The feature data obtained by preprocessing and data fusion of the source domain resting data, the source domain motor imagery data and the target subject resting data is obtained by respectively performing channel screening, bandpass filtering, downsampling, extracting frequency domain features and performing feature fusion on the source domain resting data, the source domain motor imagery data and the target subject resting data.

6. The method for decoding EEG signals based on data enhancement according to claim 1, characterized in that: The BAdaptNet network model includes a deep attention feature extraction layer, a dynamic convolution layer, a multiple convolution feature fusion layer and an output layer; wherein the deep attention feature extraction layer uses the SE attention mechanism to screen channels, enabling the transfer learning network to perform dynamic channel feature recalibration to improve the network's representation ability; the dynamic convolution layer uses dynamic convolution to adapt to the non-stationary characteristics of EEG signals; the multiple convolution feature fusion layer improves the model's information capture capability by fusing features from multiple convolution layers.

7. The method for decoding EEG signals based on data enhancement according to claim 6, characterized in that: The generated target subject motor imagery features are used to perform transfer learning and retraining on the pre-trained BAdaptNet network model, and a personalized EEG signal decoding model adapted to the target subject is obtained through transfer learning fine-tuning, including: Freeze the weight parameters of the deep attention feature extraction layer and dynamic convolution layer in the pre-trained BAdaptNet network model, and only adjust the weight parameters of the multiple convolution feature fusion layer and output layer in the pre-trained BAdaptNet network model.

8. A data-enhanced electroencephalogram (EEG) signal decoding device based on the method according to any one of claims 1 to 7, characterized in that: include: a preprocessing unit configured to preprocess the resting data of the source domain, the motor imagery data of the source domain, and the resting data of the target domain to obtain preprocessed EEG features; A generator pre-training unit is configured to pre-train an RM-GAN network model including a generator and a discriminator using the pre-processed EEG features to obtain a pre-trained generator; The transfer learning model pre-training unit is configured to train the BAdaptNet network model using the pre-processed motor imagery features of the source domain to obtain a pre-trained BAdaptNet network model; a data generation unit configured to generate target subject motor imagery features using a pre-trained generator using feature data of the source domain resting data, the source domain motor imagery data, and the target subject resting data after preprocessing and data fusion; The transfer learning model retraining unit is configured to use the generated target subject's motor imagery features to perform transfer learning retraining on the pre-trained BAdaptNet network model, and obtain a personalized EEG signal decoding model adapted to the target subject through transfer learning fine-tuning; The real-time decoding unit is configured to use a personalized EEG signal decoding model to decode the current motor imagery EEG data of the target subject collected in real time to obtain the current motor intention of the target subject.

9. A computer storage medium, characterized in that The computer storage medium stores instructions, and when the instructions are executed, the EEG signal decoding method based on data enhancement according to any one of claims 1 to 7 is implemented.

10. A computing device, characterized in that It comprises a processor and a communication interface coupled to the processor; the processor is used to run a computer program or instruction to implement the EEG signal decoding method based on data enhancement as described in any one of claims 1 to 7.

Images