A method and system for generating EEG virtual channel signals by integrating spatiotemporal features

Abstract

The present invention discloses a method and system for generating EEG virtual channel signals with the fusion of time and space features. It innovatively designs a multi-layer convolutional neural network with the fusion of time and space domain features and the consistency constraint of time and space features. Through transfer learning, multiple virtual channel EEG signals are generated from a small number of channel EEG signals. The network includes a time domain feature extraction module and a space domain feature extraction module, which can not only utilize the time domain features of the EEG signals of each channel, but also obtain the correlation features between the channels, that is, realize the fusion of time and space features of EEG signals and improve the learning and expression capabilities of the network; the designed composite loss function comprehensively considers the local loss in time and frequency domains, as well as the global loss of distribution consistency, and can more comprehensively and accurately evaluate the consistency between the original signal and the generated signal, guide network training, and improve network performance. The present invention effectively solves the problem caused by the small number of brain area signals and insufficient information collected by portable EEG detectors.

Description

Technical Field

[0001] The present invention relates to the technical field of human physiological signal detection, and in particular to a method and system for generating EEG virtual channel signals by fusing spatiotemporal features. Background Art

[0002] Electroencephalogram (EEG) signals are an important parameter reflecting the state of human brain activity and are used for research on brain development and cognitive ability, diagnosis of mental illness and brain pathology, and implementation of brain-computer interfaces. To obtain EEG signals from different brain regions, most devices follow the international 10-20 system of electrode placement, typically employing 32-128 electrodes (channels) depending on the needs. Using a larger number of electrodes can lead to limitations such as bulk, complexity of fitting, prolonged signal detection time, and high cost, limiting their application. Consequently, portable, low-channel (fewer than or equal to 8 electrodes) EEG monitors have emerged. While these monitors can overcome the aforementioned limitations of non-portable, multi-channel EEG monitors, they often lack sufficient information due to the limited number of brain regions they sample. Addressing the issue of insufficient EEG signals in portable, low-channel EEG monitors is crucial. Summary of the Invention

[0003] The first purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide a method for generating EEG virtual channel signals with spatiotemporal feature fusion, which effectively solves the problem caused by the small number of brain area signals and insufficient information collected by portable low-channel EEG detectors.

[0004] The second object of the present invention is to provide an EEG virtual channel signal generation system with spatiotemporal feature fusion.

[0005] The first object of the present invention is achieved by the following technical solution: a method for generating an EEG virtual channel signal by fusion of spatiotemporal features, comprising the following steps:

[0006] 1) Multi-channel EEG signal acquisition: Use a non-portable multi-channel EEG detector to collect multi-channel EEG signals from multiple brain regions of the human body, or obtain multi-channel EEG signals from public databases; For low-channel EEG signal acquisition, determine the number of electrodes and brain region locations as needed, and use a portable low-channel EEG detector to collect low-channel EEG signals;

[0007] 2) EEG signal preprocessing, including denoising, baseline drift removal, standardization, and segmentation of all EEG signals;

[0008] 3) The preprocessed multi-channel EEG signals are randomly divided into a training set, a validation set, and a test set according to a predetermined ratio;

[0009] 4) Designing an EEG virtual channel signal generation model, which is a multi-layer convolutional neural network. The model uses the brain area of ​​a portable few-channel EEG detector as a reference. Among the multi-channel EEG signals, the EEG signal of the corresponding reference brain area is selected as input, and the multi-channel EEG signal is used as output. The front end of the multi-layer convolutional neural network includes a time domain feature extraction module and a spatial domain feature extraction module, and the back end fuses the time domain features and spatial domain features. This network structure design can not only utilize the time domain features of the EEG signals of each channel, but also obtain the correlation features between the EEG signals of each channel, realizing the fusion of the spatiotemporal features of the EEG signals, thereby improving the learning and expression capabilities of the network.

[0010] 5) Define the network loss function and design a new composite loss function that integrates local losses in the time and frequency domains and global loss of distribution consistency to accurately evaluate the consistency between the original signal and the generated signal, guide network training, and improve learning effects;

[0011] 6) Model training: Use the training set to train the designed EEG virtual channel signal generation model, use the validation set to evaluate the model performance and optimize the parameters, and use the test set to evaluate the model performance to obtain the model with the best performance;

[0012] 7) Input the preprocessed few-channel EEG signal into the model with the best performance to generate EEG virtual channel signal.

[0013] Furthermore, in step 2), first, according to the frequency range, noise type and time-frequency characteristics of the EEG signal, a corresponding filter is designed to remove noise interference and decompose it into different frequency components; then, the baseline drift is removed and the signal is standardized; finally, the signal is segmented at predetermined time intervals.

[0014] Furthermore, in step 4), a multi-layer convolutional neural network is used, and the network is designed with a time domain feature extraction module and a spatial domain feature extraction module; wherein the time domain feature extraction module extracts time domain features of each channel of the multi-channel EEG signal input into the network in a multi-layer convolution structure; and the spatial domain feature extraction module performs convolution in the channel direction to obtain correlation features between channels;

[0015] The time domain feature extraction module and the spatial domain feature extraction module adopt two structural designs: serial connection and parallel connection. Among them, the serial connection structural design is to first extract the time domain features of each channel, and then input them into the spatial domain feature extraction module, and extract the spatial domain features by performing convolution operations in the channel direction; the parallel connection structural design refers to designing two independent branches at the network input end, one of which is the time domain feature extraction module and the other is the spatial domain feature extraction module; finally, a feature fusion module is designed to fuse the two types of features.

[0016] Furthermore, the time domain feature extraction module consists of three two-dimensional convolutional layers and a bidirectional LSTM layer, which is designed to extract the time domain features of the EEG signal. The input EEG signal is 6×3000 in size. First, it passes through three two-dimensional convolutional layers, with the number of convolution kernels being 64, 128 and 256 respectively, and the convolution kernel sizes being (1,7), (1,5) and (1,3) respectively, with the step size being (1,2). Deep temporal features are extracted from different scales, and the sequence length is gradually downsampled from 3000 to 375, obtaining 256 feature maps of 6×375 in size. Subsequently, the three-dimensional vector (6×375×256) is converted into a two-dimensional vector (3 75×1536), that is, 375 time points, each of which contains 1536 features, which are then input into the bidirectional LSTM layer to further extract time domain features. The bidirectional LSTM layer can better capture the temporal dependencies in sequence data by simultaneously learning the forward and backward information of the time series. The number of features in the hidden layer is set to 256, and the output feature of each time point is 512, that is, 256 units in the forward and backward LSTM layers, and the final output dimension is 375×512; in order to be fused with the subsequent spatial features, the output of the bidirectional LSTM layer is converted to 1×375×512, that is, 512 feature maps of size 1×375 are output;

[0017] The spatial domain feature extraction module captures the relationship between EEG signals across different channels through four two-dimensional convolutional layers. The first three two-dimensional convolutional layers have 64, 128, and 256 convolution kernels, with kernel sizes of (3,3) and strides of (1,2) to gradually extract local spatial domain features and reduce the temporal dimension. Finally, a two-dimensional convolutional layer with 512 kernels and a kernel size of (6,1) is used to capture global spatial domain features across all channels without padding, outputting 512 feature maps of size 1×375.

[0018] In the feature fusion module, the temporal and spatial domain features are concatenated along the channel dimension to obtain 1024 fused feature maps of size 1×375, so that the model can simultaneously consider the temporal and spatial information of the EEG signal during the upsampling process. Subsequently, the feature map is processed by a two-dimensional deconvolution layer with 1024 convolution kernels and a convolution kernel size of (17, 1) without padding, so that the shape of the feature map is expanded to 17×375. In order to restore the original temporal length of the signal, the model uses three two-dimensional deconvolution layers to gradually upsample the feature map. The number of convolution kernels is 256, 128, and 64, and the sizes are (1, 3), (1, 5), and (1, 7), respectively, with a step size of (1, 2). Finally, the feature map size is expanded to 17×3000. Finally, a two-dimensional convolution layer with a convolution kernel number of 1 and a size of (1, 1) is used to integrate all the feature maps and output an EEG virtual channel signal with the same temporal length as the input.

[0019] Furthermore, in step 5), the network loss function L is defined as:

[0020] L = αLoss KL +βLoss TMSE +γLoss FMSE (1)

[0021] Where, Loss KL 、Loss TMSE and Loss FMSE are KL divergence loss, time domain mean square error loss and frequency domain mean square error loss respectively, α, β and γ are the weights of the corresponding loss terms respectively;

[0022] KL divergence loss KL is defined as follows:

[0023]

[0024] Where f(x) and g(x) are the two probability distributions P of the random variable x. f and P g The probability density function, x i is the i-th element in x, and N is the number of elements in x;

[0025] Time domain mean square error loss Loss TMSE The definition is as follows:

[0026]

[0027] Where, f j With g j are the intensities of the original signal f and the generated signal g at time j, respectively, and T is the signal length;

[0028] Frequency domain mean square error loss Loss FMSE The definition is as follows:

[0029]

[0030] Where F and G represent the frequency spectra of the original signal f and the generated signal g, respectively, k is the frequency, and M is the highest frequency value;

[0031] The network loss function L includes both the consistency of the original signal and the generated signal in the time domain and the consistency of the two in the frequency domain. Since the mean square error examines the local differences between the two and does not consider the differences in the overall distribution characteristics, the KL divergence loss is introduced;

[0032] Since the network loss function L takes into account both local loss and global loss in the time-frequency domain, it can enhance the model's ability to generate signals.

[0033] The second object of the present invention is achieved through the following technical solution: a system for generating EEG virtual channel signals by fusion of spatiotemporal features, which is used to implement the above-mentioned method for generating EEG virtual channel signals by fusion of spatiotemporal features, comprising:

[0034] A signal acquisition module is used to acquire multi-channel EEG signals and low-channel EEG signals; wherein, a non-portable multi-channel EEG detector is used to acquire multi-channel EEG signals from multiple brain regions of the human body, or multi-channel EEG signals are obtained from a public database; the number of electrodes and the location of brain regions are determined according to needs, and a portable low-channel EEG detector is used to acquire low-channel EEG signals;

[0035] The EEG signal preprocessing module is used to preprocess EEG signals, including denoising, removing baseline drift, standardizing and segmenting all EEG signals; the preprocessed multi-channel EEG signals are randomly divided into training set, validation set and test set according to a predetermined ratio;

[0036] The model design module is used to design an EEG virtual channel signal generation model, which is a multi-layer convolutional neural network. It uses the brain area of ​​a portable few-channel EEG detector as a reference, selects the EEG signal of the corresponding reference brain area as input, and outputs the multi-channel EEG signal; wherein, the front end of the multi-layer convolutional neural network includes a time domain feature extraction module and a space domain feature extraction module, and the back end fuses the two types of features, time domain features and space domain features. This network structure design can not only utilize the time domain features of the EEG signals of each channel, but also obtain the correlation features between the EEG signals of each channel, realizing the fusion of the time and space features of the EEG signals, thereby improving the learning and expression capabilities of the network; defining the network loss function, designing a new composite loss function, integrating the local loss in the time domain and frequency domain and the global loss of distribution consistency, accurately evaluating the consistency between the original signal and the generated signal, guiding network training, and improving the learning effect;

[0037] The optimal model generation module uses the training set to train the designed EEG virtual channel signal generation model, uses the validation set to evaluate the model performance and optimize the parameters, and uses the test set to evaluate the model performance to obtain the model with the best performance;

[0038] The signal generation module is used to input the few-channel EEG signal into the model with the best performance to generate EEG virtual channel signals.

[0039] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0040] 1. A new convolutional neural network architecture was designed, comprising a temporal and spatial feature extraction module. This architecture not only utilizes the temporal characteristics of each EEG channel but also captures the correlation features between channels, thus achieving a fusion of the temporal and spatial features of EEG signals. The features obtained are more comprehensive and effective than those extracted from a single domain, thereby improving the network's learning and expression capabilities.

[0041] 2. The designed convolutional neural network includes a time domain feature extraction module and a spatial domain feature module. The connection methods of the time domain feature extraction module and the spatial domain feature module are flexible and diverse, and can be connected in series or in parallel. In addition, other modules can be added to the structure, such as time domain long short-term memory, time domain and spatial domain attention mechanism, etc.

[0042] 3. A new composite loss function is designed, which comprehensively considers local losses in the time domain and frequency domain, as well as global loss of distribution consistency. This loss function can more accurately and effectively evaluate the consistency between the original signal and the generated signal, guide network training, further enhance the network's learning ability, and improve network performance.

[0043] 4. Compared with common non-portable multi-channel EEG detectors, the present invention only needs to collect EEG signals from a small number of channels to effectively generate EEG virtual channel signals. This significantly reduces the complexity and cost of hardware design, makes it easier to wear, improves wearing comfort and experience, speeds up acquisition time, and effectively solves the problem of insufficient information collected by portable small-channel EEG signal detectors. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Flowchart of the method of the present invention.

[0045] Figure 2 This figure shows the structure of a multi-layer convolutional neural network in which the time domain and spatial domain feature extraction modules are connected in parallel in an embodiment of the present invention. In the figure, Conv2D represents a two-dimensional convolutional layer, BiLSTM represents a bidirectional long short-term memory network (LSTM) layer, and Conv2Dtranspose represents a two-dimensional deconvolution layer.

[0046] Figure 3 Schematic diagram of the relevant brain regions involved in the embodiments of the present invention, with reference to the international 10-20 system; in the figure, the solid circle represents the brain region where the EEG signal of the portable low-channel EEG detector is collected, and the dotted circle represents the brain region where the EEG virtual channel signal is generated.

[0047] Figure 4 This is a partial comparison diagram of the original EEG channel signals and the generated EEG virtual channel signals in the test set of an embodiment of the present invention; in the figure, from top to bottom are partial EEG signals of brain regions such as F3, FZ, F4, C3, Cz, C4, T5, P3, Pz, P4 and T6.

[0048] Figure 5 This is a schematic diagram of quantitative evaluation parameters for the difference between the generated EEG virtual channel signal and the original EEG channel signal in the test set of an embodiment of the present invention, including the mean Pearson correlation coefficient and the mean square error.

[0049] Figure 6 This is a schematic diagram of the few-channel EEG signals collected by the portable few-channel EEG detector and the generated EEG virtual channel signals in an embodiment of the present invention; in the figure, from top to bottom are partial real signals of six brain regions, including forehead FP1, FP2, F7, F8, T3 and T4, as well as partial EEG signals of 11 brain regions, including F3, Fz, F4, C3, Cz, C4, T5, P3, Pz, P4 and T6, generated by the model.

[0050] Figure 7 This is an architectural diagram of the system of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0052] Example 1

[0053] like Figure 1 As shown, this embodiment discloses a method for generating an EEG virtual channel signal by fusion of spatiotemporal features, comprising the following steps:

[0054] 1) Multi-channel EEG signal acquisition: A non-portable 66-channel EEG detector (Synamps2) was used to collect multi-channel EEG signals from 66 brain regions, including FP1, FP2, and F3, from 119 people. The acquisition time for each person was 60 seconds, and the sampling frequency was set to 500 Hz. Figure 3 The 17 channels shown are used as multi-channel EEG signals; for low-channel EEG signal collection, six brain regions, namely forehead FP1, FP2, F7, F8, T3 and T4, are selected, and a portable 6-channel EEG detector is used to attach detection electrodes to the corresponding 6 brain regions to collect the corresponding 6-channel EEG signals.

[0055] 2) EEG signal preprocessing: First, a bandpass filter with a passband of 0.5 Hz to 40 Hz is designed to extract the main frequency components of the EEG signal and remove mains interference and high-frequency interference; then, baseline drift removal and signal normalization are performed; finally, the signal is segmented at 6-second time intervals.

[0056] 3) The multi-channel EEG signals are randomly divided into training set, validation set and test set in a ratio of 8:1:1.

[0057] 4) Design an EEG virtual channel signal generation model, which is a multi-layer convolutional neural network. Figure 2The network structure of the model adopts a parallel dual-branch structure, which is used to extract the time domain features and spatial domain features of the electroencephalogram (EEG) signal respectively. One branch is the time domain feature extraction module, and the other branch is the spatial domain feature extraction module. The time domain feature extraction module consists of three two-dimensional convolutional layers and a bidirectional LSTM layer, which is designed to extract the time domain features of the EEG signal. The input EEG signal is a 6×3000 size. It first passes through three two-dimensional convolutional layers with the number of convolution kernels being 64, 128, and 256 respectively. The convolution kernel sizes are (1,7), (1,5), and (1,3), and the stride length is (1,2). Deep temporal features are extracted from different scales, and the sequence length is gradually downsampled from 3000 to 375, resulting in 256 feature maps of size 6×375. Subsequently, through dimensionality rescaling, the three-dimensional vector (6×375×256) is converted to a two-dimensional vector (375×1536), representing 375 time points, each containing 1536 features. This is then fed into a bidirectional LSTM layer to further extract temporal features. By simultaneously learning both forward and backward information about the time series, the bidirectional LSTM layer can better capture temporal dependencies in sequential data. The number of features in the hidden layer is set to 256, and the output features for each time point are 512 (256 units each in the forward and backward LSTM layers), resulting in an output dimension of 375×512. To facilitate fusion with subsequent spatial features, the output of the bidirectional LSTM layer is converted to 1×375×512, outputting 512 feature maps of size 1×375.

[0058] Meanwhile, the spatial domain feature extraction module captures the relationship between EEG signals across different channels through four 2D convolutional layers. The first three 2D convolutional layers have 64, 128, and 256 kernels, respectively, with kernel sizes of (3,3) and strides of (1,2). This gradually extracts local spatial domain features and reduces the temporal dimension. Finally, a 2D convolutional layer with 512 kernels of size (6,1) is used to capture global spatial features across all channels without padding, outputting 512 feature maps of size 1×375.

[0059] Finally, a feature fusion module is designed to fuse temporal and spatial features. Within this module, temporal and spatial features are concatenated along the channel dimension, resulting in 1024 fused feature maps of size 1×375. This fusion strategy allows the model to simultaneously consider both temporal and spatial information of the EEG signal during upsampling. The feature map is then processed through a two-dimensional deconvolution layer with 1024 kernels and a kernel size of (17, 1) without padding, expanding the feature map to 17×375. To restore the original temporal length of the signal, the model progressively upsamples the feature map using three two-dimensional deconvolution layers with kernels of 256, 128, and 64, sizes of (1, 3), (1, 5), and (1, 7), respectively, and a stride of (1, 2), ultimately expanding the feature map size to 17×3000. Finally, all feature maps are integrated through a two-dimensional convolution layer with a convolution kernel of 1 and a size of (1,1), and an EEG virtual channel signal with the same length as the input time series is output.

[0060] The network input is the EEG signals of 6 brain regions, namely FP1, FP2, F7, F8, T3 and T4 in the training set, and the EEG signals of 11 brain regions, namely F3, Fz, F4, C3, Cz, C4, T5, P3, Pz, P4 and T6, as well as the EEG signals of 6 brain regions, a total of 17 brain regions, are used as the network output.

[0061] 5) Define the network loss function and design a new composite loss function that integrates local losses in the time and frequency domains and global loss of distribution consistency to accurately evaluate the consistency between the original signal and the generated signal, guide network training, and improve learning effects;

[0062] Define the network loss function L as:

[0063] L = αLoss KL +βLoss TMSE +γLoss FMSE (1)

[0064] Where, Loss KL 、Loss TMSE and Loss FMSE are KL divergence loss, time domain mean square error loss, and frequency domain mean square error loss, respectively. α, β, and γ are the weights of the corresponding loss terms. In this embodiment, the values ​​of α, β, and γ are set to 0.2, 0.5, and 0.3, respectively.

[0065] KL divergence loss KL is defined as follows:

[0066]

[0067] Where f(x) and g(x) are the two probability distributions P of the random variable x. f and P g The probability density function, x i is the i-th element in x, and N is the number of elements in x;

[0068] Time domain mean square error loss Loss TMSE The definition is as follows:

[0069]

[0070] Where, f j With g j are the intensities of the original signal f and the generated signal g at time j, respectively, and T is the signal length;

[0071] Frequency domain mean square error loss Loss FMSE The definition is as follows:

[0072]

[0073] Where F and G represent the frequency spectra of the original signal f and the generated signal g, respectively, k is the frequency, and M is the highest frequency value;

[0074] The network loss function L includes both the consistency of the original signal and the generated signal in the time domain and the consistency of the two in the frequency domain. Since the mean square error examines the local differences between the two and does not consider the differences in overall distribution characteristics, the KL divergence loss is introduced;

[0075] Since the network loss function L takes into account both local loss and global loss in the time-frequency domain, it can enhance the model's ability to generate signals.

[0076] 6) Model training: The model was trained using the training set, and model performance and parameter optimization were evaluated using the validation set using mean square error (MSE) and Pearson correlation coefficient. For model performance evaluation, the EEG signals from the test set were input into the trained network to generate EEG signals for all channels. The mean square error (MSE) and Pearson correlation coefficient were used to evaluate the difference between the generated EEG signals for each channel and the actual EEG signals for the corresponding channel to obtain the optimal model.

[0077] Figure 4A partial comparison of the original EEG channel signals in the training set and the generated virtual EEG channel signals. From top to bottom, they are the F3, Fz, F4, C3, Cz, C4, T5, P3, Pz, P4, and T6 channels. The solid line represents the real signal, and the dashed line represents the generated signal. The comparison shows that the virtual EEG channel signals are very similar to the real EEG signals, demonstrating that the model can accurately capture the dynamic characteristics of EEG signals. In particular, the generated virtual signals have a high degree of consistency with the real signals in the key feature points and waveform morphology of the frontal and parietal brain regions, indicating that the constructed model can effectively learn the characteristics of real EEG signals.

[0078] Figure 5 Figure 3 shows the mean Pearson correlation coefficient (a) and mean mean square error (b) between the generated EEG virtual channel signals and the original EEG channel signals. The generated EEG virtual channel signals exhibit high correlation with the real signals on multiple channels, particularly on channels F3, Fz, and F4, where the Pearson correlation coefficients exceed 0.9, indicating a high degree of consistency between the generated and real signals. The Pearson correlation coefficients between the real signals and the model-generated virtual signals for seven channels are higher than 0.75, and the mean Pearson correlation coefficient for all virtual channels is 0.75, demonstrating that the model can well simulate the dynamic characteristics of EEG signals for most channels. The mean square error between the real signals and the model-generated virtual signals ranges from 1.47 to 4.52, with a mean of 3.26, indicating that the model can generate EEG virtual channel signals that are very close to the real signals, demonstrating the model's good performance.

[0079] 7) Input the preprocessed few-channel EEG signal into the model with the best performance to generate EEG virtual channel signal;

[0080] Figure 6 Figure 1 shows the few-channel EEG signals and the resulting virtual EEG channel signals. From top to bottom, these are partial real signals from six brain regions: FP1, FP2, F7, F8, T3, and T4, and partial EEG signals from 11 brain regions: F3, FZ, F4, C3, CZ, C4, T5, P3, Pz, P4, and T6, generated by the model.

[0081] Example 2

[0082] This embodiment discloses a system for generating EEG virtual channel signals by fusion of spatiotemporal features, which is used to implement the method for generating EEG virtual channel signals by fusion of spatiotemporal features described in Example 1. Figure 7 As shown, including:

[0083] A signal acquisition module is used to acquire multi-channel EEG signals and low-channel EEG signals; wherein, a non-portable multi-channel EEG detector is used to acquire multi-channel EEG signals from multiple brain regions of the human body, or multi-channel EEG signals are obtained from a public database; the number of electrodes and the location of brain regions are determined according to needs, and a portable low-channel EEG detector is used to acquire low-channel EEG signals;

[0084] The EEG signal preprocessing module is used to preprocess EEG signals, including denoising, removing baseline drift, standardizing and segmenting all EEG signals; the preprocessed multi-channel EEG signals are randomly divided into training set, validation set and test set according to a predetermined ratio;

[0085] The model design module is used to design an EEG virtual channel signal generation model, which is a multi-layer convolutional neural network. It uses the brain area of ​​a portable few-channel EEG detector as a reference, selects the EEG signal of the corresponding reference brain area as input, and outputs the multi-channel EEG signal; wherein, the front end of the multi-layer convolutional neural network includes a time domain feature extraction module and a space domain feature extraction module, and the back end fuses the two types of features, time domain features and space domain features. This network structure design can not only utilize the time domain features of the EEG signals of each channel, but also obtain the correlation features between the EEG signals of each channel, realizing the fusion of the time and space features of the EEG signals, thereby improving the learning and expression capabilities of the network; defining the network loss function, designing a new composite loss function, integrating the local loss in the time domain and frequency domain and the global loss of distribution consistency, accurately evaluating the consistency between the original signal and the generated signal, guiding network training, and improving the learning effect;

[0086] The optimal model generation module uses the training set to train the designed EEG virtual channel signal generation model, uses the validation set to evaluate the model performance and optimize the parameters, and uses the test set to evaluate the model performance to obtain the model with the best performance;

[0087] The signal generation module is used to input the few-channel EEG signal into the model with the best performance to generate EEG virtual channel signals.

[0088] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for generating EEG virtual channel signals by fusion of spatiotemporal features, characterized in that: The following steps are involved: 1) Multi-channel EEG signal acquisition: using a non-portable multi-channel EEG detector to collect multi-channel EEG signals from multiple brain regions of the human body, or obtaining multi-channel EEG signals from a public database; For the collection of few-channel EEG signals, the number of electrodes and the location of brain regions are determined according to the needs, and a portable few-channel EEG detector is used to collect the few-channel EEG signals; 2) EEG signal preprocessing, including denoising, baseline drift removal, standardization, and segmentation of all EEG signals; 3) The preprocessed multi-channel EEG signals are randomly divided into a training set, a validation set, and a test set according to a predetermined ratio; 4) Designing an EEG virtual channel signal generation model, which is a multi-layer convolutional neural network. The model uses the brain area of ​​a portable few-channel EEG detector as a reference. Among the multi-channel EEG signals, the EEG signal of the corresponding reference brain area is selected as input, and the multi-channel EEG signal is used as output. The front end of the multi-layer convolutional neural network includes a time domain feature extraction module and a spatial domain feature extraction module, and the back end fuses the time domain features and spatial domain features. This network structure design can not only utilize the time domain features of the EEG signals of each channel, but also obtain the correlation features between the EEG signals of each channel, realizing the fusion of the spatiotemporal features of the EEG signals, thereby improving the learning and expression capabilities of the network. A multi-layer convolutional neural network is used, which is designed with both a time domain feature extraction module and a spatial domain feature extraction module. The time domain feature extraction module extracts time domain features for each channel of the multi-channel EEG signal input into the network in a multi-layer convolution structure. The spatial domain feature extraction module performs convolution in the channel direction to obtain correlation features between channels. The time domain feature extraction module and the spatial domain feature extraction module adopt two structural designs: serial connection and parallel connection. The serial connection structural design first extracts the time domain features of each channel, then inputs them into the spatial domain feature extraction module, and extracts the spatial domain features by performing convolution operations in the channel direction. The parallel connection structural design refers to designing two independent branches at the network input end, one of which is the time domain feature extraction module and the other is the spatial domain feature extraction module. Finally, a feature fusion module is designed to fuse the two types of features. 5) Define the network loss function and design a new composite loss function that integrates local losses in the time and frequency domains and global loss of distribution consistency to accurately evaluate the consistency between the original signal and the generated signal, guide network training, and improve learning effects; Define the network loss function L as: L=αLoss KL +βLoss TMSE +γLoss FMSE (1) Where, Loss KL 、Loss TMSE and Loss FMSE are KL divergence loss, time domain mean square error loss and frequency domain mean square error loss respectively, α, β and γ are the weights of the corresponding loss terms respectively; KL divergence loss KL is defined as follows: Where f(x) and g(x) are the two probability distributions P of the random variable x. f and P g The probability density function, x i is the i-th element in x, and N is the number of elements in x; Time domain mean square error loss Loss TMSE The definition is as follows: Where, f j With g j are the intensities of the original signal f and the generated signal g at time j, respectively, and T is the signal length; Frequency domain mean square error loss Loss FMSE The definition is as follows: Where F and G represent the frequency spectra of the original signal f and the generated signal g, respectively, k is the frequency, and M is the highest frequency value; The network loss function L includes both the consistency of the original signal and the generated signal in the time domain and the consistency of the two in the frequency domain. Since the mean square error examines the local differences between the two and does not consider the differences in the overall distribution characteristics, the KL divergence loss is introduced; Since the network loss function L considers both local loss in the time-frequency domain and global loss, it can enhance the model's ability to generate signals; 6) Model training: Use the training set to train the designed EEG virtual channel signal generation model, use the validation set to evaluate the model performance and optimize the parameters, and use the test set to evaluate the model performance to obtain the model with the best performance; 7) Input the preprocessed few-channel EEG signal into the model with the best performance to generate EEG virtual channel signal.

2. The method for generating an EEG virtual channel signal by fusion of spatiotemporal features according to claim 1, characterized in that: In step 2), first, according to the frequency range, noise type and time-frequency characteristics of the EEG signal, a corresponding filter is designed to remove noise interference and decompose it into different frequency components; Then, the baseline drift is removed and the signal is normalized; finally, the signal is segmented at predetermined time intervals.

3. The method for generating an EEG virtual channel signal by fusion of spatiotemporal features according to claim 2, characterized in that: The temporal feature extraction module consists of three two-dimensional convolutional layers and a bidirectional LSTM layer. It is designed to extract the temporal features of EEG signals. The input EEG signal is a 6×3000 size. It first passes through three two-dimensional convolutional layers with 64, 128, and 256 convolution kernels, respectively. The convolution kernel sizes are (1,7), (1,5), and (1,3), respectively, with a step size of (1,2). Deep temporal features are extracted from different scales and the sequence length is gradually downsampled from 3000 to 375, resulting in 256 feature maps of size 6×375. Subsequently, through dimensionality adjustment, the three-dimensional vector (6×375×256) is converted into a two-dimensional vector (375×1536), that is, 375 time points, each time point contains 1536 features, and then it is input into the bidirectional LSTM layer to further extract time domain features. The bidirectional LSTM layer can better capture the temporal dependencies in the sequence data by simultaneously learning the forward and reverse information of the time series. The number of features in the hidden layer is set to 256, and the output feature of each time point is 512, that is, 256 units in the forward and backward LSTM layers, and the final output dimension is 375×512; in order to fuse with the subsequent spatial features, the output of the bidirectional LSTM layer is converted to 1×375×512, that is, 512 feature maps of size 1×375 are output; The spatial domain feature extraction module captures the relationship between EEG signals across different channels through four two-dimensional convolutional layers. The first three two-dimensional convolutional layers have 64, 128, and 256 convolution kernels, with kernel sizes of (3,3) and strides of (1,2) to gradually extract local spatial domain features and reduce the temporal dimension. Finally, a two-dimensional convolutional layer with 512 kernels and a kernel size of (6,1) is used to capture global spatial domain features across all channels without padding, outputting 512 feature maps of size 1×375. In the feature fusion module, the temporal and spatial features are concatenated along the channel dimension to obtain 1024 fused feature maps of size 1×375, allowing the model to simultaneously consider the temporal and spatial information of the EEG signal during the upsampling process; Subsequently, the feature map is processed by a two-dimensional deconvolution layer with 1024 convolution kernels and a convolution kernel size of (17, 1) without padding, so that the shape of the feature map is expanded to 17×375; in order to restore the original time series length of the signal, the model uses three two-dimensional deconvolution layers to gradually upsample the feature map, with the number of convolution kernels being 256, 128 and 64, and the sizes being (1, 3), (1, 5) and (1, 7) respectively, and the step size being (1, 2), and finally the size of the feature map is expanded to 17×3000; finally, all feature maps are integrated through a two-dimensional convolution layer with a convolution kernel number of 1 and a size of (1, 1), and the output is an EEG virtual channel signal with the same time series length as the input.

4. A system for generating EEG virtual channel signals by integrating spatiotemporal features, characterized in that: A method for generating an EEG virtual channel signal for realizing the spatiotemporal feature fusion according to any one of claims 1 to 3, comprising: A signal acquisition module is used to acquire multi-channel EEG signals and low-channel EEG signals; wherein, a non-portable multi-channel EEG detector is used to acquire multi-channel EEG signals from multiple brain regions of the human body, or multi-channel EEG signals are obtained from a public database; the number of electrodes and the location of brain regions are determined according to needs, and a portable low-channel EEG detector is used to acquire low-channel EEG signals; The EEG signal preprocessing module is used to preprocess EEG signals, including denoising, removing baseline drift, standardizing and segmenting all EEG signals; the preprocessed multi-channel EEG signals are randomly divided into training set, validation set and test set according to a predetermined ratio; The model design module is used to design an EEG virtual channel signal generation model, which is a multi-layer convolutional neural network. It uses the brain area of ​​a portable few-channel EEG detector as a reference, selects the EEG signal of the corresponding reference brain area as input, and outputs the multi-channel EEG signal; wherein, the front end of the multi-layer convolutional neural network includes a time domain feature extraction module and a space domain feature extraction module, and the back end fuses the two types of features, time domain features and space domain features. This network structure design can not only utilize the time domain features of the EEG signals of each channel, but also obtain the correlation features between the EEG signals of each channel, realizing the fusion of the time and space features of the EEG signals, thereby improving the learning and expression capabilities of the network; defining the network loss function, designing a new composite loss function, integrating the local loss in the time domain and frequency domain and the global loss of distribution consistency, accurately evaluating the consistency between the original signal and the generated signal, guiding network training, and improving the learning effect; The optimal model generation module uses the training set to train the designed EEG virtual channel signal generation model, uses the validation set to evaluate the model performance and optimize the parameters, and uses the test set to evaluate the model performance to obtain the model with the best performance; The signal generation module is used to input the few-channel EEG signal into the model with the best performance to generate EEG virtual channel signals.

Images