Space-time feature fusion electroencephalogram virtual channel signal generation method and system

Abstract

The invention discloses an electroencephalogram virtual channel signal generation method and system based on spatio-temporal feature fusion, a multilayer convolutional neural network with time domain and spatial domain feature fusion and spatio-temporal feature consistency constraint is innovatively designed, and multiple virtual channel electroencephalogram signals are generated from few-channel electroencephalogram signals through transfer learning. The network comprises a time domain feature extraction module and a spatial domain feature extraction module, time domain features of electroencephalogram signals of all channels can be utilized, relevance features between all the channels can be obtained, spatial-temporal feature fusion of the electroencephalogram signals is achieved, and the learning and expression ability of the network is improved. According to the designed composite loss function, time domain and frequency domain local loss and distribution consistency global loss are comprehensively considered, the consistency of an original signal and a generated signal can be evaluated more comprehensively and accurately, network training is guided, and network performance is improved. According to the portable electroencephalogram detector, the problems caused by the fact that the number of brain region signals collected by the portable electroencephalogram detector is small and information is insufficient are effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of human physiological signal detection, and in particular to a method and system for generating an EEG virtual channel signal by fusing temporal and spatial features. Background Art

[0002] Electroencephalogram (EEG) is an important parameter that reflects the state of human brain activity. It can be used for brain development and cognitive ability research, diagnosis of mental illness and brain lesions, and realization of brain-computer interface. In order to obtain EEG signals from different brain regions, most of them refer to the international 10-20 system electrode placement method. According to the needs, 32-128 electrodes (channels) are generally used. If a large number of electrodes are used, the equipment has the disadvantages of large size, complex wearing, long signal detection time and high cost, which limits its application. Therefore, portable few-channel (less than or equal to 8 electrodes) EEG detectors came into being. Although this type of detector can overcome the above shortcomings of non-portable multi-channel EEG detectors, it will have insufficient information because it collects fewer brain regions. How to solve the problem of insufficient EEG signals of portable few-channel EEG detectors is very critical. Summary of the invention

[0003] The first purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a method for generating EEG virtual channel signals with spatiotemporal feature fusion, which effectively solves the problem of small number of brain area signals and insufficient information collected by portable few-channel EEG detectors.

[0004] The second object of the present invention is to provide an EEG virtual channel signal generation system with spatiotemporal feature fusion.

[0005] The first object of the present invention is achieved by the following technical solution: A method for generating an EEG virtual channel signal by fusion of spatiotemporal features, comprising the following steps:

[0006] 1) Multi-channel EEG signal acquisition: use a non-portable multi-channel EEG detector to collect multi-channel EEG signals from multiple brain regions of the human body, or obtain multi-channel EEG signals from a public database; for low-channel EEG signal acquisition, determine the number of electrodes and the location of brain regions according to needs, and use a portable low-channel EEG detector to collect low-channel EEG signals;

[0007] 2) EEG signal preprocessing, including denoising, removing baseline drift, standardization and segmentation of all EEG signals;

[0008] 3) randomly dividing the preprocessed multi-channel EEG signals into a training set, a validation set, and a test set according to a predetermined ratio;

[0009] 4) Designing an EEG virtual channel signal generation model, which is a multi-layer convolutional neural network, which uses the brain area of ​​a portable few-channel EEG detector as a reference, selects the EEG signal of the corresponding reference brain area as input in the multi-channel EEG signal, and uses the multi-channel EEG signal as output; wherein the front end of the multi-layer convolutional neural network includes a time domain feature extraction module and a space domain feature extraction module, and the back end fuses the two types of features, the time domain feature and the space domain feature. This network structure design can not only utilize the time domain features of the EEG signals of each channel, but also obtain the correlation features between the EEG signals of each channel, thereby realizing the fusion of the time and space features of the EEG signals, thereby improving the learning and expression capabilities of the network;

[0010] 5) Define the network loss function and design a new composite loss function that integrates local losses in the time domain and frequency domain as well as global losses in distribution consistency to accurately evaluate the consistency between the original signal and the generated signal, guide network training, and improve learning effects;

[0011] 6) Model training: use the training set to train the designed EEG virtual channel signal generation model, use the validation set to evaluate the model performance and optimize the parameters, and use the test set to evaluate the model performance to obtain the model with the best performance;

[0012] 7) Input the preprocessed few-channel EEG signal into the model with the best performance to generate EEG virtual channel signal.

[0013] Furthermore, in step 2), firstly, a corresponding filter is designed according to the frequency range, noise type and time-frequency characteristics of the EEG signal to remove noise interference and decompose it into different frequency components; then, the baseline drift is removed and the signal is standardized; finally, the signal is segmented at predetermined time intervals.

[0014] Further, in step 4), a multi-layer convolutional neural network is used, and the network is designed with a time domain feature extraction module and a space domain feature extraction module; wherein the time domain feature extraction module extracts time domain features for each channel of the multi-channel EEG signal input into the network in a multi-layer convolution structure; and the space domain feature extraction module performs convolution in the channel direction to obtain correlation features between channels;

[0015] The time domain feature extraction module and the space domain feature extraction module adopt two structural designs: serial connection and parallel connection. Among them, the structural design of the serial connection is to first extract the time domain features of each channel, and then input them into the space domain feature extraction module, and extract the space domain features by performing convolution operation in the channel direction; the structural design of the parallel connection refers to designing two independent branches at the network input end, one of which is the time domain feature extraction module, and the other is the space domain feature extraction module; finally, a feature fusion module is designed to fuse the two types of features.

[0016] Furthermore, the time domain feature extraction module is composed of three two-dimensional convolutional layers and one bidirectional LSTM layer, which is designed to extract the time domain features of the EEG signal. The EEG signal with a size of 6×3000 is input. First, it passes through three two-dimensional convolutional layers, with the number of convolution kernels being 64, 128 and 256 respectively, the sizes of convolution kernels being (1,7), (1,5) and (1,3) respectively, and the step sizes being (1,2). The deep temporal features are extracted from different scales, and the sequence length is gradually downsampled from 3000 to 375, so as to obtain 256 feature maps with a size of 6×375. Subsequently, the three-dimensional vector (6×375×256) is converted into a two-dimensional vector (3 75×1536), that is, 375 time points, each time point contains 1536 features, which are then input into the bidirectional LSTM layer to further extract time domain features. The bidirectional LSTM layer can better capture the temporal dependencies in the sequence data by simultaneously learning the forward and reverse information of the time series. The number of features in the hidden layer is set to 256, and the output feature of each time point is 512, that is, 256 units in the forward and backward LSTM layers, and the final output dimension is 375×512; in order to merge with the subsequent spatial features, the output of the bidirectional LSTM layer is converted to 1×375×512, that is, 512 feature maps of size 1×375 are output;

[0017] The spatial domain feature extraction module captures the relationship between EEG signals of different channels through four two-dimensional convolutional layers. The number of convolution kernels of the first three two-dimensional convolutional layers is 64, 128 and 256 respectively, the size of the convolution kernel is (3,3), and the step size is (1,2) to gradually extract local spatial domain features and reduce the temporal dimension. Finally, a two-dimensional convolutional layer with a convolution kernel number of 512 and a size of (6,1) is used to capture global spatial domain features across all channels without padding, and 512 feature maps of size 1×375 are output.

[0018] In the feature fusion module, the time domain features and the spatial domain features are concatenated along the channel dimension to obtain 1024 fused feature maps of size 1×375, so that the model can simultaneously consider the temporal and spatial information of the EEG signal during the upsampling process; then, the feature map is processed by a two-dimensional deconvolution layer with 1024 convolution kernels and a convolution kernel size of (17,1) without padding, so that the shape of the feature map is expanded to 17×375; in order to restore the original temporal length of the signal, the model uses three two-dimensional deconvolution layers to gradually upsample the feature map, with the number of convolution kernels being 256, 128 and 64, and the sizes being (1,3), (1,5) and (1,7) respectively, and the step size being (1,2), and finally the size of the feature map is expanded to 17×3000; finally, a two-dimensional convolution layer with a convolution kernel number of 1 and a size of (1,1) is used to integrate all the feature maps and output an EEG virtual channel signal with the same temporal length as the input.

[0019] Further, in step 5), the network loss function L is defined as:

[0020] L=αLoss KL +βLoss TMSE +γLoss FMSE (1)

[0021] In the formula, Loss KL 、Loss TMSE and Loss FMSE They are KL divergence loss, time domain mean square error loss and frequency domain mean square error loss respectively, α, β and γ are the weights of the corresponding loss terms respectively;

[0022] KL divergence loss KL is defined as follows:

[0023]

[0024] In the formula, f9x) and g(x) are the two probability distributions P of the random variable x. f and P g The probability density function, x i is the i-th element in x, and N is the number of elements in x;

[0025] Time domain mean square error loss Loss TMSE The definition is as follows:

[0026]

[0027] In the formula, f j With g j are the strengths of the original signal f and the generated signal g at time j, respectively, and T is the signal length;

[0028] Frequency domain mean square error loss Loss FMSE The definition is as follows:

[0029]

[0030] In the formula, F and G represent the frequency spectrum of the original signal f and the generated signal g respectively, k is the frequency, and M is the highest frequency value;

[0031] The network loss function L includes both the consistency of the original signal and the generated signal in the time domain and the consistency of the two in the frequency domain. Since the mean square error examines the local differences between the two and does not consider the differences in overall distribution characteristics, the KL divergence loss is introduced;

[0032] Since the network loss function L takes into account both the local loss in the time-frequency domain and the global loss, it can enhance the model's ability to generate signals.

[0033] The second object of the present invention is achieved by the following technical solution: a system for generating an EEG virtual channel signal with spatiotemporal feature fusion, which is used to implement the above-mentioned method for generating an EEG virtual channel signal with spatiotemporal feature fusion, comprising:

[0034] A signal acquisition module is used to acquire multi-channel EEG signals and few-channel EEG signals; wherein a non-portable multi-channel EEG detector is used to acquire multi-channel EEG signals of multiple brain regions of the human body, or multi-channel EEG signals are acquired from a public database; the number of electrodes and the location of brain regions are determined according to needs, and a portable few-channel EEG detector is used to acquire few-channel EEG signals;

[0035] The EEG signal preprocessing module is used to preprocess the EEG signals, including denoising, removing baseline drift, standardizing and segmenting all EEG signals; the preprocessed multi-channel EEG signals are randomly divided into a training set, a validation set and a test set according to a predetermined ratio;

[0036] A model design module is used to design an EEG virtual channel signal generation model, which is a multi-layer convolutional neural network. The model uses the brain area of ​​a portable few-channel EEG detector as a reference, selects the EEG signal of the corresponding reference brain area as input, and uses the multi-channel EEG signal as output; wherein the front end of the multi-layer convolutional neural network includes a time domain feature extraction module and a space domain feature extraction module, and the back end fuses the two types of features, the time domain feature and the space domain feature. This network structure design can not only utilize the time domain features of the EEG signals of each channel, but also obtain the correlation features between the EEG signals of each channel, thereby realizing the fusion of the time and space features of the EEG signals, thereby improving the learning and expression capabilities of the network; defining the network loss function, designing a new composite loss function, integrating the local loss in the time domain and frequency domain and the global loss of distribution consistency, accurately evaluating the consistency between the original signal and the generated signal, guiding network training, and improving the learning effect;

[0037] The optimal model generation module uses the training set to train the designed EEG virtual channel signal generation model, uses the validation set to evaluate the model performance and optimize the parameters, and uses the test set to evaluate the model performance to obtain the model with the best performance;

[0038] The signal generation module is used to input the few-channel EEG signal into the model with the best performance to generate EEG virtual channel signal.

[0039] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0040] 1. A new convolutional neural network structure is designed, including a time domain feature extraction module and a spatial domain feature extraction module. This structure can not only utilize the time domain features of the EEG signals of each channel, but also obtain the correlation features between the channels, that is, it realizes the fusion of the time and space features of the EEG signals. The features obtained are more comprehensive and effective than the single domain features, thereby improving the learning and expression capabilities of the network.

[0041] 2. The designed convolutional neural network includes a time domain feature extraction module and a spatial domain feature module. The connection method between the time domain feature extraction module and the spatial domain feature module is flexible and diverse, and can be connected in series or in parallel. In addition, other modules can be added to the structure, such as time domain long short-term memory, time domain and spatial domain attention mechanism, etc.

[0042] 3. A new composite loss function is designed, which comprehensively considers the local losses in the time domain and frequency domain, as well as the global loss of distribution consistency. This loss function can more accurately and effectively evaluate the consistency between the original signal and the generated signal, guide the network training, further enhance the network's learning ability, and improve network performance.

[0043] 4. Compared with common non-portable multi-channel EEG detectors, the present invention only needs to collect EEG signals from a small number of channels to effectively generate EEG virtual channel signals. This not only significantly reduces the complexity and cost of hardware design, is easy to wear, improves wearing comfort and experience, speeds up the acquisition time, but also effectively solves the problem of insufficient information collection by portable small-channel EEG signal detectors. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Flow chart of the method of the present invention.

[0045] Figure 2 It is the structure of a multi-layer convolutional neural network in which the time domain and spatial domain feature extraction modules are connected in parallel in an embodiment of the present invention; in the figure, Conv2D represents a two-dimensional convolutional layer, BiLSTM represents a bidirectional long short-term memory network (Long Short-Term Memory, LSTM) layer, and Conv2Dtranspose represents a two-dimensional deconvolution layer.

[0046] Figure 3 With reference to the international 10-20 system, a schematic diagram of the relevant brain regions involved in the embodiments of the present invention is shown; in the figure, the solid circle is the brain region for collecting EEG signals of the portable few-channel EEG detector, and the dotted circle is the brain region for generating EEG virtual channel signals.

[0047] Figure 4 This is a partial comparison diagram of the original EEG channel signals and the generated EEG virtual channel signals in the test set of an embodiment of the present invention; in the figure, from top to bottom are partial EEG signals of brain regions such as F3, FZ, F4, C3, Cz, C4, T5, P3, Pz, P4 and T6.

[0048] Figure 5 This is a schematic diagram of quantitative evaluation parameters for the difference between the generated EEG virtual channel signal and the original EEG channel signal in the test set of an embodiment of the present invention, including the mean of the Pearson correlation coefficient and the mean of the mean square error.

[0049] Figure 6 This is a schematic diagram of the few-channel EEG signals collected by the portable few-channel EEG detector and the generated EEG virtual channel signals in an embodiment of the present invention; in the figure, from top to bottom are partial real signals of six brain regions, namely forehead FP1, FP2, F7, F8, T3 and T4, and partial EEG signals of 11 brain regions, namely F3, Fz, F4, C3, Cz, C4, T5, P3, Pz, P4 and T6, generated by the model.

[0050] Figure 7 Schematic diagram of the system of the present invention. DETAILED DESCRIPTION

[0051] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0052] Example 1

[0053] like Figure 1 As shown, this embodiment discloses a method for generating an EEG virtual channel signal by fusion of spatiotemporal features, comprising the following steps:

[0054] 1) Multi-channel EEG signal acquisition: A non-portable 66-channel EEG detector (Synamps2) was used to collect multi-channel EEG signals from FP1, FP2, and F3 of 119 people, with a total of 66 brain regions. The acquisition time for each person was 60 seconds, and the sampling frequency was set to 500 Hz. Figure 3 The 17 channels shown are used as multi-channel EEG signals; for low-channel EEG signal collection, six brain regions, namely, forehead FP1, FP2, F7, F8, T3 and T4, are selected, and a portable 6-channel EEG detector is used to attach detection electrodes to the corresponding 6 brain regions to collect corresponding 6-channel EEG signals.

[0055] 2) EEG signal preprocessing: first design a bandpass filter with a passband of 0.5 Hz to 40 Hz to extract the main frequency components of the EEG signal and remove mains interference and high-frequency interference; then, perform baseline drift removal and signal standardization processing; finally, segment the signal at 6-second time intervals.

[0056] 3) The multi-channel EEG signals are randomly divided into training set, validation set and test set in a ratio of 8:1:1.

[0057] 4) Design a virtual EEG channel signal generation model, which is a multi-layer convolutional neural network. Figure 2The network structure of the model adopts a parallel dual-branch structure, which is used to extract the time domain features and spatial domain features of the electroencephalogram (EEG) signal respectively. One branch is the time domain feature extraction module, and the other branch is the spatial domain feature extraction module. The time domain feature extraction module consists of three two-dimensional convolutional layers and a bidirectional LSTM layer, which is designed to extract the time domain features of the EEG signal. The input EEG signal of size 6×3000 first passes through three two-dimensional convolutional layers, with the number of convolution kernels being 64, 128, and 256 respectively, and the convolution kernel sizes being (1,7), (1,5), and (1,3), respectively, with the step size being (1,2). Deep time series features are extracted from different scales, and the sequence length is gradually downsampled from 3000 to 375, resulting in 256 feature maps of size 6×375. Subsequently, through dimensional adjustment, the three-dimensional vector (6×375×256) is converted into a two-dimensional vector (375×1536), that is, 375 time points, each containing 1536 features. It is then input into the bidirectional LSTM layer to further extract time domain features. The bidirectional LSTM layer can better capture the temporal dependencies in sequence data by simultaneously learning the forward and reverse information of the time series. Among them, the number of features in the hidden layer is set to 256, the output features of each time point are 512 (256 units in the forward and backward LSTM layers), and the final output dimension is 375×512. In order to merge with the subsequent spatial features, the output of the bidirectional LSTM layer is converted to 1×375×512, that is, 512 feature maps of size 1×375 are output.

[0058] At the same time, the spatial domain feature extraction module captures the relationship between EEG signals between different channels through four two-dimensional convolutional layers. The number of convolution kernels in the first three two-dimensional convolutional layers is 64, 128, and 256 respectively, the convolution kernel size is (3,3), and the step size is (1,2) to gradually extract local spatial domain features and reduce the temporal dimension. Finally, through a two-dimensional convolutional layer with 512 convolution kernels and a size of (6,1), global spatial features are captured across all channels without padding, and 512 feature maps of size 1×375 are output.

[0059] Finally, a feature fusion module is designed to fuse the two types of features, time domain features and spatial domain features. In the feature fusion module, the time domain features and spatial domain features are spliced ​​along the channel dimension to obtain 1024 fused feature maps of size 1×375. Through this fusion strategy, the model can simultaneously consider the temporal and spatial information of the EEG signal during the upsampling process. Subsequently, the feature map is processed by a two-dimensional deconvolution layer with 1024 convolution kernels and a convolution kernel size of (17,1) without padding, so that the shape of the feature map is expanded to 17×375. In order to restore the original temporal length of the signal, the model uses three two-dimensional deconvolution layers to gradually upsample the feature map. The number of convolution kernels is 256, 128, and 64, and the sizes are (1,3), (1,5), and (1,7), respectively. The step size is (1,2), and the feature map size is finally expanded to 17×3000. Finally, all feature maps are integrated through a two-dimensional convolution layer with a convolution kernel of 1 and a size of (1,1), and an EEG virtual channel signal with the same time series length as the input is output.

[0060] The network input is the EEG signals of 6 brain regions, namely FP1, FP2, F7, F8, T3 and T4 in the training set, and the EEG signals of 11 brain regions, namely F3, Fz, F4, C3, Cz, C4, T5, P3, Pz, P4 and T6, as well as the EEG signals of 6 brain regions, a total of 17 brain regions, are used as the network output.

[0061] 5) Define the network loss function and design a new composite loss function that integrates local losses in the time domain and frequency domain as well as global losses in distribution consistency to accurately evaluate the consistency between the original signal and the generated signal, guide network training, and improve learning effects;

[0062] Define the network loss function L as:

[0063] L=αLoss KL +βLoss TMSE +γLoss FMSE (1)

[0064] In the formula, Loss KL 、Loss TMSE and Loss FMSE are KL divergence loss, time domain mean square error loss and frequency domain mean square error loss respectively, α, β and γ are weights of corresponding loss terms respectively, in this embodiment, the values ​​of α, β and γ are set to 0.2, 0.5 and 0.3 respectively;

[0065] KL divergence loss KL is defined as follows:

[0066]

[0067] In the formula, f9x) and g(x) are the two probability distributions P of the random variable x. f and P g The probability density function, x i is the i-th element in x, and N is the number of elements in x;

[0068] Time domain mean square error loss Loss TMSE The definition is as follows:

[0069]

[0070] In the formula, f j With g j are the strengths of the original signal f and the generated signal g at time j, respectively, and T is the signal length;

[0071] Frequency domain mean square error loss Loss FMSE The definition is as follows:

[0072]

[0073] In the formula, F and G represent the frequency spectrum of the original signal f and the generated signal g respectively, k is the frequency, and M is the highest frequency value;

[0074] The network loss function L includes both the consistency of the original signal and the generated signal in the time domain and the consistency of the two in the frequency domain. Since the mean square error examines the local differences between the two and does not consider the differences in overall distribution characteristics, the KL divergence loss is introduced;

[0075] Since the network loss function L takes into account both the local loss in the time-frequency domain and the global loss, it can enhance the model's ability to generate signals.

[0076] 6) Model training: Use the training set to train the model, use the mean square error and Pearson correlation coefficient, and use the validation set to evaluate the model performance and optimize the parameters; Model performance evaluation: Input the EEG signals in the test set into the trained network to generate EEG signals of all channels. Use the mean square error and Pearson correlation coefficient to evaluate the difference between the generated EEG signals of each channel and the actual EEG signals of the corresponding channel to obtain the model with the best performance.

[0077] Figure 4Partial comparison between the original EEG channel signals in the training set and the generated EEG virtual channel signals. From top to bottom, they are F3, Fz, F4, C3, Cz, C4, T5, P3, Pz, P4 and T6 channels, where the solid line is the real signal and the dotted line is the generated signal. By comparison, it can be seen that the EEG virtual channel signals are very similar to the real EEG signals, indicating that the model can accurately capture the dynamic characteristics of the EEG signals. In particular, the key feature points and waveform shapes in the frontal and parietal brain area signals, the generated virtual signals are highly consistent with the real signals, indicating that the constructed model can effectively learn the characteristics of the real EEG signals.

[0078] Figure 5 The mean Pearson correlation coefficient (a) and mean square error (b) between the generated EEG virtual channel signal and the original EEG channel signal. It can be seen that the generated EEG virtual channel signal shows a high correlation with the real signal on multiple channels, especially in the F3, Fz and F4 channels, the Pearson correlation coefficient is more than 0.9, indicating that the generated signal is highly consistent with the real signal. Among them, the Pearson correlation coefficient between the real signal of 7 channels and the virtual signal generated by the model is higher than 0.75, and the mean Pearson correlation coefficient of all virtual channels is 0.75, which proves that the model can simulate the dynamic change characteristics of EEG signals of most channels well. The mean square error between the real signal and the virtual signal generated by the model ranges from 1.47 to 4.52, with a mean of 3.26, indicating that the model can generate EEG virtual channel signals that are very close to the real signal, reflecting the good performance of the model.

[0079] 7) Inputting the preprocessed few-channel EEG signal into the model with the best performance to generate EEG virtual channel signal;

[0080] Figure 6 The following are the few-channel EEG signals and the final generated EEG virtual channel signals. From top to bottom, they are the partial real signals of the six brain regions, namely the forehead FP1, FP2, F7, F8, T3 and T4, and the partial EEG signals of the 11 brain regions, namely F3, FZ, F4, C3, CZ, C4, T5, P3, Pz, P4 and T6, generated by the model.

[0081] Example 2

[0082] This embodiment discloses a system for generating a virtual EEG channel signal by fusion of spatiotemporal features, which is used to implement the method for generating a virtual EEG channel signal by fusion of spatiotemporal features described in Embodiment 1. Figure 7 As shown, including:

[0083] A signal acquisition module is used to acquire multi-channel EEG signals and few-channel EEG signals; wherein a non-portable multi-channel EEG detector is used to acquire multi-channel EEG signals of multiple brain regions of the human body, or multi-channel EEG signals are acquired from a public database; the number of electrodes and the location of brain regions are determined according to needs, and a portable few-channel EEG detector is used to acquire few-channel EEG signals;

[0084] The EEG signal preprocessing module is used to preprocess the EEG signals, including denoising, removing baseline drift, standardizing and segmenting all EEG signals; the preprocessed multi-channel EEG signals are randomly divided into a training set, a validation set and a test set according to a predetermined ratio;

[0085] A model design module is used to design an EEG virtual channel signal generation model, which is a multi-layer convolutional neural network. The model uses the brain area of ​​a portable few-channel EEG detector as a reference, selects the EEG signal of the corresponding reference brain area as input, and uses the multi-channel EEG signal as output; wherein the front end of the multi-layer convolutional neural network includes a time domain feature extraction module and a space domain feature extraction module, and the back end fuses the two types of features, the time domain feature and the space domain feature. This network structure design can not only utilize the time domain features of the EEG signals of each channel, but also obtain the correlation features between the EEG signals of each channel, thereby realizing the fusion of the time and space features of the EEG signals, thereby improving the learning and expression capabilities of the network; defining the network loss function, designing a new composite loss function, integrating the local loss in the time domain and frequency domain and the global loss of distribution consistency, accurately evaluating the consistency between the original signal and the generated signal, guiding network training, and improving the learning effect;

[0086] The optimal model generation module uses the training set to train the designed EEG virtual channel signal generation model, uses the validation set to evaluate the model performance and optimize the parameters, and uses the test set to evaluate the model performance to obtain the model with the best performance;

[0087] The signal generation module is used to input the few-channel EEG signal into the model with the best performance to generate EEG virtual channel signal.

[0088] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for generating EEG virtual channel signals by fusion of spatiotemporal features, characterized in that: The following steps are involved: 1) Multi-channel EEG signal acquisition: using a non-portable multi-channel EEG detector to collect multi-channel EEG signals from multiple brain regions of the human body, or obtaining multi-channel EEG signals from a public database; For the collection of few-channel EEG signals, the number of electrodes and the location of brain regions are determined according to the needs, and a portable few-channel EEG detector is used to collect few-channel EEG signals; 2) EEG signal preprocessing, including denoising, removing baseline drift, standardization and segmentation of all EEG signals; 3) randomly dividing the preprocessed multi-channel EEG signals into a training set, a validation set, and a test set according to a predetermined ratio; 4) Designing an EEG virtual channel signal generation model, which is a multi-layer convolutional neural network, which uses the brain area of ​​a portable few-channel EEG detector as a reference, selects the EEG signal of the corresponding reference brain area as input in the multi-channel EEG signal, and uses the multi-channel EEG signal as output; wherein the front end of the multi-layer convolutional neural network includes a time domain feature extraction module and a space domain feature extraction module, and the back end fuses the two types of features, the time domain feature and the space domain feature. This network structure design can not only utilize the time domain features of the EEG signals of each channel, but also obtain the correlation features between the EEG signals of each channel, thereby realizing the fusion of the time and space features of the EEG signals, thereby improving the learning and expression capabilities of the network; 5) Define the network loss function and design a new composite loss function that integrates local losses in the time domain and frequency domain as well as global losses in distribution consistency to accurately evaluate the consistency between the original signal and the generated signal, guide network training, and improve learning effects; 6) Model training: use the training set to train the designed EEG virtual channel signal generation model, use the validation set to evaluate the model performance and optimize the parameters, and use the test set to evaluate the model performance to obtain the model with the best performance; 7) Input the preprocessed few-channel EEG signal into the model with the best performance to generate EEG virtual channel signal.

2. The method for generating an EEG virtual channel signal by fusion of spatiotemporal features according to claim 1, characterized in that: In step 2), firstly, according to the frequency range, noise type and time-frequency characteristics of the EEG signal, a corresponding filter is designed to remove noise interference and decompose it into different frequency components; Then, the baseline drift is removed and the signal is normalized; finally, the signal is segmented at predetermined time intervals.

3. The method for generating an EEG virtual channel signal by fusion of spatiotemporal features according to claim 1, characterized in that: In step 4), a multi-layer convolutional neural network is used, and the network is designed with a time domain feature extraction module and a space domain feature extraction module; wherein the time domain feature extraction module extracts time domain features for each channel of the multi-channel EEG signal input into the network in a multi-layer convolution structure; and the space domain feature extraction module performs convolution in the channel direction to obtain correlation features between channels; The time domain feature extraction module and the space domain feature extraction module adopt two structural designs: serial connection and parallel connection. Among them, the structural design of the serial connection is to first extract the time domain features of each channel, and then input them into the space domain feature extraction module, and extract the space domain features by performing convolution operation in the channel direction; the structural design of the parallel connection refers to designing two independent branches at the network input end, one of which is the time domain feature extraction module, and the other is the space domain feature extraction module; finally, a feature fusion module is designed to fuse the two types of features.

4. The method for generating an EEG virtual channel signal by fusion of spatiotemporal features according to claim 3, characterized in that: The time domain feature extraction module consists of three two-dimensional convolutional layers and one bidirectional LSTM layer, and is designed to extract the time domain features of EEG signals. The input is an EEG signal of size 6×3000. First, it passes through three two-dimensional convolutional layers, with the number of convolution kernels being 64, 128, and 256, respectively, and the convolution kernel sizes being (1,7), (1,5), and (1,3), respectively, with the step length being (1,2). It extracts deep temporal features from different scales, and gradually downsamples the sequence length from 3000 to 375, obtaining 256 feature maps of size 6×375. Subsequently, through dimensional adjustment, the three-dimensional vector (6×375×256) is converted into a two-dimensional vector (375×1536), that is, 375 time points, each time point contains 1536 features, and then it is input into the bidirectional LSTM layer to further extract time domain features. The bidirectional LSTM layer can better capture the temporal dependencies in the sequence data by simultaneously learning the forward and reverse information of the time series. The number of features in the hidden layer is set to 256, and the output feature of each time point is 512, that is, 256 units in the forward and backward LSTM layers, and the final output dimension is 375×512; in order to merge with the subsequent spatial features, the output of the bidirectional LSTM layer is converted to 1×375×512, that is, 512 feature maps of size 1×375 are output; The spatial domain feature extraction module captures the relationship between EEG signals of different channels through four two-dimensional convolutional layers. The number of convolution kernels of the first three two-dimensional convolutional layers is 64, 128 and 256 respectively, the size of the convolution kernel is (3,3), and the step size is (1,2) to gradually extract local spatial domain features and reduce the temporal dimension. Finally, a two-dimensional convolutional layer with a convolution kernel number of 512 and a size of (6,1) is used to capture global spatial domain features across all channels without padding, and 512 feature maps of size 1×375 are output. In the feature fusion module, the time domain features and the spatial domain features are spliced ​​along the channel dimension to obtain 1024 fused feature maps of size 1×375, so that the model can simultaneously consider the temporal and spatial information of the EEG signal during the upsampling process; Subsequently, the feature map is processed by a two-dimensional deconvolution layer with 1024 convolution kernels and a convolution kernel size of (17, 1) without padding, so that the shape of the feature map is expanded to 17×375. In order to restore the original time series length of the signal, the model uses three two-dimensional deconvolution layers to gradually upsample the feature map. The number of convolution kernels is 256, 128 and 64, the size is (1, 3), (1, 5) and (1, 7) respectively, and the step size is (1, 2). Finally, the size of the feature map is expanded to 17×3000. Finally, a two-dimensional convolution layer with a convolution kernel number of 1 and a size of (1, 1) is used to integrate all feature maps and output an EEG virtual channel signal with the same time series length as the input.

5. The method for generating an EEG virtual channel signal by fusion of spatiotemporal features according to claim 1, characterized in that: In step 5), the network loss function L is defined as: L=αLoss KL +βLoss TMSE +γLoss FMSE (1) In the formula, Loss KL 、Loss TMSE and Loss FMSE They are KL divergence loss, time domain mean square error loss and frequency domain mean square error loss respectively, α, β and γ are the weights of the corresponding loss terms respectively; KL divergence loss KL is defined as follows: In the formula, f(x) and g(x) are the two probability distributions P of the random variable x. f and P g The probability density function, x i is the i-th element in x, and N is the number of elements in x; Time domain mean square error loss Loss TMSE The definition is as follows: In the formula, f j With g j are the strengths of the original signal f and the generated signal g at time j, respectively, and T is the signal length; Frequency domain mean square error loss Loss FMSE The definition is as follows: In the formula, F and G represent the frequency spectrum of the original signal f and the generated signal g respectively, k is the frequency, and M is the highest frequency value; The network loss function L includes both the consistency of the original signal and the generated signal in the time domain and the consistency of the two in the frequency domain. Since the mean square error examines the local differences between the two and does not consider the differences in overall distribution characteristics, the KL divergence loss is introduced; Since the network loss function L takes into account both the local loss in the time-frequency domain and the global loss, it can enhance the model's ability to generate signals.

6. A system for generating EEG virtual channel signals by integrating spatiotemporal features, characterized in that: A method for generating an EEG virtual channel signal for realizing the spatiotemporal feature fusion according to any one of claims 1 to 5, comprising: A signal acquisition module is used to acquire multi-channel EEG signals and few-channel EEG signals; wherein a non-portable multi-channel EEG detector is used to acquire multi-channel EEG signals of multiple brain regions of the human body, or multi-channel EEG signals are acquired from a public database; the number of electrodes and the location of brain regions are determined according to needs, and a portable few-channel EEG detector is used to acquire few-channel EEG signals; The EEG signal preprocessing module is used to preprocess the EEG signals, including denoising, removing baseline drift, standardizing and segmenting all EEG signals; the preprocessed multi-channel EEG signals are randomly divided into a training set, a validation set and a test set according to a predetermined ratio; A model design module is used to design an EEG virtual channel signal generation model, which is a multi-layer convolutional neural network. The model uses the brain area of ​​a portable few-channel EEG detector as a reference, selects the EEG signal of the corresponding reference brain area as input, and uses the multi-channel EEG signal as output; wherein the front end of the multi-layer convolutional neural network includes a time domain feature extraction module and a space domain feature extraction module, and the back end fuses the two types of features, the time domain feature and the space domain feature. This network structure design can not only utilize the time domain features of the EEG signals of each channel, but also obtain the correlation features between the EEG signals of each channel, thereby realizing the fusion of the time and space features of the EEG signals, thereby improving the learning and expression capabilities of the network; defining the network loss function, designing a new composite loss function, integrating the local loss in the time domain and frequency domain and the global loss of distribution consistency, accurately evaluating the consistency between the original signal and the generated signal, guiding network training, and improving the learning effect; The optimal model generation module uses the training set to train the designed EEG virtual channel signal generation model, uses the validation set to evaluate the model performance and optimize the parameters, and uses the test set to evaluate the model performance to obtain the model with the best performance; The signal generation module is used to input the few-channel EEG signal into the model with the best performance to generate EEG virtual channel signal.

Images