Task-driven dynamic interpolation method for multichannel electroencephalogram signals

Through the task-driven dynamic interpolation method, combined with deep learning and task-related characteristics, EEG signals are processed, and the problem of insufficient adaptability to specific task states and personalized electrode layout in the existing technology is solved, achieving more efficient EEG signal reconstruction and reappear.

CN119025823APending Publication Date: 2024-11-26UNIV OF ELECTRONICS SCI & TECH OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410989818.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing EEG signal processing methods ignore the EEG signal characteristics in specific task states in rehabilitation tasks and the unique EEG pattern of different patients, and have limited adaptability to non-standardized electrode layout and personalized electrode caps, resulting in the interpolation model pairs. The signal reproduction ability is insufficient.

Method used

A task-driven dynamic interpolation method is adopted to obtain EEG signals in different task states, and generate data sets using non-overlapping sliding window cutting and normalization processing to build a brain including sequence input layer, CNN neural network and long and short-term memory network. The electrical signal is interpolated to the network, and the loss function is designed to optimize the network weights, and interpolated processing is performed in combination with deep learning and task-related characteristics.

Benefits of technology

It improves the accuracy of reconstruction of EEG signals, reproduces the task state characteristics of real EEG signals, improves the spatial and temporal accuracy of signal processing, and is suitable for EEG signal interpolation under dynamic changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025823A_ABST
    Figure CN119025823A_ABST
Patent Text Reader

Abstract

The invention relates to the field of medical signal processing and artificial intelligence, in particular to a task-driven dynamic interpolation method for multichannel electroencephalogram signals. By constructing an electroencephalogram signal interpolation network, features of electroencephalogram signals in space and time dimensions are extracted, and by designing a new loss function, network weight is optimized and interpolation performance is improved. And training the network based on the new loss function to obtain a task-driven dynamic interpolation network model, and reconstructing the electroencephalogram signal to be processed by using the task-driven dynamic interpolation network model. In the constructed loss function, the data processing capability of deep learning and the EEG signal characteristics related to the specific task state are combined, so that the prior knowledge related to the task is integrated, the error analysis is carried out by utilizing the relationship between the channels and the historical task state information, the accuracy of the reconstructed electroencephalogram signal is improved, and the accuracy of the reconstructed electroencephalogram signal is improved. Task state characteristics of real electroencephalogram signals are reproduced, and powerful assistance is provided for subsequent decoding and analysis of the electroencephalogram signals.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of medical signal processing and artificial intelligence, and in particular to a task-driven dynamic interpolation method for multi-channel electroencephalogram signals. Background Art

[0002] Brain-computer interface (BCI) technology has shown great potential in promoting neural remodeling in stroke patients and improving the effectiveness of rehabilitation training. Especially in motor imagery (MI) tasks, BCI systems can help restore patients' motor functions by capturing patients' electroencephalogram (EEG) signals and converting user intentions into control signals for external devices. In this process, high-quality EEG signal acquisition and analysis is a crucial link for BCI systems to extract information from EEG. In actual BCI applications, problems such as electrode detachment, blockage, or EEG channel damage frequently occur, resulting in the loss of important signal data, compromising the integrity of EEG data, and affecting subsequent signal processing and decoding steps, thereby reducing the performance of the BCI system and the effectiveness of rehabilitation training. Therefore, it is necessary to process the original EGC signal for subsequent analysis and interpretation of the EEG signal.

[0003] Traditional EEG signal processing methods include N nearest neighbor, plane and spherical spline interpolation, and inverse distance weighting. These traditional techniques estimate the missing signal value in different ways, each with its own advantages and limitations. The N nearest neighbor method is easy to implement but not suitable for complex signal patterns; while the spline interpolation technique takes into account the spatial distribution, it requires high accuracy of the electrode layout; the inverse distance weighting method has its application in spatial analysis, but faces challenges in real-time performance and parameter selection.

[0004] With the successful application of deep learning in the field of image processing, researchers have found that deep learning methods such as GAN and its variants, as well as new technologies such as graph neural networks, that is, using complex models to adaptively learn features from a large amount of sample data to perform advanced interpolation processing on EEG signals, can improve the spatial and temporal accuracy of signal processing, optimize interpolation performance, and obtain signals that are closer to real EEG data. However, existing EEG signal processing methods based on deep learning still have the following problems:

[0005] 1) The EEG signal characteristics under specific task states in rehabilitation tasks are ignored, and these characteristics are extremely critical in intention decoding and personalized rehabilitation.

[0006] 2) The physiological characteristics of EEG signals are ignored and the unique EEG patterns of different patients are not considered.

[0007] 3) The adaptability to non-standardized electrode layouts or personalized electrode caps is limited, and the sensitivity is low in dynamic situations, such as motor imagery tasks.

[0008] The existence of the above problems leads to the interpolation model's insufficient ability to reproduce signals and poor ability to reconstruct real EEG signals. Summary of the invention

[0009] The purpose of the present invention is to provide a task-driven dynamic interpolation method for multi-channel EEG signals to improve the accuracy of reconstructed EEG signals, achieve task-state feature reproduction of real EEG signals, and provide powerful assistance for subsequent decoding and analysis of EEG signals.

[0010] To achieve the above object, the present invention adopts the following technical solution:

[0011] A task-driven dynamic interpolation method for multi-channel electroencephalogram signals comprises the following steps:

[0012] Step 1, obtain EEG signals of different task states; for each task state: use non-overlapping sliding windows to cut all EEG signals, obtain several EEG signal segments with equal distances and perform normalization processing to generate the EEG data set of the task state;

[0013] Step 2: construct an EEG signal interpolation network. The EEG signal interpolation network model includes a sequential input layer, a three-layer CNN neural network, a long short-term memory network, and a regression output layer connected in sequence;

[0014] Step 3: Construct a loss function for the EEG signal interpolation network, wherein the loss function includes a reconstruction error term L recon , task-dependent error term L task and channel coordination error term L cc ;

[0015] The reconstruction error term L recon for:

[0016]

[0017] The task-state related error term L task for:

[0018]

[0019] The channel coordination error term is:

[0020]

[0021] The loss function L of the EEG signal interpolation network is:

[0022] L=L recon +L task +L cc

[0023] In the formula, y iIt is the original real signal of the damaged channel; is the reconstructed signal output by the EEG signal interpolation network, N is the number of sampling points; M is the number of features, θ(y i ) is the task-state feature of the original real signal of the damaged channel, is the task-state feature of the reconstructed signal output by the interpolation network; p are the remaining channels whose channel correlation coefficient with the damaged channel (channel to be interpolated) is greater than 0.6, and P is the number of these channels. is the reconstructed signal output by the interpolation network.

[0024] Step 4: According to the task requirements, the corresponding task data set obtained in step 1 and the loss function designed in step 3 are used to train and optimize the EEG signal interpolation network to obtain a task-driven dynamic interpolation (T-DDI) network model; the task-driven dynamic interpolation network model can be used to reconstruct the EEG signal to be processed.

[0025] Furthermore, the step 1 is to obtain EEG signals of different task states from public data, and the public data sets include: "EEG Motor Movement / Imagery Dataset" public data set, and SSVEP public EEG data set provided by Tsinghua University.

[0026] Furthermore, the normalization method used in step 1 is the Min-Max normalization method, which performs normalization on several non-overlapping EEG signal segments so that their values ​​are between the interval [-1, 1].

[0027] Furthermore, the sequence input layer of the EEG signal interpolation network receives the EEG signal data set of the corresponding task state according to the task requirements;

[0028] Each layer of the three-layer CNN neural network is composed of a CNN convolutional layer, a BatchNormalization layer, and a ReLU layer connected in sequence. The convolutional layer is used to extract the spatial features of the EEG signal, the BatchNormalization layer is used to perform batch data normalization, and the ReLU layer is used to perform nonlinear activation operations on the output of the BatchNormalization layer.

[0029] The long short-term memory (LSTM) network consists of a LSMT layer, a Drop out layer, and a fully connected layer connected in sequence. The LSMT layer is used to capture the long-term and short-term time domain features of the EEG signal. The Drop out layer is used to prevent overfitting during the training process. The fully connected layer contains the same number of neurons as the output dimension.

[0030] The regression output layer is used to output the interpolation result of the EEG signal interpolation network, that is, the reconstructed EEG signal.

[0031] The EEG signal to be processed is interpolated according to the long-term and short-term time domain features of the EEG signal provided by the long short-term memory (LSTM) network to complete the EEG signal reconstruction.

[0032] Furthermore, the training optimization strategy of the EEG signal interpolation network in step 4 is the Adam optimizer.

[0033] The above method can also be used to process various physiological electrical signals that are susceptible to interference, such as electrocardiogram signals and electromyography signals.

[0034] The present invention provides a task-driven dynamic interpolation method for multi-channel EEG signals, which extracts the characteristics of EEG signals in spatial and temporal dimensions by constructing an EEG signal interpolation network, and optimizes the network weights and improves the interpolation performance by designing a new loss function. The above network is trained based on the new loss function to obtain a task-driven dynamic interpolation network model, which can be used to complete the reconstruction of the EEG signals to be processed. Since the loss function designed by the present invention combines the data processing capabilities of deep learning and the EEG signal characteristics related to a specific task state, it not only integrates the task-related prior knowledge, but also uses the relationship between channels and historical task state information for error analysis, thereby improving the accuracy of the reconstructed EEG signals, reproducing the task state characteristics of the real EEG signals, and providing powerful assistance for the subsequent decoding and analysis of EEG signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Task-driven dynamic interpolation network structure diagram;

[0036] Figure 2 Time domain illustration of interpolation results on the motor imagery dataset;

[0037] Figure 3 Time domain illustration of the interpolation results on the steady-state visual stimulation dataset. DETAILED DESCRIPTION

[0038] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and embodiments.

[0039] like Figure 1 As shown, this embodiment provides a task-driven dynamic interpolation method for multi-channel EEG signals, comprising the following steps:

[0040] Step 1: Obtain EEG signals of different task states; for each task state: use a non-overlapping sliding window to cut all EEG signals, obtain several equidistant EEG signal segments and perform normalization processing to generate the EEG data set of the task state.

[0041] This embodiment uses the public data set "EEG Motor Movement / Imagery Dataset" and the steady-state visual evoked potential (SSVEP) public EEG data set provided by Tsinghua University, namely "Benchmark Dataset". Among them, the public data set "EEG Motor Movement / Imagery Dataset" covers the electrical signals of three different brain states: resting state, left and right hand movement imagination state, and both hands and feet movement imagination state. This data set is configured with 64 EEG acquisition channels according to the 10-10 international standard system, and the original data includes the corresponding EEG signals collected from 109 subjects. This embodiment selects the data of the first 10 subjects to construct a data set of motor imagination task state, specifically:

[0042] 1) Each subject provided 1-minute EEG recordings in three different brain states, with a sampling rate of 160 Hz;

[0043] 2) For each state: Use a sliding window of 1 second to slice the data and generate 60 sets of non-overlapping data segments. Each data segment has 160 sampling points and 64 channels, so the data dimension of each segment is 160×64. The data dimension provided by each subject is 3 (number of states) × 60 (number of groups in each state) × 160 (number of sampling points) × 64 (number of channels).

[0044] 3) The data of the 10 subjects were merged and then normalized using the Min-Max normalization method, and the normalized results were mapped to the interval [-1, 1]; thus completing the construction of the motor imagery task state dataset.

[0045] The public dataset "Benchmark Dataset" is used to extract two specific EEG signals: resting state and flicker task state at 8Hz frequency. This dataset is configured with 64 EEG acquisition channels according to the international 10-20 system standard, and the original data comes from 35 subjects. In this embodiment, the data of the first 10 subjects are selected from this dataset to construct a dataset focusing on the steady-state visual stimulation task state, as follows:

[0046] 1) For each subject, record the resting EEG signal and the EEG signal of the flicker task state at a frequency of 8 Hz. Record 6 sets of data for each state, and the duration of each set of data is 6 seconds. In order to eliminate the data deviation that may be caused by eye movement and flicker observation fatigue, this embodiment extracts the 2nd to 4th seconds of each set of records as valid data, and the sampling rate of each set of records is 1000 Hz.

[0047] 2) Use a 1-second sliding window to segment the data, and generate 2 1-second non-overlapping segments from each valid 2-second data set, so a total of 12 groups of data segments are obtained in each state, that is, 6 groups of records × 2 segments per group. Each data segment contains 1000 sampling points and 64 channels, thus forming a data dimension of 1000×64. That is, the data dimension provided by each subject can be expressed as 2 (number of states) × 12 (number of segments in each state) × 1000 (number of sampling points per segment) × 64 (number of channels).

[0048] 3) The data of the 10 subjects were combined and then normalized using the Min-Max normalization method, and the normalized results were mapped to the interval [-1, 1] to complete the construction of the steady-state visual stimulation task data set.

[0049] The EEG signals in the motor imagery task state data and the steady-state visual stimulation task state data set are respectively subjected to random channel deletion to simulate the abandonment of channel data after channel damage during actual acquisition. In this embodiment, the number of randomly missing channels is 1, 2, 3, 4, 8, and 10, respectively, and all the missing channel data are set to 0. The data set after the random channel deletion is used as the final data set and divided: 80% of the motor imagery task state data set is used as the training set of the model, 10% is used as the test set of the model, and 10% is used as the verification set of the model; 80% of the steady-state visual stimulation task state data set is used as the training set of the model, 10% is used as the test set of the model, and 10% is used as the verification set of the model.

[0050] Step 2: Construct an EEG signal interpolation network to extract the time and space domain features of the EEG signal. The EEG signal interpolation network model includes a sequential input layer, a three-layer CNN neural network, a long short-term memory network, and a regression output layer.

[0051] The sequence input layer is used to receive the EEG signal data set of the required task state; its input dimension is 65, the first 64 dimensions represent 64 channels, and the 65th dimension is the average time-frequency domain characteristics of the EEG signal under the corresponding task state.

[0052] Each layer of the three-layer CNN neural network is composed of a CNN convolutional layer, a BatchNormalization layer and a ReLU layer connected in sequence. The CNN convolutional layer has a convolution kernel size of 5*5 and a number of 16, which performs a convolution operation on the input data to extract the spatial domain features of the EEG signal. The BatchNormalization layer follows the convolution layer and is used to perform batch data standardization to reduce internal covariate shift, help accelerate the convergence speed during training, and improve the stability of the model. The ReLU layer is used to perform nonlinear activation operations on the output of the BatchNormalization layer, so that the network can learn and represent more complex patterns.

[0053] The long short-term memory (LSTM) network is composed of a LSMT layer, a Drop out layer, and a fully connected layer connected in sequence. The LSMT layer contains 256 gated units for capturing the long-term and short-term time domain features of EEG signals. The Drop out layer is used to prevent overfitting during training. The fully connected layer contains the same number of neurons as the output dimension.

[0054] The output size of the regression output layer is 1×the number of sampling points, and its output is the interpolation result of the EEG signal interpolation network, that is, the reconstructed EEG signal.

[0055] Step 3: Construct a loss function for the EEG signal interpolation network, wherein the loss function includes a reconstruction error term L recon , task-dependent error term L task and channel coordination error term L cc ;

[0056] The reconstruction error term L recon for:

[0057]

[0058] The task-state related error term L task for:

[0059]

[0060] The channel coordination error term is:

[0061]

[0062] In the formula, y i It is the original real signal of the damaged channel; is the reconstructed signal output by the EEG signal interpolation network, N is the number of sampling points; M is the number of features, θ(y i ) is the task-state feature of the original real signal of the damaged channel, The task-state characteristics of the reconstructed signal output by the EEG signal interpolation network; p are the remaining channels whose channel correlation coefficient with the damaged channel is greater than 0.6, P is the number of these channels, It is the reconstructed signal output by the EEG signal interpolation network.

[0063] The loss function L used in this embodiment can be obtained by weighted summing the above three error terms:

[0064] L=L recon +L task +L cc

[0065] In this embodiment, the initial weight coefficients of the EEG signal interpolation network are set to 0.3, 0.4, and 0.3 respectively.

[0066] Step 4: According to the task requirements, based on the training set in the corresponding task data set obtained in step 1 and the loss function designed in step 3, the Adam optimizer is used to train and optimize the EEG signal interpolation network constructed in step 2.

[0067] In this embodiment, the maximum number of training rounds is set to 100, and the initial learning rate is 5e-3. In order to avoid overfitting, the learning rate is multiplied by a decay factor of 0.1 every 25 training rounds. The loss function Loss is calculated and back-propagated to the network for optimization. When the loss function converges or reaches the maximum number of iterations, the training is stopped to obtain the optimal network model. The task-driven dynamic interpolation network model can be used to complete the reconstruction of the EEG signal to be processed. The interpolation results of this embodiment on the motor imagery dataset can be found in Figure 2 The interpolation results on the steady-state visual stimulation dataset can be found in Figure 3 .

[0068] In order to verify the performance of the task-driven dynamic interpolation method for multi-channel EEG signals, this implementation compared it with some classic traditional methods and deep learning methods. During the verification process, the evaluation indicators used were the correlation coefficient between the reconstructed signal and the original real signal, and the root mean square error after reconstruction. The calculation formula of the correlation coefficient (CC) is:

[0069]

[0070] The calculation formula for normalized root mean square error (NMSE) is:

[0071]

[0072] In the formula, y i is the original true signal of the damaged channel, The reconstructed signal output by the EEG signal interpolation network.

[0073] The verification results are shown in Table 1, Table 2, Table 3 and Table 4:

[0074] Table 1 Comparison of EEG interpolation network performance - Within-Subject

[0075]

[0076] Table 1 shows the performance comparison of the EEG signal interpolation network when a single channel data is missing. In the experiment, one channel is randomly missing each time and the missing channel is different each time, and the average result of 10 times is counted. The motor imagery data set is divided into three types of data: imagining left and right hands, imagining both hands and feet, and resting state data; the steady-state visual stimulation data set is divided into two types of data: 8Hz stimulation and resting state; because the variance of different types of data results in the same data set is in the order of 10e-4, the difference is very small, so the average result of the data is presented.

[0077] Table 2 Comparison of EEG interpolation network performance - Cross-Subject

[0078]

[0079]

[0080] Table 2 tests Cross-Subject, that is, the model is trained using the data of 80% of the subjects under the same task state (that is, in this embodiment, the data of 8 subjects are finally used to train the model). After obtaining the general model, the network performance is verified using the data of the subjects who did not participate in the training. It can be seen that the task-driven dynamic interpolation method of this embodiment has the best generalization performance.

[0081] Table 3 Comparison of interpolation performance under different numbers of missing channels - correlation coefficient

[0082]

[0083]

[0084] Table 4 Comparison of interpolation performance under different numbers of missing channels - normalized root mean square error

[0085]

[0086] Tables 3 and 4 show the comparison of interpolation performance based on the motor imagery dataset for different numbers of missing channels. Each time, n (n=1, 2, 3, 4, 8, 10) channels are randomly missing, and the missing channels are not exactly the same each time. A total of ten experiments are conducted and the average results of the ten experiments are calculated.

[0087] In summary, the interpolation method of this embodiment can be used as a preprocessing method for EEG signals for automatic EEG signal interpolation tasks, laying the foundation for subsequent analysis and application of EEG signals. When implemented, it is embedded before the EEG signal processing algorithm to realize real-time processing of EEG signal interpolation tasks. In addition, the task-driven dynamic interpolation method for multi-channel EEG signals of this embodiment is also applicable to the processing of various physiological electrical signals that are susceptible to interference, such as electrocardiogram signals and electromyography signals.

[0088] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A task-driven dynamic interpolation method for multi-channel EEG signals, characterized in that: The following steps are involved: Step 1: Obtain EEG signals of different task states; for each task state: use a non-overlapping sliding window to cut all EEG signals, obtain several EEG signal segments with equal distances and perform normalization processing to generate the EEG data set of the task state; Step 2: construct an EEG signal interpolation network. The EEG signal interpolation network model includes a sequential input layer, a three-layer CNN neural network, a long short-term memory network, and a regression output layer connected in sequence; Step 3: Construct a loss function for the EEG signal interpolation network, wherein the loss function includes a reconstruction error term L recon , task-dependent error term L task and channel coordination error term L cc ; The reconstruction error term L recon for: The task-state related error term L task for: The channel coordination error term is: The loss function L of the EEG signal interpolation network is: L=L recon +L task +L cc In the formula, y i It is the original real signal of the damaged channel; is the reconstructed signal output by the EEG signal interpolation network, N is the number of sampling points; M is the number of features, θ(y i ) is the task-state feature of the original real signal of the damaged channel, The task-state characteristics of the reconstructed signal output by the interpolation network; y p are the remaining channels whose channel correlation coefficient with the damaged channel (channel to be interpolated) is greater than 0.6, and P is the number of channels. Step 4: According to the task requirements, the corresponding task data set obtained in step 1 and the loss function designed in step 3 are used to train and optimize the EEG signal interpolation network to obtain a task-driven dynamic interpolation (T-DDI) network model; the task-driven dynamic interpolation network model can be used to reconstruct the EEG signal to be processed.

2. According to claim 1, a task-driven dynamic interpolation method for multi-channel EEG signals is characterized in that: The step 1 is to obtain EEG signals of different task states from the public data set "EEG Motor Movement / Imagery Dataset" and the SSVEP public EEG data set provided by Tsinghua University.

3. The task-driven dynamic interpolation method for multi-channel EEG signals according to claim 1, characterized in that: The normalization processing method adopted in step 1 is the Min-Max normalization method, which performs normalization processing on several non-overlapping EEG signal segments so that their values ​​are between the interval of [-1, 1].

4. The task-driven dynamic interpolation method for multi-channel EEG signals according to claim 1, characterized in that: The sequence input layer of the EEG signal interpolation network is used to receive the EEG signal data set of the required task state; Each layer of the three-layer CNN neural network is composed of a CNN convolutional layer, a BatchNormalization layer, and a ReLU layer connected in sequence. The convolutional layer is used to extract the spatial features of the EEG signal, the BatchNormalization layer is used to perform batch data normalization, and the ReLU layer is used to perform nonlinear activation operations on the output of the BatchNormalization layer. The long short-term memory network is composed of LSMT layers, Drop out layers and fully connected layers connected in sequence. The LSMT layer is used to capture the long-term and short-term time domain features of EEG signals. The Drop out layer is used to prevent overfitting during training. The fully connected layer contains the same number of neurons as the output dimension. The regression output layer is used to output the interpolation result of the EEG signal interpolation network, that is, the reconstructed EEG signal.

5. The task-driven dynamic interpolation method for multi-channel EEG signals according to claim 1, characterized in that: The training optimization strategy of the EEG signal interpolation network in step 4 is the Adam optimizer.

6. A task-driven dynamic interpolation method for multi-channel EEG signals according to any one of claims 1 to 5, characterized in that; This method can also be used to process other physiological electrical signals besides EEG signals.