Walking imagination electroencephalogram feature generation method, system and computer program product

Abstract

The application discloses a walking imagination electroencephalogram feature generation method and system and a computer program product. The method uses a deep reinforcement learning method to generate features of walking imagination electroencephalogram, so as to expose more data features and obtain a more reliable extended dataset of walking electroencephalogram data. The method comprises the following steps: collecting an original dataset; inputting the original dataset into a variational autoencoder to obtain a feature map; introducing an attention mechanism to the feature map to obtain a modified feature vector; inputting the modified feature vector into a generator of a generative adversarial network; learning the spatial distribution of walking imagination electroencephalogram data; training to obtain a classified electroencephalogram feature generation model; mixing data samples generated by the electroencephalogram feature generation model into the original dataset; and generating an extended dataset of walking electroencephalogram data.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical assistance technology, and more specifically, to a method, system, and computer program product for generating brainwave features of walking imagination. Background Technology

[0002] Deep reinforcement learning is one of the hottest research areas in artificial intelligence. Its rapid development has led to its widespread application in fields such as intelligent robot control, robot vision, and autonomous driving. In a standard deep reinforcement learning framework, an agent obtains feedback and interacts with its environment by perceiving information and performing actions. Specifically, the agent uses sensors to perceive its state in the environment, then selects an action to execute through its own decision-making process. As the agent's state changes in the environment, it simultaneously perceives feedback signals provided by the environment.

[0003] Deep reinforcement learning has been widely applied in the field of medical rehabilitation assistive robots. Among these applications, deep learning algorithms for gait imagery brainwaves are crucial for controlling medical rehabilitation exoskeleton robots. However, the complex, lengthy, and easily affected by external environmental interference processes in gait imagery EEG data acquisition hinder the expansion of effective gait imagery EEG data. Here, research on transforming the original data is one method to address the data shortage, generating new data with a distribution similar to the original data. This includes traditional methods and machine learning-based methods. Traditional data expansion methods include geometric transformations (translation, reverse rotation, etc.) and noise reduction. However, EEG data is time-series related, and operations such as translation, flipping, and rotation cannot be performed on it, thus traditional methods struggle to achieve ideal results. In deep reinforcement learning applications in brain-computer interface scenarios, problems such as insufficient training samples and a lack of effective data exist, affecting the training results of the discriminative model. In EEG recognition research, some researchers first extract features from the EEG signal and then add Gaussian noise to the features to generate new feature samples. This approach can, to some extent, expand the number of EEG data samples, making the trained model more stable. Adversarial generative networks can generate data that conforms to the distribution of the original dataset, and this approach is widely used in few-shot learning. Summary of the Invention

[0004] Due to the aforementioned deficiencies in existing technologies, this invention provides a method, system, and computer program product for generating walking imagination EEG features. Addressing the problems with existing walking imagination EEG datasets applicable to medical brain-computer interface walking assistance robots, this invention employs deep reinforcement learning to generate walking imagination EEG features, thereby exposing more of its data characteristics and augmenting the collected original dataset.

[0005] To achieve the above objectives, the present invention provides a method for generating EEG features of walking imagery, characterized by comprising the following steps:

[0006] Step S1: Collect EEG data of the subjects’ walking imagination and filter the collected EEG data to obtain the raw dataset.

[0007] Step S2: Input the original dataset into the variational autoencoder to obtain the feature map;

[0008] Step S3: An attention mechanism is introduced into the feature map to obtain channel attention weights; the attention weights are multiplied by the input features to obtain a feature map with added attention.

[0009] Step S4: Input the attention-added feature map into the generator of the generative adversarial network (GAN) to learn the spatial distribution of the walking imagery EEG data and train to obtain a classified EEG feature generation model; wherein, the attention mechanism is introduced in the training process.

[0010] Step S5: Mix the data samples generated by the EEG feature generation model into the original dataset to generate an extended dataset of walking EEG data.

[0011] This invention addresses the problem of expanding walking imagination EEG data. It utilizes generative adversarial network learning and introduces an attention mechanism to generate features and expand the dataset, which has broad application prospects in the construction of walking imagination EEG datasets and brain-computer interface robots.

[0012] Furthermore, in step S1, the acquired EEG data is filtered by selecting a 7-40Hz bandpass filter to reduce interference from irrelevant signals in the data, and a 50Hz notch filter to reduce power frequency interference.

[0013] Further, in step S2, the variational autoencoder samples sample X for each channel. k Matching Gaussian distribution And optimize the parameter μ using a neural network. k and For μ and σ 2 Loss function for training for:

[0014] In the formula, N e This represents the number of electrode channels used in EEG data acquisition. KL loss is selected to measure the difference between a Gaussian distribution and a standard normal distribution.

[0015] Furthermore, in step S3, the structure of the attention generation channel is a fully connected neural network and a sigmoid function; the input data is processed by convolutional layers in the MLP to extract shallow features, then max pooling layers are used to reduce the dimensionality of the data, and finally dense blocks and average pooling layers are used to extract deep features; the sigmoid function is used to map the channel attention weights.

[0016] Furthermore, in step S4, the generative adversarial network has two discriminators D and one generator G; the two discriminators D are composed of neural networks with the same structure but without sharing parameters; both the generator G and the discriminators D introduce an attention mechanism.

[0017] Furthermore, the discriminator D and the generator G use different learning rates. This feature can further improve training performance.

[0018] Furthermore, the discriminator D and the generator G are selected from any one of deep residual networks, fully connected neural networks, convolutional neural networks, and temporal neural networks.

[0019] Furthermore, the discriminator D and the generator G are selected as deep residual networks; the adversarial loss function of the generator G and the discriminator D is:

[0020]

[0021] In the formula, V(D,G) is the adversarial loss between the generator G and the discriminator D. For the input data distribution, To obey The average of all input data s1 from the distribution is taken with respect to log2D(s1), P z (z) represents the input noise distribution. To conform to the noise distribution P z The data of (z) is averaged over log2(1-D(G(z))), where D(x1) is the discriminant's judgment result on the original data feature map, and D(G(z)) is the probability that the discriminant judges the generator input noise feature map as true.

[0022] The loss function is designed as follows:

[0023]

[0024]

[0025] In the formula, l G Let l be the loss function of the generator. DLet D(G(s1)) be the loss function of the discriminator, where D(G(s1)) is the probability that the discriminator classifies the generator's output feature map as true, and D(G(z)) is the probability that the discriminator classifies the generator's input noise feature map as true.

[0026] On the other hand, the present invention provides a system for generating brainwave features of walking imagination, characterized in that it includes:

[0027] The data acquisition module is used to collect EEG data of the subjects' walking imagination to obtain the raw dataset;

[0028] A feature map generation module is used to generate feature maps from the original dataset using a variational autoencoder.

[0029] The channel attention module is used to obtain the channel attention weights of the feature map and then obtain the corrected feature vector;

[0030] The training module is used to train the modified feature vector through a generative adversarial network model to obtain a classified EEG feature generation model.

[0031] An extended dataset generation module is used to mix the data samples generated by the EEG feature generation model into the original dataset to generate an extended dataset of walking EEG data.

[0032] The training process of the training module incorporates a channel attention module.

[0033] In another aspect, the present invention provides a computer program product, characterized in that, when the computer program is executed by a processor, it implements the above-described method for generating brainwave features of walking imagination.

[0034] Compared with the prior art, the above invention has the following advantages or beneficial effects:

[0035] (1) This invention utilizes a variational autoencoder to extract the modified feature vector hidden inside the channel, and then generates the modified feature vector from it. Compared with the method of directly using GAN, it has certain improvements in robustness and generation quality.

[0036] (2) This invention utilizes a GAN with a channel attention module to acquire EEG data features in the walking imagery mode, performs feature learning on each sampling channel, and learns the importance of each channel to the walking EEG features during training, thereby improving training efficiency to a certain extent.

[0037] (3) This invention generates features on the original collected walking imagery EEG data, further expanding the amount of available data and providing convenience for the construction of walking imagery EEG datasets. Attached Figure Description

[0038] The invention, its features and advantages will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0039] Figure 1 This is a step diagram of a method for generating brainwave features of walking imagination in one embodiment of the present invention;

[0040] Figure 2 This is a flowchart of a method for generating brainwave features of walking imagination in one embodiment of the present invention.

[0041] Figure 3 This is a schematic diagram of the structure of a walking imagination EEG feature generation system in one embodiment of the present invention. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the invention.

[0043] In the following detailed description, numerous specific details are set forth to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that well-known algorithms and models are not shown in detail to avoid obscuring the spirit of the invention.

[0044] Furthermore, the terms “comprising,” “including,” and any variations thereof are intended to cover non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units that is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0045] Example 1

[0046] See Figure 1 and Figure 2 This embodiment provides a method for generating EEG features of walking imagination, characterized by the following steps:

[0047] Step S1: Collect EEG data of the subjects’ walking imagination and filter the collected EEG data to obtain the raw dataset.

[0048] As a preferred technical solution, specifically, data is collected under a suitable acquisition environment and a standardized walking imagery EEG acquisition paradigm. The sampling frequency and acquisition channels are determined, as are the EEG acquisition frequency band and sampling duration. EEG data is characterized by high temporal resolution but low spatial resolution, making it susceptible to external noise interference. Common interferences originate from the experimental environment and other electrical signals within the body. As an example, the sampling frequency is 1000Hz, the electrode caps use the international 10 / 20 system electrode placement method, and the 10 electrode channels acquired are F1, F2, FC1, FC2, FC3, FC4, C1, C2, C3, and C4, with the reference electrode being Cz. A 50Hz notch filter is created to reduce power frequency interference, and a bandpass filter is created to extract the EEG components in the 7-40Hz frequency band from the selected EEG data to obtain the input feature map S. For the walking imagery EEG in this embodiment, the focus is more on the 7-40Hz frequency band of EEG signals in the data. This is because the brainwave signals in this frequency band are related to motor sensation, reducing interference from irrelevant signals such as electromyography and electrooculography in the data. The input feature map S has dimension N. e ×N t A matrix, where N e N represents the number of electrode channels used for sampling. t This represents the number of sampling points per channel. Each channel in the input feature map X is sampled as X. k .

[0049] Step S2: Input the original dataset into a variational autoencoder (VAE) to obtain a feature map.

[0050] Specifically, as an example, for each sample X, it is input into the VAE to obtain a feature map S. During this process, the VAE samples sample X for each channel. k Matching Gaussian distribution Approximate μ by optimizing parameters using a neural network. k and For μ and σ 2 The loss function for training is as follows:

[0051]

[0052]

[0053]

[0054] Where N e The number of electrode channels sampled is also the dimension of the feature map S.

[0055] Step S3: The feature map is introduced with an attention mechanism to obtain channel attention weights; the attention weights are multiplied with the input features to obtain a feature map with added attention.

[0056] Due to the multi-channel time-series nature of EEG, a channel attention mechanism module (CAM) is introduced. As a preferred technical solution, the attention generation channel is further structured as a fully connected neural network (MLP) and a sigmoid function. Input data is processed in the MLP through convolutional layers to extract shallow features, then max-pooling layers are used to reduce the dimensionality of the data, and finally dense block and average pooling layers are used to extract deep features. The sigmoid function is used to map the channel attention weights.

[0057] The attention mechanism here is used to channel-weight the data before training the generative adversarial network. The feature map S is passed through the channel attention module to obtain the channel attention weights C(S). Multiplying these weights by the feature map yields the modified feature map S1 with added attention. Here, C represents the mapping function of the channel attention module, which can be obtained using the sigmoid function.

[0058] Specifically, as an example, step S3 first performs global pooling and average pooling on the input feature map, then learns the channel features through the MLP, and finally calculates the channel attention feature vector C(S) using the sigmoid function. The channel attention feature C(S) is calculated as follows:

[0059] C(S)=σ(MLP(AvgPool(S))+MLP(MaxPool(S))

[0060] In the formula, σ represents the Sigmoid function, S is the original feature map, MLP represents the fully connected neural network operation, AvgPool represents the average pooling operation, and AvgPool represents the max pooling operation.

[0061] The channel attention module obtains the channel attention feature weights C(S), which are multiplied by the input feature map. The weights are then multiplied by the feature map to obtain the modified attention-added feature map S1 = C(S)·S. The attention-added feature map S1 is used to evaluate the importance of each channel to the walking EEG features.

[0062] Step S4: Input the attention-added feature map into the generator of the GAN to learn the spatial distribution of the walking imagery EEG data and train to obtain a classified EEG feature generation model; wherein, the attention mechanism is introduced during the training process.

[0063] Specifically, as an example, a GAN has two discriminators D and one generator G. Both discriminators D are composed of neural networks with the same structure but sharing parameters. One discriminator D assigns high scores to samples that conform to the real data distribution; the other discriminator rewards samples generated by the generator with high scores. Through a game between the generator and the two discriminators D, the distribution of synthetic samples generated by the generator G infinitely approximates the distribution of the real data.

[0064] Generative Adversarial Networks (GANs) train two models simultaneously: a generator G and a discriminator D. The goal of a GAN is to train an effective discriminator D, and then, based on D, train a good generator G. The objective of generator G is to cause discriminator D to make as many mistakes as possible, eventually converging to a unique solution—that is, G completely covers the original data distribution, and D cannot distinguish between data generated by G and real data. The entire process is an end-to-end iterative training of G and D alternately. An attention mechanism is introduced into this process; the attention mechanism is a mechanism that filters out a small amount of important information from a large amount of data. The adversarial features generated by the attention mechanism can more effectively train a good generator G. The attention mechanism is not only applied to generator G; both generator G and discriminator D use the attention mechanism. To improve training performance, different learning rates are used in discriminator D and generator G.

[0065] The objective functions for generator G and discriminator D are:

[0066]

[0067] In the formula, V(D,G) is the adversarial loss between the generator G and the discriminator D. For the input data distribution, To obey The average of all input data s1 from the distribution is taken with respect to log2D(s1), P z (z) represents the input noise distribution. To conform to the noise distribution P z The data of (z) is averaged over log2(1-D(G(z))), where D(s1) is the discriminator's judgment result on the original data feature map, and D(G(z)) is the probability that the discriminator judges the generator's input noise feature map as true.

[0068] As a preferred technical solution, furthermore: the generator G and discriminator D can be selected from fully connected neural networks, convolutional neural networks, and temporal neural networks, etc. In this example, a deep residual network is selected as the generator and discriminator. The loss function is designed as follows:

[0069]

[0070]

[0071] In the formula, l G Let l be the loss function of the generator. D Let D(G(s1)) be the loss function of the discriminator, where D(G(s1)) is the probability that the discriminator classifies the generator's output feature map as true, and D(G(z)) is the probability that the discriminator classifies the generator's input noise feature map as true. After introducing the attention mechanism, the enhanced generator trained using existing EEG data is used to train the model again. The features generated by GAN... This approach can effectively enhance the generated samples, improving their quality. The attention mechanism is not only applicable to the generator G; both the generator G and the discriminator D utilize it. To improve training performance, different learning rates are used in the discriminator D and the generator G. For example, the discriminator D uses a fixed learning rate of 2e-4, while the generator G uses a fixed learning rate of 1e-3 or a rate that decreases by a decay factor after a certain number of iterations. An attention module is inserted before the average pooling operation in the deep residual networks of both the generator G and the discriminator D. The channel attention features are calculated in the same way as in step S3, ensuring that the residual network output is the result of multiplying the channel attention weights. This improves the robustness of the training and generation processes and the quality of the generated samples.

[0072] Step S5: Mix the data samples generated by the EEG feature generation model into the original dataset to generate an expanded dataset of walking EEG data. As an example, a simple random method is used to mix the features generated by the walking EEG feature generation model... By mixing the generated features with real data X, the distinction between generated and original features is no longer made, thus enabling the generation of an extended dataset of walking EEG data.

[0073] Example 2

[0074] See Figure 3 This embodiment provides a system for generating EEG features of walking imagery, including:

[0075] The data acquisition module 100 is used to collect EEG data of the subjects' walking imagination to obtain the raw dataset;

[0076] Feature map generation module 110 is used to generate feature maps from the original dataset using a variational autoencoder;

[0077] The channel attention module 120 is used to obtain the channel attention weights of the feature map and then obtain the corrected feature vector;

[0078] Training module 130 is used to train the modified feature vector through a generative adversarial network model to obtain a classified EEG feature generation model;

[0079] The extended dataset generation module 140 is used to mix the data samples generated by the EEG feature generation model into the original dataset to generate an extended dataset of walking EEG data.

[0080] The training process of the training module incorporates a channel attention module.

[0081] The process by which the system generates walking imagery EEG features and obtains an extended dataset includes steps S1 to S5 as described in Example 1.

[0082] Example 3

[0083] This embodiment provides a computer program product, which, when executed by a processor, implements the method for generating brainwave features of walking imagination as described in Embodiment 1.

[0084] The computer program product may be stored in a computer-readable storage medium. The computer-readable storage medium may include, but is not limited to, floppy disks, optical disks, CD-ROMs (Read-Only Optical Disk Memory), magneto-optical disks, ROMs (Read-Only Memory), RAMs (Random Access Memory), EPROMs (Erasable Programmable Read-Only Memory), EEPROMs (Electrically Erasable Programmable Read-Only Memory), magnetic cards or optical cards, flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions. The computer-readable storage medium may be a product not connected to a computer device or a component used in a computer device.

[0085] Those skilled in the art will understand that the various methods or computer programs of the above embodiments can be implemented by a smart terminal including one or more memories and one or more processors; the one or more memories are used to store the computer program; the one or more processors are connected to the memories and are used to run the computer program to perform the walking imagination EEG feature generation method as described in Embodiment 1.

[0086] Optionally, the memory may include, but is not limited to, high-speed random access memory and non-volatile memory. For example, one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices; the processor 42 may include, but is not limited to, a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0087] Optionally, the processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0088] In summary, this invention provides a method, system, and computer program product for generating walking-image EEG features. It employs deep reinforcement learning to generate features from walking-image EEG data, thereby exposing more of its data characteristics and obtaining a more reliable extended dataset of walking EEG data. The method includes the following steps: acquiring an original dataset; inputting the original dataset into a feature map obtained by a variational autoencoder; introducing an attention mechanism into the feature map to obtain a modified feature vector; inputting the modified feature vector into a generator of a generative adversarial network to learn the spatial distribution of the walking-image EEG data, training a classification EEG feature generation model; and mixing the data samples generated by the EEG feature generation model into the original dataset to generate an extended dataset of walking EEG data.

[0089] Those skilled in the art should understand that variations can be implemented by combining existing technology with the above embodiments, which will not be elaborated here. Such variations do not affect the essence of the present invention, and will not be elaborated here either.

[0090] The preferred embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and the devices and structures not described in detail should be understood as being implemented in a conventional manner in the art. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the present invention. This does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the present invention's technical solutions still fall within the protection scope of the present invention.

[0091] Those skilled in the art will recognize that the units, i.e., algorithm steps, of the various examples described in connection with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

Claims

1. A method for generating EEG features of walking imagination, characterized in that, Includes the following steps: Step S1: Collect EEG data of the subjects’ walking imagination and filter the collected EEG data to obtain the raw dataset. Step S2: Input the original dataset into the variational autoencoder to obtain the feature map; Step S3: An attention mechanism is introduced into the feature map to obtain channel attention weights; the attention weights are multiplied by the input features to obtain a feature map with added attention. Step S4: Input the attention-added feature map into the generator of the generative adversarial network to learn the spatial distribution of the walking imagery EEG data and train to obtain a classified EEG feature generation model; wherein, an attention mechanism is introduced during the training process. Step S5: Mix the data samples generated by the EEG feature generation model into the original dataset to generate an extended dataset of walking EEG data; In step S1, the collected EEG data is filtered by selecting a 7-40Hz bandpass filter to reduce interference from irrelevant signals in the data and a 50Hz notch filter to reduce power frequency interference. In step S3, the structure of the attention generation channel is a fully connected neural network and a sigmoid function. The input data is processed by convolutional layers in the MLP to extract shallow features, then max pooling layers are used to reduce the dimensionality of the data, and finally dense blocks and average pooling layers are used to extract deep features. The sigmoid function is used to map the channel attention weights. Step S3 first performs global pooling and average pooling on the input feature map, then the MLP learns the channel features, and finally uses the sigmoid function to calculate the channel attention feature vector. Among them, channel attention features Calculated as follows: In the formula, This represents the Sigmoid function. This is the original feature map. This represents the operation of a fully connected neural network. This indicates the average pooling operation. This represents the max pooling operation; In step S4, the generative adversarial network has two discriminators D and one generator G; the two discriminators D are composed of neural networks with the same structure but not sharing parameters; both the generator G and the discriminators D introduce an attention mechanism; the discriminators D and the generator G use different learning rates; an attention module is inserted before the average pooling operation in the deep residual network of the generator G and the discriminator D, and the channel attention features are calculated in the same way as in step S3, so that the output of the residual network is the result of multiplying the channel attention weights.

2. The method for generating EEG features of walking imagination according to claim 1, characterized in that, In step S2, the variational autoencoder samples each channel. Matching Gaussian distribution And optimize parameters through neural networks and ;right and Loss function for training for: ； ； In the formula, This represents the number of electrode channels used in EEG data acquisition.

3. The method for generating EEG features of walking imagination according to claim 1, characterized in that, The discriminator D and generator G are selected from any one of deep residual networks, fully connected neural networks, convolutional neural networks, and temporal neural networks.

4. The method for generating EEG features of walking imagination according to claim 1, characterized in that, The discriminator D and generator G are selected from deep residual networks; the adversarial loss function of the generator G and discriminator D is: In the formula, For generator With discriminator The antagonistic losses, For the input data distribution, To obey All input data of the distribution right Take the average value. For the input noise distribution, To conform to noise distribution Data Take the average value. The discriminator's judgment result on the original data feature map. This represents the probability that the discriminator classifies the noise feature map input to the generator as true. The loss function is designed as follows: In the formula, Let the loss function of the generator be... Let be the loss function of the discriminator, where Let be the probability that the discriminator classifies the generator's output feature map as true. This represents the probability that the discriminator classifies the noise feature map input to the generator as true.

5. A system for generating EEG features related to walking imagery, characterized in that, A method for generating EEG features of walking imagery as described in any one of claims 1 to 4, comprising: The data acquisition module is used to collect EEG data of the subjects' walking imagination to obtain the raw dataset; A feature map generation module is used to generate feature maps from the original dataset using a variational autoencoder. The channel attention module is used to obtain the channel attention weights of the feature map and then obtain the corrected feature vector; The training module is used to train the modified feature vector through a generative adversarial network model to obtain a classified EEG feature generation model. An extended dataset generation module is used to mix the data samples generated by the EEG feature generation model into the original dataset to generate an extended dataset of walking EEG data. The training process of the training module incorporates a channel attention module.

6. A computer program product, characterized in that, When the computer program is executed by the processor, it implements the method for generating EEG features of walking imagination as described in any one of claims 1 to 4.

Images