Motion recognition method, device and equipment based on electroencephalogram signals and storage medium

Abstract

The invention is suitable for the technical field of action recognition, and provides an action recognition method, device and equipment based on electroencephalogram signals and a medium, the method comprises the steps that data preprocessing is conducted on original electroencephalogram signals, and initial time domain signals are obtained; performing data enhancement on the initial time domain signal through a generative adversarial network to obtain a target time domain signal; segmenting the target time domain signal by an electroencephalogram electrode channel to obtain segmented data, and performing short-time Fourier transform on the segmented data to obtain a target time-frequency domain signal corresponding to the target time domain signal; respectively carrying out feature extraction processing on the target time domain signal and the target time-frequency domain signal to obtain a time domain action feature corresponding to the target time domain signal and a time-frequency domain action feature corresponding to the target time-frequency domain signal, and carrying out feature fusion on the time-frequency domain action feature and the time domain action feature to obtain a target time domain action feature and a target time-frequency domain action feature; fusion features are obtained; and recognizing an action corresponding to the electroencephalogram time domain signal according to the fusion feature.

Description

Technical Field

[0001] The present application belongs to the field of motion recognition technology, and in particular, relates to a method, device, equipment and storage medium for motion recognition based on electroencephalogram signals. Background Art

[0002] The brain is the material basis of all advanced human behaviors. It is composed of a large number of nerve cells, synapses and glial cells. These nerve cells are constantly carrying out spontaneous, rhythmic and comprehensive electrical activities. The generated electric field is transmitted through the volume conductor to form a potential distribution on the scalp. This potential signal with time as the axis is the electroencephalogram (EEG) signal. EEG signals are widely used as a non-invasive EEG signal that records voltage fluctuations in brain neurons. Using the motor imagery experimental paradigm to decode EEG signals can achieve the purpose of identifying human movements. In recent years, some deep learning methods have been applied to the field of EEG signal decoding, which has improved the overall accuracy of action recognition using EEG signals.

[0003] Because EEG signals are complex and have a high-dimensional structure. Therefore, advanced deep learning algorithms are needed to process and decode these complex brain data. A single deep learning method, such as using only convolutional neural networks (CNN), can no longer meet the requirements for EEG signal recognition accuracy. It is necessary to explore the combination of deep learning methods with different characteristics to better extract the spatiotemporal features of EEG signals. In addition, most deep learning methods only consider extracting features in the feature domain of one data for classification. The most commonly used feature is the time domain feature, which is the original data of the EEG signal. However, the features of other data domains, such as the frequency domain and the time-frequency domain, are ignored, which affects the recognition accuracy. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, device and storage medium for action recognition based on EEG signals, which can solve the problem of insufficient recognition accuracy caused by ignoring the features of other data domains in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a method for action recognition based on EEG signals, comprising:

[0006] Perform data preprocessing on the original EEG signal to obtain the initial time domain signal;

[0007] Performing data enhancement on the initial time domain signal by generating an adversarial network to obtain a target time domain signal;

[0008] Segmenting the target time domain signal by EEG electrode channels to obtain segmented data, and performing short-time Fourier transform on the segmented data to obtain a target time-frequency domain signal corresponding to the target time domain signal;

[0009] Performing feature extraction processing on the target time domain signal and the target time-frequency domain signal respectively to obtain a time domain action feature corresponding to the target time domain signal and a time-frequency domain action feature corresponding to the target time-frequency domain signal, and performing feature fusion on the time-frequency domain action feature and the time domain action feature to obtain a fusion feature;

[0010] The action corresponding to the EEG time domain signal is identified according to the fusion feature.

[0011] Optionally, the step of performing data preprocessing on the original electroencephalogram signal to obtain an initial time domain signal includes:

[0012] According to the data interception rule in the data preprocessing, an EEG time domain signal of a preset length is intercepted from the original EEG signal, and the EEG time domain signal of the preset length is segmented by time steps to obtain a set of segmented time domain signals;

[0013] According to the filtering rule in the data preprocessing, each fragment time domain signal in the fragment time domain signal set is filtered by a fifth-order Bavost bandpass filter to obtain a filtered signal;

[0014] According to the normalization rule in the data preprocessing, normalization processing is performed on each electrode channel of the filtered signal to obtain the initial time domain signal.

[0015] Optionally, before the step of performing data enhancement on the initial time domain signal by generating an adversarial network to obtain a target time domain signal, the step further includes:

[0016] The initial time domain signal is used as training data, and the generator and discriminator in the generative adversarial network are alternately trained through the training data until the output data of the trained generator meets the preset conditions, and the trained generator is used as the generative model.

[0017] Optionally, the step of performing data enhancement on the initial time domain signal by generating an adversarial network to obtain a target time domain signal includes:

[0018] Inputting randomly generated noise data into the generation model to obtain artificial data of a predetermined multiple;

[0019] The artificial data and the initial time domain signal are combined to complete data enhancement and obtain a target time domain signal.

[0020] Optionally, the step of performing feature extraction processing on the target time domain signal and the target time-frequency domain signal respectively to obtain a time domain action feature corresponding to the target time domain signal and a time-frequency domain action feature corresponding to the target time-frequency domain signal comprises:

[0021] Inputting the target time domain signal into a preset EEGNet sub-model in the preset feature model to obtain a time domain action feature corresponding to the target time domain signal;

[0022] The target time-frequency domain signal is input into the Transformer sub-model in the preset feature model to obtain the time-frequency domain action feature corresponding to the target time-frequency domain signal.

[0023] Optionally, the preset EEGNet sub-model includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a separable convolutional layer, and a point-by-point convolutional layer. The convolution kernel size of the first convolutional layer is a preset size, the convolution kernel size of the second convolutional layer is 1 / 4 of the preset size, and the convolution kernel size of the third convolutional layer is 1 / 16 of the preset size. The discard rates of the first convolutional layer, the second convolutional layer, and the third convolutional layer are all 0.3, and the first convolutional layer, the second convolutional layer, the third convolutional layer, the separable convolutional layer, and the point-by-point convolutional layer are connected in series in sequence.

[0024] Optionally, the step of fusing the time-frequency domain motion feature with the time-domain motion feature to obtain a fused feature includes:

[0025] The time-frequency domain action feature and the time-domain action feature are input into the feature fusion submodule in the preset feature model, and the feature fusion submodule performs feature fusion on the time-frequency domain action feature and the time-domain action feature according to the feature fusion method of the feature level to obtain the fused feature.

[0026] On the other hand, an embodiment of the present application provides a motion recognition device based on EEG signals, the device comprising:

[0027] A data acquisition module is used to perform data preprocessing on the original EEG signal to obtain an initial time domain signal;

[0028] A data enhancement module, used to perform data enhancement on the initial time domain signal through a generative adversarial network to obtain a target time domain signal;

[0029] A data conversion module, used to segment the target time domain signal according to the EEG electrode channel to obtain segmented data, and perform short-time Fourier transform on the segmented data to obtain a target time-frequency domain signal corresponding to the target time domain signal;

[0030] A feature extraction and fusion module is used to perform feature extraction processing on the target time domain signal and the target time-frequency domain signal respectively, to obtain a time domain action feature corresponding to the target time domain signal and a time-frequency domain action feature corresponding to the target time-frequency domain signal, and to perform feature fusion on the time-frequency domain action feature and the time domain action feature to obtain a fusion feature;

[0031] The recognition module is used to recognize the action corresponding to the EEG time domain signal according to the fusion feature.

[0032] In a third aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned EEG signal-based action recognition method when executing the computer program.

[0033] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for action recognition based on EEG signals as described above is implemented.

[0034] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0035] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0036] The original EEG signal is preprocessed to obtain an initial time domain signal; the initial time domain signal is enhanced by a generative adversarial network to obtain a target time domain signal; the target time domain signal is segmented by EEG electrode channels to obtain segmented data, and the segmented data is short-time Fourier transformed to obtain a target time-frequency domain signal corresponding to the target time domain signal; the target time domain signal and the target time-frequency domain signal are subjected to feature extraction processing respectively to obtain a time domain action feature corresponding to the target time domain signal and a time-frequency domain action feature corresponding to the target time-frequency domain signal, and the time-frequency domain action feature and the time-domain action feature are subjected to feature fusion to obtain a fusion feature; the action corresponding to the EEG time domain signal is identified according to the fusion feature. Generative adversarial networks are used for data enhancement to avoid the problem of inaccurate model training due to insufficient data. By converting time domain signal data into time-frequency domain signal data, the frequency signal features that change over time are obtained and utilized, thereby improving classification accuracy and better versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 This is an application scenario diagram of the action recognition method based on EEG signals provided in one embodiment of the present application;

[0039] Figure 2 is a flowchart of an action recognition method based on EEG signals provided in one embodiment of the present application;

[0040] Figure 3 is a flowchart of an action recognition method based on EEG signals provided in another embodiment of the present application;

[0041] Figure 4 It is a structural schematic diagram of an action recognition device based on EEG signals provided in one embodiment of the present application. DETAILED DESCRIPTION

[0042] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0043] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0044] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0045] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0046] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0047] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0048] The method for action recognition based on EEG signals provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In an application environment, the EEG signal acquisition device 100 communicates with the server 200, the server 120 obtains the original EEG signal sent by the EEG signal acquisition device 100, performs data preprocessing on the original EEG signal to obtain an initial time domain signal; performs data enhancement on the initial time domain signal by generating an adversarial network to obtain a target time domain signal; the target time domain signal is segmented according to the EEG electrode channel to obtain segmented data, and the segmented data is short-time Fourier transformed to obtain a target time-frequency domain signal corresponding to the target time domain signal; feature extraction processing is performed on the target time domain signal and the target time-frequency domain signal respectively to obtain a time domain action feature corresponding to the target time domain signal, and a time-frequency domain action feature corresponding to the target time-frequency domain signal, and the time-frequency domain action feature and the time domain action feature are feature fused to obtain a fused feature; and the action corresponding to the EEG time domain signal is identified according to the fused feature. By using a generative adversarial network for data enhancement, the problem of inaccurate model training due to insufficient data is avoided. By converting time-domain signal data into time-frequency domain signal data, the frequency signal characteristics that change over time are obtained and utilized, thereby improving classification accuracy and better versatility.

[0049] Figure 2 A schematic flowchart of the method for action recognition based on EEG signals provided in the present application is shown. As an example but not a limitation, the method can be applied to the above-mentioned server 200, that is, the server 200 is used as a server for action recognition based on EEG signals.

[0050] The present application provides an action recognition method based on EEG signals, comprising:

[0051] S101, performing data preprocessing on the original electroencephalogram signal to obtain an initial time domain signal;

[0052] In a possible implementation, the step of performing data preprocessing on the original electroencephalogram signal to obtain an initial time domain signal includes:

[0053] According to the data interception rule in the data preprocessing, an EEG time domain signal of a preset length is intercepted from the original EEG signal, and the EEG time domain signal of the preset length is segmented by time steps to obtain a set of segmented time domain signals;

[0054] According to the filtering rule in the data preprocessing, each fragment time domain signal in the fragment time domain signal set is filtered by a fifth-order Bavost bandpass filter to obtain a filtered signal;

[0055] According to the normalization rule in the data preprocessing, normalization processing is performed on each electrode channel of the filtered signal to obtain the initial time domain signal.

[0056] Exemplarily, an end time point T2 and a start time point T1 are selected, and during the motor imagery experiment (T2-T1, preset duration), sampling is performed at 250 Hz and divided into (T2-T1)×250 time steps to obtain a set of fragmented time domain signals. The signal data is filtered using a fifth-order Bavost bandpass filter (filtering rule) from 4 Hz to 30 Hz, and then each electrode channel is normalized. The initial time domain signal, the original EEG signal is the time domain data.

[0057] S102, performing data enhancement on the initial time domain signal by generating an adversarial network to obtain a target time domain signal;

[0058] In a possible implementation manner, before the step of performing data enhancement on the initial time domain signal by generating an adversarial network to obtain a target time domain signal, the step further includes:

[0059] The initial time domain signal is used as training data, and the generator and discriminator in the generative adversarial network are alternately trained through the training data until the output data of the trained generator meets the preset conditions, and the trained generator is used as the generative model.

[0060] For example, a generative adversarial network is a generative model that obtains a lot of training samples from a training library to learn the probability distribution generated by these training cases. In this example, the trained generator will be able to generate new data that is very similar to the real data.

[0061] The specific implementation method is to let the generator network and the discriminator network compete with each other. The generator network continuously captures the probability distribution of real signal data in the training library and transforms the input random noise into new samples (that is, fake data). The discriminator network can observe real and fake data at the same time to determine whether the data is real.

[0062] In a possible implementation, the step of performing data enhancement on the initial time domain signal by generating an adversarial network to obtain a target time domain signal includes:

[0063] Inputting randomly generated noise data into the generation model to obtain artificial data of a predetermined multiple;

[0064] The artificial data and the initial time domain signal are combined to complete data enhancement and obtain a target time domain signal.

[0065] Exemplarily, the training signal data is sent to the GAN network for training to obtain the GAN generation model and the discriminant model. After the discriminator and the generator confront each other, some noise is randomly generated and input into the trained generation model to generate 25 times the artificial signal data (artificial data). The artificial data and the initial time domain signal are merged to complete data enhancement and obtain the target time domain signal.

[0066] S103, segmenting the target time domain signal according to the EEG electrode channel to obtain segmented data, and performing short-time Fourier transform on the segmented data to obtain a target time-frequency domain signal corresponding to the target time domain signal;

[0067] Exemplarily, the time domain signal data is converted into time-frequency domain signal data through short-time Fourier transform, so as to better extract the spatiotemporal characteristics of the EEG signal and avoid ignoring the time-frequency domain characteristics and affecting the recognition accuracy.

[0068] S104, performing feature extraction processing on the target time domain signal and the target time-frequency domain signal respectively to obtain a time domain action feature corresponding to the target time domain signal and a time-frequency domain action feature corresponding to the target time-frequency domain signal, and performing feature fusion on the time-frequency domain action feature and the time domain action feature to obtain a fusion feature;

[0069] In a possible implementation, the step of performing feature extraction processing on the target time domain signal and the target time-frequency domain signal respectively to obtain a time domain action feature corresponding to the target time domain signal and a time-frequency domain action feature corresponding to the target time-frequency domain signal includes:

[0070] Inputting the target time domain signal into a preset EEGNet sub-model in the preset feature model to obtain a time domain action feature corresponding to the target time domain signal;

[0071] The target time-frequency domain signal is input into the Transformer sub-model in the preset feature model to obtain the time-frequency domain action feature corresponding to the target time-frequency domain signal.

[0072] Exemplarily, the preset feature model is constructed by presetting an EEGNet sub-model and a Transformer sub-model, combining the spatiotemporal convolution capability of the preset EEGNet network model, and using the Tansformer network model to establish long-term dependencies between features, thereby achieving higher classification accuracy and better versatility than similar deep learning methods.

[0073] In a possible implementation, the step of fusing the time-frequency domain motion feature with the time-domain motion feature to obtain a fused feature includes:

[0074] The time-frequency domain action feature and the time-domain action feature are input into the feature fusion submodule in the preset feature model, and the feature fusion submodule performs feature fusion on the time-frequency domain action feature and the time-domain action feature according to the feature fusion method of the feature level to obtain the fused feature.

[0075] Exemplarily, the feature fusion method at the feature level can be understood as fusing high- and low-level features. Since high-level semantic features are close to the output but have low resolution, and high-resolution features are close to the input but have low semantic levels, both high-level features and low-level features have their own weaknesses. The segmentation results of the high-level features in the second row retain the large semantic structure, but the small structure is seriously lost; the segmentation results of the low-level features in the third row retain rich details, but the semantic category prediction is very poor. Therefore, the feature fusion method at the feature level makes the obtained fusion features better.

[0076] S105: Identify the action corresponding to the EEG time domain signal according to the fusion feature.

[0077] Exemplarily, the fused features are classified using a Softmax classifier and are ultimately divided into four types of actions: left hand, right hand, feet, and tongue.

[0078] The original EEG signal is preprocessed to obtain an initial time domain signal; the initial time domain signal is enhanced by a generative adversarial network to obtain a target time domain signal; the target time domain signal is segmented by EEG electrode channels to obtain segmented data, and the segmented data is short-time Fourier transformed to obtain a target time-frequency domain signal corresponding to the target time domain signal; the target time domain signal and the target time-frequency domain signal are subjected to feature extraction processing respectively to obtain a time domain action feature corresponding to the target time domain signal and a time-frequency domain action feature corresponding to the target time-frequency domain signal, and the time-frequency domain action feature and the time-domain action feature are subjected to feature fusion to obtain a fusion feature; the action corresponding to the EEG time domain signal is identified according to the fusion feature. Generative adversarial networks are used for data enhancement to avoid the problem of inaccurate model training due to insufficient data. By converting time domain signal data into time-frequency domain signal data, the frequency signal features that change over time are obtained and utilized, thereby improving classification accuracy and better versatility.

[0079] In a possible implementation, the preset EEGNet sub-model includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a separable convolutional layer, and a point-by-point convolutional layer. The convolution kernel size of the first convolutional layer is a preset size, the convolution kernel size of the second convolutional layer is 1 / 4 of the preset size, and the convolution kernel size of the third convolutional layer is 1 / 16 of the preset size. The discard rates of the first convolutional layer, the second convolutional layer, and the third convolutional layer are all 0.3, and the first convolutional layer, the second convolutional layer, the third convolutional layer, the separable convolutional layer, and the point-by-point convolutional layer are connected in series in sequence.

[0080] Exemplarily, the existing EEGNet network model is used as the basis for improvement, the first convolution layer of the existing EEGNet network model is expanded to three layers, the original-size convolution kernel, 1 / 4 original-size convolution kernel, and 16 / 1 original-size convolution kernel in the existing EEGNet network model are used to perform parallel convolution on the first convolution layer, the second convolution layer, and the third convolution layer, and the drop rate of each layer is increased to 0.3, while retaining the separable convolution layer and point-by-point convolution layer of the existing EEGNet network model

[0081] like Figure 3As shown, after receiving the start command, the EEG time domain signal data (equivalent to the above-mentioned original EEG signal) is obtained, and the EEG time domain signal data is divided into time steps of 250 Hz according to the time period T2-T1 (end time point-start time point) during the motor imagination period to obtain a set of fragmented time domain signals, and the signal data is filtered using a fifth-order Bavost bandpass filter (filtering rule) from 4 Hz to 30 Hz, and then each electrode channel is normalized to obtain the initial time domain signal, and the original EEG signal is the time domain data. Channel normalization is used to reduce the risk of overfitting. The initial time domain signal, the original EEG signal is the time domain data, and a generative adversarial network (GAN) is used to send the training signal data to the GAN network for training to obtain a generative model and a discriminant model of the GAN, and some noise is randomly generated and input into the trained generative model to generate 25 times the artificial signal. Data, merging the artificial data and the initial time domain signal to complete data enhancement, obtaining a target time domain signal, segmenting the target time domain signal by EEG electrode channels, and using short-time Fourier transform (STFT) to change the input mode of the segmented target time domain signal to obtain time-frequency domain signal data (target time-frequency domain signal), respectively inputting the time domain signal data and the time-frequency domain signal data into a preset feature model (new model) that combines an improved EEGNet model (preset EEGNet sub-model) and a Transformer model (Transformer sub-model), using the new model to respectively extract time domain action features and time-frequency domain action features, using a feature-level feature fusion method to fuse the features extracted from the two signal domains, using a Softmax classifier for classification, and finally dividing them into four categories of actions, left hand, right hand, feet and tongue.

[0082] On the other hand, an embodiment of the present application provides a motion recognition device based on EEG signals, the device comprising:

[0083] The data acquisition module 201 is used to perform data preprocessing on the original EEG signal to obtain an initial time domain signal;

[0084] A data enhancement module 202 is used to perform data enhancement on the initial time domain signal by generating an adversarial network to obtain a target time domain signal;

[0085] The data transformation module 203 is used to segment the target time domain signal according to the EEG electrode channel to obtain segmented data, and perform short-time Fourier transformation on the segmented data to obtain a target time-frequency domain signal corresponding to the target time domain signal;

[0086] The feature extraction and fusion module 204 is used to perform feature extraction processing on the target time domain signal and the target time-frequency domain signal respectively, obtain the time domain action feature corresponding to the target time domain signal, and the time-frequency domain action feature corresponding to the target time-frequency domain signal, and perform feature fusion on the time-frequency domain action feature and the time domain action feature to obtain a fusion feature;

[0087] The identification module 205 is used to identify the action corresponding to the EEG time domain signal according to the fusion feature.

[0088] In the third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following is achieved: performing data preprocessing on the original electroencephalogram signal to obtain an initial time domain signal; performing data enhancement on the initial time domain signal by generating an adversarial network to obtain a target time domain signal; segmenting the target time domain signal according to the EEG electrode channel to obtain segmented data, and performing short-time Fourier transform on the segmented data to obtain a target time-frequency domain signal corresponding to the target time domain signal; performing feature extraction processing on the target time domain signal and the target time-frequency domain signal, respectively, to obtain time domain action features corresponding to the target time domain signal, and time-frequency domain action features corresponding to the target time-frequency domain signal, and performing feature fusion on the time-frequency domain action features and the time domain action features to obtain fused features; and a step of identifying the action corresponding to the EEG time domain signal according to the fused features.

[0089] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements: performing data preprocessing on the original electroencephalogram signal to obtain an initial time domain signal; performing data enhancement on the initial time domain signal by generating an adversarial network to obtain a target time domain signal; segmenting the target time domain signal according to the EEG electrode channel to obtain segmented data, and performing short-time Fourier transform on the segmented data to obtain a target time-frequency domain signal corresponding to the target time domain signal; performing feature extraction processing on the target time domain signal and the target time-frequency domain signal, respectively, to obtain time domain action features corresponding to the target time domain signal, and time-frequency domain action features corresponding to the target time-frequency domain signal, and performing feature fusion of the time-frequency domain action features and the time domain action features to obtain fused features; and the step of identifying the action corresponding to the EEG time domain signal according to the fused features.

[0090] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0091] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0092] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0093] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment / terminal equipment and methods can be implemented in other ways. For example, the device / network equipment / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0094] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0095] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for motion recognition based on EEG signals, characterized in that: include: Perform data preprocessing on the original EEG signal to obtain the initial time domain signal; Performing data enhancement on the initial time domain signal by generating an adversarial network to obtain a target time domain signal; Segmenting the target time domain signal by EEG electrode channels to obtain segmented data, and performing short-time Fourier transform on the segmented data to obtain a target time-frequency domain signal corresponding to the target time domain signal; Performing feature extraction processing on the target time domain signal and the target time-frequency domain signal respectively to obtain a time domain action feature corresponding to the target time domain signal and a time-frequency domain action feature corresponding to the target time-frequency domain signal, and performing feature fusion on the time-frequency domain action feature and the time domain action feature to obtain a fusion feature; The action corresponding to the EEG time domain signal is identified according to the fusion feature.

2. The method for motion recognition based on EEG signals as claimed in claim 1, characterized in that: The step of performing data preprocessing on the original electroencephalogram signal to obtain an initial time domain signal comprises: According to the data interception rule in the data preprocessing, an EEG time domain signal of a preset length is intercepted from the original EEG signal, and the EEG time domain signal of the preset length is segmented by time steps to obtain a set of segmented time domain signals; According to the filtering rule in the data preprocessing, each fragment time domain signal in the fragment time domain signal set is filtered by a fifth-order Bavost bandpass filter to obtain a filtered signal; According to the normalization rule in the data preprocessing, normalization processing is performed on each electrode channel of the filtered signal to obtain the initial time domain signal.

3. The method for motion recognition based on EEG signals as claimed in claim 1, characterized in that: Before the step of performing data enhancement on the initial time domain signal by generating an adversarial network to obtain a target time domain signal, the method further includes: The initial time domain signal is used as training data, and the generator and discriminator in the generative adversarial network are alternately trained through the training data until the output data of the trained generator meets the preset conditions, and the trained generator is used as the generative model.

4. The method for motion recognition based on EEG signals as claimed in claim 3, characterized in that: The step of performing data enhancement on the initial time domain signal by generating an adversarial network to obtain a target time domain signal comprises: Inputting randomly generated noise data into the generation model to obtain artificial data of a predetermined multiple; The artificial data and the initial time domain signal are combined to complete data enhancement and obtain a target time domain signal.

5. The method for motion recognition based on EEG signals as claimed in claim 1, characterized in that: The step of performing feature extraction processing on the target time domain signal and the target time-frequency domain signal respectively to obtain a time domain action feature corresponding to the target time domain signal and a time-frequency domain action feature corresponding to the target time-frequency domain signal comprises: Inputting the target time domain signal into a preset EEGNet sub-model in the preset feature model to obtain a time domain action feature corresponding to the target time domain signal; The target time-frequency domain signal is input into the Transformer sub-model in the preset feature model to obtain the time-frequency domain action feature corresponding to the target time-frequency domain signal.

6. The method for motion recognition based on EEG signals as claimed in claim 5, characterized in that: The preset EEGNet sub-model includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a separable convolutional layer, and a point-by-point convolutional layer. The convolution kernel size of the first convolutional layer is a preset size, the convolution kernel size of the second convolutional layer is 1 / 4 of the preset size, and the convolution kernel size of the third convolutional layer is 1 / 16 of the preset size. The discard rates of the first convolutional layer, the second convolutional layer, and the third convolutional layer are all 0.

3. The first convolutional layer, the second convolutional layer, the third convolutional layer, the separable convolutional layer, and the point-by-point convolutional layer are connected in series in sequence.

7. The method for motion recognition based on EEG signals as claimed in claim 5, characterized in that: The step of fusing the time-frequency domain motion feature and the time-domain motion feature to obtain a fused feature comprises: The time-frequency domain action feature and the time-domain action feature are input into the feature fusion submodule in the preset feature model, and the feature fusion submodule performs feature fusion on the time-frequency domain action feature and the time-domain action feature according to the feature fusion method of the feature level to obtain the fused feature.

8. An action recognition device based on EEG signals, characterized in that: The device comprises: A data acquisition module is used to perform data preprocessing on the original EEG signal to obtain an initial time domain signal; A data enhancement module, used to perform data enhancement on the initial time domain signal through a generative adversarial network to obtain a target time domain signal; A data conversion module, used to segment the target time domain signal according to the EEG electrode channel to obtain segmented data, and perform short-time Fourier transform on the segmented data to obtain a target time-frequency domain signal corresponding to the target time domain signal; A feature extraction and fusion module is used to perform feature extraction processing on the target time domain signal and the target time-frequency domain signal respectively, to obtain a time domain action feature corresponding to the target time domain signal and a time-frequency domain action feature corresponding to the target time-frequency domain signal, and to perform feature fusion on the time-frequency domain action feature and the time domain action feature to obtain a fusion feature; The recognition module is used to recognize the action corresponding to the EEG time domain signal according to the fusion feature.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images