EEG signal cross-device decoding and classification model training method, decoding and classification method, and device

Abstract

The present application provides a method for training a cross-device decoding and classification model for EEG signals, a decoding and classification method, and a device. The training method includes: pre-training a latent space deep neural network corresponding to a classification model based on a generative adversarial network based on pre-trained feature vectors and decoding type labels corresponding to each EEG signal segment; using the network as a discriminator in the classification model, and inputting the feature matrix of the EEG signal segment into an autoencoder based on a convolutional neural network in the classification model to train the classification model, and training the classification model to be a cross-device decoding and classification model for EEG signals that outputs the decoding type of the EEG signal segment and the type of the EEG signal acquisition device. The present application can solve the problem of poor reliability and effectiveness of decoding and classification models trained using EEG signals from devices across manufacturers, and improve the data utilization rate of EEG signals used for training.

Description

Technical Field

[0001] The present application relates to the fields of electroencephalogram (EEG) technology and deep learning technology, and in particular to a method for training a cross-device EEG signal decoding and classification model, a decoding and classification method, and a device. Background Art

[0002] Brain-computer interfaces (BCIs), an emerging technology that establishes a direct connection between the brain and external devices, enabling interaction between brain signals and external devices, offer new hope for the rehabilitation of stroke patients. Within BCI technology, methods for capturing brain information are categorized as non-invasive, semi-invasive, and invasive, depending on the placement of the electrodes relative to the cerebral cortex. These methods increase in difficulty and risk. Currently, non-invasive signal acquisition methods, represented by electroencephalograms (EEGs), are the mainstream research. These methods capture spontaneous biopotential information from the brain through electrodes and then amplify and process the data. These data reflect the spontaneous and rhythmic changes in brain cells' potentials during different mental activities. Furthermore, compared to other implantable electrodes, EEG technology, due to its unique non-invasive nature, eliminates the need for surgical considerations or survival issues, enabling it to cover a wider range of users and scenarios. The most widely studied scenario is the perception of motor imagery.

[0003] Motor imagery (MI) is a mental training method that simulates motor movements or activity scenes in the mind. By capturing motor imagery data with EEG technology and then decoding it through a series of processing steps (for example, distinguishing between resting state and movement of the left and right hands, feet, and legs), it can be used for downstream rehabilitation tasks. This method can not only improve athlete performance but also assist in patient rehabilitation, further demonstrating the huge potential of brain-computer interface technology in the field of rehabilitation.

[0004] However, while EEG-based motor imagery decoding tasks have achieved significant success in research and practice, significant limitations remain in data utilization and model training. First, most EEG-based decoding models rely on signals collected by a single device, making them difficult to generalize to data collected by devices from different vendors. Because hardware parameters such as electrodes and amplifiers vary between vendors, even the same imagery signal from the same subject can exhibit significant data distribution differences. This distributional disparity clearly hinders machine learning models from decoding tasks. Even if the original model retains excellent decoding performance after retraining on a dataset from a new device, this translates to increased computational resources and training time, increasing training costs. Furthermore, this approach inherently forces the model to prioritize learning from later data without retaining information learned from previous data. This results in the model becoming sensitive to a single data set after each training session, leading to data dependency. This can lead to poor performance in real-world scenarios involving data from multiple vendors. In addition, although there have been a few studies on the decoding and classification of EEG cross-device data in motor imagery tasks, since they learn the mapping relationship of device data distribution, they are limited by the chain relationship in scenarios where data crosses more than two devices, which increases the cost of use.

[0005] Therefore, there is an urgent need to design a method that can solve the problems of poor reliability and effectiveness of decoding and classification models trained with EEG signals from cross-vendor devices and low data utilization of EEG signals used for training. Summary of the Invention

[0006] In view of this, embodiments of the present application provide a method for training an EEG signal cross-device decoding and classification model, a decoding and classification method, and a device to eliminate or improve one or more defects in the prior art.

[0007] One aspect of the present application provides a method for training an EEG signal cross-device decoding and classification model, comprising:

[0008] For each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, based on the pre-trained feature vector corresponding to each EEG signal segment and the decoding type label corresponding to each EEG signal segment, a latent space deep neural network corresponding to the classification model based on the generative adversarial network is pre-trained to obtain a pre-trained deep neural network for predicting the decoding type of the pre-trained feature vector;

[0009] The pre-trained deep neural network is used as the discriminator in the classification model based on the generative adversarial network, and the feature matrix of each EEG signal segment corresponding to each type of the EEG signal acquisition device is respectively input into the autoencoder based on the convolutional neural network corresponding to each type of the EEG signal acquisition device in the classification model to train the classification model, and then the classification model is trained to be an EEG signal cross-device decoding classification model for outputting the classification result data corresponding to the signal segment including the decoding type and the EEG signal acquisition device type according to the feature matrix of the signal segment, wherein each of the autoencoders is connected to the discriminator.

[0010] In some embodiments of the present application, before pre-training the latent space deep neural network corresponding to the classification model based on the generative adversarial network based on the pre-trained feature vectors corresponding to each of the EEG signal segments and the decoding type labels corresponding to each of the EEG signal segments, the method further includes:

[0011] Using the feature matrices corresponding to the respective EEG signal segments in the respective signal segment groups corresponding to the respective EEG signal acquisition device types, respectively training the convolutional neural network-based autoencoders corresponding to the respective EEG signal acquisition device types in the classification model based on the generative adversarial network, so that each of the autoencoders is used to respectively compress the input feature matrices and output compressed feature data corresponding to each of the feature matrices;

[0012] According to the compressed feature data corresponding to each type of the EEG signal acquisition device, a pre-trained feature vector corresponding to each of the EEG signal segments is obtained.

[0013] In some embodiments of the present application, obtaining the pre-trained feature vectors corresponding to the respective EEG signal segments according to the respective compressed feature data corresponding to the respective EEG signal acquisition device types includes: grouping the respective compressed feature data according to the respective EEG signal acquisition device types to obtain compressed feature groups corresponding to the respective EEG signal acquisition device types;

[0014] Selecting one of the compressed feature groups as a benchmark group, and selecting compressed feature data having the same decoding type label as that corresponding to each compressed feature data in the benchmark group from the other compressed feature groups except the benchmark group, to form a plurality of parent sample data corresponding to each compressed feature data in the benchmark group;

[0015] The average vector corresponding to each of the parent sample data is obtained as a pre-trained feature vector, and the Euclidean distance corresponding to each of the parent sample data is determined, and the decoding type label of the feature matrix to which the parent sample data corresponding to each of the pre-trained feature vectors belongs is used as the decoding type label of each of the pre-trained feature vectors.

[0016] In some embodiments of the present application, before using the feature matrices corresponding to the respective EEG signal segments in the respective signal segment groups corresponding to the respective EEG signal acquisition device types to respectively train the convolutional neural network-based autoencoders corresponding to the respective EEG signal acquisition device types in the classification model based on the generative adversarial network, the method further includes:

[0017] Preprocessing the original EEG signals corresponding to each EEG signal acquisition device type to obtain signal segment groups corresponding to each EEG signal acquisition device type, each including EEG signal segments, wherein each signal segment group corresponding to the same EEG signal acquisition device type has a one-to-one correspondence with each preset decoding type label;

[0018] For each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, the feature matrix for representing the time domain features and frequency domain features of the same dimension of each EEG signal segment is extracted respectively, and the EEG signal acquisition device type corresponding to the EEG signal segment is added to the decoding type label corresponding to the EEG signal segment to which each feature matrix belongs.

[0019] In some embodiments of the present application, for each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, extracting a feature matrix for each EEG signal segment for representing the time domain features and frequency domain features of the same dimension, including:

[0020] Creating a sliding window, and segmenting each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type each time the sliding window slides, to obtain sub-signal segments having a length equal to the window width of the sliding window; and performing preset time domain and frequency domain feature extraction steps on the sub-signal segments to obtain one-dimensional features corresponding to each of the sub-signal segments for representing time domain features and frequency domain features;

[0021] The one-dimensional features obtained during each sliding of the sliding window are stacked to obtain a feature matrix corresponding to the electroencephalogram signal segment composed of each sub-signal segment.

[0022] In some embodiments of the present application, the time domain and frequency domain feature extraction steps include:

[0023] Performing a preset spatial correlation pooling step on each channel segment of each channel of the current sub-signal segment, so as to use the target feature vector corresponding to the sub-signal segment obtained by the spatial correlation pooling step as the time domain feature vector corresponding to the sub-signal segment;

[0024] Furthermore, Fourier transform is performed on each channel segment of each channel corresponding to the sub-signal segment, and then a preset spatial correlation pooling step is performed on each channel segment after the Fourier transform, so as to use the target feature vector corresponding to the sub-signal segment obtained by the spatial correlation pooling step as the frequency domain feature vector corresponding to the sub-signal segment; and the time domain feature vector and the frequency domain feature vector corresponding to the sub-signal segment are connected to obtain a one-dimensional feature corresponding to the sub-signal segment.

[0025] In some embodiments of the present application, the spatial correlation pooling step includes:

[0026] Converting each channel segment of each channel of the current sub-signal segment into a discrete point vector of each channel based on an average interval time sampling method;

[0027] Determining the Pearson correlation coefficient between every two channels in each of the channels according to the discrete point vectors of each channel, so as to obtain a correlation coefficient matrix corresponding to the sub-signal segments;

[0028] and determining the electrode coordinates of each channel of the current sub-signal segment based on a Cartesian coordinate algorithm, and determining the Euclidean distance between every two electrode coordinates to obtain a Euclidean distance matrix corresponding to the sub-signal segment;

[0029] determining, according to the correlation coefficient matrix and the Euclidean distance matrix corresponding to the sub-signal segments, an average spatial correlation index corresponding to each of the channels of the sub-signal segments;

[0030] Average pooling processing is performed on the average spatial correlation index corresponding to each of the channels of the sub-signal segment to obtain a target feature vector corresponding to the sub-signal segment.

[0031] Another aspect of the present application provides a method for cross-device decoding and classification of EEG signals, comprising:

[0032] Extracting a feature matrix representing time domain features and frequency domain features of the same dimension from the EEG signal segment corresponding to the current target EEG signal;

[0033] The feature matrix corresponding to the target EEG signal is input into a preset EEG signal cross-device decoding classification model, so that the EEG signal cross-device decoding classification model outputs classification result data corresponding to the target EEG signal, which includes the decoding type and the EEG signal acquisition device type, wherein the EEG signal cross-device decoding classification model is pre-trained based on the EEG signal cross-device decoding classification model training method.

[0034] A third aspect of the present application provides an apparatus for training a cross-device EEG signal decoding classification model, comprising: a pre-training module for pre-training a latent space deep neural network corresponding to a classification model based on a generative adversarial network, for each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, based on a pre-trained feature vector corresponding to each EEG signal segment and a decoding type label corresponding to each EEG signal segment, to obtain a pre-trained deep neural network for predicting the decoding type of the pre-trained feature vector;

[0035] A training module is used to use the pre-trained deep neural network as a discriminator in a classification model based on a generative adversarial network, and input the feature matrices of each EEG signal segment corresponding to each type of EEG signal acquisition device into the autoencoder based on a convolutional neural network corresponding to each type of EEG signal acquisition device in the classification model to train the classification model, and then train the classification model to be an EEG signal cross-device decoding classification model that outputs the classification result data corresponding to the signal segment, which includes the decoding type and the EEG signal acquisition device type, based on the feature matrix of the signal segment, wherein each of the autoencoders is connected to the discriminator.

[0036] A fourth aspect of the present application provides an apparatus for cross-device decoding and classification of EEG signals, comprising:

[0037] A feature extraction module is used to extract a feature matrix representing time domain features and frequency domain features of the same dimension from the EEG signal segment corresponding to the current target EEG signal;

[0038] The model classification module is used to input the feature matrix corresponding to the target EEG signal into a preset EEG signal cross-device decoding classification model, so that the EEG signal cross-device decoding classification model outputs classification result data corresponding to the target EEG signal, which includes the decoding type and the EEG signal acquisition device type, wherein the EEG signal cross-device decoding classification model is pre-trained based on the EEG signal cross-device decoding classification model training method.

[0039] The fifth aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements the EEG signal cross-device decoding and classification model training method, and / or implements the EEG signal cross-device decoding and classification method.

[0040] The sixth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the EEG signal cross-device decoding and classification model training method, and / or implements the EEG signal cross-device decoding and classification method.

[0041] The seventh aspect of the present application provides a computer program product, including a computer program and / or instructions, which, when executed by a processor, implement the EEG signal cross-device decoding and classification model training method, and / or implement the EEG signal cross-device decoding and classification method.

[0042] The present application provides a method for training a cross-device decoding classification model for EEG signals. The method pre-trains a latent space deep neural network corresponding to a classification model based on a generative adversarial network based on pre-trained feature vectors and decoding type labels corresponding to each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, so as to obtain a pre-trained deep neural network for predicting the decoding type of the pre-trained feature vector; the pre-trained deep neural network is used as a discriminator in a classification model based on a generative adversarial network, and the feature matrices of each EEG signal segment corresponding to each EEG signal acquisition device type are respectively input into the autoencoder based on a convolutional neural network corresponding to each EEG signal acquisition device type in the classification model to train the classification model. The classification model is then trained to be a cross-device decoding and classification model for EEG signals, which outputs classification result data corresponding to the signal segment, including the decoding type and the EEG signal acquisition device type, based on the feature matrix of the signal segment. Each of the autoencoders is connected to the discriminator, which can solve the problem of poor reliability and effectiveness of the decoding classification model trained with EEG signals from cross-vendor devices, effectively improve the data utilization rate of EEG signals for training and reduce the cost of data use, effectively save the computing resources and time cost required for the model training process and reduce data dependence, improve the generalization ability of the decoding classification model, and thus effectively improve the reliability, effectiveness and efficiency of the EEG signal decoding classification model training process, and thus improve the effectiveness and reliability of using the trained decoding classification model to identify the decoding type and acquisition device type of EEG signal segments.

[0043] Additional advantages, purposes, and features of the present application will be described in part in the following description and will become apparent to those skilled in the art upon study of the following or may be learned from practice of the present application. The purposes and other advantages of the present application may be achieved and obtained by the structures specifically pointed out in the specification and drawings.

[0044] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings described herein are intended to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. The components in the drawings are not drawn to scale, but are only for the purpose of illustrating the principles of the present application. In order to facilitate the illustration and description of some parts of the present application, the corresponding parts in the drawings may be enlarged, that is, they may become larger than other components in the exemplary device actually manufactured according to the present application. In the drawings:

[0046] Figure 1 This is a first flow chart of a method for training a cross-device EEG signal decoding and classification model in one embodiment of the present application.

[0047] Figure 2 This is a schematic diagram showing an example of the architecture of a classification model based on a generative adversarial network provided in this application.

[0048] Figure 3 This is a second flow chart of the method for training a cross-device EEG signal decoding and classification model in one embodiment of the present application.

[0049] Figure 4 This is a third flow chart of the method for training a cross-device EEG signal decoding and classification model in one embodiment of the present application.

[0050] Figure 5 This is a flowchart of the time domain and frequency domain feature extraction steps in the EEG signal cross-device decoding and classification model training method in one embodiment of the present application.

[0051] Figure 6 This is a flowchart of the spatial correlation pooling step in the EEG signal cross-device decoding and classification model training method in one embodiment of the present application.

[0052] Figure 7 This is a flowchart of a method for training a cross-device EEG signal decoding and classification model in an application example of the present application.

[0053] Figure 8This is a schematic diagram illustrating an example of the execution architecture of S2 in the EEG signal cross-device decoding and classification model training method in an application example of the present application.

[0054] Figure 9 This is a schematic diagram illustrating an example of the execution architecture of S3 in the EEG signal cross-device decoding and classification model training method in an application example of the present application.

[0055] Figure 10 2 is a flow chart of a method for cross-device decoding and classification of EEG signals in one embodiment of the present application.

[0056] Figure 11 This is a structural diagram of an apparatus for training a cross-device EEG signal decoding and classification model in one embodiment of the present application.

[0057] Figure 12 Schematic diagram of the structure of the cross-device EEG signal decoding and classification device in one embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail in conjunction with the embodiments and drawings. Here, the illustrative embodiments of this application and their descriptions are used to explain this application, but are not intended to limit this application.

[0059] It should also be noted here that in order to avoid obscuring the present application due to unnecessary details, the accompanying drawings only show structures and / or processing steps that are closely related to the scheme according to the present application, while other details that are not closely related to the present application are omitted.

[0060] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0061] It should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0062] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0063] In order to effectively solve the current problems of poor model cross-device capabilities and serious waste of data information in EEG motor imagery decoding tasks, the embodiments of the present application respectively provide a method for training a cross-device decoding and classification model for EEG signals, a cross-device decoding and classification model training device for executing the cross-device decoding and classification model training method for EEG signals, a cross-device decoding and classification method for EEG signals, a cross-device decoding and classification device for executing the cross-device decoding and classification method for EEG signals, a physical device, a computer-readable storage and a computer program product. By combining the idea of a generative adversarial network (GAN) with an autoencoder based on a convolutional neural network to construct a decoding and classification model, the defect of poor performance when using data from cross-vendor devices is solved, thereby achieving the purpose of improving data utilization.

[0064] The details are described in detail through the following examples.

[0065] Based on this, the embodiment of the present application provides a method for training a cross-device EEG signal decoding classification model that can be implemented by a cross-device EEG signal decoding classification model training device, see Figure 1 The EEG signal cross-device decoding and classification model training method specifically includes the following contents:

[0066] Step 100: For each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, based on the pre-trained feature vector corresponding to each EEG signal segment and the decoding type label corresponding to each EEG signal segment, the latent space deep neural network corresponding to the classification model based on the adversarial generative network is pre-trained to obtain a pre-trained deep neural network for predicting the decoding type of the pre-trained feature vector.

[0067] In one embodiment of the present application, there are at least two types of EEG signal acquisition devices, and each type of EEG signal acquisition device can be represented by a unique manufacturer identifier or model of the different EEG signal acquisition devices.

[0068] It is understood that each signal segment group corresponding to the same EEG signal acquisition device type corresponds to each preset decoding type label. For example, if there are five decoding type labels, then there are also five signal segment groups corresponding to a certain EEG signal acquisition device type, and each group uniquely corresponds to a decoding type label. It should also be noted that the decoding type label corresponding to the signal segment group means that each EEG signal segment in the signal segment group is assigned the same decoding type label.

[0069] In one or more embodiments of the present application, the EEG signal segment can be abbreviated as a signal segment or an EEG signal segment. The decoding type label includes labels for representing different decoding types, and the decoding type includes at least a motor imagery state and a resting state. The motor imagery state can include at least imagining the movement of the left hand, right hand, left foot, or right foot, etc.

[0070] In step 100, the pre-trained feature vector As the latent space layer of the classifier (i.e., classification model), a network composed of the latent space layer and multiple layers of neural networks sequentially connected to the latent space layer is a latent space deep neural network.

[0071] Specifically, first freeze the autoencoder layer in the classification model and directly convert the pre-trained feature vector The latent space input is fed into the first network layer after the autoencoder layer. After passing through the multi-layer neural network, the predicted label (i.e., the decoding type of the pre-trained feature vector) is output. The cross-entropy loss is calculated, and the latent space deep neural network is continuously optimized until the latent space deep neural network converges. It is important to note that at this point, the neural network only has a prediction function, learning the model in terms of decoding prediction, rather than the discriminator role in the adversarial generative network (GAN). Therefore, the evaluation of the device model is ignored when calculating the loss. After training is complete, the latent space deep network model parameters are retained and used as the pre-trained deep neural network.

[0072] In one or more embodiments of the present application, the EEG signal acquisition device may be referred to simply as a device.

[0073] Step 200: Use the pre-trained deep neural network as the discriminator in the classification model based on the generative adversarial network, and input the feature matrix of each EEG signal segment corresponding to each EEG signal acquisition device type into the autoencoder based on the convolutional neural network corresponding to each EEG signal acquisition device type in the classification model to train the classification model, and then train the classification model to be a cross-device EEG signal decoding classification model for outputting the classification result data of the decoding type and EEG signal acquisition device type corresponding to the signal segment according to the feature matrix of the signal segment, wherein each of the autoencoders is connected to the discriminator.

[0074] In step 200, see Figure 2 The classification model based on the generative adversarial network can specifically include an autoencoder based on a convolutional neural network corresponding to each type of the EEG signal acquisition device (i.e. Figure 2 ), and each of the autoencoders is connected to its own latent space layer (i.e., the pre-trained feature vector ), each of the latent space layers is connected to an upsampling layer, the upsampling layer is connected to the discriminator, the discriminator classifies and discriminates the output features of the upsampling layer, and passes the discrimination loss to the autoencoder for optimizing the autoencoder parameters. Here, "device 1 feature matrix" refers to the feature matrix of each EEG signal segment corresponding to one type of EEG signal acquisition device, and "device 2 feature matrix" refers to the feature matrix of each EEG signal segment corresponding to another type of EEG signal acquisition device (feature matrix can also be called feature vector matrix).

[0075] Specifically, the data distribution between devices from different manufacturers is quite different, which is also the difficulty of the cross-device problem. In order to reduce the difference between the distributions, a discriminator based on the GAN model idea is introduced for cross-device training. By allowing the discriminator to determine the device source of the data in the latent space, the feature extraction process of the autoencoder for the data features is continuously optimized, so as to achieve the purpose that the latent space features cannot allow the discriminator to distinguish the device source, thereby reducing the data distribution of different devices and completing the task of cross-device decoding and classification of the model. The specific training process is: First, the feature vector matrix Matrix of device 1 and device 2 is converted into i A 20% validation set is reserved. Several batches are then divided for each device, and the model is trained alternately using the batches from the two devices. Unlike pre-training, during formal training, the latent space deep neural network acts as a discriminator, outputting predictions about the device from which the feature originated and the decoding task status. The calculated loss function is also passed to the upstream autoencoder for parameter optimization. This training process allows the autoencoder to learn information that reduces differences in data distribution across devices while ensuring that information that improves decoding accuracy is not lost.

[0076] From the above description, it can be seen that the EEG signal cross-device decoding and classification model training method provided in the embodiment of the present application can solve the problem of poor reliability and effectiveness of the decoding and classification model trained using EEG signals from cross-vendor devices, can effectively improve the data utilization rate of EEG signals for training and reduce data usage costs, can effectively save the computing resources and time costs required for the model training process and reduce data dependence, can improve the generalization ability of the decoding and classification model, and thus can effectively improve the reliability, effectiveness and efficiency of the EEG signal decoding and classification model training process, and thus can improve the effectiveness and reliability of using the trained decoding and classification model to identify the decoding type and acquisition device type of EEG signal segments.

[0077] In order to further improve the convergence ability of the EEG signal cross-device decoding classification model, in the EEG signal cross-device decoding classification model training method provided in the embodiment of the present application, see Figure 3The method for training a cross-device EEG signal decoding and classification model includes the following contents before step 100:

[0078] Step 010: Using the feature matrices corresponding to each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, respectively train the autoencoders based on convolutional neural networks corresponding to each of the EEG signal acquisition device types in the classification model based on the generative adversarial network, so that each of the autoencoders is used to compress the input feature matrix and output the compressed feature data corresponding to each of the feature matrices.

[0079] Specifically, in order to speed up the convergence of the model, this application adds a pre-training step before model training. Unlike conventional pre-training data sets, this application proposes a new spatial weighted data enhancement algorithm for generating pre-training features, the purpose of which is to use coarse fusion features to preheat the model parameters instead of random initialization. The specific steps are: initialize several autoencoders based on convolutional neural networks according to the number of cross-devices, and then use all the data from the same device to train the corresponding autoencoder for data compression, and calculate the reconstruction loss according to the following formula And continuously optimize the model parameters until the model converges. Then this application uses the compressed features (i.e. compressed feature data) of all data as h1...h i , whose dimension is recorded as e. After training is completed, all the model parameters of the autoencoder are retained.

[0080] Among them, M i represents the original feature matrix; Represents the reconstructed feature matrix obtained after the encoding and decoding process of the self-encoding.

[0081] Step 020: Obtain the pre-trained feature vector corresponding to each of the EEG signal segments based on the compressed feature data corresponding to each of the EEG signal acquisition device types.

[0082] In order to further improve the effectiveness and reliability of obtaining the pre-trained feature vectors corresponding to each of the EEG signal segments, and to further improve the convergence ability of the EEG signal cross-device decoding classification model, in a method for training an EEG signal cross-device decoding classification model provided in an embodiment of the present application, see Figure 4 Step 020 of the EEG signal cross-device decoding and classification model training method specifically includes the following content:

[0083] Step 021: Grouping the compressed feature data according to the types of the EEG signal acquisition devices to obtain compressed feature groups corresponding to the types of the EEG signal acquisition devices.

[0084] Step 022: Select one from each of the compressed feature groups as a baseline group, and select compressed feature data with the same decoding type label as each of the compressed feature data in the baseline group from the other compressed feature groups except the baseline group, so as to form a plurality of parent sample data corresponding to each of the compressed feature data in the baseline group.

[0085] Step 023: Obtain the average vector corresponding to each of the parent sample data as a pre-trained feature vector, determine the Euclidean distance corresponding to each of the parent sample data, and use the decoding type label of the feature matrix to which the parent sample data corresponding to each of the pre-trained feature vectors belongs as the decoding type label of each of the pre-trained feature vectors.

[0086] Specifically, the compressed feature data is classified into several groups according to different devices and labels. The compressed feature data in each compressed feature group comes from the same device and has the same label. Then, taking all groups of a certain device as the benchmark, the full sampling strategy is followed in the data of other different devices but with the same label to match all features in the benchmark group to form a pair of parent sample data. Assume that there are p hp in a benchmark group, and there are q h in the corresponding device groups with the same label. q After full sampling, this application can obtain p×q pairs of parent sample data, which are recorded as parents1...parents in this application. p×q Next, the application calculates the average vector of h in each pair as the newly generated pre-trained feature vector according to the following formula And calculate the Euclidean distance as the weight inverse of the pre-trained feature vector. If the distance between two features is smaller, it means that the fusion generated The more it helps to reduce the distribution differences across devices, thus improving The weight coefficient of , and vice versa.

[0087]

[0088] in, Represents the feature vector obtained after fusion, h p represents the pth compressed feature vector in device group 1, h q Represents the qth compressed feature vector in device group 2. Represents h p The i-th element of the vector, Represents h q The i-th element of the vector, e, represents the dimension of the h vector. It should be noted that the device group numbers 1 and 2 are only used to distinguish different devices in this description and are not actual numbers.

[0089] Finally, this application will generate several pre-trained feature vectors Its classification task label is consistent with the label of the parent sample, and the device label is randomly selected from the parent samples.

[0090] In the existing methods, there is a waste of resources due to the low utilization rate of EEG data. As biological data, EEG has the defects of high difficulty and high cost of collection. Due to the differences in manufacturers' equipment, data set merging has great limitations. For example, different public data sets are limited by the difference in the number of acquisition channels (number of electrodes). In the data set merging stage, a full conditional matching method is usually used to narrow the range of available data, or a channel number backward compatibility method is used to directly delete additional channel information for data with more channels. In addition, in the data feature extraction stage, the common method is to extract coarse-grained feature information in the time domain and frequency domain based on single-channel information. Spatial information is also limited to exploring the position distribution of channels, ignoring the dynamic relationship between brain activity and channel position from the perspective of temporal evolution. Obviously, extracting fine-grained features through improved methods is beneficial to improving the model's decoding ability of EEG motor imagery from the perspective of improving data utilization and information mining rate.

[0091] Based on this, in order to fully mine data information and improve the information mining rate, in the EEG signal cross-device decoding classification model training method provided in the embodiment of the present application, see Figure 3 The EEG signal cross-device decoding and classification model training method further specifically includes the following contents before step 010:

[0092] Step 030: Preprocess the original EEG signals corresponding to each EEG signal acquisition device type to obtain signal segment groups corresponding to each EEG signal acquisition device type, each of which contains EEG signal segments, wherein each signal segment group corresponding to the same EEG signal acquisition device type corresponds one-to-one to each preset decoding type label.

[0093] In an example of step 030, first, use equipment from different manufacturers to collect EEG signals of the subjects during motor imagery (imagining moving the left hand, right hand, left foot, right foot, etc.) and resting state, or use an internationally published EEG signal data set for motor imagery. Then use a general preprocessing process to uniformly preprocess all data, specifically including artifact removal, bandpass filtering, data segmentation and baseline correction. The data segmentation step is to segment the data according to the experimental task or stimulus event. After the preprocessing is completed, several groups of EEG signal segments will be obtained for each subject, and each segment is only a motor imagery task state or resting state. Assume that after the preprocessing is completed, there are M1 groups of EEG signal segments from device 1, M2 groups of EEG signal segments from device 2, and so on. It should be noted that the length and number of channels of the EEG signal segments vary from device to device and are not exactly the same.

[0094] Step 040: For each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, extract the feature matrix of each EEG signal segment for representing the time domain features and frequency domain features of the same dimension, and add the EEG signal acquisition device type corresponding to the EEG signal segment to the decoding type label corresponding to the EEG signal segment to which each feature matrix belongs.

[0095] In order to further fully mine data information and improve the information mining rate, in the EEG signal cross-device decoding classification model training method provided in the embodiment of the present application, see Figure 4 Step 040 of the EEG signal cross-device decoding and classification model training method specifically includes the following content:

[0096] Step 041: creating a sliding window, and each time the sliding window slides, segmenting each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type to obtain sub-signal segments with a length equal to the window width of the sliding window; and performing preset time domain and frequency domain feature extraction steps on the sub-signal segments to obtain one-dimensional features corresponding to each of the sub-signal segments for representing time domain features and frequency domain features;

[0097] Step 042: stacking the one-dimensional features obtained during each sliding of the sliding window to obtain a feature matrix corresponding to the EEG signal segments formed by the sub-signal segments.

[0098] Step 043: Add the EEG signal acquisition device type corresponding to the EEG signal segment to the decoding type label corresponding to the EEG signal segment to which each of the feature matrices belongs.

[0099] In order to further improve the effectiveness and reliability of time domain and frequency domain feature extraction, in the EEG signal cross-device decoding classification model training method provided in the embodiment of the present application, see Figure 5 The time domain and frequency domain feature extraction steps in the EEG signal cross-device decoding and classification model training method specifically include the following:

[0100] Step 01: performing a preset spatial correlation pooling step on each channel segment of each channel of the current sub-signal segment, so as to use the target feature vector corresponding to the sub-signal segment obtained by the spatial correlation pooling step as the time domain feature vector corresponding to the sub-signal segment;

[0101] And, step 02: performing Fourier transform on each of the channel segments of each channel corresponding to the sub-signal segment, and then performing a preset spatial correlation pooling step on each of the channel segments after the Fourier transform, so as to use the target feature vector corresponding to the sub-signal segment obtained by the spatial correlation pooling step as the frequency domain feature vector corresponding to the sub-signal segment;

[0102] Step 03: Connect the time domain feature vector and the frequency domain feature vector corresponding to the sub-signal segment to obtain the one-dimensional feature corresponding to the sub-signal segment.

[0103] In order to further improve the effectiveness and reliability of spatial correlation pooling, in the EEG signal cross-device decoding classification model training method provided in the embodiment of the present application, see Figure 6 The spatial correlation pooling step in the EEG signal cross-device decoding classification model training method specifically includes the following contents:

[0104] Step 04: converting each channel segment of each channel of the current sub-signal segment into a discrete point vector of each channel based on the average interval time sampling method;

[0105] Step 05: determining the Pearson correlation coefficient between every two channels in each channel according to the discrete point vectors of each channel, so as to obtain a correlation coefficient matrix corresponding to the sub-signal segment;

[0106] And, step 06: determining the electrode coordinates of each channel of the current sub-signal segment based on a Cartesian coordinate algorithm, and determining the Euclidean distance between every two electrode coordinates to obtain a Euclidean distance matrix corresponding to the sub-signal segment; step 07: determining the average spatial correlation index corresponding to each of the channels of the sub-signal segment according to the correlation coefficient matrix and the Euclidean distance matrix corresponding to the sub-signal segment;

[0107] Step 08: performing average pooling processing on the average spatial correlation index corresponding to each of the channels of the sub-signal segment to obtain a target feature vector corresponding to the sub-signal segment.

[0108] Specifically, in a specific example of step 040, the EEG signal segment obtained in step 030 is not suitable for directly performing the motor imagery decoding and classification task due to its high dimensionality and lack of information extraction. It needs to be further processed to obtain a feature matrix. The specific steps are as follows: Assume that the first EEG signal segment in M1 is taken as m1. m1 has n channels, denoted as C1...C n . First, in order to extract fine-grained features and capture the dynamic changes of the time-frequency-space dimensions, this application uses a dynamic sliding time window segmentation algorithm different from the static fixed time window segmentation algorithm. Suppose a sliding window Win is created with a window width of w seconds and a sliding step of s seconds. It should be noted that in order to capture feature information more finely, w is generally set to less than 1 second. At the same time, in order to better capture dynamically changing information and reduce the differences between adjacent windows, each next sliding window needs to overlap with the previous window, so s is generally set to half of w. This application records each slide as Slide i Each slide will obtain a sub-signal segment of length w corresponding to the EEG signal segment for all channels. The i-th channel segment of the n-th channel is recorded as This application takes all channel data in the first window as an example (denoted as ), each channel is a continuous band, and then based on the spatial correlation pooling algorithm newly proposed in this application, a feature vector that integrates spatial relationships and dynamic changes is extracted from the time domain and frequency domain.

[0109] The following takes the time domain feature extraction process as an example to explain the specific steps of the algorithm in detail. First, use the average interval time sampling method to The continuous band is converted into a discrete point vector Then calculate the Pearson correlation coefficient r between the two channels a and b according to the following formula a，b ; Get the correlation coefficient matrix Among them They are and The mean of .

[0110]

[0111] Where T is the number of discrete sampling time points, u t represents the amplitude value of channel a of a certain segment at time point t, v trepresents the amplitude value of channel b of a certain segment at time point t, They are and The mean value, r a，b Represents the Pearson correlation coefficient between channels a and b.

[0112] Then, based on the Cartesian coordinate algorithm, a coordinate system is established with the center of the head as the three-dimensional origin coordinate to obtain the electrode coordinates of each channel. Then, the Euclidean distance D between the two electrodes is calculated according to the following formula: 1，2 ...D a，b .

[0113] Where (x a ,y a , z a ) represents the three-dimensional coordinates of channel a, (x b ,y b , z b ) represents the three-dimensional coordinates of channel b. D a，b Represents the Euclidean distance between channels a and b.

[0114] After obtaining the correlation coefficient and Euclidean distance between two electrodes, the average spatial correlation index Dr1...Dr is calculated for each channel according to the following formula: n This index describes the correlation between the spatial position of each channel data and the other channel data in the time domain. The larger the value, the more similar the electrode is to the distant channel in the time domain of this window, and the smaller the value, the opposite is true.

[0115] Among them, Dr. a represents the average spatial correlation index of channel a. D a，i represents the Euclidean distance between channel a and channel i, r a，i represents the Pearson correlation coefficient between channel a and channel i. n represents the number of electrode channels.

[0116] Next, since different manufacturers' equipment or different acquisition strategies will result in different dimensions of the vector composed of the average spatial correlation index, in order to facilitate the pre-training of the subsequent model, a pooling algorithm is further added to scale the vector to a uniform dimension. Specifically, the average pooling idea in the convolutional neural network is used for reference. Starting from the first channel, the average value is taken from the index including itself and the adjacent index on the right as an element of the output feature Dr′1, and so on to obtain the next round of features. Repeat the above steps until the feature dimensions of all devices are unified. At this point, the application will obtain a time domain feature vector (that is, the target feature vector as the time domain feature vector) TV1 of uniform dimension in the time domain of the time window. In addition, Transform from time domain to frequency domain, and then extract the frequency domain feature vector of unified dimension (i.e., the target feature vector as the frequency domain feature vector) FV1 by the same steps. After the above steps, this application connects TV1 and FV1 together to obtain the one-dimensional feature TFV1 of Slide1. By continuously sliding the window, this application can obtain several TFVs i . Finally, by putting these TFV i Stacked together, this application extracts a feature matrix Matrix1 for the m1 signal. By analogy, this application can obtain several Matrix i , they have the same matrix dimensions even if they come from different devices. Furthermore, all feature labels are re-encoded, adding device type information to the original task type label. For example, in a binary classification task, the original task type label for device 1 is 1, indicating that the EEG signal is in the resting state. After adding device type information, it becomes (1, 1), and so on.

[0117] To further illustrate the above embodiment, this application also provides an application example of a method for training a cross-device EEG signal decoding and classification model. Figure 7 The application examples of the EEG signal cross-device decoding and classification model training method specifically include the following:

[0118] S1. EEG signal preprocessing: Obtain the subject's EEG signal data and perform preprocessing. The signal data of devices from different manufacturers use a unified preprocessing method.

[0119] Specifically, first, use equipment from different manufacturers to collect EEG signals of subjects during motor imagery (imagining moving the left hand, right hand, left foot, right foot, etc.) and at rest, or use an internationally published EEG signal dataset for motor imagery. Then use a common preprocessing process to uniformly preprocess all data, including artifact removal, bandpass filtering, data segmentation, and baseline correction. The data segmentation step is to segment the data according to the experimental task or stimulus event. After the preprocessing is completed, several groups of EEG signal segments will be obtained for each subject, and each segment is only a motor imagery task state or resting state. Assume that in the data after preprocessing, there are M1 group EEG signal segments from device 1, M2 group EEG signal segments from device 2, and so on. It should be noted that the length and number of channels of the EEG signal segments vary from device to device and are not exactly the same.

[0120] S2. Time-frequency dual feature extraction with fine-grained fusion of spatial information: In order to fully exploit the information that is beneficial to the decoding and classification tasks, fine-grained feature vectors are extracted from the preprocessing results obtained in S1 from three levels: time domain, frequency domain, and spatial relationship. Specifically, the overall signal is divided into several signal segments based on the dynamic sliding window algorithm, and then a newly proposed spatial correlation pooling algorithm is used to extract feature vectors from the fused spatial relationship and dynamically changing time domain and frequency domain.

[0121] Specifically, see Figure 8 The EEG signal segment data obtained for S1 is not suitable for direct motor imagery decoding and classification due to its high dimensionality and lack of information extraction. It needs to be further processed to obtain a feature vector. The specific steps are as follows: Assume that the first EEG signal segment in M1 is taken as m1. m1 has n channels, denoted as C1...C n . First, in order to extract fine-grained features and capture the dynamic changes of the time-frequency-space dimensions, this application uses a dynamic sliding time window segmentation algorithm instead of a static fixed time window segmentation algorithm. Suppose a sliding window Win is created with a window width of w seconds and a sliding step of s seconds. It should be noted that in order to capture feature information more finely, w is generally set to less than 1 second. At the same time, in order to better capture dynamically changing information and reduce the differences between adjacent windows, each next sliding window needs to overlap with the previous window, so s is generally set to half of w. This application records each slide as Slide i Each slide will obtain a sub-signal segment of length w corresponding to the EEG signal segment for all channels. The i-th channel segment of the n-th channel is recorded as This application takes all channel data in the first window as an example (denoted as ), each channel is a continuous band, and then based on the spatial correlation pooling algorithm newly proposed in this application, a feature vector that integrates spatial relationships and dynamic changes is extracted from the time domain and frequency domain.

[0122] The following takes the time domain feature extraction process as an example to explain the specific steps of the algorithm in detail. First, use the average interval time sampling method to The continuous band is converted into a discrete point vector Then calculate the Pearson correlation coefficient r between the two channels a and b according to formula (1) and formula (2) a，b ; Get the correlation coefficient matrix Among them They are and The mean of .

[0123]

[0124] Then, based on the Cartesian coordinate algorithm, a coordinate system is established with the center of the head as the three-dimensional origin coordinate to obtain the electrode coordinates of each channel. Then, the Euclidean distance D between the two electrodes is calculated according to formula (3): 1，2 ...D a，b .

[0125]

[0126] After obtaining the correlation coefficient and Euclidean distance between two electrodes, the average spatial correlation index Dr1...Dr is calculated for each channel according to formula (4). n This index describes the correlation between the spatial position of each channel data and the other channel data in the time domain. The larger the value, the more similar the electrode is to the distant channel in the time domain of this window, and the smaller the value, the opposite is true.

[0127]

[0128] Next, since different manufacturers' equipment or different acquisition strategies will result in different dimensions of the vector composed of the average spatial correlation index, in order to facilitate the pre-training of the subsequent model, a pooling algorithm is further added to scale the vector to a uniform dimension. Specifically, the average pooling idea in the convolutional neural network is used for reference. Starting from the first channel, the average value is taken from the index including itself and the adjacent index on the right as an element of the output feature Dr′1, and so on to obtain the next round of features. Repeat the above steps until the feature dimensions of all devices are unified. At this point, the application will obtain a time domain feature vector (that is, the target feature vector as the time domain feature vector) TV1 of uniform dimension in the time domain of the time window. In addition, Transform from time domain to frequency domain, and then extract the frequency domain feature vector of unified dimension (i.e., the target feature vector as the frequency domain feature vector) FV1 by the same steps. After the above steps, this application connects TV1 and FV1 together to obtain the one-dimensional feature TFV1 of Slide1. By continuously sliding the window, this application can obtain several TFVs i . Finally, by putting these TFV i Stacked together, this application extracts a feature matrix Matrix1 for the m1 signal. By analogy, this application can obtain several Matrix i , they have the same matrix dimensions even if they come from different devices. Furthermore, all feature labels are re-encoded, adding device type information to the original task type label. For example, in a binary classification task, the original task type label for device 1 is 1, indicating that the EEG signal is in the resting state. After adding device type information, it becomes (1, 1), and so on.

[0129] S3. Generate pre-trained features based on the spatially weighted data augmentation algorithm: First, train an autoencoder based on a convolutional neural network to reduce the feature dimensions of different devices to the same level in the latent space. Then, use the spatially weighted data augmentation algorithm in the latent space to generate new features for pre-training by pairing features of different devices with the same label. The task labels of the new features are consistent with the task labels of the original features.

[0130] Specifically, see Figure 9 In order to speed up the convergence of the model, this application adds a pre-training step before model training. Different from conventional pre-training data sets, this application proposes a new spatial weighted data enhancement algorithm for generating pre-training features, the purpose of which is to use coarse fusion features to preheat the model parameters instead of random initialization. The specific steps are: initialize several autoencoders based on convolutional neural networks according to the number of cross-devices, and then use all the data of the same device to train the corresponding autoencoders for data compression, calculate the reconstruction loss according to formula (5) and continuously optimize the model parameters until the model converges. Then this application uses the compressed features of all data as h1…h i , whose dimension is recorded as e. After training is completed, all the model parameters of the autoencoder are retained.

[0131] Then, the compressed feature data is classified into several groups according to different devices and labels. The data in each compressed feature group comes from the same device and has the same label. Then, taking all groups of a certain device as the benchmark, the full sampling strategy is followed in the data of other different devices but with the same label to match all features in the benchmark group to form a pair of parent sample data. Assuming that there are p h in a benchmark group p , corresponding to q h in different device groups with the same label q After full sampling, this application can obtain p×q pairs of parent sample data, which are recorded as parents1…parents in this application. p×q Next, the application calculates the average vector of h in each pair as the newly generated pre-trained feature vector according to formula (6) And calculate the Euclidean distance as the weight inverse of the pre-trained feature vector. If the distance between two features is smaller, it means that the fusion generated The more it helps to reduce the distribution differences across devices, thus improving The weight coefficient of , and vice versa.

[0132]

[0133] Finally, this application will generate several pre-trained feature vectors Its classification task label is consistent with the label of the parent sample, and the device label is randomly selected from the parent samples.

[0134] S4. Pretrain the latent space deep learning model: Use the pretrained features generated in S3 to pretrain the model. Specifically, freeze the autoencoder layers in the model and use the newly generated pretrained features to train only the deep neural network in the latent space. The predicted output at this point only reflects the model's performance in decoding and classifying the EEG signal (resting state vs. motor imagery).

[0135] Specifically, we use the pre-trained feature vector generated in step 3 Pre-train the model's latent space deep neural network. The specific steps are: first freeze the autoencoder layer in the model, and then directly This is fed into the first layer of the latent space network as input. After passing through multiple layers of the neural network, the predicted label is output. Cross-entropy loss is calculated, and the model is continuously optimized until convergence. It's important to note that at this point, the neural network only performs prediction functions, not the discriminator role in GANs. Instead, the model learns the decoding and prediction aspects. Therefore, the device model is ignored when calculating the loss. The latent space deep network model parameters are retained after training.

[0136] S5: Train a cross-device model based on adversarial thinking. The feature matrix extracted in S2 is used as training data, and the model pre-trained in S4 is used for formal training to test the model's performance in the cross-device decoding and classification task. The predicted output reflects both the model's performance in decoding and classifying the EEG signal and its performance in identifying the device type.

[0137] Specifically, the data distribution between devices from different manufacturers is quite different, which is also the difficulty of the cross-device problem. In order to reduce the difference between the distributions, a discriminator based on the GAN model idea is introduced for cross-device training. By allowing the discriminator to determine the device source of the data in the latent space, the feature extraction process of the autoencoder for the data features is continuously optimized, so as to achieve the purpose that the latent space features cannot allow the discriminator to distinguish the device source, thereby reducing the data distribution of different devices and completing the task of cross-device decoding and classification of the model. The specific training process is: First, the feature vector matrix Matrix of device 1 and device 2 is converted into iA 20% validation set is reserved. Several batches are then divided for each device, and the model is trained alternately using the batches from the two devices. Unlike pre-training, during formal training, the latent space deep neural network acts as a discriminator, outputting predictions about the device from which the feature originated and the decoding task status. The calculated loss function is also passed to the upstream autoencoder for parameter optimization. This training process allows the autoencoder to learn information that reduces differences in data distribution across devices while ensuring that information that improves decoding accuracy is not lost.

[0138] Based on the EEG signal cross-device decoding and classification model training method provided in the above embodiment, this application also provides an embodiment of the EEG signal cross-device decoding and classification method, see Figure 10 The cross-device EEG signal decoding and classification method specifically includes the following contents:

[0139] Step 300: extracting a feature matrix for representing time domain features and frequency domain features of the same dimension from the EEG signal segment corresponding to the current target EEG signal.

[0140] It can be understood that the extraction content of the feature matrix performed in step 300 can be implemented in the manner of step 040 in the EEG signal cross-device decoding and classification model training method.

[0141] Step 400: Input the feature matrix corresponding to the target EEG signal into a preset EEG signal cross-device decoding classification model, so that the EEG signal cross-device decoding classification model outputs classification result data corresponding to the target EEG signal, which includes the decoding type and the EEG signal acquisition device type, wherein the EEG signal cross-device decoding classification model is pre-trained based on the EEG signal cross-device decoding classification model training method.

[0142] The EEG signal cross-device decoding and classification model training method mentioned in the embodiment of the EEG signal cross-device decoding and classification method provided in this application can be specifically used to adopt the processing flow of the embodiment of the EEG signal cross-device decoding and classification model training method in the above embodiment. Its function will not be repeated here, and you can refer to the detailed description of the above-mentioned EEG signal cross-device decoding and classification model training method embodiment.

[0143] From the above description, it can be seen that the cross-device EEG signal decoding and classification method provided in the embodiment of the present application can improve the effectiveness and reliability of using the trained decoding classification model to identify the decoding type and acquisition device type of EEG signal segments.

[0144] From the software level, this application also provides an EEG signal cross-device decoding classification model training device for executing all or part of the EEG signal cross-device decoding classification model training method, see Figure 11The EEG signal cross-device decoding and classification model training device specifically includes the following contents:

[0145] A pre-training module 10 is configured to pre-train a latent space deep neural network corresponding to a classification model based on a generative adversarial network for each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, based on a pre-trained feature vector corresponding to each EEG signal segment and a decoding type label corresponding to each EEG signal segment, to obtain a pre-trained deep neural network for predicting the decoding type of the pre-trained feature vector;

[0146] The training module 20 is used to use the pre-trained deep neural network as a discriminator in a classification model based on a generative adversarial network, and input the feature matrices of each EEG signal segment corresponding to each type of the EEG signal acquisition device into the autoencoder based on a convolutional neural network corresponding to each type of the EEG signal acquisition device in the classification model to train the classification model, and then train the classification model to be a cross-device EEG signal decoding classification model that outputs the classification result data corresponding to the signal segment, which includes the decoding type and the EEG signal acquisition device type, based on the feature matrix of the signal segment, wherein each of the autoencoders is connected to the discriminator.

[0147] The embodiment of the EEG signal cross-device decoding classification model training device provided in this application can be specifically used to execute the processing flow of the embodiment of the EEG signal cross-device decoding classification model training method in the above-mentioned embodiment. Its functions will not be repeated here, and you can refer to the detailed description of the above-mentioned EEG signal cross-device decoding classification model training method embodiment.

[0148] The part of the EEG signal cross-device decoding classification model training device that performs EEG signal cross-device decoding classification model training can be completed in the server or in the client device. The specific selection can be based on the processing power of the client device and the limitations of the user's usage scenario. This application is not limited to this. If all operations are completed in the client device, the client device may also include a processor for the specific processing of the EEG signal cross-device decoding classification model training.

[0149] The client device may include a communication module (i.e., a communication unit) that can establish a communication connection with a remote server to implement data transmission with the server. The server may include a server on the task scheduling center side, and in other implementation scenarios, may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0150] The server and the client device may communicate using any suitable network protocol, including network protocols that have not yet been developed as of the filing date of this application. Examples of such network protocols include TCP / IP, UDP / IP, HTTP, and HTTPS. Furthermore, examples of such network protocols include RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols, which are used on top of the aforementioned protocols.

[0151] From the above description, it can be seen that the EEG signal cross-device decoding and classification model training device provided in the embodiment of the present application can solve the problem of poor reliability and effectiveness of the decoding and classification model trained using EEG signals from cross-vendor devices, can effectively improve the data utilization rate of EEG signals for training and reduce data usage costs, can effectively save the computing resources and time costs required for the model training process and reduce data dependence, can improve the generalization ability of the decoding and classification model, and thus can effectively improve the reliability, effectiveness and efficiency of the EEG signal decoding and classification model training process, and thus can improve the effectiveness and reliability of using the trained decoding and classification model to identify the decoding type and acquisition device type of EEG signal segments.

[0152] From the software level, this application also provides an EEG signal cross-device decoding and classification device for executing all or part of the EEG signal cross-device decoding and classification method, see Figure 12 The EEG signal cross-device decoding and classification device specifically includes the following contents:

[0153] A feature extraction module 30 is used to extract a feature matrix representing time domain features and frequency domain features of the same dimension from the EEG signal segment corresponding to the current target EEG signal;

[0154] It can be understood that the feature matrix extraction performed by the feature extraction module 30 can be implemented in the manner of step 040 in the EEG signal cross-device decoding classification model training method.

[0155] The model classification module 40 is used to input the feature matrix corresponding to the target EEG signal into a preset EEG signal cross-device decoding classification model, so that the EEG signal cross-device decoding classification model outputs classification result data corresponding to the target EEG signal, which includes the decoding type and the EEG signal acquisition device type, wherein the EEG signal cross-device decoding classification model is pre-trained based on the EEG signal cross-device decoding classification model training method.

[0156] The embodiment of the EEG signal cross-device decoding and classification apparatus provided in this application can be specifically used to execute the processing flow of the embodiment of the EEG signal cross-device decoding and classification method in the above-mentioned embodiment. Its functions will not be repeated here, and reference can be made to the detailed description of the above-mentioned EEG signal cross-device decoding and classification method embodiment.

[0157] From the above description, it can be seen that the cross-device EEG signal decoding and classification device provided in the embodiment of the present application can improve the effectiveness and reliability of using the trained decoding classification model to identify the decoding type and acquisition device type of EEG signal segments.

[0158] The present application also provides an electronic device that may include a processor, a memory, a receiver, and a transmitter. The processor is used to execute the EEG signal cross-device decoding and classification model training method and / or the aforementioned EEG signal cross-device decoding and classification method mentioned in the above embodiments, wherein the processor and the memory may be connected via a bus or other means, with bus connection being used as an example. The receiver may be connected to the processor and the memory via a wired or wireless manner.

[0159] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0160] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as the program instructions / modules corresponding to the EEG signal cross-device decoding and classification model training method in the embodiments of the present application. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the EEG signal cross-device decoding and classification model training method and / or the aforementioned EEG signal cross-device decoding and classification method in the above-mentioned method embodiments.

[0161] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0162] The one or more modules are stored in the memory, and when executed by the processor, perform the EEG signal cross-device decoding and classification model training method and / or the aforementioned EEG signal cross-device decoding and classification method in the embodiment.

[0163] In some embodiments of the present application, the user equipment may include a processor, a memory and a transceiver unit, and the transceiver unit may include a receiver and a transmitter. The processor, memory, receiver and transmitter may be connected through a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0164] As an implementation method, the functions of the receiver and transmitter in this application can be considered to be implemented through a transceiver circuit or a dedicated transceiver chip, and the processor can be considered to be implemented through a dedicated processing chip, a processing circuit or a general-purpose chip.

[0165] As another implementation method, it is possible to use a general-purpose computer to implement the server provided in the embodiments of the present application. That is, the program code for implementing the functions of the processor, receiver, and transmitter is stored in a memory, and the general-purpose processor implements the functions of the processor, receiver, and transmitter by executing the code in the memory.

[0166] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned EEG signal cross-device decoding and classification model training method and / or the aforementioned EEG signal cross-device decoding and classification method. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the art.

[0167] An embodiment of the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the aforementioned EEG signal cross-device decoding and classification model training method, and / or implements the steps of the aforementioned EEG signal cross-device decoding and classification method.

[0168] It should be understood by those skilled in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of this application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0169] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0170] In this application, the features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace the features of other embodiments.

[0171] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Those skilled in the art will appreciate that various modifications and variations of the present embodiment are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A method for training a cross-device EEG signal decoding and classification model, characterized in that: include: For each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, based on the pre-trained feature vector corresponding to each EEG signal segment and the decoding type label corresponding to each EEG signal segment, a latent space deep neural network corresponding to the classification model based on the generative adversarial network is pre-trained to obtain a pre-trained deep neural network for predicting the decoding type of the pre-trained feature vector; The pre-trained deep neural network is used as the discriminator in the classification model based on the generative adversarial network, and the feature matrix of each EEG signal segment corresponding to each type of the EEG signal acquisition device is respectively input into the autoencoder based on the convolutional neural network corresponding to each type of the EEG signal acquisition device in the classification model to train the classification model, and then the classification model is trained to be an EEG signal cross-device decoding classification model for outputting the classification result data corresponding to the signal segment including the decoding type and the EEG signal acquisition device type according to the feature matrix of the signal segment, wherein each of the autoencoders is connected to the discriminator.

2. The method for training an EEG signal cross-device decoding and classification model according to claim 1, characterized in that: Before pre-training the latent space deep neural network corresponding to the classification model based on the generative adversarial network based on the pre-trained feature vectors corresponding to the respective EEG signal segments and the decoding type labels corresponding to the respective EEG signal segments, the method further includes: Using the feature matrices corresponding to the respective EEG signal segments in the respective signal segment groups corresponding to the respective EEG signal acquisition device types, respectively training the convolutional neural network-based autoencoders corresponding to the respective EEG signal acquisition device types in the classification model based on the generative adversarial network, so that each of the autoencoders is used to respectively compress the input feature matrices and output compressed feature data corresponding to each of the feature matrices; According to the compressed feature data corresponding to each type of the EEG signal acquisition device, a pre-trained feature vector corresponding to each of the EEG signal segments is obtained.

3. The method for training an EEG signal cross-device decoding and classification model according to claim 2, wherein: The step of obtaining the pre-trained feature vectors corresponding to the respective EEG signal segments according to the respective compressed feature data corresponding to the respective EEG signal acquisition device types includes: Grouping the compressed feature data according to the type of each EEG signal acquisition device to obtain compressed feature groups corresponding to each type of the EEG signal acquisition device; Selecting one of the compressed feature groups as a benchmark group, and selecting compressed feature data having the same decoding type label as that corresponding to each compressed feature data in the benchmark group from the other compressed feature groups except the benchmark group, to form a plurality of parent sample data corresponding to each compressed feature data in the benchmark group; The average vector corresponding to each of the parent sample data is obtained as a pre-trained feature vector, and the Euclidean distance corresponding to each of the parent sample data is determined, and the decoding type label of the feature matrix to which the parent sample data corresponding to each of the pre-trained feature vectors belongs is used as the decoding type label of each of the pre-trained feature vectors.

4. The method for training an EEG signal cross-device decoding and classification model according to claim 2, wherein: Before respectively training the autoencoders based on convolutional neural networks corresponding to the respective EEG signal acquisition device types in the classification model based on the generative adversarial network using the feature matrices corresponding to the respective EEG signal segments in the respective signal segment groups corresponding to the respective EEG signal acquisition device types, the method further includes: Preprocessing the original EEG signals corresponding to each EEG signal acquisition device type to obtain signal segment groups corresponding to each EEG signal acquisition device type, each including EEG signal segments, wherein each signal segment group corresponding to the same EEG signal acquisition device type has a one-to-one correspondence with each preset decoding type label; For each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, the feature matrix for representing the time domain features and frequency domain features of the same dimension of each EEG signal segment is extracted respectively, and the EEG signal acquisition device type corresponding to the EEG signal segment is added to the decoding type label corresponding to the EEG signal segment to which each feature matrix belongs.

5. The method for training an EEG signal cross-device decoding and classification model according to claim 4, characterized in that: The method comprises: creating a sliding window, and segmenting each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type each time, to obtain sub-signal segments having a length equal to the window width of the sliding window; and performing preset time domain and frequency domain feature extraction steps on the sub-signal segments to obtain one-dimensional features corresponding to each sub-signal segment for representing the time domain features and the frequency domain features; The one-dimensional features obtained during each sliding of the sliding window are stacked to obtain a feature matrix corresponding to the electroencephalogram signal segment composed of each sub-signal segment.

6. The method for training an EEG signal cross-device decoding and classification model according to claim 5, characterized in that: The time domain and frequency domain feature extraction steps include: Performing a preset spatial correlation pooling step on each channel segment of each channel of the current sub-signal segment, so as to use the target feature vector corresponding to the sub-signal segment obtained by the spatial correlation pooling step as the time domain feature vector corresponding to the sub-signal segment; Furthermore, Fourier transform is performed on each channel segment of each channel corresponding to the sub-signal segment, and then a preset spatial correlation pooling step is performed on each channel segment after the Fourier transform, so as to use the target feature vector corresponding to the sub-signal segment obtained by the spatial correlation pooling step as the frequency domain feature vector corresponding to the sub-signal segment; and the time domain feature vector and the frequency domain feature vector corresponding to the sub-signal segment are connected to obtain a one-dimensional feature corresponding to the sub-signal segment.

7. The method for training an EEG signal cross-device decoding and classification model according to claim 6, characterized in that: The spatial correlation pooling step includes: Converting each channel segment of each channel of the current sub-signal segment into a discrete point vector of each channel based on an average interval time sampling method; Determining the Pearson correlation coefficient between every two channels in each of the channels according to the discrete point vectors of each channel, so as to obtain a correlation coefficient matrix corresponding to the sub-signal segments; and determining the electrode coordinates of each channel of the current sub-signal segment based on a Cartesian coordinate algorithm, and determining the Euclidean distance between every two electrode coordinates to obtain a Euclidean distance matrix corresponding to the sub-signal segment; determining, according to the correlation coefficient matrix and the Euclidean distance matrix corresponding to the sub-signal segments, an average spatial correlation index corresponding to each of the channels of the sub-signal segments; Average pooling is performed on the average spatial correlation index corresponding to each of the channels of the sub-signal segment to obtain a target feature vector corresponding to the sub-signal segment.

8. A cross-device decoding and classification method for EEG signals, characterized in that: include: Extracting a feature matrix representing time domain features and frequency domain features of the same dimension from the EEG signal segment corresponding to the current target EEG signal; The feature matrix corresponding to the target EEG signal is input into a preset EEG signal cross-device decoding classification model, so that the EEG signal cross-device decoding classification model outputs classification result data corresponding to the target EEG signal, which includes the decoding type and the EEG signal acquisition device type, wherein the EEG signal cross-device decoding classification model is pre-trained based on the EEG signal cross-device decoding classification model training method as described in any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the EEG signal cross-device decoding and classification model training method as described in any one of claims 1 to 7, and / or implements the EEG signal cross-device decoding and classification method as described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is processed When executed by the device, the method for training a cross-device EEG signal decoding and classification model according to any one of claims 1 to 7 is implemented, and / or, Implement the cross-device EEG signal decoding and classification method as described in claim 8.

Images