Electroencephalogram signal cross-device decoding classification model training method, decoding classification method and device

Abstract

The invention provides an electroencephalogram signal cross-equipment decoding classification model training method, a decoding classification method and equipment. The training method comprises the following steps: pre-training a hidden space deep neural network corresponding to a classification model based on an adversarial generative network based on a pre-training feature vector and a decoding type label corresponding to each electroencephalogram signal segment; using the network as a discriminator in a classification model, and inputting the feature matrix of the electroencephalogram signal fragment into an auto-encoder based on the convolutional neural network in the classification model to train the classification model; and training the classification model into an electroencephalogram signal cross-device decoding classification model for outputting a decoding type of electroencephalogram signal fragments and an electroencephalogram signal acquisition device type. According to the method, the problem that a decoding classification model trained by electroencephalogram signals of cross-manufacturer equipment is poor in reliability and effectiveness can be solved, and the data utilization rate of the electroencephalogram signals for training is increased.

Description

Technical Field

[0001] The present application relates to the fields of electroencephalogram (EEG) technology and deep learning technology, and in particular to a method for training a cross-device decoding and classification model for EEG signals, a decoding and classification method, and a device. Background Art

[0002] As an emerging technology that can establish a direct connection between the brain and external devices and realize the interaction between brain signals and external devices, brain-computer interface has brought new hope for the rehabilitation of stroke patients. In brain-computer interface technology, the way to capture brain information is divided into non-invasive, semi-invasive and invasive methods according to the placement of the electrode relative to the cerebral cortex, and its difficulty and danger increase step by step. At present, non-invasive signal acquisition methods represented by electroencephalogram (EEG) are the mainstream of research. It captures the spontaneous biopotential information of the brain through electrodes and obtains data after a series of steps such as amplification and processing. It reflects the spontaneous and rhythmic potential changes of cells in the brain under different thinking activities. And compared with other implantable electrodes, EEG technology benefits from its unique advantage of non-invasiveness. There is no problem of considering the surgical conditions of the subjects and affecting the survival rate, which enables it to cover more users and scenarios. Among them, the most studied scenario is the perception of motor imagery.

[0003] Motor imagery (MI) is a mental training method that simulates motor movements or activity scenes in the mind. The motor imagery data captured by EEG technology and then decoded through a series of processing steps (such as distinguishing between resting state and movement of left and right hands, feet and legs) can be used for downstream rehabilitation tasks. This method can not only be used to improve athlete performance, but also assist patient rehabilitation, which further demonstrates the great potential of brain-computer interface technology in the field of rehabilitation.

[0004] However, although the EEG-based motor imagery decoding task has achieved great success in research and practice at this stage, there are still many limitations in data utilization and model training. First of all, most of the decoding models based on EEG data rely on signals collected by a single device, and the generalization ability of the model is difficult to apply to data collected by another manufacturer's device. Since the parameters of hardware devices such as electrodes and amplifiers vary from manufacturer to manufacturer, even the same imagination signal of the same subject will have large data distribution differences. The distribution difference is obviously not conducive to the model's decoding task for machine learning models. Even if the original model still has excellent decoding ability after training on the new device data set again, it also means that more computing resources and longer time are required, thereby increasing the training cost. At the same time, the essence of this method is to always let the model strengthen the learning of later data without considering the retention of information learned from previous data. This will make the model always sensitive to a single data after each training and generate data dependence. If there are multi-vendor device data in the real scene, it will perform poorly. In addition, although there are a few studies on the decoding and classification of EEG cross-device data in motor imagery tasks, since they learn the mapping relationship of device data distribution, they are limited by the chain relationship in scenarios with data across more than two devices, which increases the cost of use.

[0005] Therefore, there is an urgent need to design a method that can solve the problems of poor reliability and effectiveness of decoding and classification models trained with EEG signals from cross-vendor devices and low data utilization of EEG signals used for training. Summary of the invention

[0006] In view of this, the embodiments of the present application provide a method for training an EEG signal cross-device decoding and classification model, a decoding and classification method and a device to eliminate or improve one or more defects existing in the prior art.

[0007] One aspect of the present application provides a method for training an EEG signal cross-device decoding classification model, comprising: For each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, based on the pre-trained feature vector corresponding to each EEG signal segment and the decoding type label corresponding to each EEG signal segment, a latent space deep neural network corresponding to the classification model based on the generative adversarial network is pre-trained to obtain a pre-trained deep neural network for predicting the decoding type of the pre-trained feature vector; The pre-trained deep neural network is used as the discriminator in the classification model based on the generative adversarial network, and the feature matrices of each EEG signal segment corresponding to each type of the EEG signal acquisition device are respectively input into the autoencoder based on the convolutional neural network corresponding to each type of the EEG signal acquisition device in the classification model to train the classification model, and then the classification model is trained to be an EEG signal cross-device decoding classification model for outputting the classification result data of the decoding type and the EEG signal acquisition device type corresponding to the signal segment according to the feature matrix of the signal segment, wherein each of the autoencoders is connected to the discriminator.

[0008] In some embodiments of the present application, before pre-training the latent space deep neural network corresponding to the classification model based on the generative adversarial network based on the pre-trained feature vectors corresponding to each of the EEG signal segments and the decoding type labels corresponding to each of the EEG signal segments, the method further includes: Using the feature matrices corresponding to the respective EEG signal segments in the respective signal segment groups corresponding to the respective EEG signal acquisition device types, respectively training the autoencoders based on convolutional neural networks corresponding to the respective EEG signal acquisition device types in the classification model based on the generative adversarial network, so that the respective autoencoders are used to perform data compression on the input feature matrices and output compressed feature data corresponding to the respective feature matrices; According to each of the compressed feature data corresponding to each of the types of the electroencephalogram signal acquisition devices, a pre-trained feature vector corresponding to each of the electroencephalogram signal segments is obtained.

[0009] In some embodiments of the present application, the obtaining of the pre-trained feature vectors corresponding to the respective EEG signal segments according to the respective compressed feature data corresponding to the respective EEG signal acquisition device types comprises: grouping the respective compressed feature data according to the respective EEG signal acquisition device types to obtain compressed feature groups corresponding to the respective EEG signal acquisition device types; Selecting one of the compressed feature groups as a reference group, and selecting compressed feature data having the same decoding type label as that corresponding to each compressed feature data in the reference group from the other compressed feature groups except the reference group, so as to form a plurality of parent sample data corresponding to each compressed feature data in the reference group; The average vector corresponding to each of the parent sample data is obtained as a pre-trained feature vector, and the Euclidean distance corresponding to each of the parent sample data is determined, and the decoding type label of the feature matrix to which the parent sample data corresponding to each of the pre-trained feature vectors belongs is used as the decoding type label of each of the pre-trained feature vectors.

[0010] In some embodiments of the present application, before using the feature matrices corresponding to the respective EEG signal segments in the respective signal segment groups corresponding to the respective EEG signal acquisition device types to respectively train the convolutional neural network-based autoencoders corresponding to the respective EEG signal acquisition device types in the classification model based on the generative adversarial network, the method further includes: Preprocessing the original EEG signals corresponding to each type of EEG signal acquisition device to obtain signal segment groups corresponding to each type of EEG signal acquisition device, each of which contains EEG signal segments, wherein each of the signal segment groups corresponding to the same type of EEG signal acquisition device corresponds to each preset decoding type label one by one; For each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, the feature matrix for representing the time domain features and frequency domain features of the same dimension of each EEG signal segment is extracted respectively, and the EEG signal acquisition device type corresponding to the EEG signal segment is added to the decoding type label corresponding to the EEG signal segment to which each feature matrix belongs.

[0011] In some embodiments of the present application, for each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, extracting a feature matrix for representing the time domain features and frequency domain features of the same dimension of each EEG signal segment, respectively, includes: A sliding window is created, and each time the sliding window slides, each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type is segmented to obtain a sub-signal segment with a length equal to the window width of the sliding window; and a preset time domain and frequency domain feature extraction step is performed on the sub-signal segment to obtain a one-dimensional feature corresponding to each of the sub-signal segments for representing the time domain feature and the frequency domain feature; The one-dimensional features obtained during each sliding of the sliding window are stacked to obtain a feature matrix corresponding to the electroencephalogram signal segments formed by the sub-signal segments.

[0012] In some embodiments of the present application, the time domain and frequency domain feature extraction step includes: For each channel segment of each channel of the current sub-signal segment, a preset spatial correlation pooling step is performed to use the target feature vector corresponding to the sub-signal segment obtained by the spatial correlation pooling step as the time domain feature vector corresponding to the sub-signal segment; And, performing Fourier transform on each of the channel segments of each channel corresponding to the sub-signal segment, and then performing a preset spatial correlation pooling step on each channel segment after the Fourier transform, so as to use the target feature vector corresponding to the sub-signal segment obtained by the spatial correlation pooling step as the frequency domain feature vector corresponding to the sub-signal segment; connecting the time domain feature vector and the frequency domain feature vector corresponding to the sub-signal segment to obtain a one-dimensional feature corresponding to the sub-signal segment.

[0013] In some embodiments of the present application, the spatial correlation pooling step includes: Based on the average interval time sampling method, converting each channel segment of each channel of the current sub-signal segment into a discrete point vector of each channel; Determine the Pearson correlation coefficient between every two channels in each of the channels according to the discrete point vectors of each channel, so as to obtain a correlation coefficient matrix corresponding to the sub-signal segments; and, determining the electrode coordinates of each channel of the current sub-signal segment based on a Cartesian coordinate algorithm, and determining the Euclidean distance between every two electrode coordinates, so as to obtain a Euclidean distance matrix corresponding to the sub-signal segment; Determining, according to the correlation coefficient matrix and the Euclidean distance matrix corresponding to the sub-signal segments, the average spatial correlation index corresponding to each of the channels of the sub-signal segments; Average pooling is performed on the average spatial correlation index corresponding to each of the channels of the sub-signal segment to obtain a target feature vector corresponding to the sub-signal segment.

[0014] Another aspect of the present application provides a method for cross-device decoding and classification of EEG signals, comprising: Extracting a feature matrix of the EEG signal segment corresponding to the current target EEG signal, which is used to represent the time domain features and frequency domain features of the same dimension; The feature matrix corresponding to the target EEG signal is input into a preset EEG signal cross-device decoding classification model, so that the EEG signal cross-device decoding classification model outputs classification result data corresponding to the target EEG signal, which includes a decoding type and an EEG signal acquisition device type, wherein the EEG signal cross-device decoding classification model is pre-trained based on the EEG signal cross-device decoding classification model training method.

[0015] The third aspect of the present application provides an apparatus for training a cross-device decoding classification model of an electroencephalogram signal, comprising: a pre-training module for pre-training a latent space deep neural network corresponding to a classification model based on a generative adversarial network based on pre-trained feature vectors corresponding to each electroencephalogram signal segment in each signal segment group corresponding to each type of electroencephalogram signal acquisition device, and obtaining a pre-trained deep neural network for predicting the decoding type of the pre-trained feature vector; A training module is used to use the pre-trained deep neural network as a discriminator in a classification model based on a generative adversarial network, and input the feature matrices of each EEG signal segment corresponding to each type of the EEG signal acquisition device into the autoencoder based on a convolutional neural network corresponding to each type of the EEG signal acquisition device in the classification model to train the classification model, and then train the classification model to be a cross-device EEG signal decoding classification model that outputs the classification result data of the decoding type and the EEG signal acquisition device type corresponding to the signal segment according to the feature matrix of the signal segment, wherein each of the autoencoders is connected to the discriminator.

[0016] The fourth aspect of the present application provides an EEG signal cross-device decoding and classification device, comprising: A feature extraction module, used to extract a feature matrix representing time domain features and frequency domain features of the same dimension from an EEG signal segment corresponding to the current target EEG signal; A model classification module is used to input the feature matrix corresponding to the target EEG signal into a preset EEG signal cross-device decoding classification model, so that the EEG signal cross-device decoding classification model outputs classification result data corresponding to the target EEG signal, which includes a decoding type and an EEG signal acquisition device type, wherein the EEG signal cross-device decoding classification model is pre-trained based on the EEG signal cross-device decoding classification model training method.

[0017] The fifth aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor implements the EEG signal cross-device decoding and classification model training method, and / or implements the EEG signal cross-device decoding and classification method.

[0018] The sixth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the EEG signal cross-device decoding and classification model training method, and / or implements the EEG signal cross-device decoding and classification method.

[0019] The seventh aspect of the present application provides a computer program product, including a computer program and / or instructions, which, when executed by a processor, implement the EEG signal cross-device decoding and classification model training method, and / or implement the EEG signal cross-device decoding and classification method.

[0020] The present application provides a method for training a cross-device decoding classification model for EEG signals. The method pre-trains a latent space deep neural network corresponding to a classification model based on a generative adversarial network based on pre-trained feature vectors corresponding to each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, and a decoding type label corresponding to each EEG signal segment, so as to obtain a pre-trained deep neural network for predicting the decoding type of the pre-trained feature vector; the pre-trained deep neural network is used as a discriminator in a classification model based on a generative adversarial network, and the feature matrices of each EEG signal segment corresponding to each EEG signal acquisition device type are respectively input into the autoencoders based on convolutional neural networks corresponding to each EEG signal acquisition device type in the classification model to train the classification model. The classification model is then trained to be a cross-device decoding and classification model for EEG signals, which outputs classification result data corresponding to the signal segment, including the decoding type and the type of EEG signal acquisition device, based on the feature matrix of the signal segment. Each of the autoencoders is connected to the discriminator, which can solve the problem of poor reliability and effectiveness of the decoding and classification model trained with EEG signals from cross-vendor devices, effectively improve the data utilization rate of EEG signals for training and reduce the cost of data use, effectively save the computing resources and time cost required for the model training process and reduce data dependence, improve the generalization ability of the decoding and classification model, and thus effectively improve the reliability, effectiveness and efficiency of the EEG signal decoding and classification model training process, and thus improve the effectiveness and reliability of using the trained decoding and classification model to identify the decoding type and acquisition device type of EEG signal segments.

[0021] Additional advantages, purposes, and features of the present application will be partially described in the following description, and will become partially apparent to those skilled in the art after studying the following, or may be learned from the practice of the present application. The purposes and other advantages of the present application can be achieved and obtained by the structures specifically pointed out in the specification and the drawings.

[0022] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. The components in the drawings are not drawn to scale, but are only for the purpose of illustrating the principles of the present application. In order to facilitate the illustration and description of some parts of the present application, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present application. In the drawings: Figure 1 This is a first flow chart of a method for training a cross-device EEG signal decoding and classification model in one embodiment of the present application.

[0024] Figure 2 A schematic diagram showing an example architecture of a classification model based on a generative adversarial network provided in this application.

[0025] Figure 3 This is a second flow chart of the method for training a cross-device EEG signal decoding and classification model in one embodiment of the present application.

[0026] Figure 4 This is a third flow chart of the method for training a cross-device EEG signal decoding and classification model in one embodiment of the present application.

[0027] Figure 5 This is a flowchart of the time domain and frequency domain feature extraction steps in the EEG signal cross-device decoding classification model training method in one embodiment of the present application.

[0028] Figure 6 Schematic diagram of the flow of the spatial correlation pooling step in the EEG signal cross-device decoding classification model training method in one embodiment of the present application.

[0029] Figure 7 This is a flow chart of a method for training a cross-device EEG signal decoding and classification model in an application example of the present application.

[0030] Figure 8 This is a schematic diagram showing an example of the execution architecture of S2 in the EEG signal cross-device decoding and classification model training method in an application example of the present application.

[0031] Fig. 9 This is a schematic diagram showing an example of the execution architecture of S3 in the EEG signal cross-device decoding and classification model training method in an application example of the present application.

[0032] Fig.10 Schematic diagram of the flow of a method for cross-device decoding and classification of EEG signals in one embodiment of the present application.

[0033] Fig.11 This is a structural diagram of an apparatus for training a cross-device EEG signal decoding and classification model in one embodiment of the present application.

[0034] Fig.12 Schematic diagram of the structure of an apparatus for cross-device decoding and classification of EEG signals in one embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the implementation modes and the accompanying drawings. Here, the illustrative implementation modes and descriptions of the present application are used to explain the present application, but are not intended to limit the present application.

[0036] It should also be noted here that in order to avoid obscuring the present application due to unnecessary details, only the structures and / or processing steps closely related to the scheme according to the present application are shown in the accompanying drawings, while other details that are not very relevant to the present application are omitted.

[0037] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0038] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0039] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0040] In order to effectively solve the current problems of poor cross-device model capabilities and serious waste of data information based on EEG motor imagery decoding tasks, the embodiments of the present application respectively provide a method for training a cross-device decoding and classification model for EEG signals, a cross-device decoding and classification model training device for executing the cross-device decoding and classification model training method for EEG signals, a cross-device decoding and classification method for EEG signals, an EEG cross-device decoding and classification device for executing the cross-device decoding and classification method for EEG signals, a physical device, a computer-readable storage and a computer program product. By combining the idea of ​​a generative adversarial network (GAN) with an autoencoder based on a convolutional neural network to construct a decoding and classification model, the defect of poor performance when using data from cross-vendor devices is solved, thereby achieving the purpose of improving data utilization.

[0041] The details are described in detail through the following examples.

[0042] Based on this, the embodiment of the present application provides a method for training a cross-device decoding classification model of an electroencephalogram signal, which can be implemented by a cross-device decoding classification model training device of an electroencephalogram signal, see Figure 1 The EEG signal cross-device decoding classification model training method specifically includes the following contents: Step 100: For each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, based on the pre-trained feature vector corresponding to each EEG signal segment and the decoding type label corresponding to each EEG signal segment, the latent space deep neural network corresponding to the classification model based on the generative adversarial network is pre-trained to obtain a pre-trained deep neural network for predicting the decoding type of the pre-trained feature vector.

[0043] In one embodiment of the present application, there are at least two types of EEG signal acquisition devices, and each type of EEG signal acquisition device can be represented by a unique manufacturer identifier or model of the different EEG signal acquisition devices.

[0044] It can be understood that each signal segment group corresponding to the same EEG signal acquisition device type corresponds to each preset decoding type label. For example, if there are five decoding type labels, then there are also five signal segment groups corresponding to a certain EEG signal acquisition device type, and each group uniquely corresponds to a decoding type label. It should also be noted that the decoding type label corresponding to the signal segment group means that each EEG signal segment in the signal segment group is provided with the same decoding type label.

[0045] In one or more embodiments of the present application, the EEG signal segment can be abbreviated as a signal segment or an EEG signal segment. The decoding type label includes labels for representing different decoding types, and the decoding type includes at least a motor imagination state and a resting state. The motor imagination state can include at least imagining the activity of the left hand, right hand, left foot or right foot, etc.

[0046] In step 100, the pre-trained feature vector As the latent space layer of the classifier (i.e., classification model), the network composed of the latent space layer and the multi-layer neural networks sequentially connected to the latent space layer is a latent space deep neural network.

[0047] Specifically, first freeze the autoencoder layer in the classification model and directly convert the pre-trained feature vector As the input of the latent space layer to the first layer of the network after the autoencoder layer, the predicted label (i.e. the decoding type of the pre-trained feature vector) is output after passing through the multi-layer neural network, the cross entropy loss loss is calculated, and the latent space deep neural network is continuously optimized until the latent space deep neural network converges. It should be noted that at this time, the neural network only has a prediction function, which is the learning of the model in decoding prediction, rather than the discriminator role in the adversarial generative network GAN. Therefore, the evaluation of the device model is ignored when calculating the loss. After the training is completed, the parameters of the latent space deep network model are retained, and it is used as a pre-trained deep neural network at this time.

[0048] In one or more embodiments of the present application, the EEG signal acquisition device may be referred to as a device.

[0049] Step 200: Use the pre-trained deep neural network as the discriminator in the classification model based on the generative adversarial network, and input the feature matrices of each EEG signal segment corresponding to each type of the EEG signal acquisition device into the autoencoder based on the convolutional neural network corresponding to each type of the EEG signal acquisition device in the classification model to train the classification model, and then train the classification model to be a cross-device EEG signal decoding classification model that outputs the classification result data of the decoding type and the EEG signal acquisition device type corresponding to the signal segment according to the feature matrix of the signal segment, wherein each of the autoencoders is connected to the discriminator.

[0050] In step 200, see Figure 2 The classification model based on the generative adversarial network may specifically include an autoencoder based on a convolutional neural network corresponding to each type of the EEG signal acquisition device (i.e. Figure 2 The encoder in the figure), and each of the autoencoders is connected to its own latent space layer (i.e., the pre-trained feature vector ), each of the latent space layers is connected to an upsampling layer, the upsampling layer is connected to the discriminator, the discriminator classifies and discriminates the output features of the upsampling layer, and passes the discrimination loss to the autoencoder for optimizing the autoencoder parameters. Among them, "device 1 feature matrix" refers to the feature matrix of each EEG signal segment corresponding to one type of EEG signal acquisition device, and "device 2 feature matrix" refers to the feature matrix of each EEG signal segment corresponding to another type of EEG signal acquisition device (the feature matrix can also be called a feature vector matrix).

[0051] Specifically, the data distribution between devices from different manufacturers is quite different, which is also the difficulty of the cross-device problem. In order to reduce the difference between the distributions, a discriminator based on the GAN model idea is introduced for cross-device training. By allowing the discriminator to determine the device source of the data in the latent space, the feature extraction process of the autoencoder for data features is continuously optimized, so as to achieve the purpose that the latent space features cannot allow the discriminator to distinguish the source of the device, thereby reducing the data distribution of different devices and completing the task of model cross-device decoding and classification. The specific training process is: First, the feature vector matrix Matrix of device 1 and device 2 is converted into i20% is reserved as a validation set, and then several batches are divided for each device, and then the batches of the two devices are alternately used to train the model. Unlike pre-training, during the formal training process, the deep neural network in the latent space acts as a discriminator, and the output is to predict which device the feature comes from and the decoding task status. The calculated loss function will be passed to the upstream autoencoder for parameter optimization. This training process allows the autoencoder to learn information that can reduce the differences between data distributions across devices, while ensuring that information that improves decoding accuracy is not lost.

[0052] From the above description, it can be seen that the EEG signal cross-device decoding and classification model training method provided in the embodiment of the present application can solve the problem of poor reliability and effectiveness of the decoding and classification model trained using EEG signals from cross-vendor devices, can effectively improve the data utilization rate of EEG signals used for training and reduce data usage costs, can effectively save the computing resources and time costs required for the model training process and reduce data dependence, can improve the generalization ability of the decoding and classification model, and thus can effectively improve the reliability, effectiveness and efficiency of the EEG signal decoding and classification model training process, and thus can improve the effectiveness and reliability of using the trained decoding and classification model to identify the decoding type and acquisition device type of EEG signal segments.

[0053] In order to further improve the convergence ability of the EEG signal cross-device decoding classification model, in a method for training an EEG signal cross-device decoding classification model provided in an embodiment of the present application, see Figure 3 The method for training a cross-device EEG signal decoding classification model includes the following contents before step 100: Step 010: Using the feature matrices corresponding to each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, respectively train the convolutional neural network-based autoencoders corresponding to each of the EEG signal acquisition device types in the classification model based on the generative adversarial network, so that each of the autoencoders is used to compress the input feature matrix and output the compressed feature data corresponding to each of the feature matrices.

[0054] Specifically, in order to speed up the convergence of the model, this application adds a pre-training step before model training. Different from conventional pre-training data sets, this application proposes a new spatial weighted data enhancement algorithm for generating pre-training features, the purpose of which is to use coarse fusion features to preheat the model parameters instead of random initialization. The specific steps are: initialize several autoencoders based on convolutional neural networks according to the number of cross-devices, and then use all the data from the same device to train the corresponding autoencoder for data compression, and calculate the reconstruction loss according to the following formula And continuously optimize the model parameters until the model converges. Then this application uses the compressed features (i.e., compressed feature data) for all data as h1...h i , whose dimension is recorded as e. After training, all the model parameters of the autoencoder are retained.

[0055] Among them, M i represents the original feature matrix; Represents the reconstructed feature matrix obtained after the encoding and decoding process of the self-encoding.

[0056] Step 020: According to each of the compressed feature data corresponding to each of the types of the EEG signal acquisition devices, obtain the pre-trained feature vector corresponding to each of the EEG signal segments.

[0057] In order to further improve the effectiveness and reliability of obtaining the pre-trained feature vectors corresponding to each of the EEG signal segments, and to further improve the convergence ability of the EEG signal cross-device decoding classification model, in a method for training an EEG signal cross-device decoding classification model provided in an embodiment of the present application, see Figure 4 , step 020 of the EEG signal cross-device decoding classification model training method specifically includes the following contents: Step 021: Grouping each of the compressed feature data according to the type of each of the EEG signal acquisition devices to obtain a compressed feature group corresponding to each of the types of the EEG signal acquisition devices.

[0058] Step 022: Select one from each of the compressed feature groups as a baseline group, and select compressed feature data with the same decoding type label as that corresponding to each of the compressed feature data in the baseline group from the other compressed feature groups except the baseline group, so as to form a plurality of parent sample data corresponding to each of the compressed feature data in the baseline group.

[0059] Step 023: Obtain the average vector corresponding to each of the parent sample data as a pre-trained feature vector, determine the Euclidean distance corresponding to each of the parent sample data, and use the decoding type label of the feature matrix to which the parent sample data corresponding to each of the pre-trained feature vectors belongs as the decoding type label of each of the pre-trained feature vectors.

[0060] Specifically, the compressed feature data is classified into several groups according to different devices and labels. The compressed feature data in each compressed feature group comes from the same device and has the same label. Then, taking all groups of a certain device as the benchmark, the complete sampling strategy is followed in the data of other different devices but with the same label to match all features in the benchmark group to form a pair of parent sample data. Assuming that there are p hp in a benchmark group, there are q h corresponding to the same label but different device groups. q After full sampling, this application can obtain p×q pairs of parent sample data, which are recorded as parents1...parents in this application. p×q Next, the application calculates the average vector of h in each pair as the newly generated pre-trained feature vector according to the following formula: The Euclidean distance is calculated as the inverse of the weight of the pre-trained feature vector. If the distance between two features is smaller, it means that the fusion generated The more it helps to reduce the distribution differences across devices, thus improving The weight coefficient of , and vice versa.

[0061] in, represents the feature vector obtained after fusion, h p represents the pth compressed feature vector in device group 1, h q Represents the qth compressed feature vector in device group 2. Represents h p The i-th element of the vector, Represents h q The i-th element of the vector, e represents the dimension of the h vector. It should be noted that the device group numbers 1 and 2 are only used to distinguish different devices in the description here, and are not real numbers.

[0062] Finally, this application will generate several pre-trained feature vectors Its classification task label is consistent with the label of the parent sample, and the device label is randomly selected from the parent samples.

[0063] In the existing methods, there is a waste of resources due to the low utilization rate of EEG data. As biological data, EEG has the defects of high difficulty and high cost of collection. With the support of differences in manufacturers' equipment, data set merging has great limitations. For example, different public data sets are limited by the difference in the number of acquisition channels (number of electrodes). In the data set merging stage, the full conditional matching method is usually used to narrow the range of available data or the channel number backward compatibility method is used to directly delete the additional channel information for data with more channels. In addition, in the data feature extraction stage, the common method is to extract coarse-grained feature information in the time domain and frequency domain from single channel information, and the spatial information is also limited to exploring the location distribution of the channel, ignoring the mining of the dynamic relationship between brain activity and channel position from the perspective of time series evolution. Obviously, extracting fine-grained features through improved methods is conducive to improving the model's decoding ability for EEG motor imagery from the perspective of improving data utilization and information mining rate.

[0064] Based on this, in order to fully mine data information and improve the information mining rate, in a method for training a cross-device EEG signal decoding classification model provided in an embodiment of the present application, see Figure 3 The EEG signal cross-device decoding classification model training method further specifically includes the following contents before step 010: Step 030: Preprocess the original EEG signals corresponding to each EEG signal acquisition device type to obtain signal segment groups corresponding to each EEG signal acquisition device type, each of which contains EEG signal segments, wherein each of the signal segment groups corresponding to the same EEG signal acquisition device type corresponds one-to-one to each preset decoding type label.

[0065] In an example of step 030, firstly, the EEG signals of the subjects in motor imagery (imagining the left hand, right hand, left foot, right foot, etc.) and resting state are collected using equipment from different manufacturers, or an internationally disclosed EEG signal data set of motor imagery is used. Then all the data are uniformly preprocessed using a general preprocessing process, specifically including artifact removal, bandpass filtering, data segmentation and baseline correction. The data segmentation step is to segment the data according to the experimental task or stimulus event. After the preprocessing is completed, several groups of EEG signal segments will be obtained for each subject, and each segment is only a motor imagery task state or resting state. Assume that after the preprocessing is completed, there are M1 groups of EEG signal segments from device 1, M2 groups of EEG signal segments from device 2, and so on. It should be noted that the length and number of channels of the EEG signal segments vary from device to device and are not exactly the same.

[0066] Step 040: For each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, extract the feature matrix for representing the time domain features and frequency domain features of the same dimension of each EEG signal segment, and add the EEG signal acquisition device type corresponding to the EEG signal segment to the decoding type label corresponding to the EEG signal segment to which each feature matrix belongs.

[0067] In order to further fully mine data information and improve the information mining rate, in a method for training a cross-device EEG signal decoding classification model provided in an embodiment of the present application, see Figure 4 Step 040 of the EEG signal cross-device decoding classification model training method specifically includes the following contents: Step 041: creating a sliding window, and segmenting each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type each time the sliding window slides, so as to obtain a sub-signal segment with a length equal to the window width of the sliding window; and performing a preset time domain and frequency domain feature extraction step on the sub-signal segment, so as to obtain a one-dimensional feature corresponding to each of the sub-signal segments for representing the time domain feature and the frequency domain feature; Step 042: stacking the one-dimensional features obtained at each sliding of the sliding window to obtain a feature matrix corresponding to the EEG signal segments formed by the sub-signal segments.

[0068] Step 043: Add the type of EEG signal acquisition device corresponding to the EEG signal segment to which each of the feature matrices belongs to the decoding type label corresponding to the EEG signal segment.

[0069] In order to further improve the effectiveness and reliability of time domain and frequency domain feature extraction, in a method for training a cross-device EEG signal decoding classification model provided in an embodiment of the present application, see Figure 5 The time domain and frequency domain feature extraction steps in the EEG signal cross-device decoding classification model training method specifically include the following contents: Step 01: performing a preset spatial correlation pooling step on each channel segment of each channel of the current sub-signal segment, so as to use a target feature vector corresponding to the sub-signal segment obtained by the spatial correlation pooling step as a time domain feature vector corresponding to the sub-signal segment; And, step 02: performing Fourier transform on each of the channel segments of each channel corresponding to the sub-signal segment, and then performing a preset spatial correlation pooling step on each channel segment after the Fourier transform, so as to use the target feature vector corresponding to the sub-signal segment obtained by the spatial correlation pooling step as the frequency domain feature vector corresponding to the sub-signal segment; Step 03: Connect the time domain feature vector and the frequency domain feature vector corresponding to the sub-signal segment to obtain a one-dimensional feature corresponding to the sub-signal segment.

[0070] In order to further improve the effectiveness and reliability of spatial correlation pooling, in a method for training a cross-device EEG signal decoding classification model provided in an embodiment of the present application, see Figure 6 The spatial correlation pooling step in the EEG signal cross-device decoding classification model training method specifically includes the following contents: Step 04: converting each channel segment of each channel of the current sub-signal segment into a discrete point vector of each channel based on the average interval time sampling method; Step 05: determining the Pearson correlation coefficient between every two channels in each of the channels according to the discrete point vectors of each channel, so as to obtain a correlation coefficient matrix corresponding to the sub-signal segments; And, step 06: determining the electrode coordinates of each channel of the current sub-signal segment based on a Cartesian coordinate algorithm, and determining the Euclidean distance between every two electrode coordinates to obtain a Euclidean distance matrix corresponding to the sub-signal segment; step 07: determining the average spatial correlation index corresponding to each channel of the sub-signal segment according to the correlation coefficient matrix and the Euclidean distance matrix corresponding to the sub-signal segment; Step 08: Performing average pooling processing on the average spatial correlation index corresponding to each of the channels of the sub-signal segment to obtain a target feature vector corresponding to the sub-signal segment.

[0071] Specifically, in a specific example of step 040, the EEG signal segment obtained in step 030 is not suitable for directly performing the decoding and classification task of motor imagery due to its high dimension and unextracted information, and needs to be further processed to obtain a feature matrix. The specific steps are: Assume that the first EEG signal segment in M1 is taken as m1. m1 has n channels, denoted as C1...C n. First, in order to extract fine-grained features and capture the dynamic changes of the time-frequency-space dimensions, this application uses a dynamic sliding time window segmentation algorithm different from the static fixed time window segmentation algorithm. Suppose a sliding window Win is created with a window width of w seconds and a sliding step of s seconds. It should be noted that in order to capture feature information more finely, w is generally set to less than 1 second. At the same time, in order to better capture dynamically changing information and reduce the differences between adjacent windows, each next sliding window needs to overlap with the previous window, so s is generally set to half of w. This application records each slide as Slide i Each slide will obtain a w-length sub-signal segment corresponding to the EEG signal segment for all channels. The i-th channel segment of the n-th channel is recorded as This application takes all channel data in the first window as an example (denoted as ), each channel is a continuous band, and then based on the spatial correlation pooling algorithm newly proposed in this application, a feature vector that integrates spatial relationships and dynamic changes is extracted from the time domain and frequency domain.

[0072] The following takes the time domain feature extraction process as an example to explain the specific steps of the algorithm in detail. First, use the average interval time sampling method to The continuous band is converted into a discrete point vector Then the Pearson correlation coefficient r between the two channels a and b is calculated according to the following formula a，b ; Get the correlation coefficient matrix Among them They are and The mean of .

[0073] Where T is the number of discrete sampling time points, u t represents the amplitude value of channel a of a certain segment at time point t, v t represents the amplitude value of channel b of a certain segment at time point t, They are and The mean value, r a，b Represents the Pearson correlation coefficient between channels a and b.

[0074] Then, based on the Cartesian coordinate algorithm, a coordinate system is established with the center of the top of the head as the three-dimensional origin coordinate to obtain the electrode coordinates of each channel. Then, the Euclidean distance D between two electrodes is calculated according to the following formula: 1，2 ...D a，b .

[0075] Where (xa ,y a , z a ) represents the three-dimensional coordinates of channel a, (x b ,y b , z b ) represents the three-dimensional coordinates of channel b. a，b Represents the Euclidean distance between channels a and b.

[0076] After obtaining the correlation coefficient and Euclidean distance between two electrodes, the average spatial correlation index Dr1...Dr is calculated for each channel according to the following formula: n , this index describes the correlation of each channel data and the other channel data in the time domain with the fusion of spatial position. The larger the value, the more similar the electrode is to the distant channel in the time domain of this window, and the smaller the value, the opposite is true.

[0077] Among them, Dr. a represents the average spatial correlation index of channel a. D a，i represents the Euclidean distance between channel a and channel i, r a，i represents the Pearson correlation coefficient between channel a and channel i. n represents the number of electrode channels.

[0078] In the next step, since different manufacturers' equipment or different acquisition strategies will result in different dimensions of the vector composed of the average spatial correlation index, in order to facilitate the pre-training of the subsequent model, a pooling algorithm is further added to scale the vector to a unified dimension. Specifically, the average pooling idea in the convolutional neural network is borrowed. Starting from the first channel, the average value of the index including itself and the adjacent index on the right is taken as an element of the output feature Dr′1, and so on to obtain the next round of features. Repeat the above steps until the feature dimensions of all devices are unified. At this point, the present application will obtain a time domain feature vector of unified dimension in the time domain of the time window (that is, the target feature vector as the time domain feature vector) TV1. In addition, Transform from time domain to frequency domain, and then extract the frequency domain feature vector of unified dimension (i.e., the target feature vector as the frequency domain feature vector) FV1 by the same steps. After the above steps, the present application connects TV1 and FV1 together to obtain the one-dimensional feature TFV1 of Slide1. By continuously sliding the window, the present application can obtain several TFV i . Finally, by combining these TFV i Stacked together, this application extracts a feature matrix Matrix1 for the m1 signal. By analogy, this application can obtain several Matrix i, they have the same matrix dimensions even if they come from different devices. In addition, the labels of all features are re-encoded, and the device type information is added based on the original task type label. For example, in a binary classification task, the original task type label of device 1 is 1, which means that the EEG signal is in a resting state. After adding the device type information, it becomes (1, 1), and so on.

[0079] To further illustrate the above embodiment, the present application also provides an application example of a method for training a cross-device EEG signal decoding classification model. Figure 7 The application examples of the EEG signal cross-device decoding classification model training method specifically include the following contents: S1. EEG signal preprocessing: Obtain the EEG signal data of the subject and perform preprocessing. The signal data of devices from different manufacturers adopt a unified preprocessing method.

[0080] Specifically, firstly, use equipment from different manufacturers to collect EEG signals of subjects during motor imagery (imagining moving the left hand, right hand, left foot, right foot, etc.) and at rest, or use an internationally published EEG signal data set for motor imagery. Then use a general preprocessing process to uniformly preprocess all data, including artifact removal, bandpass filtering, data segmentation, and baseline correction. The data segmentation step is to segment the data according to the experimental task or stimulus event. After the preprocessing is completed, several groups of EEG signal segments will be obtained for each subject, and each segment is only in one motor imagery task state or resting state. Assume that after the preprocessing, there are M1 groups of EEG signal segments from device 1, M2 groups of EEG signal segments from device 2, and so on. It should be noted that the length and number of channels of the EEG signal segments vary from device to device and are not exactly the same.

[0081] S2. Time-frequency dual feature extraction with fine-grained fusion of spatial information: In order to fully explore the information that is beneficial to the decoding classification task, the preprocessing results obtained in S1 are used to extract fine-grained feature vectors from three levels: time domain, frequency domain, and spatial relationship. Specifically, the overall signal is divided into several signal segments based on the dynamic sliding window algorithm, and then a newly proposed spatial correlation pooling algorithm is used to extract feature vectors from the fused spatial relationship and dynamically changing time domain and frequency domain.

[0082] Specifically, see Figure 8 , the EEG signal segment data obtained for S1 is not suitable for directly performing the decoding and classification task of motor imagery due to its high dimension and unextracted information, and needs to be further processed to obtain the feature vector. The specific steps are: Assume that the first EEG signal segment in M1 is recorded as m1. m1 has n channels, recorded as C1...C n. First, in order to extract fine-grained features and capture the dynamic changes of the time-frequency-space dimensions, this application uses a dynamic sliding time window segmentation algorithm different from the static fixed time window segmentation algorithm. Suppose a sliding window Win is created with a window width of w seconds and a sliding step of s seconds. It should be noted that in order to capture feature information more finely, w is generally set to less than 1 second. At the same time, in order to better capture dynamically changing information and reduce the differences between adjacent windows, each next sliding window needs to overlap with the previous window, so s is generally set to half of w. This application records each slide as Slide i Each slide will obtain a w-length sub-signal segment corresponding to the EEG signal segment for all channels. The i-th channel segment of the n-th channel is recorded as This application takes all channel data in the first window as an example (denoted as ), each channel is a continuous band, and then based on the spatial correlation pooling algorithm newly proposed in this application, a feature vector that integrates spatial relationships and dynamic changes is extracted from the time domain and frequency domain.

[0083] The following takes the time domain feature extraction process as an example to explain the specific steps of the algorithm in detail. First, use the average interval time sampling method to The continuous band is converted into a discrete point vector Then, the Pearson correlation coefficient r between the two channels a and b is calculated according to formula (1) and formula (2) a，b ; Get the correlation coefficient matrix Among them They are and The mean of .

[0084] Then, based on the Cartesian coordinate algorithm, a coordinate system is established with the center of the top of the head as the three-dimensional origin coordinate to obtain the electrode coordinates of each channel. Then, the Euclidean distance D between two electrodes is calculated according to formula (3): 1，2 ...D a，b .

[0085] After obtaining the correlation coefficient and Euclidean distance between two electrodes, the average spatial correlation index Dr1...Dr is calculated for each channel according to formula (4): n , this index describes the correlation of each channel data and the other channel data in the time domain with the fusion of spatial position. The larger the value, the more similar the electrode is to the distant channel in the time domain of this window, and the smaller the value, the opposite is true.

[0086] In the next step, since different manufacturers' equipment or different acquisition strategies will result in different dimensions of the vector composed of the average spatial correlation index, in order to facilitate the pre-training of the subsequent model, a pooling algorithm is further added to scale the vector to a unified dimension. Specifically, the average pooling idea in the convolutional neural network is borrowed. Starting from the first channel, the average value of the index including itself and the adjacent index on the right is taken as an element of the output feature Dr′1, and so on to obtain the next round of features. Repeat the above steps until the feature dimensions of all devices are unified. At this point, the present application will obtain a time domain feature vector of unified dimension in the time domain of the time window (that is, the target feature vector as the time domain feature vector) TV1. In addition, Transform from time domain to frequency domain, and then extract the frequency domain feature vector of unified dimension (i.e., the target feature vector as the frequency domain feature vector) FV1 by the same steps. After the above steps, the present application connects TV1 and FV1 together to obtain the one-dimensional feature TFV1 of Slide1. By continuously sliding the window, the present application can obtain several TFV i . Finally, by combining these TFV i Stacked together, this application extracts a feature matrix Matrix1 for the m1 signal. By analogy, this application can obtain several Matrix i , they have the same matrix dimensions even if they come from different devices. In addition, the labels of all features are re-encoded, and the device type information is added based on the original task type label. For example, in a binary classification task, the original task type label of device 1 is 1, which means that the EEG signal is in a resting state. After adding the device type information, it becomes (1, 1), and so on.

[0087] S3. Generate pre-trained features based on spatial weighted data enhancement algorithm: First, train the autoencoder based on convolutional neural network to reduce the feature dimensions of different devices to be consistent in the latent space. Then, use the spatial weighted data enhancement algorithm in the latent space to generate new features for pre-training by pairing features of different devices with the same label. The task labels of the new features are consistent with the task labels of the original features.

[0088] Specifically, see Fig. 9In order to speed up the convergence of the model, this application adds a pre-training step before model training. Different from conventional pre-training data sets, this application proposes a new spatial weighted data enhancement algorithm for generating pre-training features. The purpose is to use coarse fusion features to warm up the model parameters instead of random initialization. The specific steps are: initialize several autoencoders based on convolutional neural networks according to the number of cross-devices, and then use all the data of the same device to train the corresponding autoencoder for data compression, calculate the reconstruction loss according to formula (5) and continuously optimize the model parameters until the model converges. Then this application uses the compressed features of all data as h1…h i , whose dimension is recorded as e. After training, all the model parameters of the autoencoder are retained.

[0089] Then, the compressed feature data is classified into several groups according to different devices and labels. The data in each compressed feature group comes from the same device and has the same label. Then, taking all groups of a certain device as the benchmark, the complete sampling strategy is followed in the data of other different devices but with the same label to match all features in the benchmark group to form a pair of parent sample data. Assuming that there are p h in a benchmark group p , corresponding to the same label in different equipment groups there are q h q After full sampling, this application can obtain p×q pairs of parent sample data, which are recorded as parents1…parents in this application. p×q Next, the application calculates the average vector of h in each pair according to formula (6) as the newly generated pre-trained feature vector The Euclidean distance is calculated as the inverse of the weight of the pre-trained feature vector. If the distance between two features is smaller, it means that the fusion generated The more it helps to reduce the distribution differences across devices, thus improving The weight coefficient of , and vice versa.

[0090] Finally, this application will generate several pre-trained feature vectors Its classification task label is consistent with the label of the parent sample, and the device label is randomly selected from the parent samples.

[0091] S4, pre-training latent space deep learning model: Use the pre-training features generated in S3 to pre-train the model. Specifically, freeze the autoencoder layer in the model, and use the newly generated pre-training features to train only the deep neural network in the latent space. At this time, the predicted output only reflects the performance of the model in decoding and classifying the EEG signal (resting state or motor imagery).

[0092] Specifically, we use the pre-trained feature vector generated in step 3 Pre-train the model's latent space deep neural network. The specific steps are: first freeze the autoencoder layer in the model, and directly As the input of the latent space layer to the first layer of the network, and then after passing through the multi-layer neural network, the predicted label is output, the cross entropy loss is calculated, and the model is continuously optimized until the model converges. It should be noted that at this time, the neural network only has a prediction function, not the discriminator role in GAN, but the learning of the model in decoding prediction, so the evaluation of the device model is ignored when calculating the loss. The parameters of the latent space deep network model are retained after training is completed.

[0093] S5, training a cross-device model based on adversarial thinking. Use the feature matrix extracted by S2 as training data, and use the model pre-trained by S4 for formal training to test the performance of the model in the cross-device decoding and classification task. At this time, the predicted output reflects both the performance of the model in decoding and classification of the EEG signal and the recognition performance of the device type.

[0094] Specifically, the data distribution between devices from different manufacturers is quite different, which is also the difficulty of the cross-device problem. In order to reduce the difference between the distributions, a discriminator based on the GAN model idea is introduced for cross-device training. By allowing the discriminator to determine the device source of the data in the latent space, the feature extraction process of the autoencoder for data features is continuously optimized, so as to achieve the purpose that the latent space features cannot allow the discriminator to distinguish the source of the device, thereby reducing the data distribution of different devices and completing the task of model cross-device decoding and classification. The specific training process is: First, the feature vector matrix Matrix of device 1 and device 2 is converted into i 20% is reserved as a validation set, and then several batches are divided for each device, and then the batches of the two devices are alternately used to train the model. Unlike pre-training, during the formal training process, the deep neural network in the latent space acts as a discriminator, and the output is to predict which device the feature comes from and the decoding task status. The calculated loss function will be passed to the upstream autoencoder for parameter optimization. This training process allows the autoencoder to learn information that can reduce the differences between data distributions across devices, while ensuring that information that improves decoding accuracy is not lost.

[0095] Based on the EEG signal cross-device decoding classification model training method provided in the above embodiment, the present application also provides an embodiment of an EEG signal cross-device decoding classification method, see Fig.10 The cross-device decoding and classification method of EEG signals specifically includes the following contents: Step 300: extracting a feature matrix for representing time domain features and frequency domain features of the same dimension from an EEG signal segment corresponding to the current target EEG signal.

[0096] It can be understood that the extraction content of the feature matrix performed in the step 300 can be implemented by adopting the step 040 in the EEG signal cross-device decoding classification model training method.

[0097] Step 400: Input the feature matrix corresponding to the target EEG signal into a preset EEG signal cross-device decoding classification model, so that the EEG signal cross-device decoding classification model outputs classification result data corresponding to the target EEG signal, which includes a decoding type and an EEG signal acquisition device type, wherein the EEG signal cross-device decoding classification model is pre-trained based on the EEG signal cross-device decoding classification model training method.

[0098] The EEG signal cross-device decoding and classification model training method mentioned in the embodiment of the EEG signal cross-device decoding and classification method provided in the present application can be specifically used for the processing flow of the embodiment of the EEG signal cross-device decoding and classification model training method in the above embodiment. Its function will not be repeated here, and reference can be made to the detailed description of the above-mentioned EEG signal cross-device decoding and classification model training method embodiment.

[0099] From the above description, it can be seen that the cross-device EEG signal decoding and classification method provided in the embodiment of the present application can improve the effectiveness and reliability of using the trained decoding and classification model to identify the decoding type and acquisition device type of EEG signal segments.

[0100] From the software level, the present application also provides an EEG signal cross-device decoding classification model training device for executing all or part of the EEG signal cross-device decoding classification model training method, see Fig.11 The EEG signal cross-device decoding classification model training device specifically includes the following contents: A pre-training module 10 is used to pre-train a latent space deep neural network corresponding to a classification model based on a generative adversarial network for each EEG signal segment in each signal segment group corresponding to each type of EEG signal acquisition device, based on a pre-trained feature vector corresponding to each EEG signal segment and a decoding type label corresponding to each EEG signal segment, so as to obtain a pre-trained deep neural network for predicting a decoding type of the pre-trained feature vector; The training module 20 is used to use the pre-trained deep neural network as a discriminator in a classification model based on a generative adversarial network, and input the feature matrices of each EEG signal segment corresponding to each type of the EEG signal acquisition device into the autoencoder based on a convolutional neural network corresponding to each type of the EEG signal acquisition device in the classification model to train the classification model, and then train the classification model to be a cross-device decoding classification model for EEG signals that outputs the classification result data corresponding to the signal segment, including the decoding type and the EEG signal acquisition device type, based on the feature matrix of the signal segment, wherein each of the autoencoders is connected to the discriminator.

[0101] The embodiment of the EEG signal cross-device decoding classification model training device provided in the present application can be specifically used to execute the processing flow of the embodiment of the EEG signal cross-device decoding classification model training method in the above-mentioned embodiment. Its functions will not be repeated here, and reference can be made to the detailed description of the above-mentioned EEG signal cross-device decoding classification model training method embodiment.

[0102] The part of the EEG signal cross-device decoding classification model training device that performs the EEG signal cross-device decoding classification model training can be completed in the server or in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application is not limited to this. If all operations are completed in the client device, the client device may also include a processor for specific processing of the EEG signal cross-device decoding classification model training.

[0103] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0104] The server and the client device may communicate with each other using any suitable network protocol, including network protocols that have not yet been developed on the date of filing this application. The network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Of course, the network protocols may also include, for example, RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols used on top of the above protocols.

[0105] From the above description, it can be seen that the EEG signal cross-device decoding and classification model training device provided in the embodiment of the present application can solve the problem of poor reliability and effectiveness of the decoding and classification model trained with EEG signals from cross-vendor devices, can effectively improve the data utilization rate of EEG signals used for training and reduce the cost of data use, can effectively save the computing resources and time cost required for the model training process and reduce data dependence, can improve the generalization ability of the decoding and classification model, and thus can effectively improve the reliability, effectiveness and efficiency of the EEG signal decoding and classification model training process, and thus can improve the effectiveness and reliability of using the trained decoding and classification model to identify the decoding type and acquisition device type of EEG signal segments.

[0106] From the software level, the present application also provides an EEG signal cross-device decoding and classification device for executing all or part of the EEG signal cross-device decoding and classification method, see Fig.12 The EEG signal cross-device decoding and classification device specifically includes the following contents: A feature extraction module 30, used to extract a feature matrix representing time domain features and frequency domain features of the same dimension from an EEG signal segment corresponding to the current target EEG signal; It can be understood that the feature matrix extraction performed by the feature extraction module 30 can be implemented by adopting the step 040 in the EEG signal cross-device decoding classification model training method.

[0107] The model classification module 40 is used to input the feature matrix corresponding to the target EEG signal into a preset EEG signal cross-device decoding classification model, so that the EEG signal cross-device decoding classification model outputs classification result data corresponding to the target EEG signal, which includes a decoding type and an EEG signal acquisition device type, wherein the EEG signal cross-device decoding classification model is pre-trained based on the EEG signal cross-device decoding classification model training method.

[0108] The embodiment of the apparatus for cross-device decoding and classification of EEG signals provided in the present application can be specifically used to execute the processing flow of the embodiment of the method for cross-device decoding and classification of EEG signals in the above-mentioned embodiment. Its functions will not be repeated here, and reference may be made to the detailed description of the embodiment of the method for cross-device decoding and classification of EEG signals.

[0109] From the above description, it can be seen that the cross-device EEG signal decoding and classification device provided in the embodiment of the present application can improve the effectiveness and reliability of using the trained decoding and classification model to identify the decoding type and acquisition device type of EEG signal segments.

[0110] The embodiment of the present application also provides an electronic device, which may include a processor, a memory, a receiver and a transmitter, wherein the processor is used to execute the EEG signal cross-device decoding classification model training method and / or the aforementioned EEG signal cross-device decoding classification method mentioned in the above embodiment, wherein the processor and the memory may be connected via a bus or other means, with the bus connection being taken as an example. The receiver may be connected to the processor and the memory via a wired or wireless manner.

[0111] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0112] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the program instructions / modules corresponding to the EEG signal cross-device decoding classification model training method in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, the EEG signal cross-device decoding classification model training method and / or the aforementioned EEG signal cross-device decoding classification method in the above method embodiment.

[0113] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0114] The one or more modules are stored in the memory, and when executed by the processor, perform the EEG signal cross-device decoding and classification model training method and / or the aforementioned EEG signal cross-device decoding and classification method in the embodiment.

[0115] In some embodiments of the present application, the user equipment may include a processor, a memory, and a transceiver unit, which may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected through a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0116] As an implementation method, the functions of the receiver and the transmitter in the present application can be considered to be implemented through a transceiver circuit or a dedicated chip for transceiver, and the processor can be considered to be implemented through a dedicated processing chip, a processing circuit or a general chip.

[0117] As another implementation method, it is possible to use a general-purpose computer to implement the server provided in the embodiment of the present application, that is, to store the program code for implementing the functions of the processor, receiver, and transmitter in a memory, and the general-purpose processor implements the functions of the processor, receiver, and transmitter by executing the code in the memory.

[0118] The embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned EEG signal cross-device decoding classification model training method and / or the aforementioned EEG signal cross-device decoding classification method. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0119] An embodiment of the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the aforementioned EEG signal cross-device decoding and classification model training method, and / or implements the steps of the aforementioned EEG signal cross-device decoding and classification method.

[0120] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0121] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0122] In this application, the features described and / or illustrated for one embodiment may be used in the same manner or in a similar manner in one or more other embodiments, and / or combined with or replace the features of other embodiments. The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training a cross-device EEG signal decoding classification model, characterized in that: include: For each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, based on the pre-trained feature vector corresponding to each EEG signal segment and the decoding type label corresponding to each EEG signal segment, a latent space deep neural network corresponding to the classification model based on the generative adversarial network is pre-trained to obtain a pre-trained deep neural network for predicting the decoding type of the pre-trained feature vector; The pre-trained deep neural network is used as the discriminator in the classification model based on the generative adversarial network, and the feature matrices of each EEG signal segment corresponding to each type of the EEG signal acquisition device are respectively input into the autoencoder based on the convolutional neural network corresponding to each type of the EEG signal acquisition device in the classification model to train the classification model, and then the classification model is trained to be an EEG signal cross-device decoding classification model for outputting the classification result data of the decoding type and the EEG signal acquisition device type corresponding to the signal segment according to the feature matrix of the signal segment, wherein each of the autoencoders is connected to the discriminator.

2. The method for training a cross-device EEG signal decoding classification model according to claim 1, characterized in that: Before pre-training the latent space deep neural network corresponding to the classification model based on the generative adversarial network based on the pre-trained feature vectors corresponding to each of the EEG signal segments and the decoding type labels corresponding to each of the EEG signal segments, the method further includes: Using the feature matrices corresponding to the respective EEG signal segments in the respective signal segment groups corresponding to the respective EEG signal acquisition device types, respectively training the autoencoders based on convolutional neural networks corresponding to the respective EEG signal acquisition device types in the classification model based on the generative adversarial network, so that the respective autoencoders are used to perform data compression on the input feature matrices and output compressed feature data corresponding to the respective feature matrices; According to each of the compressed feature data corresponding to each of the types of the electroencephalogram signal acquisition devices, a pre-trained feature vector corresponding to each of the electroencephalogram signal segments is obtained.

3. The method for training a cross-device EEG signal decoding classification model according to claim 2, characterized in that: The step of obtaining the pre-trained feature vectors corresponding to the respective EEG signal segments according to the respective compressed feature data corresponding to the respective EEG signal acquisition device types comprises: Grouping the compressed feature data according to the types of the EEG signal acquisition devices to obtain compressed feature groups corresponding to the types of the EEG signal acquisition devices; Selecting one of the compressed feature groups as a reference group, and selecting compressed feature data having the same decoding type label as that corresponding to each compressed feature data in the reference group from the other compressed feature groups except the reference group, so as to form a plurality of parent sample data corresponding to each compressed feature data in the reference group; The average vector corresponding to each of the parent sample data is obtained as a pre-trained feature vector, and the Euclidean distance corresponding to each of the parent sample data is determined, and the decoding type label of the feature matrix to which the parent sample data corresponding to each of the pre-trained feature vectors belongs is used as the decoding type label of each of the pre-trained feature vectors.

4. The method for training a cross-device EEG signal decoding classification model according to claim 2, characterized in that: Before respectively training the autoencoders based on convolutional neural networks corresponding to the types of EEG signal acquisition devices in the classification model based on the generative adversarial network using the feature matrices corresponding to the respective EEG signal segments in the respective signal segment groups corresponding to the types of EEG signal acquisition devices, the method further includes: Preprocessing the original EEG signals corresponding to each type of EEG signal acquisition device to obtain signal segment groups corresponding to each type of EEG signal acquisition device, each of which contains EEG signal segments, wherein each of the signal segment groups corresponding to the same type of EEG signal acquisition device corresponds to each preset decoding type label one by one; For each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, the feature matrix for representing the time domain features and frequency domain features of the same dimension of each EEG signal segment is extracted respectively, and the EEG signal acquisition device type corresponding to the EEG signal segment is added to the decoding type label corresponding to the EEG signal segment to which each feature matrix belongs.

5. The method for training a cross-device EEG signal decoding classification model according to claim 4, characterized in that: The method comprises: for each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type, extracting a feature matrix for representing the time domain features and frequency domain features of each EEG signal segment for the same dimension, including: creating a sliding window, and segmenting each EEG signal segment in each signal segment group corresponding to each EEG signal acquisition device type at each sliding of the sliding window to obtain a sub-signal segment having a length equal to the window width of the sliding window; and performing a preset time domain and frequency domain feature extraction step on the sub-signal segment to obtain a one-dimensional feature for representing the time domain features and the frequency domain features corresponding to each of the sub-signal segments; The one-dimensional features obtained during each sliding of the sliding window are stacked to obtain a feature matrix corresponding to the electroencephalogram signal segments formed by the sub-signal segments.

6. The method for training a cross-device EEG signal decoding classification model according to claim 5, characterized in that: The time domain and frequency domain feature extraction steps include: For each channel segment of each channel of the current sub-signal segment, a preset spatial correlation pooling step is performed to use the target feature vector corresponding to the sub-signal segment obtained by the spatial correlation pooling step as the time domain feature vector corresponding to the sub-signal segment; And, performing Fourier transform on each of the channel segments of each channel corresponding to the sub-signal segment, and then performing a preset spatial correlation pooling step on each channel segment after the Fourier transform, so as to use the target feature vector corresponding to the sub-signal segment obtained by the spatial correlation pooling step as the frequency domain feature vector corresponding to the sub-signal segment; connecting the time domain feature vector and the frequency domain feature vector corresponding to the sub-signal segment to obtain a one-dimensional feature corresponding to the sub-signal segment.

7. The method for training a cross-device EEG signal decoding classification model according to claim 6, characterized in that: The spatial correlation pooling step comprises: Based on the average interval time sampling method, converting each channel segment of each channel of the current sub-signal segment into a discrete point vector of each channel; Determine the Pearson correlation coefficient between every two channels in each of the channels according to the discrete point vectors of each channel, so as to obtain a correlation coefficient matrix corresponding to the sub-signal segments; and, determining the electrode coordinates of each channel of the current sub-signal segment based on a Cartesian coordinate algorithm, and determining the Euclidean distance between every two electrode coordinates, so as to obtain a Euclidean distance matrix corresponding to the sub-signal segment; Determining, according to the correlation coefficient matrix and the Euclidean distance matrix corresponding to the sub-signal segments, the average spatial correlation index corresponding to each of the channels of the sub-signal segments; Average pooling is performed on the average spatial correlation index corresponding to each of the channels of the sub-signal segment to obtain a target feature vector corresponding to the sub-signal segment.

8. A method for cross-device decoding and classification of EEG signals, characterized in that: include: Extracting a feature matrix of the EEG signal segment corresponding to the current target EEG signal, which is used to represent the time domain features and frequency domain features of the same dimension; The feature matrix corresponding to the target EEG signal is input into a preset EEG signal cross-device decoding classification model, so that the EEG signal cross-device decoding classification model outputs classification result data corresponding to the target EEG signal, which includes a decoding type and an EEG signal acquisition device type, wherein the EEG signal cross-device decoding classification model is pre-trained based on the EEG signal cross-device decoding classification model training method as described in any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the EEG signal cross-device decoding and classification model training method as described in any one of claims 1 to 7, and / or implements the EEG signal cross-device decoding and classification method as described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is processed When executed by the device, the method for training a cross-device EEG signal decoding and classification model according to any one of claims 1 to 7 is implemented, and / or, Implement the cross-device EEG signal decoding and classification method as described in claim 8.

Images