Driving method and system based on generative adversarial network and brain-computer interface

Abstract

The invention discloses a driving method and system based on a generative adversarial network and a brain-computer interface. The method comprises the following steps: acquiring original visual steady-state evoked electroencephalogram data of a user; inputting the original visual steady-state evoked electroencephalogram data into a generative adversarial network fused by a convolution-loop network for processing to obtain enhanced visual steady-state electroencephalogram data, fusing the original visual steady-state evoked electroencephalogram data and the enhanced visual steady-state electroencephalogram data, and performing feature decoding to obtain decoded electroencephalogram signal features; and compiling the decoded electroencephalogram signal characteristics into a brain-controlled driving operation instruction to control a corresponding vehicle. According to the method, the generative adversarial network for enhancing the visual steady-state evoked electroencephalogram data is constructed, the decoding efficiency of various driving operations in the brain-controlled driving scene is improved, and the stability of the brain-controlled driving process is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and brain-computer interface technology, and in particular to a driving method and system based on a generative adversarial network and a brain-computer interface. Background Art

[0002] The brain-controlled driving method is to achieve direct control of the vehicle by the brain through a brain-computer interface. Brain-controlled driving has very important practical value for both disabled and healthy people. It can effectively improve the user's control authority and reduce the user's workload while ensuring safety.

[0003] In the current research field of brain-controlled driving methods, how to improve the decoding efficiency of EEG data features to improve driving stability and safety is an important topic. With the increasing application of deep learning methods in the classification and understanding of image, text and other data, it has also shown great application potential in the decoding of EEG data. However, the EEG data decoding method based on deep learning faces the problem of severe shortage of EEG data, which will lead to overfitting problems when training high-performance decoding models in brain-controlled driving scenarios, greatly reducing the practicality and applicability of brain-controlled driving systems designed using such methods.

[0004] As a method of data amplification in the field of deep learning, data augmentation technology can effectively solve the problem of insufficient data when training neural networks. It has very wide applications in the field of computer vision. Researchers have successfully generated new data that can be used through simple methods such as rotation, flipping, shifting, cropping, and scaling. However, unlike image data, it is difficult for existing technologies to enhance EEG data through methods such as cropping and scaling. At the same time, because visual homeostasis induces unique characteristics of EEG data in the time-frequency domain, whether the generated new data can show the expected characteristics in the time-frequency domain is also a major problem in data enhancement.

[0005] In general, the current use of deep learning methods in the field of brain-controlled driving to achieve EEG data feature extraction and classification faces the problem of insufficient EEG data. However, unlike image data, EEG data cannot be generated using simple methods such as rotation, flipping, shifting, cropping, and scaling to generate new usable data. This greatly reduces the practicality and applicability of using deep learning methods for EEG data feature extraction and classification and brain-controlled driving operations. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a driving method and system based on a generative adversarial network and a brain-computer interface. According to the time-frequency characteristics of visual steady-state induced EEG data, a generative adversarial network is introduced to enhance the EEG data, and then a deep learning method is used to train the original and enhanced visual steady-state induced EEG data to improve the decoding efficiency of EEG features and enhance the safety and stability of brain-controlled driving.

[0007] On the one hand, to achieve the above-mentioned purpose, the present invention provides a driving method based on a generative adversarial network and a brain-computer interface, comprising:

[0008] Obtaining the user's original visual steady-state evoked EEG data;

[0009] Inputting the original visual steady-state evoked EEG data into a convolutional-recurrent network fused generative adversarial network for processing to obtain enhanced visual steady-state EEG data, wherein the convolutional-recurrent network fused generative adversarial network is obtained by training a training set, and the training set is the initial visual steady-state evoked EEG data;

[0010] fusing the original visual steady-state evoked EEG data with the enhanced visual steady-state EEG data, and decoding them to obtain decoded EEG signal features;

[0011] The decoded EEG signal features are compiled into brain-controlled driving operation instructions to control the corresponding vehicle.

[0012] Preferably, the generative adversarial network fused with the convolution-recurrent network comprises: a generator subnetwork and a discriminator subnetwork, wherein the generator subnetwork is used to build a time-space domain deconvolution network according to the time-frequency characteristics of the visual steady-state evoked potential, and enhance the input EEG data; the discriminator subnetwork is used to build a space-time domain convolution-recurrent network according to the time-frequency characteristics of the visual steady-state evoked potential, and perform true and false discrimination on the enhanced EEG data.

[0013] Preferably, the convolutional-recurrent network fused generative adversarial network selects Wasserstein distance as the basic loss function, and introduces a penalty term to improve the corresponding loss function.

[0014] Preferably, obtaining the decoded EEG signal features includes:

[0015] The original visual steady-state evoked EEG data and the enhanced visual steady-state EEG data are fused, the fused data are trained using a deep learning model to obtain visual steady-state evoked EEG features, the visual steady-state evoked EEG features are decoded, and the decoded EEG signal features are obtained.

[0016] On the other hand, to achieve the above-mentioned purpose, the present invention also provides a driving system based on a generative adversarial network and a brain-computer interface, comprising:

[0017] EEG acquisition module: used to collect the user's original visual steady-state induced EEG characteristics;

[0018] EEG data processing module: used for inputting the original visual steady-state induced EEG features into the generative adversarial network fused with the convolutional-recurrent network for processing, obtaining enhanced visual steady-state EEG data, and fusing the original visual steady-state induced EEG features with the enhanced visual steady-state EEG data for decoding, obtaining decoded EEG signal features, and compiling them into brain-controlled driving operation instructions, which are input to the corresponding vehicle for control;

[0019] Vehicle data processing module: used to receive vehicle control instructions and update vehicle speed and position information;

[0020] Driving and stimulation interface display module: used to present the visual steady-state EEG evoked source and vehicle driving interface in real time.

[0021] Preferably, the convolution-recurrent network fused generative adversarial network includes a generator subnetwork and a discriminator subnetwork, wherein the generator subnetwork is used to enhance the input EEG data, and the discriminator subnetwork is used to distinguish true from false on the enhanced data through a bidirectional long short-term memory network.

[0022] Preferably, the generator subnetwork comprises:

[0023] A Gaussian noise generating unit is used to generate single-channel Gaussian noise according to the characteristic of the original visual steady-state induced EEG data containing Gaussian noise components, wherein the single-channel Gaussian noise is obtained through N fully connected layers to generate a first Gaussian noise;

[0024] A time domain deconvolution unit is used to perform time domain deconvolution on the single-channel Gaussian noise according to the frequency characteristics of the original visual steady-state evoked EEG data, in which the power spectrum and signal-to-noise ratio are significantly improved and have a significant response at the corresponding visual stimulation frequency, to generate a first single-channel EEG signal;

[0025] A spatial deconvolution unit is used to perform spatial deconvolution on the single-channel EEG signal according to the spatial characteristics of the original visual steady-state evoked EEG data that are most obviously responded to in the occipital region of the brain, so as to generate a first multi-channel enhanced visual steady-state evoked EEG signal;

[0026] Wherein, the time domain deconvolution unit is located before the spatial domain deconvolution unit.

[0027] Preferably, the discriminator subnetwork comprises:

[0028] Spatial convolution unit: used for performing a spatial convolution operation on the enhanced visual steady-state evoked EEG data according to the spatial characteristics of the original visual steady-state evoked EEG data that have the most obvious response in the occipital area of ​​the brain, to generate a second single-channel EEG signal;

[0029] A time domain convolution unit is used to perform a time domain convolution operation on the second single-channel EEG signal according to the frequency characteristics of the original visual steady-state evoked EEG data with significantly improved power spectrum and signal-to-noise ratio at the corresponding visual stimulation frequency, so as to generate a third single-channel EEG data;

[0030] Recurrent network unit: used for inputting the third single-channel EEG data into the recurrent neural network for enhancement processing and flattening operations;

[0031] Fully connected and discriminant unit: used to distinguish true from false for the enhanced EEG data through N fully connected layers;

[0032] Wherein, the spatial domain convolution unit is located before the temporal domain convolution unit.

[0033] A computer device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the driving method.

[0034] A computer-readable storage medium stores a computer program / instruction thereon, which implements the steps of a driving method when executed by a processor.

[0035] Compared with the prior art, the present invention has the following advantages and technical effects:

[0036] The present invention realizes brain-controlled driving through visual steady-state induced brain-computer interface technology. In addition, a generative adversarial network is used to enhance the EEG data to achieve better decoding efficiency and improve safety and stability during driving. In addition, the present invention has good scalability, and the generative adversarial network used for EEG data enhancement can be combined with other types of brain-computer interface paradigms to achieve applications in more stimulation paradigms and more usage scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0038] Figure 1 A flow chart of a driving method based on a generative adversarial network and a brain-computer interface according to an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of an electroencephalogram-induced stimulation source according to an embodiment of the present invention;

[0040] Figure 3 Schematic diagram of the generator subnetwork structure of an embodiment of the present invention;

[0041] Figure 4 Schematic diagram of the structure of the discriminator subnetwork according to an embodiment of the present invention;

[0042] Figure 5 A schematic diagram of a generative adversarial network training and EEG data enhancement process according to an embodiment of the present invention;

[0043] Figure 6 A frequency domain schematic diagram of generating visual steady-state evoked EEG according to an embodiment of the present invention;

[0044] Figure 7 A schematic diagram of changes in EEG feature decoding efficiency after fusing original and enhanced EEG data and training using a deep learning model in an embodiment of the present invention;

[0045] Figure 8 Schematic diagram of the structure of a driving system based on a generative adversarial network and a brain-computer interface according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0047] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0048] First, the technical terms in this embodiment are explained:

[0049] 1. Generative Adversarial Networks:

[0050] The generative adversarial network consists of a generative model and a discriminative model. The generative model is responsible for capturing the distribution of sample data, while the discriminative model is generally a binary classifier that determines whether the input is real data or a generated sample. The optimization process of this model is a "binary minimax game" problem. During training, one of the parties (the discriminative network or the generative network) is fixed, and the parameters of the other model are updated. The iterations are repeated alternately, and finally, the generative model can estimate the distribution of sample data. The emergence of generative adversarial networks has greatly promoted the research on unsupervised learning and image generation.

[0051] 2. Brain-computer interface

[0052] Brain-Machine Interface (BMI; Brain Computer Interface, BCI) refers to a direct connection created between the human or animal brain and external devices to achieve information exchange between the brain and the device.

[0053] Brain-computer interface, sometimes also called "brain port" direct neural interface or "brain-machine fusion perception" brain-machine interface, is a direct connection pathway established between the human or animal brain (or brain cell culture) and external devices. In the case of a one-way brain-computer interface, the computer either receives commands from the brain or sends signals to the brain (such as video reconstruction), but cannot send and receive signals at the same time. A two-way brain-computer interface allows two-way information exchange between the brain and external devices.

[0054] Brain-computer interface is a direct communication channel between the brain and external devices. Its signal comes from the central nervous system and does not rely on the peripheral nerve and muscle system during transmission. It is often used to assist, enhance, and repair the sensory-motor function of the human body or improve human-computer interaction capabilities.

[0055] 3. Visual steady-state induced EEG data:

[0056] Evoked potentials are nerve action potentials or postsynaptic potentials generated when the brain is stimulated by pulsed current, flashes, changing images, or continuous sounds. This brain potential has a certain latency period, and the length of the latency period is related to the stimulation site, the distance from the recording site, and the conduction speed of the nerve impulse.

[0057] There are three main types of evoked potentials: visual evoked potential, auditory evoked potential, and tactile evoked potential. Visual evoked potential is a natural response signal of the brain to visual stimulation. According to the frequency of visual stimulation, visual evoked potential can be roughly divided into three categories: transient visual evoked potential (TVEP), steady-state visual evoked potential (SSVEP), and pseudo-random code visual evoked potential.

[0058] The visual stimulation frequency of transient visual evoked potential is generally less than 4Hz, and the time interval between stimulations is relatively long, which can ensure the complete presentation of each EEG evoked potential. It is generally used in visual research. Compared with transient visual evoked potential, the time interval between flashes of SSVEP stimulation modules can be much smaller, but adjacent stimuli will overlap, resulting in overlapping of the induced response signals in time.

[0059] Researchers believe that the various neural networks distributed in the brain have their own inherent resonant frequencies. Under normal conditions, these neural networks are not synchronized with each other, and are chaotic and irregular. The EEG signals at this time are spontaneous EEG. When a constant frequency of external visual stimulation is applied, the neural network consistent with the stimulation frequency or harmonic frequency will resonate, causing the brain's potential activity to change significantly at the stimulation frequency or harmonic frequency, thereby generating SSVEP signals.

[0060] From the perspective of physiological characteristics, different functional areas of the brain have their own division of labor, and the sensory, motor, and cognitive modules in different cortical areas are independent of each other. However, the various functional modules cooperate with each other to form an organic whole. In the process of the brain processing sensory information, multiple modules work in parallel. The BCI system based on SSVEP signals determines the brain's thinking activities by detecting the EEG signals in the occipital visual area.

[0061] The visual stimulus sources commonly used to induce SSVEP signals mainly include light stimulus sources, graphic stimulus sources, and pattern flip stimulus sources. Light stimulus sources mainly use light sources such as LEDs, fluorescent lamps, and xenon lamps that are modulated at a certain frequency as visual stimuli. Light source stimulators generally contain a precise circuit control system that can generate various flashing patterns of different frequencies through control. The parameters of the light source are described by a modulation depth, which is related to the amplitude of the induced SSVEP signal.

[0062] The graphic stimulus source mainly utilizes the flickering stimulation of the computer display screen, that is, making graphics such as a single simple square, complex multiple squares, pictures of different colors, etc., and the graphics flicker at a certain frequency.

[0063] The pattern flip stimulus source is also generated on the display screen, but the difference is that the pattern flip stimulus source mainly uses two graphics to appear alternately at a certain flip rate to present visual stimulation. At present, the commonly used graphic patterns are black and white checkerboard and grid.

[0064] In the experiment, when the stimulation frequency is between 1-100Hz, SSVEP signals can be induced. In the BCI system based on SSVEP, the commonly used stimulation frequency range is 4-50Hz. Among them, the frequency band of stimulation frequency of 1-12Hz is often called the low frequency band; the frequency band of stimulation frequency of 12-30Hz is the medium frequency band; and the frequency band of 30-50Hz is the high frequency band.

[0065] When the stimulation frequency is 10Hz, SSVEP produces the largest response; secondly, the stimulation frequencies that can induce SSVEP signals more obviously are concentrated in the mid-frequency band, while the stimulation frequencies in the high-frequency band can induce the smallest response. Since the SSVEP signals induced by stimulation frequencies in the mid-to-low frequency band can produce a larger response, the stimulation frequencies currently used by the BCI system based on SSVEP are mainly concentrated in the mid-to-low frequency band (4-30Hz). The stimulation frequency in this frequency band has certain disadvantages: visual stimulation sources that flicker at this frequency band are more likely to cause visual fatigue in the subjects; when the stimulation frequency of the visual stimulation source is in this frequency band, it is easy to induce epilepsy in the subjects.

[0066] The frequencies of the alpha and beta bands of human spontaneous brain waves are 8-13Hz and 13-30Hz respectively, which will overlap with the stimulation frequency in this frequency band, thus causing false detection.

[0067] There are many disadvantages in the low-mid frequency band, but this frequency band can make the induced SSVEP signal produce the maximum response, and these disadvantages can be solved by enhancing the energy of the SSVEP signal. Therefore, many studies are also based on the low-mid frequency band.

[0068] This embodiment proposes a driving method based on a generative adversarial network and a brain-computer interface, such as Figure 1 ,include:

[0069] Obtaining the user's original visual steady-state evoked EEG data;

[0070] Inputting the original visual steady-state evoked EEG data into the convolution-recurrent network fused generative adversarial network for processing to obtain enhanced visual steady-state EEG data, wherein the convolution-recurrent network fused generative adversarial network training set is trained to obtain the original visual steady-state evoked EEG data;

[0071] fusing the original visual steady-state evoked EEG data with the enhanced visual steady-state EEG data, and decoding them to obtain decoded EEG signal features;

[0072] The decoded EEG signal features are compiled into brain-controlled driving operation instructions and input into the corresponding vehicle for control.

[0073] This embodiment realizes brain-controlled driving through visual steady-state induced brain-computer interface technology. In addition, a generative adversarial network is used to enhance the EEG data to achieve better decoding efficiency and improve safety and stability during driving.

[0074] Furthermore, in this embodiment, obtaining the user's original visual steady-state induced EEG data includes: EEG induction and EEG acquisition sub-steps.

[0075] The EEG induction sub-step is used to induce the user's visual steady-state induced EEG. It is necessary to encode the stimulus source considering the number of driving instructions, and present the encoded visual steady-state induced stimulus source to the user, such as Figure 2 As shown, this embodiment combines the four basic driving instructions of turning left, turning right, accelerating, and decelerating to construct an EEG induced stimulation source.

[0076] The EEG acquisition sub-step is used to acquire the induced visual steady-state induced EEG. The EEG acquisition sub-step requires the user to acquire the induced raw EEG data when viewing the stimulus source.

[0077] Furthermore, the generative adversarial network of the convolution-recurrent network fusion includes: a generator subnetwork and a discriminator subnetwork, wherein the generator subnetwork is used to build a time-space domain deconvolution network according to the time-frequency characteristics of the visual steady-state evoked potential, and enhance the input EEG data; the discriminator subnetwork is used to build a space-time domain convolution-recurrent network according to the time-frequency characteristics of the visual steady-state evoked potential, and perform true and false discrimination on the enhanced EEG data.

[0078] Specifically, the steps for building a generative adversarial network of convolution-recurrent network fusion are to build an applicable generative adversarial network according to the time-frequency characteristics of the acquired visual steady-state evoked EEG data, specifically: generator subnetwork building substep, discriminator subnetwork building substep and optimization function building substep.

[0079] In this embodiment, the length of a single trial of the original visual steady-state evoked EEG data is 8×1024 (N ch ×N t ), N ch N is the number of channels for collecting EEG (including channels PO7, PO3, POZ, PO4, PO8, O1, Oz and O2), t The length of EEG data is 1024 sampling points, the acquisition time is 4s, and the sampling rate is 256Hz.

[0080] The generator sub-network construction sub-step is used to build a generative adversarial network generator for visual steady-state induced EEG data enhancement according to the time-frequency characteristics of the original visual steady-state induced EEG data, such as Figure 3 Schematic diagram of the generator subnetwork structure of this embodiment.

[0081] The generator sub-network construction step is to first generate N through the Gaussian noise generation unit t0 is 100, i.e., Gaussian noise of length 1×100, and is mapped to length N through N=3 fully connected layers t1is 256, that is, the length is Gaussian noise of 1×256. The time domain deconvolution unit performs time domain deconvolution through a time domain deconvolution kernel with k of 1×128 to generate N t2 is 1024, that is, a signal with a length of 1×1024; in the spatial deconvolution unit, a spatial deconvolution kernel with k being 8×1 is used to perform spatial deconvolution, and finally N ch is 8, N t2 The enhanced EEG data is 1024.

[0082] The discriminator sub-network is used to build a generative adversarial network discriminator for visual steady-state induced EEG data enhancement based on the time-frequency characteristics of the original visual steady-state induced EEG data, such as Figure 4 The left sub-figure is a schematic diagram of the discriminator sub-network structure of the embodiment of the present application. The discriminator sub-network construction sub-step first performs a spatial domain convolution operation on the EEG data (including the original EEG data and the enhanced EEG data), and the convolution kernel size k is 1×1, generating a signal with a length of 1×1024; in the time domain convolution unit, a time domain convolution kernel with a k of 1×10 is used to perform time domain convolution to generate N t1 is 512, that is, a signal with a length of 1×512; then the signal enters the recurrent neural network. In this embodiment, the recurrent neural network uses a bidirectional long short-term memory network, and the relevant parameters are Figure 4 The right sub-image is shown, and finally three fully connected layers are used for true / false discrimination.

[0083] Furthermore, the convolutional-recurrent network fusion generative adversarial network selects Wasserstein distance as the basic loss function and introduces penalty terms to improve the corresponding loss function.

[0084] Specifically, according to the time domain and frequency domain characteristics of visual steady-state evoked EEG data enhancement, in order to make the frequency response of the enhanced visual steady-state evoked EEG data closer to the real EEG data, the Wasserstein distance is selected as the basic loss function, and considering the nonlinear characteristics of EEG data, to prevent the "gradient explosion" and other problems in the subsequent network training and data enhancement steps, the penalty term is introduced to improve the corresponding loss function, and then a complete convolution-recurrent network fusion generative adversarial network for visual steady-state evoked EEG data enhancement is built. The formula is as follows:

[0085]

[0086] In the formula, E x～px [D(x)] is the distribution of the original visual steady-state evoked EEG data, E x～pg [D(x)] is used to enhance the distribution of EEG data, and a penalty term λE is introduced x～ppenalty max(0,||▽xD(x)||-1)dx}.

[0087] As another embodiment of the present invention, the training sub-step of the generative adversarial network is to use the original visual stability induced EEG data to train the convolution-recurrent generative adversarial network. The process includes first separating the original EEG data according to specific frequency stimulation codes, and then sequentially inputting them into the training convolution-recurrent generative adversarial network, such as Figure 5 It is a schematic diagram of the generative adversarial network training and EEG data enhancement process in the embodiment. The left sub-figure is a schematic diagram of the training process, and the right sub-figure is a diagram of the changes in the EEG spectrum generated during the training process. It can be seen from the right sub-figure that as the training process progresses, the frequency characteristics of the generated EEG data are getting closer and closer to the real EEG data.

[0088] Specifically, the trained generative adversarial network is used to generate visual steady-state evoked EEG data with specific frequency stimulus encoding, such as Figure 6 The spectrum comparison diagram of the visual steady-state evoked EEG data generated for the embodiment and the original visual steady-state evoked EEG data shows that for a single stimulation frequency, the generated EEG data and the original EEG data have obvious correspondence at the corresponding frequency.

[0089] Furthermore, the decoded EEG signal features are obtained, including:

[0090] The original visual steady-state evoked EEG data and the enhanced visual steady-state EEG data are fused, the fused data are trained using a deep learning model to obtain visual steady-state evoked EEG features, the visual steady-state evoked EEG features are decoded, and the decoded EEG signal features are obtained.

[0091] Specifically, the visual steady-state evoked EEG feature decoding sub-step is to fuse the original and enhanced EEG data, use the deep learning model for training, and perform efficient decoding of the visual steady-state evoked EEG features, such as Figure 7 As shown, it is a schematic diagram of the change in EEG feature decoding efficiency after data fusion. This embodiment uses two classic deep learning networks SSVEPNet and EEGNet for verification. It can be seen that under different time window lengths, the EEG feature decoding efficiency after data fusion is improved to varying degrees.

[0092] The decoded EEG signal features are compiled into corresponding brain-controlled driving operation instructions, and then output to the vehicle to complete the corresponding operations. In this embodiment, the driving operations include turning left, turning right, accelerating and decelerating.

[0093] This embodiment also provides a driving system based on a generative adversarial network and a brain-computer interface, such as Figure 8 ,include:

[0094] EEG acquisition module: used to collect the user's original visual steady-state induced EEG characteristics;

[0095] EEG data processing module: used for inputting the original visual steady-state induced EEG features into the generative adversarial network fused with the convolutional-recurrent network for processing, obtaining enhanced visual steady-state EEG data, and fusing the original visual steady-state induced EEG features with the enhanced visual steady-state EEG data for decoding, obtaining decoded EEG signal features, and compiling them into brain-controlled driving operation instructions, which are input to the corresponding vehicle for control;

[0096] Vehicle data processing module: used to receive vehicle control instructions and update vehicle speed and position information;

[0097] Driving and stimulation interface display module: used to present the visual steady-state EEG evoked source and vehicle driving interface in real time.

[0098] Furthermore, the generative adversarial network of the convolutional-recurrent network fusion includes a generator subnetwork and a discriminator subnetwork, wherein the generator subnetwork is used to enhance the input EEG data, and the discriminator subnetwork is used to distinguish the true from the false of the enhanced data through a bidirectional long short-term memory network.

[0099] Specifically, the generator subnetwork includes:

[0100] Gaussian noise generation unit: used to generate single-channel Gaussian noise according to the characteristics of the original visual steady-state induced EEG data containing Gaussian noise components, and its length is 1×N t0 , after passing through N fully connected layers, it becomes a length of 1×N t1 Gaussian noise (i.e., the first Gaussian noise), where N t1 is the time domain length of the generated single-channel Gaussian noise, N t0 is the time domain length of the initial Gaussian noise.

[0101] Time domain deconvolution unit: used to perform time domain deconvolution on the generated single-channel Gaussian noise according to the frequency characteristics of the original visual steady-state evoked EEG data with obvious response to the corresponding visual stimulus frequency, and finally generate a single-channel EEG signal (i.e., the first single-channel EEG signal) with a length of 1×N t2 , N t2 is the time domain length of the single-channel EEG data after time domain deconvolution;

[0102] Spatial deconvolution unit: According to the spatial characteristics of the original visual steady-state evoked EEG data, which has the most obvious response in the occipital area of ​​the brain, the generated single-channel EEG data is spatially deconvolved to finally generate a multi-channel EEG signal (i.e., the first multi-channel enhanced visual steady-state evoked EEG signal), whose length is N ch ×N t2 , to form a complete generator unit, N t2is the slice length of the user's original visual steady-state evoked EEG data (also the time domain length of the single-channel EEG data after time domain deconvolution), N ch The number of channels of the user's original visual steady-state evoked EEG data collected.

[0103] The time domain deconvolution unit is located before the spatial domain deconvolution unit. In this embodiment, N is required. t0 ≤N t1 ≤N t2 .

[0104] Furthermore, the discriminator subnetwork includes:

[0105] Spatial convolution unit: used to evoke the spatial characteristics of the original visual steady-state induced EEG data in the occipital area of ​​the brain, which is the most obvious response, for the length N ch ×N t2 The EEG data or the original EEG data is convolved in the spatial domain to form a second single-channel EEG signal with a length of 1×N t2 ;;

[0106] Time domain convolution unit: According to the frequency characteristics of the original visual steady-state evoked EEG data, which has a significant response to the corresponding visual stimulus frequency, the length of 1×N t2 The single-channel EEG data is convolved in the time domain to generate the third single-channel EEG data with a length of 1×N t1 ;

[0107] Recurrent network unit: It is used to convert the length of 1×N into a recurrent network unit according to the time characteristics of the original visual steady-state evoked EEG data, which has obvious brain response around 0.25-0.5s after the physical stimulus appears. t1 The third single-channel EEG data is put into the recurrent neural network for processing and flattening;

[0108] Fully connected and discriminant unit: used to distinguish true from false for the enhanced EEG data through N fully connected layers;

[0109] Among them, the spatial domain convolution unit is located before the temporal domain convolution unit.

[0110] In this embodiment, the relevant parameter N t2 ≥N t1 ≥N t0 , and requires N t2 The length of the slice of the original EEG data collected from the user.

[0111] Specifically, the original visual steady-state evoked EEG data and the enhanced visual steady-state EEG data are fused, the fused data is trained using a deep learning model to obtain visual steady-state evoked EEG features, the visual steady-state evoked EEG features are decoded, and the decoded EEG signal features are obtained.

[0112] Driving and stimulation interface display module: used to present the visual steady-state EEG evoked source and vehicle driving interface in real time.

[0113] Specifically, the decoded EEG signal features are compiled into corresponding brain-controlled driving operation instructions, and then output to the vehicle to complete the corresponding operations. In this embodiment, the driving operations include turning left, turning right, accelerating and decelerating.

[0114] As another implementation, the driving and stimulation interface display module is also used to realize the real-time display of the stimulation and driving interface. EEG acquisition requires collecting the original EEG data induced when the user is watching the stimulation source.

[0115] This embodiment also provides a computer device, including a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the above-mentioned driving method.

[0116] This embodiment also provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of the driving method when the computer program / instruction is executed by a processor.

[0117] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A driving method based on a generative adversarial network and a brain-computer interface, characterized in that: include: Obtaining the user's original visual steady-state evoked EEG data; Inputting the original visual steady-state evoked EEG data into a convolutional-recurrent network fused generative adversarial network for processing to obtain enhanced visual steady-state EEG data, wherein the convolutional-recurrent network fused generative adversarial network is obtained by training a training set, and the training set is the initial visual steady-state evoked EEG data; fusing the original visual steady-state evoked EEG data with the enhanced visual steady-state EEG data, and decoding them to obtain decoded EEG signal features; The decoded EEG signal features are compiled into brain-controlled driving operation instructions to control the corresponding vehicle.

2. The driving method based on generative adversarial network and brain-computer interface according to claim 1, characterized in that: The generative adversarial network of the convolution-recurrent network fusion includes: a generator subnetwork and a discriminator subnetwork, wherein the generator subnetwork is used to build a time-space domain deconvolution network according to the time-frequency characteristics of visual steady-state evoked potentials, and enhance the input EEG data; the discriminator subnetwork is used to build a space-time domain convolution-recurrent network according to the time-frequency characteristics of visual steady-state evoked potentials, and perform true and false discrimination on the enhanced EEG data.

3. The driving method based on generative adversarial network and brain-computer interface according to claim 1, characterized in that: The convolution-recurrent network fusion generative adversarial network selects Wasserstein distance as the basic loss function and introduces a penalty term to improve the corresponding loss function.

4. The driving method based on generative adversarial network and brain-computer interface according to claim 1, characterized in that: Obtaining the decoded EEG signal features, including: The original visual steady-state evoked EEG data and the enhanced visual steady-state EEG data are fused, the fused data are trained using a deep learning model to obtain visual steady-state evoked EEG features, the visual steady-state evoked EEG features are decoded, and the decoded EEG signal features are obtained.

5. A driving system based on a generative adversarial network and a brain-computer interface, characterized in that: include: EEG acquisition module: used to collect the user's original visual steady-state induced EEG characteristics; EEG data processing module: used for inputting the original visual steady-state induced EEG features into the generative adversarial network fused with the convolutional-recurrent network for processing, obtaining enhanced visual steady-state EEG data, and fusing the original visual steady-state induced EEG features with the enhanced visual steady-state EEG data for decoding, obtaining decoded EEG signal features, and compiling them into brain-controlled driving operation instructions, which are input to the corresponding vehicle for control; Vehicle data processing module: used to receive vehicle control instructions and update vehicle speed and position information; Driving and stimulation interface display module: used to present the visual steady-state EEG evoked source and vehicle driving interface in real time.

6. The driving system according to claim 5, characterized in that: The convolution-recurrent network fused generative adversarial network includes a generator subnetwork and a discriminator subnetwork, wherein the generator subnetwork is used to enhance the input EEG data, and the discriminator subnetwork is used to distinguish the true from the false of the enhanced data through a bidirectional long short-term memory network.

7. The driving system according to claim 6, characterized in that: The generator sub-network includes: A Gaussian noise generating unit is used to generate single-channel Gaussian noise according to the characteristic of the original visual steady-state induced EEG data containing Gaussian noise components, wherein the single-channel Gaussian noise is obtained through N fully connected layers to generate a first Gaussian noise; A time domain deconvolution unit is used to perform time domain deconvolution on the single-channel Gaussian noise according to the frequency characteristics of the original visual steady-state evoked EEG data, in which the power spectrum and signal-to-noise ratio are significantly improved and have a significant response at the corresponding visual stimulation frequency, to generate a first single-channel EEG signal; A spatial deconvolution unit is used to perform spatial deconvolution on the single-channel EEG signal according to the spatial characteristics of the original visual steady-state evoked EEG data that are most obviously responded to in the occipital region of the brain, so as to generate a first multi-channel enhanced visual steady-state evoked EEG signal; Wherein, the time domain deconvolution unit is located before the spatial domain deconvolution unit.

8. The driving system according to claim 6, characterized in that: The discriminator subnetwork includes: Spatial convolution unit: used for performing a spatial convolution operation on the enhanced visual steady-state evoked EEG data according to the spatial characteristics of the original visual steady-state evoked EEG data that have the most obvious response in the occipital area of ​​the brain, to generate a second single-channel EEG signal; A time domain convolution unit is used to perform a time domain convolution operation on the second single-channel EEG signal according to the frequency characteristics of the original visual steady-state evoked EEG data with significantly improved power spectrum and signal-to-noise ratio at the corresponding visual stimulation frequency, so as to generate a third single-channel EEG data; Recurrent network unit: used for inputting the third single-channel EEG data into the recurrent neural network for enhancement processing and flattening operations; Fully connected and discriminant unit: used to distinguish true from false for the enhanced EEG data through N fully connected layers; Wherein, the spatial domain convolution unit is located before the temporal domain convolution unit.

9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the driving method according to any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the driving method according to any one of claims 1 to 4 are implemented.

Images