Electroencephalogram coding method based on generative adversarial technology

Abstract

An electroencephalogram coding method based on a generative adversarial technology comprises the following steps: (1) designing an electroencephalogram signal acquisition experiment normal form in combination with a visual stimulation scene requirement, and completing acquisition of a tested electroencephalogram signal by utilizing electroencephalogram signal acquisition equipment according to an experiment normal form process to obtain original electroencephalogram signal data corresponding to a visual stimulation image; (2) designing an electroencephalogram feature extraction model based on a compact space-time convolutional neural network, and extracting electroencephalogram feature data related to the visual stimulation image from the electroencephalogram signals from the original electroencephalogram signal data; (3) on the basis of the visual stimulation image and the electroencephalogram feature data, constructing an electroencephalogram mapping model based on a deep residual-cyclic regression network, and simulating representation association of the electroencephalogram feature data to the visual stimulation image under the brain vision effect; and (4) a generative adversarial mechanism is introduced into the electroencephalogram mapping model, a conditional depth generative network structure is constructed, training is performed by using the visual stimulation image and the extracted electroencephalogram feature data, an electroencephalogram coding model based on visual-electroencephalogram features is finally obtained, and the electroencephalogram coding model is used for performing electroencephalogram coding on the input visual stimulation image.

Description

Technical Field

[0001] The invention relates to an electroencephalogram (EEG) coding method based on generative adversarial technology, and belongs to the technical field of intelligent information processing. Background Art

[0002] The human visual system has a significantly better recognition effect than the current best machine vision system under statistical learning theory. In particular, when dealing with perception problems in complex environments, traditional methods encounter great difficulties. The human visual system has a strong anti-interference ability and can complete visual recognition tasks in various complex environments. Therefore, how to study target recognition and computer vision algorithms from the perspective of visual cognition has become a very important and challenging task.

[0003] Starting from the expression rules of brain nerve activity under visual stimulation, the present invention analyzes the EEG information encoding and decoding mode under visual effects, makes full use of the data analysis ability of the neural network model, uses brain nerve electrophysiology theory and intelligent information processing methods to study the change mechanism of EEG information under complex visual stimulation, and establishes an EEG encoding method under visual stimulation, thereby providing a technical approach to achieve accurate and effective identification of targets of interest under complex backgrounds, and is also expected to provide solutions for brain-computer intelligence fusion and human-computer interaction. Summary of the invention

[0004] The technical problem solved by the present invention is: to provide an EEG coding method based on generative adversarial technology for the problem of analyzing the representation pattern of EEG signals under visual stimulation.

[0005] The technical solution of the present invention is: an EEG encoding method based on generative adversarial technology, comprising:

[0006] Design an EEG signal acquisition experimental paradigm based on the requirements of the visual stimulation scene, and use the EEG signal acquisition equipment to complete the acquisition of the subject's EEG signals according to the experimental paradigm process to obtain the original EEG signal data corresponding to the visual stimulation image;

[0007] Design an EEG feature extraction model based on a compact spatiotemporal convolutional neural network to extract EEG feature data related to visual stimulus images from the original EEG signal data;

[0008] Based on the visual stimulus images and EEG feature data, an EEG mapping model based on a deep residual-recurrent regression network is constructed to simulate the representation association of EEG feature data with visual stimulus images under the action of brain vision;

[0009] The generative adversarial mechanism is introduced into the EEG mapping model, a conditional deep generative network structure is constructed, and visual stimulation images and extracted EEG feature data are used for training, and finally an EEG encoding model based on visual-EEG features is obtained. The EEG encoding model is used to perform EEG encoding on the input visual stimulation images.

[0010] Preferably, the EEG signal acquisition experimental paradigm adopts the RSVP experimental paradigm, in which the subject's task is to pay attention to the visual stimulation picture, the target picture accounts for 5%-10% of the visual stimulation picture, and the visual stimulation picture presentation frequency is 10-16Hz.

[0011] Preferably, the EEG feature data related to the visual stimulation image in the EEG signal is extracted in the following manner:

[0012] Design an EEG decoding and recognition model, which includes module 1, module 2 and a classification module;

[0013] Module 1 performs two convolution steps in sequence, where the first convolution is F1 temporal filters, a 2D convolution with a kernel size of (1, 64), and a kernel length of half the data sampling rate, outputting F1 feature maps containing EEG signals at different bandpass frequencies. The second convolution is a deep convolution learning spatial filter with a size of (C, 64), operating separately on each feature map output by the first convolution. Afterwards, the Dropout technique is used to randomly discard the individual feature outputs (the Dropout probability is set to 0.25), and finally, a pooling layer with an average size of (1, 4) is used for downsampling. The output of this module is input to module 2; C is the number of leads of the collected original EEG signal data;

[0014] Module 2 includes a separable convolution, F2 (1, 1) point convolutions, and an average pooling layer for dimensionality reduction connected in series;

[0015] In the classification module, the output of the average pooling layer in module 2 is directly passed to the softmax output of the specified number of classification categories to achieve EEG signal decoding;

[0016] The original EEG signal data is used to train the EEG decoding and recognition model, and the remaining layers of the trained EEG decoding and recognition model except the final classification module are used as the EEG feature extraction model based on the compact spatiotemporal convolutional neural network. The original EEG signal data is input into the EEG feature extraction model, and the output sequence obtained is used as the EEG feature data under the corresponding visual stimulation.

[0017] Preferably, F1 is set to 8 or 16, and F2 is set to 16 or 32.

[0018] Preferably, the EEG mapping model based on the deep residual-recurrent regression network is constructed in the following manner:

[0019] Input the visual stimulus image into the ResNet network to obtain image features including global features and local features;

[0020] After the ResNet network, LSTM and fully connected structures are introduced as regression layers to regress image features to EEG features, thus realizing the mapping between image features and EEG features.

[0021] Among them, the number of output neurons in the regression layer is the same as the dimension of the EEG feature data obtained by the EEG feature extraction model.

[0022] Preferably, the EEG mapping model based on the deep residual-recurrent regression network includes 49 convolutional layers, 1 recurrent neural network layer, and 1 fully connected layer.

[0023] Preferably, the generative adversarial mechanism is introduced into the EEG mapping generation process in the following manner to construct a conditional deep generative network model:

[0024] The EEG mapping model based on the deep residual-recurrent regression network structure is used as the generator in the generative adversarial mechanism to generate simulated EEG feature data under relevant visual stimulation images;

[0025] Construct a multi-layer deep neural network structure as a discriminator to distinguish between the generated simulated EEG feature data and the extracted EEG feature data;

[0026] The conditional deep generation network model is jointly composed of the generator and the discriminator.

[0027] Preferably, a loss function is used as a qualitative indicator of the training progress of the conditional deep generative network model; wherein the loss function of the generator is an adversarial loss function that introduces the Wasserstein distance, and the loss function of the discriminator is a mean square error loss function.

[0028] Preferably, the trained generator is used as an EEG encoding model.

[0029] Compared with the prior art, the present invention has the following beneficial effects: the present invention quantitatively analyzes the EEG signal representation pattern under visual stimulation, studies the correlation between visual stimulation and EEG signals, establishes the coding pattern of human brain neural information under visual stimulation, explores the human visual representation method based on EEG signals, supports the goal of visual scene understanding driven by brain information, and is expected to embed the human brain visual cognition model into the machine, providing a key technical foundation for realizing brain-machine hybrid intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1Schematic diagram of the EEG coding technology framework based on generative adversarial technology

[0031] Figure 2 Flowchart of RSVP-based EEG signal acquisition experimental paradigm DETAILED DESCRIPTION

[0032] The following is combined with Figure 1 The present invention is described in detail with examples. The present invention provides an EEG coding method based on generative adversarial technology, the steps are as follows:

[0033] (1) Experimental paradigm design for EEG signal acquisition under rapid sequence visual presentation

[0034] The present invention designs an effective electroencephalogram (EEG) signal acquisition experimental paradigm in combination with the visual stimulation scene requirements, and uses an EEG signal acquisition device to complete the acquisition of the subject's EEG signal.

[0035] Rapid Serial Visual Presentation (RSVP) is a commonly used experimental paradigm to measure attention and cognitive processing ability. In this experimental paradigm, the subjects need to watch a series of rapidly presented visual stimulus images in a short period of time and find the target stimulus among them. In the RSVP experiment, the subject's task is to pay attention to the target image and press the button as soon as possible after it appears. By analyzing the subject's EEG response, it can be determined whether the subject has noticed the target image.

[0036] The present invention designs a rapid serial visual presentation experimental paradigm in which each image is presented for a certain period of time. The visual stimulation images use target images and non-target images in a complex environment to study the perception ability of the human visual system to the relevant visual stimulation target images of interest. The advantages of the RSVP experimental paradigm are simplicity, efficiency, and ease of operation, and it is suitable for a variety of application scenarios, such as image retrieval, safety detection, cognitive assessment, etc. When designing the RSVP experimental paradigm, it is usually necessary to consider factors such as stimulus type, stimulus duration, stimulus interval, target stimulus, target stimulus frequency, and response mode.

[0037] Aiming at the EEG encoding scenario under visual stimulation, an RSVP EEG signal acquisition experimental paradigm under relevant visual stimulation is designed. The visual stimulation images select the visual stimulation target images and non-target images of interest according to the task requirements; the visual stimulation picture presentation frequency is set to 12.5ms (the setting range of the present invention is 10-16Hz); the stimulation interval is set to 0 according to the conventional RSVP paradigm, that is, there is no interval between the two visual stimulation images, and the next visual stimulation image appears immediately after the presentation of one image; the proportion of target images in the visual stimulation images is set to 5% (generally set to 5%-10%), that is, there are only 5 target images in every 100 visual stimulation images, and the others are non-target images; the reaction mode is set to the subject counting the number of targets to avoid interference from motion signals generated by key presses.

[0038] In this experiment, participants wore electrode caps and sat 1 meter away from a 1920*1080 resolution monitor. In order to reduce the influence of eye movement during the experiment, a fixed cross mark was added in the middle of the visual stimulus image as the fixation point. The subjects pressed the space bar on the keyboard to start viewing the image sequence. The interval between the two sequences was controlled by the subjects. Pressing the space bar started the next sequence of images.

[0039] (2) Rapid sequence presentation of EEG signal acquisition

[0040] The general characteristics of neurophysiological signals are:

[0041] ①The signal amplitude is weak and needs to be amplified by tens of thousands times;

[0042] ②The signal frequency range is low (within 100Hz), and high-frequency noise needs to be filtered out;

[0043] ③ The signal source impedance is high (there is a resistance of several thousand ohms between the scalp and the skull), which requires the preamplifier circuit to have a very high input impedance, generally greater than 10 megohms;

[0044] ④The power frequency interference of the power supply is common mode interference, and the amplifier common mode suppression needs to be relatively high;

[0045] ⑤Low signal-to-noise ratio and strong randomness.

[0046] Considering these characteristics, the EEG signal acquisition system mainly includes: amplification, filtering circuit design, analog-to-digital converter design, etc. It is generally set to 3-stage amplification: the first stage (preamplification): requires higher input impedance, higher common-mode rejection ratio, low offset voltage and drift, low noise and other performance, and can use instrument amplifiers, in-phase parallel differential amplifier circuits. The second stage (intermediate stage amplification) and the third stage (final stage amplification): need to meet high gain and low noise, low offset voltage and drift, good stability and output swing. The amplification process is interspersed with filtering processes: high-pass, low-pass, and band-stop filtering.

[0047] The experiment uses an electroencephalograph specially used for measuring and recording electroencephalograms to collect electroencephalogram signals. The working process is as follows: the electrodes placed on the scalp can detect weak EEG signals, usually 5-100μV, which are coupled to the differential amplifier through the electrode leads for amplification by about 10000-100000 times, and then converted into digital signals through the data acquisition card in the computer host, and finally the signal data is recorded by the recording system on the matching PC.

[0048] In this case, the EEG acquisition equipment used was a 64-lead device from Australia's NeuroScan SynAmps. In a laboratory environment specially designed for weak electrophysiological signal acquisition and shielded from the outside world, EEG signals were acquired using standardized and reasonable operating methods: ① To ensure good conductivity between the scalp and the sensor, the subject's hair was washed before acquisition; ② The EEG acquisition equipment was connected and debugged, and the amplifier, AD converter, data transmission optical fiber and data recording notebook were correctly connected so that EEG signals could be effectively acquired; ③ The subject was assisted in wearing the EEG cap to ensure that the electrode position matched the corresponding area of ​​the scalp; ④ The experimental process was explained to the subject to confirm that the subject had a full understanding of the experimental process and operation; ⑤ The experiment began. During the signal recording process, the experimenter always paid attention to the subject's reaction, prevented and promptly dealt with emergencies such as equipment failure and subject discomfort during the experiment, and supervised the smooth progress of the entire experiment; ⑥ After the experiment, the equipment was sorted out and experimental records were kept.

[0049] (3) EEG feature extraction based on compact spatiotemporal convolutional neural network

[0050] In order to make full use of the multi-lead spatial relationship and time-inherent information in the original EEG signal data, the present invention designs an EEG feature extraction method based on a compact spatiotemporal convolutional neural network to achieve effective extraction of EEG features under relevant visual stimulation. First, an EEG signal recognition and decoding model is designed, and its overall structural parameters are shown in Table 1. C is the number of leads (64 in the present invention) and T is the number of time points of the EEG sequence corresponding to each visual stimulation image (120 in the present invention). The model includes three parts: module 1, module 2 and classification module.

[0051] Table 1 Structural parameters of EEG signal recognition and decoding model under visual stimulation

[0052]

[0053]

[0054] In module 1, two convolution steps are performed in sequence. F1 temporal filters (set to 8 in the present invention), 2D convolution with a convolution kernel size of (1, 64), and a convolution kernel length of half the data sampling rate, output F1 feature maps containing EEG signals at different bandpass frequencies. Setting the length of the temporal convolution kernel to half the sampling rate can capture frequency information of 2Hz and above. Then, a depthwise convolution of size (C, 1) is used to learn the spatial filter. The deep convolution is not fully connected to all previous feature maps, but by operating each input feature map separately, independent feature extraction of the feature map is achieved. Compared with ordinary convolution, the amount of computation and the amount of trainable parameters of the deep convolution will also be greatly reduced. The convolution calculation of the deep convolution layer is similar to the spatial filtering calculation in the CSP algorithm, which is a weighted filtering of the signals of each channel, so the output of this layer becomes a one-dimensional signal. This operation provides a direct method to learn its spatial filter for each temporal filter, so that frequency-specific spatial filters can be effectively extracted. The parameter D controls the number of spatial filters to be learned for each feature map (D = 1 is shown in the figure). After that, batch normalization is applied along the feature map dimension, exponential linear units (ELU) are applied, and the dropout technique is used (the dropout probability is set to 0.25). A pooling layer with an average size of (1, 4) is used to reduce the sampling rate of the signal to 32Hz. We also regularize each spatial filter weight by using a maximum norm constraint of 1 on it; ||w|| 2 <1.

[0055] In module 2, separable convolution is used, that is, deep convolution (the kernel size in the table is (1, 16), representing 500ms of EEG activity at 32Hz), followed by F2 (set to 8 in the present invention) pointwise convolutions of size 1. The main advantages of separable convolution are: ① reducing the number of parameters of feature mapping; ② explicitly decoupling the relationship within the feature map (Depthwise) and between feature maps (Pointwise) by first learning a (1, 1) kernel that integrates each feature map separately, and then optimizing the combined output. For EEG signals, SeparableConvolution will learn how to integrate the kernels of each feature map in the time dimension (depthwise convolution) and how to optimize the combination (Pointwise Convolution) to separate them. This operation is very useful for EEG signals because different feature maps represent information on different time scales of the data. Thereafter, an average pooling layer of size (1, 8) is connected for data dimensionality reduction.

[0056] In the classification module, the output result of the average pooling layer of the above module 2 is directly passed to the softmax output of the specified number of classification categories to achieve EEG signal decoding.

[0057] The original EEG signal data is used to train the EEG decoding and recognition model, and the remaining layers of the trained EEG decoding and recognition model except the final classification module are used as the EEG feature extraction model based on the compact spatiotemporal convolutional neural network. The original EEG signal data is input into the EEG feature extraction model, and the obtained sequence is used as the EEG feature data under the corresponding visual stimulation.

[0058] (4) Construction of EEG mapping model based on deep residual-recurrent regression network

[0059] In order to obtain the EEG mapping representation method under visual stimulation, the present invention constructs an EEG mapping model based on a deep residual-recurrent network, maps complex visual stimulation images to EEG feature data, and simulates the representation mode of visual stimulation images by EEG information in different regions under the action of brain vision.

[0060] This part of the design is based on the deep residual-recurrent regression network EEG mapping model, which extracts effective EEG features from the input visual stimulation image. The intermediate parameters and feature map size of the residual-recurrent network are shown in Table 2, where the ResNet structure applies residual learning to each layer, and solves the problem of reduced accuracy as the network deepens training by introducing shortcut connections. The network structure proposed in the present invention is based on the ResNet network, and introduces a regression network layer that adapts to EEG features, so as to build a deep residual-recurrent mapping model for EEG signal mapping under visual stimulation images. In the EEG mapping model structure of the deep residual-recurrent network, the ResNet-50 model is used as a visual-EEG feature extractor, and the overall structure consists of 49 convolutional layers, 1 recurrent neural network layer, and 1 fully connected layer.

[0061] Table 2. EEG mapping model structure parameters and feature map scale

[0062]

[0063]

[0064] The calculation process of the EEG mapping model is mainly divided into two steps. First, the visual stimulus image is input into the ResNet network to obtain image features including global features and local features; after the ResNet network, LSTM and fully connected structure are introduced as regression layers to regress image features to EEG features, realizing the mapping of image features and EEG features; among them, the number of output neurons in the regression layer is the same as the dimension of the EEG feature data obtained by the EEG feature extraction model.

[0065] (5) Construction of EEG coding model based on generative adversarial

[0066] This section introduces the generative adversarial mechanism into the EEG mapping generation process, and develops a generative adversarial mapping method based on visual-EEG features with strong generalization ability. Using the visual feature learning model under the EEG signal source, the key information representation of complex visual stimuli is embedded in the generative adversarial network, and the EEG signal mapping and generation process under related stimuli is constrained, aiming to achieve mapping under visual stimulation and generate simulated EEG information that conforms to the visual cognitive mechanism of the human brain, and describe and display the stimulus semantic expression to the subject in the form of EEG. The content of the present invention is intended to enable EEG signals to convey more value, allowing the generative model to capture visual information related to the visual stimulus image, and characterize related samples, and obtain effective information under visual stimulation from the generated simulated EEG.

[0067] Generative Adversarial Networks (GANs) is a deep generative model inspired by the idea of ​​two-person zero-sum games in game theory. Its network structure consists of a discriminator network (Discriminator, D) and a generator network (Generator, G). The purpose of the discriminator D is to judge as accurately as possible whether the input data comes from the real data distribution or the generated distribution; while the purpose of the generator G is to learn a distribution that is consistent with the real data distribution and generate samples that are indistinguishable from the real ones. The core idea of ​​GANs is to use the adversarial training strategy between the discriminator and the generator network to force the network's generated distribution to be infinitely close to the real data distribution.

[0068] Traditional generative adversarial networks can generate corresponding dimensional data that obeys the probability distribution of the training set based on low-dimensional random variables, but the generated data that conforms to the expected probability distribution is also random, and the generated data cannot be controlled. In the present invention, the generation of EEG signals under visual stimulation requires parsing and mapping according to the content of the visual stimulation image and generating simulated EEG feature data under the corresponding rules. To this end, the present invention constructs a conditional deep generative network model based on the traditional generative adversarial network, which uses the visual stimulation image as input to parse and map out complete simulated EEG feature data.

[0069] The generative adversarial mechanism is introduced into the EEG mapping generation process, and a conditional deep generative network model is constructed. The generator uses the EEG mapping model based on the deep residual-recurrent regression network structure constructed above to generate simulated EEG feature data under relevant visual stimulation images; the discriminator uses a multi-layer deep neural network structure to achieve the discrimination between the generated simulated EEG feature data and the extracted EEG feature data. The conditional deep generative network model is composed of the generator and the discriminator.

[0070] The present invention uses the loss function as a qualitative indicator of the training progress of the conditional deep generative network model. The focus of generative adversarial network training is to balance the generator and the discriminator. If the discriminator is too strong, the loss function will not decrease, the generator will not learn anything, and the quality of the generated information will not be improved. In view of the fact that the GAN model structure is generally prone to gradient diffusion, which leads to unstable training, the parameters cannot be updated for a long time, and the network cannot converge, the present invention designs an adversarial loss function based on the Wasserstein distance as the loss function of the generator. Since the Wasserstein distance has superior smoothing characteristics relative to the KL divergence and the JS divergence, the gradient disappearance problem can be solved in theory. By mathematically transforming the Wasserstein distance into a solvable form, and using a discriminator neural network with a limited parameter value range to maximize this form, the Wasserstein distance can be approximated. WGAN not only solves the problem of unstable training, but also provides a reliable training progress indicator, and this indicator is indeed highly correlated with the quality of the generated samples. In addition, the loss function of the discriminator is the mean square error loss function between the simulated EEG features and the extracted EEG feature data.

[0071] The generative adversarial network structure proposed in the present invention no longer needs to balance the training degree of the generator and the discriminator, which greatly improves the stability of GAN training: the better the discriminator evaluation index is trained, the more beneficial it is to improve the generator; it avoids the phenomenon of model collapse and demonstrates excellent robustness.

[0072] The present invention uses the trained generator as an EEG coding model based on visual-EEG features, and the EEG coding model is used to perform EEG coding on the input visual stimulation image.

[0073] Although the present invention has been disclosed as above in the form of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

[0074] Parts not described in detail in the present invention belong to common knowledge of those skilled in the art.

Claims

1. An EEG encoding method based on generative adversarial technology, characterized in that include: Design an EEG signal acquisition experimental paradigm based on the requirements of the visual stimulation scene, and use the EEG signal acquisition equipment to complete the acquisition of the subject's EEG signals according to the experimental paradigm process to obtain the original EEG signal data corresponding to the visual stimulation image; Design an EEG feature extraction model based on a compact spatiotemporal convolutional neural network to extract EEG feature data related to visual stimulus images from the original EEG signal data; Based on the visual stimulus images and EEG feature data, an EEG mapping model based on a deep residual-recurrent regression network is constructed to simulate the representation association of EEG feature data with visual stimulus images under the action of brain vision; The generative adversarial mechanism is introduced into the EEG mapping model, a conditional deep generative network structure is constructed, and visual stimulation images and extracted EEG feature data are used for training, and finally an EEG encoding model based on visual-EEG features is obtained. The EEG encoding model is used to perform EEG encoding on the input visual stimulation images.

2. The method according to claim 1, characterized in that: The EEG signal acquisition experimental paradigm adopts the RSVP experimental paradigm. Under this paradigm, the subject's task is to pay attention to the visual stimulation pictures, the target pictures account for 5%-10% of the visual stimulation pictures, and the visual stimulation pictures are presented at a frequency of 10-16 Hz.

3. The method according to claim 1, characterized in that: The EEG feature data related to the visual stimulus image in the EEG signal is extracted in the following way: Design an EEG decoding and recognition model, the model comprising module 1, module 2 and a classification module; in Module 1 performs two convolution steps in sequence, where the first convolution is F1 temporal filters, a 2D convolution with a convolution kernel size of (1, 64), and a convolution kernel length of half the data sampling rate, outputting F1 feature maps containing EEG signals at different bandpass frequencies. The second convolution is a deep convolution learning spatial filter with a size of (C, 64), operating on each feature map output by the first convolution separately; then, the Dropout technique is used to randomly discard the individual feature outputs, and finally, a pooling layer with an average size of (1, 4) is used for downsampling; the output of this module is input to module 2; C is the number of leads of the collected original EEG signal data; Module 2 includes a separable convolution, F2 (1, 1) point convolutions, and an average pooling layer for dimensionality reduction connected in series; In the classification module, the output of the average pooling layer in module 2 is directly passed to the softmax output of the specified number of classification categories to achieve EEG signal decoding; The original EEG signal data is used to train the EEG decoding and recognition model, and the remaining layers of the trained EEG decoding and recognition model except the final classification module are used as the EEG feature extraction model based on the compact spatiotemporal convolutional neural network. The original EEG signal data is input into the EEG feature extraction model, and the output sequence obtained is used as the EEG feature data under the corresponding visual stimulation.

4. The method according to claim 3, characterized in that: F1 is set to 8 or 16, and F2 is set to 16 or 32.

5. The method according to claim 1, characterized in that: The EEG mapping model based on the deep residual-recurrent regression network is constructed in the following way: Input the visual stimulus image into the ResNet network to obtain image features including global features and local features; After the ResNet network, LSTM and fully connected structures are introduced as regression layers to regress image features to EEG features, thus realizing the mapping between image features and EEG features. Among them, the number of output neurons in the regression layer is the same as the dimension of the EEG feature data obtained by the EEG feature extraction model.

6. The method according to claim 5, characterized in that: The EEG mapping model based on the deep residual-recurrent regression network contains 49 convolutional layers, 1 recurrent neural network layer and 1 fully connected layer.

7. The method according to claim 1, characterized in that: The generative adversarial mechanism was introduced into the EEG mapping generation process in the following way, and a conditional deep generative network model was constructed: The EEG mapping model based on the deep residual-recurrent regression network structure is used as the generator in the generative adversarial mechanism to generate simulated EEG feature data under relevant visual stimulation images; Construct a multi-layer deep neural network structure as a discriminator to distinguish between the generated simulated EEG feature data and the extracted EEG feature data; The conditional deep generation network model is jointly composed of the generator and the discriminator.

8. The method according to claim 7, characterized in that: The loss function is used as a qualitative indicator of the training progress of the conditional deep generative network model; among them, the loss function of the generator is an adversarial loss function that introduces the Wasserstein distance, and the loss function of the discriminator is the mean square error loss function.

9. The method according to claim 7, characterized in that: The trained generator is used as the EEG encoding model.

Images