A method for image reconstruction using EEG signals and visual features

Abstract

The present invention discloses a method for image reconstruction using EEG signals and visual features, comprising the following steps: Step 1: constructing an EEG signal dataset; Step 2: performing brain lateralization on the original EEG signals in the EEG signal dataset; Step 3: constructing an EEG signal encoding module based on an attention mechanism to extract key information related to visual features from the EEG signals that have undergone brain lateralization processing, thereby providing a more refined and accurate feature expression for visual processing in the brain; Step 4: constructing an intra-brain visual image reconstruction module for generating visual stimulus images of corresponding categories based on the extracted key visual features; Step 5: experimental setup and training, simultaneously training the generator and discriminator in a minimax game environment to solve the problem of overfitting during training. The present invention utilizes the dual guidance of EEG and visual features to train a deep convolutional generative adversarial network, which can effectively reconstruct visual images in the brain.

Description

Technical Field

[0001] The present invention belongs to the technical field of image reconstruction, and in particular relates to a method for image reconstruction using electroencephalogram (EEG) signals and visual features. Background Art

[0002] Traditional complex visual image reconstruction methods often fail to consider both the characteristics of the brain and human vision, resulting in reconstructed images that are inconsistent with the characteristics of object details perceived by the human eye and the global information of image categories. They also fail to incorporate different scale characteristics into the distortion features, resulting in low accuracy and clarity in the reconstructed visual images.

[0003] An existing complex visual image reconstruction method based on a deep codec dual model does not consider the impact of external noise and artifacts on the analysis process of the collected EEG signals during model training. The designed algorithm is too cumbersome, and the reconstructed visual image quality is poor and severely distorted compared with the original image.

[0004] Patent application publication number CN108573512A, entitled "A Complex Visual Image Reconstruction Method Based on a Deep Encoder-Decoder Dual Model," discloses a complex visual image reconstruction method based on a deep encoder-decoder dual model. This method belongs to the field of visual scene reconstruction technology in biomedical image brain decoding. First, functional magnetic resonance (fMRI) signals are collected while viewing a large number of natural images. Four network models are then established: 1. An encoding model, which uses a convolutional neural network to encode natural images into voxel signals in the visual area; 2. A decoding model, which uses a convolutional neural network and a deconvolutional neural network to decode the voxel signals into natural images; 3. A natural image discrimination model, which distinguishes between real and fake images; and 4. A visual area response model, which distinguishes between real and fake signals. By training these four designed models, visual scene images can be reconstructed from brain signals. This method solves the problem of direct conversion between natural scenes and brain signals for the first time, enabling practical applications in brain-computer interface scenarios.

[0005] The shortcomings of this method are that it uses functional magnetic resonance imaging equipment for signal acquisition, which is difficult to operate and has high experimental costs. It ignores the interference of environmental noise on image reconstruction, the decoded features cannot be clearly mapped to the corresponding type of subject, and does not consider the differences in the perception levels of the left and right hemispheres of the human brain to different stimulation patterns, resulting in low quality of the reconstructed image. Summary of the Invention

[0006] In order to overcome the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a method for image reconstruction using EEG signals and visual features. By using the dual guidance of EEG features and visual features to train a deep convolutional generative adversarial network, the visual images in the brain can be effectively reconstructed.

[0007] The method first introduces a soft attention mechanism and brain lateralization to assess the functional differences between the left and right brain. Then, it incorporates the attention gate into the traditional long-short-term memory network to improve the correlation of the data, thereby completing the classification task more effectively. Finally, using these EEG and image features, it constructs a model based on end-to-end EEG signal and brain visual image reconstruction to improve the performance of the artificial intelligence system.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] A method for image reconstruction using EEG signals and visual features comprises the following steps:

[0010] Step 1: Construct an EEG signal dataset; the EEG signal dataset includes original EEG signals;

[0011] Step 2: Perform brain lateralization on the original EEG signals in the EEG signal dataset to reflect the different divisions of labor between the left and right hemispheres of the brain;

[0012] Step 3: Construct an EEG signal encoding module based on the attention mechanism to extract key visual features related to visual features from the EEG signal S after brain lateralization processing, providing more refined and accurate feature expression for visual processing in the brain;

[0013] Step 4: Construct a brain visual image reconstruction module to generate visual stimulus images of corresponding categories based on the extracted key visual features;

[0014] Step 5: Experimental setup and training. The generator G(z|y) and the discriminator D(x|y,h) are trained simultaneously in a minimax game environment to address the overfitting problem during training. The deep convolutional generative adversarial network is trained in two stages to improve the visualization effect.

[0015] The step 1 is specifically as follows:

[0016] The EEG data collected in the experiment are event-related electrical signals triggered by object images as the stimulus source. Images with a more eye-catching subject and less background interference are selected as visual stimuli, highlighting different salient targets to induce different visual stimuli in the subjects.

[0017] The visual stimuli selected were from different image categories in the image dataset. All images used had their black edges removed and were converted into 500×500 resolution images. In this experiment, the subjects needed to view different image categories.

[0018] Among them, the sampling frequency of the EEG device is 1000 Hz. Sampling sequences with less than 480 sampling points and greater than 500 sampling points are excluded, and sampling points with an amplitude of 0 are added to sequences with a length of 480-500 to unify them into a sampling sequence length of 500; the first 20 and last 40 sampling points of the EEG signal corresponding to each image are discarded, and the 440 sampling points recorded in 20-460ms are used as valid EEG data for experiments. The processed EEG data set is divided into training set, validation set and test set in a 4:1:1 ratio, and the EEG signals of the same image corresponding to all subjects are placed in the same set.

[0019] The step 2 is specifically as follows:

[0020] The original EEG signal input in the EEG signal dataset is recorded as Where i represents the number of channels, l ch =128 means that there are 128 channels in total for EEG signals. i represents the EEG signal of the i-th channel;

[0021] The EEG signals are decomposed into three categories, and the left hemisphere electrode group E is divided according to the electrode area. [l] , intermediate electrode group E [m] and right hemisphere electrode group E [r] , where the left and right hemisphere electrodes are symmetrical, and each channel in the left hemisphere electrode group corresponds to each channel in the right hemisphere electrode group, obtaining the left and right hemisphere difference d j :

[0022]

[0023] in Indicates the corresponding left and right hemisphere channels, j∈[1,l g ] indicates the channel number, l g Indicates the number of channels in the left hemisphere; finally, the difference matrix and the intermediate electrode group E [m] Multiply to get a variable S;

[0024] S=[D T E [m]T ] (2)

[0025] And take S as input.

[0026] The step 3 is specifically as follows:

[0027] Step 3.1: Long-Short-Term Network Unit with added attention module;

[0028] Forget Gate f t It consists of a Sigmoid function and a point multiplication operation, and its input includes the output c of the previous momentt-1 and the current input x t , the output is a value between 0 and 1, which is used to control which information in the cell state needs to be forgotten;

[0029] Input gate i t It consists of a Sigmoid function and a point multiplication operation, and its input includes the output h of the previous moment t-1 and the current input x t , and a Tanh function for calculating the current candidate cell state, the output of which is a value between 0 and 1, which is used to control the input of new information and the retention of old information;

[0030] Output gate o t The gating unit is used to control which information in the cell state needs to be output. It adjusts and filters the information processed by the cell state and only outputs the information that is useful for the current task.

[0031] When the signal s of all channels at time t is input, the input-output relationship is calculated by equations (3) to (6):

[0032] a t =ReLU(W a s t +H a h t-1 +b a ) (3)

[0033] f t =δ(W f a t +H f h t-1 +b f ) (4)

[0034] u t =δ(W u a t +H u h t-1 +b u ) (5)

[0035] o t =δ(W o a t +H o h t-1 +b o ) (6)

[0036] where for k∈{a,f,u,o}, W k is the weight matrix that maps the layer to the four gates, U k is the weight matrix connecting the previous output unit state to the four gates, bk is the bias vector; using a soft attention gate, the EEG signals input from different channels are fully connected to the nodes in the attention gate, and the size of the weight matrix W depends on the number of channels and nodes in the attention gate;

[0037] Based on the result of formula (6), the output of the long-term and short-term network cell unit c t It can be obtained from formula (7):

[0038] c t =f t *c t-1 +u t *(tanh(H c h t-1 +W c a t +b c )) (7)

[0039] Where W c is the weight matrix that maps the layer input to the candidate replacement memory cells, f t is the output of the forget gate, c t-1 is the output of the long-term and short-term network cell unit at time t-1, u t is the weight matrix connecting the previous output unit state to the four gates, H c is the weight matrix connecting the output state of the previous unit to the candidate replacement memory unit, a t is the cell state at time t, b c is the deviation vector, and the function tanh() represents the hyperbolic tangent;

[0040] Step 3.2: Build a bidirectional long short-term memory network;

[0041] Use the long short-term network unit with the attention module added in step 3.1 to build a bidirectional long short-term memory network;

[0042] The bidirectional LSTM network consists of two LSTM layers: a forward LSTM network that processes the input sequence in chronological order, and a backward LSTM network that processes the input sequence in reverse chronological order. At each time step, the input data enters both the forward and backward LSTM networks simultaneously, and their hidden states are calculated and synthesized into a vector as the output for the current time step. This ensures that the output layer captures the complete past and future contextual information for each sample point in the output sequence.

[0043] The final output of a bidirectional long-term short-term network layer is a vector composed of all outputs Indicated by, where y t It can be calculated by formula (8):

[0044]

[0045] Where W y is the weight matrix mapping the hidden layer to the output layer, b y is the bias vector of the output layer, σ y It is the Sigmoid activation function used in the output layer, and only the last element of the output vector is considered when making predictions

[0046] The step 4 is specifically as follows:

[0047] Step 4.1: Build a deep convolutional generative adversarial neural network (DCGAN);

[0048] The deep convolutional generative adversarial neural network includes a generator and a discriminator. The network combines the EEG signal S and the image code to construct an effective reverse transfer image.

[0049] Step 4.2: The generator receives as input a 1×1×256 combined vector, which includes the 1×1×128 EEG vector y extracted by the visual feature encoder and the 1×1×128 random noise z;

[0050] Step 4.3: Reshape the 1×1×256 input vector into a four-dimensional vector as the input to the next five layers, where each layer includes three operations: deconvolution, batch normalization, and ReLU operations;

[0051] Step 4.4: Each layer doubles the spatial dimension of the input vector and halves its number of channels. The last channel outputs a compressed 128×128×3 RGB color image between -1 and 1 through the Tanh function.

[0052] Step 4.5: The discriminator network takes a concatenated input vector consisting of the 128×128×3 image and its associated 128-dimensional encoder output h;

[0053] Step 4.6: Reshape the input vector into a 4D vector and then feed it into a sequence of five layers. Each layer contains three operations, including convolution, batch normalization, and ReLU operations, which halve the spatial dimension of the input vector and double its number of channels.

[0054] Step 4.7 concatenates the output of the last convolutional layer with the associated 128-dimensional EEG signal map y, which is flattened at the last layer and then input into a single Sigmoid output to generate a visual stimulus image of the corresponding category.

[0055] The step 5 is specifically as follows:

[0056] Step 5.1: The experiment is divided into two stages to train the deep convolutional generative adversarial network;

[0057] In the first stage, the proposed deep framework is trained only on image sets of the same type, ensuring that each image set contains at least 1,000 representative samples. In the first 100 epochs of training, no EEG data is used. During this stage, the combination of the EEG feature vector y and the image important feature vector h is set as a one-hot vector according to the category and expanded to the same dimension as the image feature vector. After 200,000 iterations, the pre-trained model is saved.

[0058] In the second stage, the pre-trained model was fine-tuned using 2,000 images from the visual stimulation experiment and their corresponding EEG signal feature vectors;

[0059] 2,000 images and their corresponding EEG signal feature vectors were input into a pre-trained model. The generator and discriminator networks in the pre-trained model were used and combined to build a complete deep convolutional generative adversarial network model. The final output was a conditional vector used to generate image samples of the corresponding category.

[0060] Step 5.2: Set the discriminator learning rate and generator learning rate in the deep convolutional generative adversarial network model to 0.0004 and 0.0001 respectively;

[0061] Step 5.3: Train the generator G(z|y) and the discriminator D(x|y,h) simultaneously in the minimax game environment. The overall objective function V(D,G) is calculated according to the formula:

[0062]

[0063] where p data (x) refers to the distribution of target images, p z (x) refers to the noise sample distribution, E refers to the expectation operator, D(x|y,h) refers to the discrimination result of the target image under the constraint of the combination vector of the EEG signal feature vector y and the important image feature vector h, G(z|y) refers to the process of generating an image using the noise vector z and the combination vector of the EEG signal feature vector y and the important image feature vector h, and D(G(z|y)|y,h) refers to the discrimination result of the image generated by the noise vector z and the EEG signal feature vector y under the constraint of the combination vector of the EEG signal feature vector y and the important image feature vector h;

[0064] The loss function L of the discriminator D And the generator loss function L GThe hinge loss function is used, in which the generator loss function is modified by adding shrinkage loss to the adversarial loss, and the discriminator loss function L D And the generator loss function L G The specific information is shown in formula (10) and formula (11):

[0065] L D =-E[min(0,-1+(D(x|y,h))]-E[min(0,-1-(D(G(z)|y,h))] (10)

[0066]

[0067] Where α and β are two weighting coefficients that balance the contribution of combat loss and shrinkage loss.

[0068] Beneficial effects of the present invention:

[0069] This paper proposes a new image reconstruction method, which uses the dual guidance of EEG features and visual features to generate adversarial network training, which can effectively reconstruct visual images in the brain.

[0070] The present invention proposes an EEG feature visual extraction network based on brain lateralization and attention mechanism. This network can extract key information related to visual features from EEG signals on the basis of considering the differences between the left and right hemispheres of the brain and the visual attention mechanism, providing a more refined and accurate feature expression for visual processing in the brain.

[0071] The present invention uses image features as one of the conditions for generating adversarial networks and uses them together with brain signal features as dual conditions for image reconstruction, effectively improving the quality and reliability of generated images and providing a new tool and method for studying the visual processing mechanism in the brain.

[0072] This paper uses EEG and visual features to guide deep convolutional generative adversarial network training, introduces a soft attention mechanism and brain lateralization to assess the functional differences between left and right brain, and incorporates attention gates into traditional long-short-term memory networks to improve the correlation of data, thereby effectively reconstructing visual images in the brain. It is used to address the existing EEG-based classification algorithms for reconstructing visual stimuli in the brain, where noise and artifacts mixed in the EEG signals cause sample oscillation and model instability, resulting in unclear category recognition, poor quality of reconstructed visual images, and low visual classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 Schematic diagram of the long-short term network unit with an added attention module of the present invention.

[0074] Figure 2 Schematic diagram of the bidirectional long short-term memory network of the present invention.

[0075] Figure 3 This is a schematic diagram of the DCGAN network structure diagram of the present invention.

[0076] Figure 4 To generate a schematic diagram for image quality comparison. DETAILED DESCRIPTION

[0077] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0078] Step 1: Build an EEG signal dataset:

[0079] The EEG data collected in the experiment consisted of event-related electrical signals (ERS) triggered by images of objects. Visual stimuli were selected, highlighting distinct salient objects with minimal background interference, to elicit different visual responses in the subjects. The final visual stimuli were drawn from a dataset of 40 image categories, each containing 50 images, for a total of 2,000 images. All 2,000 images were stripped of black edges and converted to a 500×500 resolution. In this experiment, six subjects viewed 40 categories of images, each containing 50 images, for a total of approximately 12,000 images. The EEG equipment sampling frequency was 1,000 Hz. To standardize the data format and eliminate interference from bad samples, sequences with fewer than 480 or greater than 500 sampling points were excluded. Sequences between 480 and 500 were then padded with zero-amplitude sampling points to bring the sequence length to 500. To eliminate the influence of the previous image sequence on the memory of the current image sequence, the first 20 and last 40 sampling points of the EEG signal corresponding to each image were discarded, and the 440 sampling points recorded between 20 and 460 ms were used as the effective EEG data for the experiment. In order to effectively avoid overfitting when using the processed EEG dataset to train the algorithm, and to verify the generalization ability of the algorithm and conduct the final performance evaluation, the processed EEG dataset was divided into a training set, a validation set, and a test set in a 4:1:1 ratio, and the EEG signals corresponding to the same image of all subjects were placed in the same set.

[0080] Step 2: Perform brain lateralization on the raw EEG signals:

[0081] The original EEG signal input is recorded as Where i represents the number of channels, l ch =128 means that the EEG signal has a total of 128 channels. In order to achieve the effect of brain lateralization, the EEG signal is further decomposed into three categories, and the left hemisphere electrode group E is divided according to the area of the electrode. [l] , intermediate electrode group E [m]and right hemisphere electrode group E [r] , because the left and right hemisphere electrodes are symmetrical, each channel in the left hemisphere electrode group corresponds to each channel in the right hemisphere electrode group, so the left and right hemisphere difference d can be obtained by (1) j :

[0082]

[0083] in Indicates the corresponding left and right hemisphere channels, j∈[1,l g ] indicates the channel number, l g Indicates the number of channels in the left hemisphere. Finally, the difference matrix and the intermediate electrode group E [m] Multiplying together gives us a variable S.

[0084] S=[D T E [m]T ] (2)

[0085] And take S as input.

[0086] Step 3: Construct an EEG signal encoding module based on the attention mechanism:

[0087] This module is designed to extract key information related to visual features from the EEG signal S that has been processed by brain lateralization, providing a more refined and accurate feature expression for visual processing in the brain.

[0088] 3.1 Long-term short-term network unit with added attention module; such as Figure 1 As shown;

[0089] A soft attention gate is used in the input part. The EEG signals input from different channels are fully connected to the nodes in the attention gate. The size of the weight matrix W depends on the number of channels and nodes in the attention gate.

[0090] Forget Gate f t It consists of a Sigmoid function and a point multiplication operation, and its input includes the output c of the previous moment t-1 and the current input x t , the output is a value between 0 and 1, which is used to control which information in the cell state needs to be forgotten.

[0091] Input gate i t It consists of a Sigmoid function and a point multiplication operation, and its input includes the output h of the previous moment t-1 and the current input x t, and a Tanh function for calculating the current candidate cell state, the output is a value between 0 and 1, which is used to control the input of new information and the retention of old information.

[0092] Output gate o t The gating unit is used to control which information in the cell state needs to be output. Its main function is to adjust and filter the information processed by the cell state and only output information that is useful for the current task.

[0093] When the signal s of all channels at time t is input, the input-output relationship can be calculated using equations (3) to (6):

[0094] a t =ReLU(W a s t +H a h t-1 +b a ) (3)

[0095] f t =δ(W f a t +H f h t-1 +b f ) (4)

[0096] u t =δ(W u a t +H u h t-1 +b u ) (5)

[0097] o t =δ(W o a t +H o h t-1 +b o ) (6)

[0098] where for k∈{a,f,u,o}, W k Is whether to map the layer to the weight matrix of the four gates, U k is the weight matrix connecting the previous output unit state to the four gates, b k is the bias vector. In order to optimize the process of adding the attention mechanism, a soft attention gate is used. The EEG signals input from different channels are fully connected to the nodes in the attention gate. Therefore, the size of the weight matrix W depends on the number of channels and nodes in the attention gate.

[0099] Based on the results of the above formula, the output of the long-term and short-term network cell unit c t It can be obtained from formula (7):

[0100] c t =f t *c t-1 +u t *(tanh(H c h t-1 +W c a t +b c )) (7)

[0101] Where W c is a weight matrix that maps layer inputs to candidates for replacement memory cells. c is the weight matrix connecting the output state of the previous unit to the candidate replacement memory unit, b c is the bias vector. The function tanh() represents the hyperbolic tangent.

[0102] 3.2 Construct a bidirectional long short-term memory network, such as Figure 2 As shown;

[0103] Use the long short-term network unit with the attention module added in step 3.1 to build a bidirectional long short-term memory network.

[0104] The bidirectional LSTM network consists of two LSTM layers: a forward LSTM network that processes the input sequence in chronological order, and a backward LSTM network that processes the input sequence in reverse chronological order. At each time step, the input data enters both the forward and backward LSTM networks simultaneously, where their hidden states are calculated and synthesized into a vector that serves as the output for the current time step. This ensures that the output layer captures the complete past and future context for each sample point in the output sequence.

[0105] The final output of the proposed bidirectional long-term short-term network layer is a vector composed of all outputs Indicated by, where y t It can be calculated by formula (8):

[0106]

[0107] Where W y is the weight matrix mapping the hidden layer to the output layer, b y is the bias vector of the output layer, σ y Is the Sigmoid activation function used in the output layer. Only the last element of the output vector is considered when making predictions

[0108] Step 4: Construct the brain visual image reconstruction module:

[0109] 4.1 Build a deep convolutional generative adversarial neural network (DCGAN), such as Figure 3 As shown;

[0110] Specifically, the framework consists of two main components: a generator and a discriminator. By combining EEG signals with image encoding, the deep convolutional GAN framework significantly differs from traditional GAN frameworks in that it eliminates normalization layers in the generator output layer and the discriminator input layer. This mitigates sample oscillation and model instability, significantly improving the stability of GAN training and the quality of generated images.

[0111] 4.2 The generator network receives as input a combined vector of size 1×1×256, which includes the EEG vector y of size 1×1×128 extracted by the visual feature encoder and the random noise z of size 1×1×128.

[0112] 4.3 Reshape the 1×1×256 input vector into a four-dimensional vector as the input of the next five layers, where each layer includes three operations: deconvolution, batch normalization, and ReLU operations.

[0113] 4.4 Each layer doubles the spatial dimension of the input vector and halves the number of channels. The last channel outputs a compressed 128×128×3 RGB color image between -1 and 1 through the Tanh function.

[0114] 4.5 The discriminator network takes a concatenated input vector consisting of a 128×128×3 image and its associated 128-dimensional encoder output h.

[0115] 4.6 reshapes the input vector into a four-dimensional vector and then feeds it into a sequence of five layers, each of which contains three operations, including convolution, batch normalization, and ReLU operations, which halve the spatial dimension of the input vector and double its number of channels.

[0116] 4.7 Concatenate the output of the last convolutional layer with the associated 128-dimensional EEG signal map y. The last layer is flattened and then input into a single sigmoid output.

[0117] Step 5: Experimental Setup and Training

[0118] 5.1 The experiment trains a deep convolutional generative adversarial network in two stages. In the first stage, the proposed deep framework is trained only on image sets from the same category, ensuring that each image set contains at least 1000 representative samples. During the first 100 epochs of training, no EEG data is used. During this stage, the combination of the EEG feature vector y and the image's important feature vector h is set as a one-hot vector based on the category and expanded to the same dimension as the image feature vector. After 200,000 iterations, the pre-trained model is saved. In the second stage, the pre-trained model is fine-tuned using 2000 images and their corresponding EEG feature vectors from the visual stimulation experiment. Specifically, the 2000 images and their corresponding EEG feature vectors are input to the model. The generator and discriminator networks from the pre-trained model are then combined to construct the full deep convolutional generative adversarial network model. The final output is a conditional vector, which is used to generate image samples of the corresponding category.

[0119] 5.2 To address the difficult balance between the generator and the discriminator, the experiment employed a two-timescale update rule (TTUR). This technique uses two different learning rates to update the neural network parameters to balance the learning rates between the generator and discriminator updates, guiding the generator to produce more realistic samples. Specifically, the discriminator learning rate was set to 0.0004 and the generator learning rate to 0.0001. This allows fewer generator steps to be used in each discriminator step, thereby better balancing the learning rates between the two and improving the generator's performance.

[0120] 5.3 To improve the visualization effect, the experiment chooses to train the generator G(z|y) and the discriminator D(x|y,h) simultaneously in the minimax game environment. The overall objective function V(D,G) can be calculated according to the formula:

[0121]

[0122] where p data (x) refers to the distribution of target images, p z (x) refers to the noise sample distribution, E refers to the expectation operator, D(x|y,h) refers to the discrimination result of the target image under the constraint of the combination vector of the EEG signal feature vector y and the important image feature vector h, G(z|y) refers to the process of generating an image using the noise vector z and the combination vector of the EEG signal feature vector y and the important image feature vector h, and D(G(z|y)|y,h) refers to the discrimination result of the image generated by the noise vector z and the EEG signal feature vector y under the constraint of the combination vector of the EEG signal feature vector y and the important image feature vector h.

[0123] The loss function L of the discriminator D And the generator loss function L G The hinge loss function is used, in which the generator loss function is modified by adding shrinkage loss to the adversarial loss, and the discriminator loss function L D And the generator loss function L G The specific information is shown in formula (10) and formula (11):

[0124] L D =-E[min(0,-1+(D(x|y,h))]-E[min(0,-1-(D(G(z)|y,h))] (10)

[0125]

[0126] Where α and β are two weighting coefficients that balance the contribution of combat loss and shrinkage loss.

[0127] 5.4 The technical effects of the present invention are described in conjunction with simulation experiments:

[0128] In order to evaluate the quality of visualization of the proposed dual-condition deep convolutional generative adversarial network and compare the visualization output images with the images generated by existing work under the same experimental setting, the generation methods involved are:

[0129] 1) VAE: a visual image reconstruction method based on variational autoencoders;

[0130] 2) cProGAN: an improved generative adversarial network visual image reconstruction method based on conditional progressive growth;

[0131] 3) VG-GAN: A visual image reconstruction method based on EEG signal feature representation based on visual cues and conditional generative adversarial networks;

[0132] 4) VG-GAN-VC: A visual image reconstruction method that adds a visual consistency maintenance term based on VG-GAN.

[0133] To quantify the contribution of our proposed deep framework, we calculated the IS scores for 50,000 generated images from the proposed methods, i.e., 1,250 images per category. The following table summarizes the IS scores of our proposed deep framework compared to existing state-of-the-art techniques.

[0134] Table 5.1 Comparison of IS scores of various generative models

[0135]

[0136]

[0137] As can be seen from the table, the IS value of the visual image reconstructed by the deep framework proposed in this invention reaches 6.77, while the IS value of the visual image reconstructed by VAE based on EEG signals is 4.49, the IS score of the visual image reconstructed by cProGAN is 5.07, the IS score of the visual image reconstructed by VG-GAN is 5.45, and the IS score of the visual image reconstructed by VG-GAN-VC is 6.26.

[0138] Compared with other existing studies on reconstructing visual stimuli within the brain, the dual-condition deep convolutional generative adversarial network method of this invention performs well in image generation, with a certain degree of improvement in clarity. This demonstrates that the proposed deep bounding box can convert EEG signal features into meaningful, category-relevant images, and the image reconstruction quality is good.

[0139] Figure 4 The paper presents a comparison of the image quality of three categories: pumpkin lanterns, giant pandas, and airplanes generated by VAE, VG-GAN, and the proposed DCGAN. From a subjective perspective, the proposed method produces visual stimulus images with higher clarity and more recognizable category entities. This has important application value for improving the expressiveness of human-computer interaction and analyzing human visual characteristics.

Claims

1. A method for image reconstruction using EEG signals and visual features, characterized in that: The following steps are included: Step 1: Build an EEG signal dataset; The EEG signal dataset includes original EEG signals; Step 2: Perform brain lateralization on the original EEG signals in the EEG signal dataset to reflect the different divisions of labor between the left and right hemispheres of the brain; Step 3: Construct an EEG signal encoding module based on the attention mechanism to extract key visual features related to visual features from the EEG signal S after brain lateralization processing; Use the long short-term network unit with an added attention module to build a bidirectional long short-term memory network; Step 4: Construct a brain visual image reconstruction module to generate visual stimulus images of corresponding categories based on the extraction of key visual features; build a deep convolutional generative adversarial neural network; Step 5: Experimental setup and training: The generator G(z|y) and the discriminator D(x|y,h) are trained simultaneously in a minimax game environment to address the overfitting problem during training. The deep convolutional generative adversarial network is trained in two stages to improve visualization. The step 2 is specifically as follows: The original EEG signal input in the EEG signal dataset is recorded as Where i represents the number of channels, l ch =128 means that there are 128 channels in total for EEG signals. i represents the EEG signal of the i-th channel; The EEG signals are decomposed into three categories, and the left hemisphere electrode group E is divided according to the electrode area. [l] , intermediate electrode group E [m] and right hemisphere electrode group E [r] , where the left and right hemisphere electrodes are symmetrical, and each channel in the left hemisphere electrode group corresponds to each channel in the right hemisphere electrode group, obtaining the left and right hemisphere difference d j : in Indicates the corresponding left and right hemisphere channels, j∈[1,l g ] indicates the channel number, l g Indicates the number of channels in the left hemisphere; finally, the difference matrix and the intermediate electrode group E [m] Multiply to get a variable S; S=[D T E [m]T ] (2) And take S as input.

2. The method for image reconstruction using EEG signals and visual features according to claim 1, characterized in that: The step 1 is specifically as follows: The visual stimuli were derived from different image categories in the image dataset. All images were converted to 500×500 resolution images with black edges removed. In this experiment, subjects were required to view images of different categories. Among them, the sampling frequency of the EEG device is 1000 Hz. Sampling sequences with less than 480 sampling points and greater than 500 sampling points are excluded, and sampling points with an amplitude of 0 are added to sequences with a length of 480-500 to unify them into a sampling sequence length of 500; the first 20 and last 40 sampling points of the EEG signal corresponding to each image are discarded, and the 440 sampling points recorded in 20-460ms are used as valid EEG data for experiments. The processed EEG data set is divided into training set, validation set and test set in a 4:1:1 ratio, and the EEG signals of the same image corresponding to all subjects are placed in the same set.

3. The method for image reconstruction using EEG signals and visual features according to claim 1, characterized in that: The step 3 is specifically as follows: Step 3.1: Long-Short-Term Network Unit with added attention module; Forget Gate f t It consists of a Sigmoid function and a point multiplication operation, and its input includes the output c of the previous moment t-1 and the current input x t , the output is a value between 0 and 1, which is used to control which information in the cell state needs to be forgotten; Input gate i t It consists of a Sigmoid function and a point multiplication operation, and its input includes the output h of the previous moment t-1 and the current input x t , and a Tanh function for calculating the current candidate cell state, the output of which is a value between 0 and 1, which is used to control the input of new information and the retention of old information; Output gate o t The gating unit is used to control which information in the cell state needs to be output. It adjusts and filters the information processed by the cell state and only outputs the information that is useful for the current task. When the signal s of all channels at time t is input, the input-output relationship is calculated by equations (3) to (6): a t =ReLU(W a s t +H a h t-1 +b a ) (3) f t =δ(W f a t +H f h t-1 +b f ) (4) u t =δ(W u a t +H u h t-1 +b u ) (5) o t =δ(W o a t +H o h t-1 +b o ) (6) where for k∈{a,f,u,o}, W k is the weight matrix that maps the layer to the four gates, U k is the weight matrix connecting the previous output unit state to the four gates, b k is the bias vector; using a soft attention gate, the EEG signals input from different channels are fully connected to the nodes in the attention gate, and the size of the weight matrix W depends on the number of channels and nodes in the attention gate.

4. The method for image reconstruction using EEG signals and visual features according to claim 3, characterized in that: Based on the result of formula (6), the output of the long-term and short-term network cell unit c t It can be obtained from formula (7): c t =f t *c t-1 +u t *(tanh(H c h t-1 +W c a t +b c )) (7) Where W c is the weight matrix that maps the layer input to the candidate replacement memory cells, f t is the output of the forget gate, c t-1 is the output of the long-term and short-term network cell unit at time t-1, u t is the weight matrix connecting the previous output unit state to the four gates, H c is the weight matrix connecting the output state of the previous unit to the candidate replacement memory unit, a t is the cell state at time t, b c is the deviation vector, and the function tanh() represents the hyperbolic tangent; Step 3.2: Build a bidirectional long short-term memory network; Use the long short-term network unit with the attention module added in step 3.1 to build a bidirectional long short-term memory network; The bidirectional LSTM network consists of two LSTM layers: a "forward LSTM network" that processes the input sequence in chronological order, and a "backward LSTM network" that processes the input sequence in reverse chronological order. At each time step, the input data enters both the forward and backward LSTM networks simultaneously, and their hidden states are calculated and synthesized into a vector as the output of the current time step, ensuring that the output layer obtains the complete past and future contextual information of each sample point in the output sequence.

5. The method for image reconstruction using EEG signals and visual features according to claim 3, characterized in that: The final output of a bidirectional long-term short-term network layer is a vector composed of all outputs Indicated by, where y t Calculated by formula (8): Where W y is the weight matrix mapping the hidden layer to the output layer, b y is the bias vector of the output layer, σ y It is the Sigmoid activation function used in the output layer, and only the last element of the output vector is considered when making predictions 6. The method for image reconstruction using EEG signals and visual features according to claim 1, characterized in that: The step 4 is specifically as follows: Step 4.1: Build a deep convolutional generative adversarial neural network (DCGAN); The deep convolutional generative adversarial neural network includes a generator and a discriminator. The network combines the EEG signal S and the image code to construct an effective reverse transfer image. Step 4.2: The generator receives as input a 1×1×256 combined vector, which includes the 1×1×128 EEG vector y extracted by the visual feature encoder and the 1×1×128 random noise z; Step 4.3: Reshape the 1×1×256 input vector into a four-dimensional vector as the input to the next five layers, where each layer includes three operations: deconvolution, batch normalization, and ReLU operations; Step 4.4: Each layer doubles the spatial dimension of the input vector and halves its number of channels. The last channel outputs a compressed 128×128×3 RGB color image between -1 and 1 through the Tanh function. Step 4.5: The discriminator network takes a concatenated input vector consisting of the 128×128×3 image and its associated 128-dimensional encoder output h; Step 4.6: Reshape the input vector into a 4D vector and then feed it into a sequence of five layers. Each layer contains three operations, including convolution, batch normalization, and ReLU operations, which halve the spatial dimension of the input vector and double its number of channels. Step 4.7 concatenates the output of the last convolutional layer with the associated 128-dimensional EEG signal map y, which is flattened at the last layer and then input into a single Sigmoid output to generate a visual stimulus image of the corresponding category.

7. The method for image reconstruction using EEG signals and visual features according to claim 1, characterized in that: The step 5 is specifically as follows: Step 5.1: The experiment is divided into two stages to train the deep convolutional generative adversarial network; In the first stage, the proposed deep framework is trained only on image sets of the same type, ensuring that each image set contains at least 1,000 representative samples. In the first 100 epochs of training, no EEG data is used. During this stage, the combination of the EEG feature vector y and the image important feature vector h is set as a one-hot vector according to the category and expanded to the same dimension as the image feature vector. After 200,000 iterations, the pre-trained model is saved. In the second stage, the pre-trained model was fine-tuned using 2,000 images from the visual stimulation experiment and their corresponding EEG signal feature vectors; Step 5.2: Set the discriminator learning rate and generator learning rate in the deep convolutional generative adversarial network model to 0.0004 and 0.0001 respectively; Step 5.3: Train the generator G(z|y) and the discriminator D(x|y,h) simultaneously in the minimax game environment. The overall objective function V(D,G) is calculated according to the formula: where p data (x) refers to the distribution of target images, p z (x) refers to the noise sample distribution, E refers to the expectation operator, D(x|y,h) refers to the discrimination result of the target image under the constraint of the combination vector of the EEG signal feature vector y and the important image feature vector h, G(z|y) refers to the process of generating an image using the noise vector z and the combination vector of the EEG signal feature vector y and the important image feature vector h, and D(G(z|y)|y,h) refers to the discrimination result of the image generated by the noise vector z and the EEG signal feature vector y under the constraint of the combination vector of the EEG signal feature vector y and the important image feature vector h.

8. The method for image reconstruction using EEG signals and visual features according to claim 7, characterized in that: In the second stage of step 5.1, the 2000 images and their corresponding EEG signal feature vectors are input into the pre-trained model, and the generator and discriminator networks in the pre-trained model are used and combined to construct a complete deep convolutional generative adversarial network model; finally, the conditional vector is output to generate image samples of the corresponding category.

9. The method for image reconstruction using EEG signals and visual features according to claim 7, characterized in that: In step 5.3, the loss function L of the discriminator D And the generator loss function L G The hinge loss function is used, in which the generator loss function is modified by adding shrinkage loss to the adversarial loss, and the discriminator loss function L D And the generator loss function L G The specific information is shown in formula (10) and formula (11): L D =-E[min(0,-1+(D(x|y,h))]-E[min(0,-1-(D(G(z)|y,h))] (10) L G =-αE[D(z|y,h)]+βl1[p G (z|y),p data (x)] (11) Where α and β are two weighting coefficients that balance the contribution of combat loss and shrinkage loss.

Images