An electroencephalogram image generation method based on autoregressive pre-training and feature alignment

Abstract

The present application relates to the technical field of electroencephalogram signal processing, in particular to a method for generating electroencephalogram images based on autoregressive pre-training and feature alignment, comprising: 1. converting the pre-processed electroencephalogram signal into electroencephalogram coding, calculating the position coding of the electroencephalogram coding, and transmitting the electroencephalogram coding with the added position coding to the electroencephalogram autoregressive encoder to obtain the latent representation; 2. training the electroencephalogram autoregressive encoder using the autoregressive pre-training method; 3. training the CLIP model using the latent representation; 4. training the denoising U-Net network in the stable diffusion model using the existing electroencephalogram-image pair dataset; 5. testing the trained electroencephalogram autoregressive encoder, the trained CLIP model and the trained denoising U-Net network, and generating the final denoised image. The present application establishes the mapping relationship between the electroencephalogram signal and the image through feature alignment, and realizes the cross-modal conversion from the electroencephalogram to the image.

Description

Technical Field

[0001] This invention relates to the field of electroencephalogram (EEG) signal processing technology, and in particular to an EEG image generation method based on autoregressive pre-training and feature alignment. Background Technology

[0002] In recent years, the development of brain-computer interface (BCI) technology has provided a new way for humans to interact with computers. Traditional BCI technology mainly focuses on using electroencephalogram (EEG) signals for tasks such as intention recognition and motor imagery; however, research on converting EEG signals into images is relatively limited. Although some studies have attempted to use deep learning methods to convert EEG signals into images, the following problems and drawbacks still exist:

[0003] a) Most existing EEG image generation methods rely on predefined image categories or templates, and the generated images are mostly abstract or simple geometric shapes, lacking diversity and realism.

[0004] b) Existing methods typically use EEG signals directly as input, failing to fully utilize the temporal characteristics and potential representations of EEG signals, resulting in low quality and accuracy of the generated images.

[0005] c) Existing methods lack effective cross-modal feature alignment mechanisms, failing to establish a good mapping relationship between EEG signals and images, which affects the relevance and consistency of generated images.

[0006] The causes of these problems and shortcomings can be summarized as follows:

[0007] a) EEG signals have characteristics such as high dimensionality and non-stationarity, making them difficult to use directly for image generation tasks.

[0008] b) The lack of large-scale, high-quality EEG-image pairing datasets limits the training and optimization of deep learning models.

[0009] c) EEG signals and images belong to different modalities, and cross-modal feature alignment and conversion present challenges.

[0010] d) Limited understanding of the cognitive mechanisms between EEG signals and visual imagery makes it difficult to establish an effective EEG-image generation model.

[0011] Therefore, a new technical solution is urgently needed to address the above problems, improve the quality, diversity, and interpretability of EEG image generation, and promote the application of brain-computer interface technology in the fields of visual imagination and image generation. Summary of the Invention

[0012] This invention provides a method for generating electroencephalograms based on autoregressive pre-training and feature alignment to solve the technical problems mentioned in the background art.

[0013] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0014] This invention provides a method for generating electroencephalogram (EEG) images based on autoregressive pre-training and feature alignment, comprising the following steps:

[0015] S1. Collect a large number of unlabeled EEG signals, preprocess them and input them into the autoregressive model, convert the preprocessed EEG signals into EEG codes, then calculate the location codes of the EEG codes, and transmit the EEG codes with added location codes to the EEG autoregressive encoder to obtain the latent representation.

[0016] S2. Input the latent representation into the EEG prediction layer to predict the EEG signal at the next moment, obtain the EEG prediction result, calculate the loss between the EEG prediction result and the preprocessed EEG signal, and use the autoregressive pre-training method to train the EEG autoregressive encoder to obtain the trained EEG autoregressive encoder.

[0017] S3. Train the CLIP model using latent representations to obtain the trained CLIP model;

[0018] S4. Train the denoising U-Net network within the stable diffusion model using the existing EEG-image dataset to obtain the trained denoising U-Net network.

[0019] S5. Test the trained EEG autoregressive encoder, the trained CLIP model, and the trained denoised U-Net network, and generate the final denoised image.

[0020] Furthermore, step S1 specifically includes the following steps:

[0021] S11. First, a large number of unlabeled EEG signals are collected and preprocessed to obtain preprocessed EEG signals. Preprocessing includes filtering, noise reduction, and segmentation.

[0022] S12. Use a one-dimensional convolutional layer to convert the preprocessed EEG signal X into EEG code;

[0023] S13. Calculate the sine and cosine outputs using sine and cosine functions of different frequencies, and concatenate them to obtain the position code of the EEG code. Then, add the position code to the EEG code.

[0024] S14. Input the EEG code with added location encoding into the EEG autoregressive encoder to obtain the latent representation Z.

[0025] Furthermore, the sine and cosine functions of different frequencies in S13 are specifically as follows:

[0026]

[0027] Where k represents the position of an object in the input sequence; d represents the dimension of the output embedding space; P(k, 2i) and P(k, 2i+1) represent two different position functions, which are used to map the position k in the input sequence to the indices (k, 2i) and (k, 2i+1) of the position matrix, respectively; n represents a user-defined scalar; and i represents the value used to map the column index to 0 ≤ i ≤ d / 2.

[0028] Furthermore, the EEG autoregressive encoder in S14 consists of six sequentially connected Transformer decoder modules. Each Transformer decoder module includes a masked multi-head attention mechanism, a first normalization layer, a feedforward module, and a second normalization layer, all connected in sequence.

[0029] Further, step S14 includes the following steps:

[0030] S141. Input the EEG code with added location encoding into the Transformer decoder module of the first layer. The EEG code is transformed into a query matrix Q, a key matrix K and a value matrix V through three different linear transformations.

[0031] Q = XW Q K = XW K V = XW V

[0032] Among them, W Q W K W V These are three different learnable weight matrices;

[0033] S142. Next, calculate the attention weight A. The calculation process for the attention weight A is expressed by the formula as follows:

[0034]

[0035] Where, d k It represents the dimension of the key, Mask represents the mask, and T represents the transpose;

[0036] S143. Use the attention weights A and the value matrix V to perform a weighted summation to obtain the output O of the masked multi-head attention mechanism in the first layer Transformer decoder module;

[0037] O = A·V;

[0038] S144. After normalizing the output O of the mask multi-head attention mechanism, input it into the feedforward module to obtain the output F of the feedforward module;

[0039] F = ReLU(O·W1+b1)·W2+b2;

[0040] Where ReLU(.) represents the activation function; W1 and b1 represent the weight matrix and bias vector of the first layer of the feedforward network, respectively. This layer is a linear transformation layer used to map the input O to an intermediate representation; W2 and b2 represent the weight matrix and bias vector of the second layer of the feedforward network, respectively. This layer is also a linear transformation layer used to map the intermediate representation after the activation function ReLU to a specified dimension.

[0041] S145. The output of the feedforward module is normalized to obtain the output B of the first layer Transformer decoder module.

[0042] B = FFNN(MMHA(X));

[0043] Where MMHA(.) represents masked multi-head attention mechanism, and FFNN(.) represents feedforward neural network;

[0044] S146. The output B of the previous Transformer decoder module is used as the input of the next Transformer decoder module to pass the output result to the next Transformer decoder module. After multiple calculations, the output of the EEG autoregressive encoder, i.e., the latent representation Z, is obtained, expressed by the following formula:

[0045] Z=Decoder6(Decoder5(...Decoder1(X));

[0046] Here, Decoder1(.) to Decoder6(.) represent the first layer Transformer decoder module to the sixth layer Transformer decoder module, respectively.

[0047] Furthermore, step S2 specifically includes the following steps:

[0048] S21. Input the latent representation Z into the EEG prediction layer to predict the EEG signal at the next moment and obtain the EEG prediction result X′.

[0049] S22. Calculate the loss L between the EEG prediction result X′ and the preprocessed EEG signal. MsE ;

[0050]

[0051] Where C represents the number of channels, U represents the length of the time series; X′(c, u) represents the predicted EEG signal value, where c represents the channel index and u represents the time point index; Label(c, u) represents the actual EEG signal label value, which corresponds to the predicted value.

[0052] S23. Repeat S1 to S2 multiple times to minimize the loss L. MSE The weight parameters of the EEG autoregressive encoder were adjusted to obtain the trained EEG autoregressive encoder.

[0053] Furthermore, step S3 specifically includes the following steps:

[0054] S31. Input the latent representation Z into two projection layers to transform it into an EEG embedding with the same dimension as the CLIP image encoder in the CLIP model.

[0055] S32. Input the real image corresponding to the EEG signal into the CLIP image encoder to obtain the image embedding;

[0056] S33. Use cosine similarity to calculate the loss L between EEG embedding and image embedding. clip ;

[0057]

[0058] Where h(τ) θ (y) represents EEG embedding, h is the projection layer, and E I It is a CLIP image encoder; τ θ (y) represents conditional information; τ θ (·) represents another projection layer, y represents the output of the EEG autoregressive encoder, i.e., the latent representation Z; I represents the image corresponding to the EEG signal;

[0059] S34. Repeat S1 to S3 multiple times to minimize the loss L. clip In this process, the weight parameters of the CLIP image encoder and the two projection layers in the CLIP model are adjusted to obtain the trained CLIP model.

[0060] Furthermore, the implementation of S4 is specifically as follows:

[0061] Using the stable diffusion loss function L SD We used existing EEG-image datasets to adjust the parameters of the cross-attention layer of the denoised U-Net network and the EEG autoregressive encoder in the stable diffusion model, resulting in the trained denoised U-Net network and the parameter-adjusted EEG autoregressive encoder. The stable diffusion model mainly consists of two stages: the diffusion process and the de-diffusion process.

[0062] Stable diffusion loss function L SD Specifically as follows:

[0063]

[0064] Wherein, noise ∈ follows a standard normal distribution. It is added during the diffusion process to generate z. t noise, ∈ θ Let z represent the noise prediction function of the denoised U-Net network with parameters θ. t τ represents the Gaussian noise used in the reverse diffusion process. θ (y) represents conditional information; t represents the time step; The formula for calculating random variables z and ∈ is given by the condition that both random variables z and ∈ follow a standard normal distribution. The expected value, that is, the sum of the random noise ∈ and the model predicted noise ∈. θ (z t ,t,τ θ The expected value of the square of the Euclidean distance between (y) and (y).

[0065] Furthermore, the cross-attention layer of the denoising U-Net network is specifically as follows:

[0066]

[0067] in, This represents the median value of the denoised U-Net network. These are different projection matrices with learnable parameters.

[0068] Furthermore, step S5 specifically includes the following steps:

[0069] S51. First, the unlabeled EEG signals are preprocessed and then input into the trained EEG autoregressive encoder to obtain the latent representation.

[0070] S52. Input the latent representation into the trained CLIP model to obtain the aligned EEG features;

[0071] S53. The aligned EEG features are introduced into the cross-attention layer of the denoising U-Net network in the stable diffusion model to control the generation of images and finally generate denoised image data.

[0072] The beneficial effects of this invention are:

[0073] 1. This invention learns the temporal characteristics and potential representations of EEG signals through autoregressive pre-training, thereby improving the ability to understand and model EEG signals.

[0074] 2. This invention establishes a mapping relationship between EEG signals and images through feature alignment, realizing cross-modal conversion from EEG to images. It achieves feature alignment with existing image generation models, thereby achieving the goal of generating images based on EEG. Attached Figure Description

[0075] Figure 1 This is a flowchart illustrating the overall process of the present invention.

[0076] Figure 2 These are partial images generated based on electroencephalography (EEG) in an embodiment of the present invention;

[0077] Figure 3 This is a flowchart illustrating the workflow of the stable diffusion model in this invention. Detailed Implementation

[0078] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many other different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0079] Reference Figure 1 This application provides a method for generating electroencephalograms (EEGs) based on autoregressive pre-training and feature alignment, comprising the following steps:

[0080] S1. Collect a large number of unlabeled EEG signals, preprocess them and input them into the autoregressive model, convert the preprocessed EEG signals into EEG codes, then calculate the location codes of the EEG codes, and transmit the EEG codes with added location codes to the EEG autoregressive encoder to obtain the latent representation.

[0081] S2. Input the latent representation into the EEG prediction layer to predict the EEG signal at the next moment, obtain the EEG prediction result, calculate the loss between the EEG prediction result and the preprocessed EEG signal, and use the autoregressive pre-training method to train the EEG autoregressive encoder to obtain the trained EEG autoregressive encoder.

[0082] S3. Train the CLIP model using latent representations to obtain the trained CLIP model;

[0083] S4. Train the denoising U-Net network within the stable diffusion model using the existing EEG-image dataset to obtain the trained denoising U-Net network.

[0084] S5. Test the trained EEG autoregressive encoder, the trained CLIP model, and the trained denoised U-Net network, and generate the final denoised image.

[0085] In some embodiments, S1 specifically includes the following steps:

[0086] S11. First, a large number of unlabeled EEG signals are collected and preprocessed to obtain preprocessed EEG signals. Preprocessing includes operations such as filtering, noise reduction, and segmentation.

[0087] Specifically, the preprocessed EEG signal is represented as X∈R C×U In this process, EEG signals with a time length greater than T are randomly truncated, EEG signals with a time series length less than U are interpolated to increase the number of sample points to make their length U, EEG signals with a channel number greater than C are randomly cropped, and EEG signals with a channel number less than C are copied and padded to make their channel number C.

[0088] S12. Use a one-dimensional convolutional layer to convert the preprocessed EEG signal X into EEG code;

[0089] S13. Calculate the sine and cosine outputs using sine and cosine functions of different frequencies, and concatenate them to obtain the position code of the EEG code. Then, add the position code to the EEG code.

[0090] S14. Input the EEG code with added sine and cosine position coding into the EEG autoregressive encoder to obtain the latent representation Z.

[0091] In some embodiments, the sine and cosine functions of different frequencies in S13 are specifically as follows:

[0092]

[0093] Where k represents the position of an object in the input sequence, and 0≤k≤U / 2; d represents the dimension of the output embedding space; P(k, 2i) and P(k, 2i+1) represent two different position functions, which are used to map the position k in the input sequence to the indices (k, 2i) and (k, 2i+1) of the position matrix, respectively; n represents a custom scalar; in this embodiment, the value of n is preferably 1000; i represents the value used to map the column index to 0≤i≤d / 2, and a single i value is mapped to both sine and cosine functions.

[0094] In some embodiments, the EEG autoregressive encoder in S14 consists of six sequentially connected Transformer decoder modules. Each Transformer decoder module includes a masked multi-head attention mechanism, a first normalization layer, a feedforward module, and a second normalization layer, which are connected in sequence.

[0095] In some embodiments, S14 includes the following steps:

[0096] S141. Input the EEG code with added location encoding into the Transformer decoder module of the first layer. The EEG code is transformed into a query matrix Q, a key matrix K and a value matrix V through three different linear transformations.

[0097] Q = XW Q K = XW K V = XW V

[0098] Among them, W Q W K W V These are three different learnable weight matrices;

[0099] S142. Next, calculate the attention weight A. The calculation process for the attention weight A is expressed by the formula as follows:

[0100]

[0101] Where, d k is the dimension of the key, Mask represents the mask, which is usually an upper triangular matrix used to mask future sequence parts, and T represents transpose;

[0102] S143. Use the attention weights A and the value matrix V to perform a weighted summation to obtain the output O of the masked multi-head attention mechanism in the first layer Transformer decoder module;

[0103] O = A·V;

[0104] S144. After normalizing the output O of the mask multi-head attention mechanism, input it into the feedforward module to obtain the output F of the feedforward module;

[0105] F = ReLU(O·W1+b1)·W2+b2;

[0106] Where ReLU(.) represents the activation function; W1 and b1 represent the weight matrix and bias vector of the first layer of the feedforward network, respectively. This layer is a linear transformation layer used to map the input O to an intermediate representation; W2 and b2 represent the weight matrix and bias vector of the second layer of the feedforward network, respectively. This layer is also a linear transformation layer used to map the intermediate representation after the activation function ReLU to a specified dimension.

[0107] S145. The output of the feedforward module is normalized to obtain the output B of the first layer Transformer decoder module.

[0108] B = FFNN(MMHA(X));

[0109] Where MMHA(.) represents masked multi-head attention mechanism, and FFNN(.) represents feedforward neural network;

[0110] S146. The output B of the previous Transformer decoder module is used as the input of the next Transformer decoder module to pass the output result to the next Transformer decoder module. After multiple calculations, the output of the EEG autoregressive encoder, i.e., the latent representation Z, is obtained, expressed by the following formula:

[0111] Z=Decoder6(Decoder5(...Decoder1(X));

[0112] Here, Decoder1(.) to Decoder6(.) represent the first layer Transformer decoder module to the sixth layer Transformer decoder module, respectively.

[0113] In some embodiments, S2 specifically includes the following steps:

[0114] S21. Input the latent representation Z into the EEG prediction layer to predict the EEG signal at the next moment and obtain the EEG prediction result X′.

[0115] S22. Calculate the loss L between the EEG prediction result X′ and the preprocessed EEG signal. MSE ;

[0116]

[0117] Where C represents the number of channels, U represents the length of the time series; X′(c, u) represents the predicted EEG signal value, where c represents the channel index and u represents the time point index; Label(c, u) represents the actual EEG signal label value, which corresponds to the predicted value.

[0118] S23. Repeat S1 to S2 multiple times to minimize the loss L. MSE The weight parameters of the EEG autoregressive encoder were adjusted to obtain the trained EEG autoregressive encoder.

[0119] This invention transforms the preprocessed EEG signal X into an EEG prediction result X′. This process predicts the time point of the EEG signal at the next moment and improves the accuracy of the prediction by minimizing the difference between the actual EEG data Label and the EEG prediction result X′.

[0120] In some embodiments, S3 specifically includes the following steps:

[0121] S31. Input the latent representation Z into two projection layers to transform it into an EEG embedding with the same dimension as the CLIP image encoder in the CLIP model.

[0122] S32. Input the real image corresponding to the EEG signal into the CLIP image encoder to obtain the image embedding;

[0123] S33. Use cosine similarity to calculate the loss L between EEG embedding and image embedding. clip ;

[0124]

[0125] Where h(τ) θ (y) represents EEG embedding, h is the projection layer, and τ θ (·) represents another projection layer, y represents the output of the EEG autoregressive encoder, i.e., the latent representation Z; E I It is a CLIP image encoder; I represents the image corresponding to the EEG signal;

[0126] S34. Repeat S1 to S3 multiple times to minimize the loss L. clip In this process, the weight parameters of the CLIP image encoder and the two projection layers in the CLIP model are adjusted to obtain the trained CLIP model.

[0127] Loss L clip The similarity between EEG feature vectors (i.e., latent representations Z) and image latent vectors is measured by calculating the cosine similarity. By optimizing the objective function, i.e., minimizing the similarity loss, a closer alignment between the EEG feature vectors and the image latent vectors can be achieved, making the EEG features more similar to text features. In this way, EEG signals, text, and images can be aligned in a unified space.

[0128] In some embodiments, S4 is specifically implemented as follows:

[0129] Reference Figure 3 The stable diffusion model mainly includes two stages: the diffusion process and the reverse diffusion process.

[0130] The diffusion process involves continuously adding Gaussian noise (z) to the real image in the image latent space. Specifically, a Vector Quantization Encoder (VQ) is first used to encode the original image data in the existing EEG-image dataset. Then, Gaussian noise is continuously added to the images until they become purely noisy images (z). t This image is random and contains no meaningful information.

[0131] The reverse diffusion process is for Gaussian noise z t The process involves continuously denoising the image to ultimately generate a realistic image corresponding to the EEG signal. Specifically, this involves processing the noisy image vector z... t The input is fed into the denoising U-Net network, and the output is the noise to be removed in this step. This involves processing the input noisy image vector z. t Subtracting the predicted noise yields the image vector z after removing some of the noise. t-1 This is what a denoising U-Net network does. The denoising U-Net network is repeatedly trained to remove all noise and generate a realistic image.

[0132] Using the stable diffusion loss function L SD We then used existing EEG-image datasets (i.e., ImageNet EEG dataset) to adjust the parameters of the cross-attention layer of the denoising U-Net network and the EEG autoregressive encoder in the stable diffusion model, resulting in the trained denoising U-Net network and the parameter-adjusted EEG autoregressive encoder.

[0133] Stable diffusion loss function L SD Specifically as follows:

[0134]

[0135] Wherein, noise ∈ follows a standard normal distribution. It is added during the diffusion process to generate z. t noise, ∈ θ Let z represent the noise prediction function of the denoised U-Net network with parameters θ. t τ represents the Gaussian noise used in the reverse diffusion process. θ (y) represents conditional information; t represents the time step; The formula for calculating random variables z and ∈ is given by the condition that both random variables z and ∈ follow a standard normal distribution. The expected value, that is, the sum of the random noise ∈ and the model predicted noise ∈. θ (z t ,t,τ θ The expected value of the square of the Euclidean distance between (y) and (y).

[0136] In some embodiments, the cross-attention layer of the denoising U-Net network is specifically as follows:

[0137]

[0138] in, This represents the median value of the denoised U-Net network. These are different projection matrices with learnable parameters.

[0139] It is important to note that during the training process, the Gaussian noise z used in the reverse diffusion process... t The noise is obtained by continuously adding noise to the image encoding during the diffusion process, while the Gaussian noise z used in the reverse diffusion process during sampling is... t It is then randomly generated.

[0140] In this embodiment, a pre-trained version 1.5 of the Stable Diffusion (SD) model is selected. Simultaneously, the Contrastive Language-Image Pre-training (CLIP) model is used to align EEG features with image features.

[0141] In some embodiments, S5 specifically includes the following steps:

[0142] S51. First, the unlabeled EEG signals are preprocessed and then input into the trained EEG autoregressive encoder to obtain the latent representation.

[0143] S52. Input the latent representation into the trained CLIP model to obtain the aligned EEG features;

[0144] S53. The aligned EEG features are introduced into the cross-attention layer of the denoising U-Net network in the stable diffusion model to control the generation of images and finally generate denoised image data.

[0145] To facilitate understanding, the following examples will be provided:

[0146] More than 30,000 unlabeled EEG data were obtained. The input dimension of each sample was [128, 512], which means there are 128 EEG channels, each channel has 512 sampling points, and the sampling frequency is 5-95Hz.

[0147] EEG data is input into an autoregressive model for training to predict the EEG signal at the next moment, and the network is continuously optimized using the loss function MSE.

[0148] After training the autoregressive model using a large amount of EEG signals—that is, after completing EEG pre-training—an EEG autoregressive encoder was obtained that can be used to extract EEG features. This encoder already has the ability to predict the EEG signal at the next moment. Next, this pre-trained EEG autoregressive encoder will be used for a specific downstream task, namely the EEG image generation task.

[0149] The dataset chosen for the image generation task was the EEG-image pair dataset, specifically the ImageNet EEG dataset, which is a collection of EEG recordings obtained from six subjects. During the experiment, subjects were simultaneously shown 2000 images of objects belonging to 40 different categories from the ImageNet dataset. Each category consisted of 50 images, each displayed for 0.5 seconds, followed by a 10-second pause after every 50 images. EEG data was recorded using a 128-channel Brainvision EEG system, generating a total of 12,000 128-channel EEG sequences. The dataset included images of various objects, such as animals, vehicles, and everyday items.

[0150] EEG data from the ImageNet EEG dataset were input into a pre-trained EEG autoregressive encoder to extract EEG features (i.e., latent representations Z). Then, the CLIP model was used to align the EEG feature vectors with their corresponding image latent vectors. The aligned EEG features were then input into the image generation model SD (i.e., a stable diffusion model) to generate the target image. Some image generation results are shown in the attached figure. Figure 2 As shown in the figure, this invention can effectively capture the semantic content of EEG signals, thereby generating images that are as realistic as possible.

[0151] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for generating electroencephalogram (EEG) images based on autoregressive pre-training and feature alignment, characterized in that, Includes the following steps: S1. Collect a large number of unlabeled EEG signals, preprocess them and input them into the autoregressive model, convert the preprocessed EEG signals into EEG codes, then calculate the location codes of the EEG codes, and transmit the EEG codes with added location codes to the EEG autoregressive encoder to obtain the latent representation. S2. Input the latent representation into the EEG prediction layer to predict the EEG signal at the next moment, obtain the EEG prediction result, calculate the loss between the EEG prediction result and the preprocessed EEG signal, and use the autoregressive pre-training method to train the EEG autoregressive encoder to obtain the trained EEG autoregressive encoder. S3. Train the CLIP model using latent representations to obtain the trained CLIP model; S4. Train the denoising U-Net network within the stable diffusion model using the existing EEG-image dataset to obtain the trained denoising U-Net network. S5. Test the trained EEG autoregressive encoder, the trained CLIP model, and the trained denoised U-Net network, and generate the final denoised image. S1 specifically includes the following steps: S11. First, a large number of unlabeled EEG signals are collected and preprocessed to obtain preprocessed EEG signals. Preprocessing includes filtering, noise reduction, and segmentation. S12. Use a one-dimensional convolutional layer to convert the preprocessed EEG signal X into EEG code; S13. Calculate the sine and cosine outputs using sine and cosine functions of different frequencies, and concatenate them to obtain the position code of the EEG code. Then, add the position code to the EEG code. S14. Input the EEG code with added location encoding into the EEG autoregressive encoder to obtain the latent representation Z; The EEG autoregressive encoder in S14 consists of six sequentially connected Transformer decoder modules. Each Transformer decoder module includes a masked multi-head attention mechanism, a first normalization layer, a feedforward module, and a second normalization layer, all connected in sequence.

2. The method for generating electroencephalograms according to claim 1, characterized in that, The sine and cosine functions of different frequencies in S13 are as follows: Where k represents the position of an object in the input sequence; d represents the dimension of the output embedding space; P(k, 2i) and P(k, 2i+1) represent two different position functions, which are used to map the position k in the input sequence to the indices (k, 2i) and (k, 2i+1) of the position matrix, respectively; n represents a user-defined scalar; and i represents the value used to map the column index to 0 ≤ i ≤ d / 2.

3. The method for generating electroencephalograms according to claim 2, characterized in that, S14 includes the following steps: S141. Input the EEG code with added location encoding into the Transformer decoder module of the first layer. The EEG code is transformed into a query matrix Q, a key matrix K and a value matrix V through three different linear transformations. Q=XW Q ,K=XW K ,V=XW V Among them, WQ, W K W V These are three different learnable weight matrices; S142. Next, calculate the attention weight A. The calculation process for the attention weight A is expressed by the formula as follows: Where, d k It represents the dimension of the key, Mask represents the mask, and T represents the transpose; S143. Use the attention weights A and the value matrix V to perform a weighted summation to obtain the output O of the masked multi-head attention mechanism in the first layer Transformer decoder module; O = A·V; S144. After normalizing the output O of the mask multi-head attention mechanism, input it into the feedforward module to obtain the output F of the feedforward module; F = ReLU(O·W1+b1)·W2+b2; Where ReLU(.) represents the activation function; W1 and b1 represent the weight matrix and bias vector of the first layer of the feedforward network, respectively; W2 and b2 represent the weight matrix and bias vector of the second layer of the feedforward network, respectively. S145. The output F of the feedforward module is normalized to obtain the output B of the first layer Transformer decoder module. B = FFNN(MMHA(X)); Where MMHA(.) represents masked multi-head attention mechanism, and FFNN(.) represents feedforward neural network; S146. The output B of the previous Transformer decoder module is used as the input of the next Transformer decoder module to pass the output result to the next Transformer decoder module. After multiple calculations, the output of the EEG autoregressive encoder, i.e., the latent representation Z, is obtained, expressed by the following formula: Z=Decoder6(Decoder5(...Decoder1(X)); Here, Decoder1(.) to Decoder6(.) represent the first layer Transformer decoder module to the sixth layer Transformer decoder module, respectively.

4. The method for generating electroencephalograms according to claim 1, characterized in that, S2 specifically includes the following steps: S21. Input the latent representation Z into the EEG prediction layer to predict the EEG signal at the next moment and obtain the EEG prediction result X'. S22. Calculate the loss L between the EEG prediction result X′ and the preprocessed EEG signal. MSE ; Where C represents the number of channels, U represents the length of the time series; X′(c, u) represents the predicted EEG signal value, where c represents the channel index and u represents the time point index; Label(c, u) represents the actual EEG signal label value, which corresponds to the predicted value. S23. Repeat S1 to S2 multiple times to minimize the loss L. MSE The weight parameters of the EEG autoregressive encoder were adjusted to obtain the trained EEG autoregressive encoder.

5. The method for generating electroencephalograms according to claim 1, characterized in that, S3 specifically includes the following steps: S31. Input the latent representation Z into two projection layers to transform it into an EEG embedding with the same dimension as the CLIP image encoder in the CLIP model. S32. Input the real image corresponding to the EEG signal into the CLIP image encoder to obtain the image embedding; S33. Use cosine similarity to calculate the loss L between EEG embedding and image embedding. clip ; Where h(τ) θ (y) represents EEG embedding, h is the projection layer, and τ θ (·) represents another projection layer, y represents the output of the EEG autoregressive encoder, i.e., the latent representation Z; E I It is a CLIP image encoder; I represents the image corresponding to the EEG signal; S34. Repeat S1 to S3 multiple times to minimize the loss L. clip In this process, the weight parameters of the CLIP image encoder and the two projection layers in the CLIP model are adjusted to obtain the trained CLIP model.

6. The method for generating electroencephalograms according to claim 5, characterized in that, The specific implementation method of S4 is as follows: Using the stable diffusion loss function L SD We used existing EEG-image datasets to adjust the parameters of the cross-attention layer of the denoised U-Net network and the EEG autoregressive encoder in the stable diffusion model, resulting in the trained denoised U-Net network and the parameter-adjusted EEG autoregressive encoder. The stable diffusion model mainly consists of two stages: the diffusion process and the de-diffusion process. Stable diffusion loss function L SD Specifically as follows: Wherein, noise ∈ follows a standard normal distribution. It is added during the diffusion process to generate z. t noise, ∈ θ Let z represent the noise prediction function of the denoised U-Net network with parameters θ. t τ represents the Gaussian noise used in the reverse diffusion process. θ (y) represents conditional information, and t represents the time step; The formula for calculating random variables z and ∈ is given by the condition that both random variables z and ∈ follow a standard normal distribution. The expected value, that is, the sum of the random noise ∈ and the model predicted noise ∈. θ (z t ,t,τ θ The expected value of the square of the Euclidean distance between (y) and (y).

7. The method for generating electroencephalograms according to claim 6, characterized in that, The specific details of the cross-attention layer of the denoising U-Net network are as follows: in, This represents the median value of the denoised U-Net network. These are different projection matrices with learnable parameters.

8. The method for generating electroencephalograms according to any one of claims 1 to 7, characterized in that, S5 specifically includes the following steps: S51. First, the unlabeled EEG signals are preprocessed and then input into the trained EEG autoregressive encoder to obtain the latent representation. S52. Input the latent representation into the trained CLIP model to obtain the aligned EEG features; S53. The aligned EEG features are introduced into the cross-attention layer of the denoising U-Net network in the stable diffusion model to control the generation of images and finally generate denoised image data.

Images