Electroencephalogram image generation method based on autoregression pre-training and feature alignment

Abstract

The invention relates to the technical field of electroencephalogram signal processing, in particular to an electroencephalogram image generation method based on autoregressive pre-training and feature alignment, which comprises the following steps: 1, converting a preprocessed electroencephalogram signal into an electroencephalogram code, calculating a position code of the electroencephalogram code, and transmitting the electroencephalogram code added with the position code to an electroencephalogram autoregressive encoder, potential characterization is obtained; 2, training an electroencephalogram autoregression encoder by using an autoregression pre-training method; 3, training the CLIP model by using the potential representation; 4, training the de-noising U-Net network in the stable diffusion model by using the existing electroencephalogram-image pair data set; and 5, testing the trained electroencephalogram autoregression encoder, the trained CLIP model and the trained de-noised U-Net network, and generating a final de-noised image. According to the method, the mapping relation between the electroencephalogram signals and the images is established through feature alignment, and cross-modal conversion from the electroencephalogram signals to the images is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of electroencephalogram (EEG) signal processing, and in particular to an EEG image generation method based on autoregressive pre-training and feature alignment. Background Art

[0002] In recent years, the development of brain-computer interface technology has provided people with a new way of human-computer interaction. Traditional brain-computer interface technology mainly focuses on using EEG signals for intention recognition, motor imagery and other tasks, but there are relatively few studies on converting EEG signals into images. Although some studies have tried to use deep learning methods to convert EEG signals into images, there are still the following problems and shortcomings:

[0003] a) Most existing EEG image generation methods rely on pre-defined image categories or templates, and the generated images are mostly abstract or simple geometric shapes, lacking diversity and authenticity.

[0004] b) Existing methods usually directly take EEG signals as input, which does not fully utilize the temporal characteristics and potential representations of EEG signals, resulting in low quality and accuracy of generated images.

[0005] c) Existing methods lack an effective cross-modal feature alignment mechanism and fail to establish a good mapping relationship between EEG signals and images, which affects the relevance and consistency of the generated images.

[0006] The causes of these problems and shortcomings can be attributed to the following points:

[0007] a) EEG signals have the characteristics of high dimensionality and non-stationarity, making them difficult to be directly used for image generation tasks.

[0008] b) The lack of large-scale, high-quality EEG-image paired datasets limits the training and optimization of deep learning models.

[0009] c) EEG signals and images belong to different modalities, and cross-modal feature alignment and conversion are challenging.

[0010] d) There is limited understanding of the cognitive mechanism between EEG signals and visual imagination, making it difficult to establish an effective EEG-image generation model.

[0011] Therefore, a new technical solution is urgently needed to solve the above problems in order to improve the quality, diversity and interpretability of EEG image generation and promote the application of brain-computer interface technology in the field of visual imagination and image generation. Summary of the invention

[0012] The present invention provides an electroencephalogram (EEG) image generation method based on autoregressive pre-training and feature alignment to solve the technical problems mentioned in the background technology.

[0013] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0014] The present invention provides an electroencephalogram image generation method based on autoregressive pre-training and feature alignment, comprising the following steps:

[0015] S1. Collect a large number of unlabeled EEG signals, preprocess them and input them into the autoregressive model, convert the preprocessed EEG signals into EEG codes, then calculate the position codes of the EEG codes, and transmit the EEG codes with position codes added to the EEG autoregressive encoder to obtain potential representations;

[0016] S2, input the potential representation into the EEG prediction layer to predict the EEG signal at the next moment, obtain the EEG prediction result, calculate the loss between the EEG prediction result and the preprocessed EEG signal, and train the EEG autoregressive encoder using the autoregressive pre-training method to obtain the trained EEG autoregressive encoder;

[0017] S3, training the CLIP model using the potential representation to obtain a trained CLIP model;

[0018] S4, using the existing EEG-image pair data set to train the denoising U-Net network in the stable diffusion model to obtain a trained denoising U-Net network;

[0019] S5. Test the trained EEG autoregressive encoder, the trained CLIP model, and the trained denoising U-Net network, and generate the final denoised image.

[0020] Furthermore, the S1 specifically includes the following steps:

[0021] S11, firstly collect a large number of unlabeled EEG signals, and preprocess them to obtain preprocessed EEG signals, wherein the preprocessing includes filtering, noise reduction, and segmentation operations;

[0022] S12, using a one-dimensional convolutional layer to convert the preprocessed EEG signal X into EEG code;

[0023] S13, respectively calculating the sine and cosine outputs through sine and cosine functions of different frequencies, and concatenating them to obtain the position code of the EEG code, and adding the position code to the EEG code;

[0024] S14. Input the EEG code with position coding added into the EEG autoregressive encoder to obtain the latent representation Z.

[0025] Furthermore, the sine and cosine functions of different frequencies in S13 are specifically as follows:

[0026]

[0027] Among them, k represents the position of the object in the input sequence; d represents the dimension of the output embedding space; P(k, 2i) and P(k, 2i+1) represent two different position functions, which are used to map the position k in the input sequence to the index (k, 2i) and (k, 2i+1) of the position matrix respectively; n represents a custom scalar; i represents the value used to map the column index to 0≤i≤d / 2.

[0028] Furthermore, the EEG autoregressive encoder in S14 is composed of six layers of sequentially connected Transformer decoder modules, and each layer of Transformer decoder modules includes a masked multi-head attention mechanism, a first normalization layer, a feedforward module and a second normalization layer connected in sequence.

[0029] Furthermore, the S14 comprises the following steps:

[0030] S141, inputting the EEG code with position code added into the Transformer decoder module of the first layer, and obtaining the query matrix Q, key matrix K and value matrix V through three different linear transformations of the EEG code;

[0031] Q=XW Q , K = XW K , V = XW V

[0032] Among them, W Q , W K , W V There are three different weight matrices that can be learned;

[0033] S142, then calculate the attention weight A. The calculation process of the attention weight A is expressed by the formula, which is as follows:

[0034]

[0035] Among them, d k is the dimension of the key, Mask represents mask, and T represents transpose;

[0036] S143, using the attention weight A and the value matrix V to perform weighted summation to obtain the output O of the masked multi-head attention mechanism in the first layer Transformer decoder module;

[0037] O = A·V;

[0038] S144, normalize the output O of the masked multi-head attention mechanism and input it into the feedforward module to obtain the output F of the feedforward module;

[0039] F = ReLU(O·W1+b1)·W2+b2;

[0040] Among them, ReLU(.) represents the activation function; W1 and b1 represent the weight matrix and bias vector of the first layer of the feedforward network respectively. This layer is a linear transformation layer used to map the input O to an intermediate representation; W2 and b2 represent the weight matrix and bias vector of the second layer of the feedforward network respectively. This layer is also a linear transformation layer used to map the intermediate representation after the activation function ReLU to the specified dimension;

[0041] S145, normalizing the output of the feedforward module to obtain the output B of the Transformer decoder module of the first layer;

[0042] B = FFNN(MMHA(X));

[0043] Among them, MMHA(.) represents the masked multi-head attention mechanism, and FFNN(.) represents the feedforward neural network;

[0044] S146, the output B of the previous layer Transformer decoder module is used as the input of the next layer Transformer decoder module to pass the output result to the next layer Transformer decoder module. After multiple calculations, the output of the EEG autoregressive encoder is obtained, that is, the potential representation Z, which is expressed by the formula, as follows:

[0045] Z=Decoder6(Decoder5(...Decoder1(X));

[0046] Among them, Decoder1(.) to Decoder6(.) represent the first-layer Transformer decoder module to the sixth-layer Transformer decoder module respectively.

[0047] Furthermore, the S2 specifically includes the following steps:

[0048] S21, input the potential representation Z into the EEG prediction layer to predict the EEG signal at the next moment, and obtain the EEG prediction result X′;

[0049] S22, calculate the loss L between the EEG prediction result X′ and the preprocessed EEG signal MsE ;

[0050]

[0051] Where C represents the number of channels, U represents the length of the time series; X′(c,u) represents the predicted EEG signal value, c represents the channel index, and u represents the time point index; Label(c,u) represents the actual EEG signal label value, corresponding to the predicted value;

[0052] S23, cycle S1 to S2 multiple times to minimize the loss L MSE , and adjust the weight parameters of the EEG autoregressive encoder to obtain the trained EEG autoregressive encoder.

[0053] Furthermore, the S3 specifically includes the following steps:

[0054] S31, input the latent representation Z into two projection layers to transform it into an EEG embedding with the same dimension as the CLIP image encoder in the CLIP model;

[0055] S32, inputting the real image corresponding to the EEG signal into the CLIP image encoder to obtain image embedding;

[0056] S33. Use cosine similarity to calculate the loss L between EEG embedding and image embedding clip ;

[0057]

[0058] Among them, h(τ θ (y)) represents EEG embedding, h is the projection layer, E I is the CLIP image encoder; τ θ (y) represents conditional information; τ θ (·) represents another projection layer, y represents the output of the EEG autoregressive encoder, i.e., the potential representation Z; I represents the image corresponding to the EEG signal;

[0059] S34, cycle S1 to S3 multiple times to minimize the loss L clip , and in this process adjust the weight parameters of the CLIP image encoder and the two projection layers in the CLIP model to obtain the trained CLIP model.

[0060] Furthermore, the implementation of S4 is specifically as follows:

[0061] With the help of the stable diffusion loss function L SD The existing EEG-image pair dataset is used to adjust the parameters of the cross-attention layer of the denoising U-Net network and the EEG autoregressive encoder in the stable diffusion model, and the trained denoising U-Net network and the EEG autoregressive encoder with adjusted parameters are obtained; the stable diffusion model mainly includes two stages, namely the diffusion process and the reverse diffusion process;

[0062] Stable diffusion loss function L SD The details are as follows:

[0063]

[0064] Among them, the noise ∈ follows the standard normal distribution is added during the diffusion process to generate z t The noise, ∈ θ represents the noise prediction function with the denoising U-Net network parameter θ, z t represents the Gaussian noise used in the inverse diffusion process, τ θ (y) represents conditional information; t represents the time step; Indicates that under the condition that the random variables z and ∈ both obey the standard normal distribution, the calculation formula The expected value of random noise ∈ and model prediction noise ∈ θ (z t , t, τ θ The expected value of the square of the Euclidean distance between (y)).

[0065] Furthermore, the cross attention layer of the denoising U-Net network is as follows:

[0066]

[0067] in, represents the median value of the denoising U-Net network, are different projection matrices with learnable parameters.

[0068] Furthermore, the S5 specifically includes the following steps:

[0069] S51, firstly, the unlabeled EEG signal is preprocessed and input into the trained EEG autoregressive encoder to obtain a latent representation;

[0070] S52, input the latent representation into the trained CLIP model to obtain the aligned EEG features;

[0071] S53. The aligned EEG features are introduced into the cross attention layer of the denoising U-Net network in the stable diffusion model to control the generation of the image and finally generate the denoised image data.

[0072] Beneficial effects of the present invention:

[0073] 1. The present invention learns the temporal characteristics and potential representations of EEG signals through autoregressive pre-training method, thereby improving the ability to understand and model EEG signals.

[0074] 2. The present invention establishes a mapping relationship between EEG signals and images through feature alignment, and realizes cross-modal conversion from EEG to images. It realizes feature alignment with the existing image generation model, thus achieving the goal of generating images based on EEG. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is a flowchart of the whole process of the present invention;

[0076] Figure 2 This is a partial image generated based on EEG in an embodiment of the present invention;

[0077] Figure 3 It is a flowchart of the working process of the stable diffusion model in the present invention. DETAILED DESCRIPTION

[0078] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Preferred embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many other different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0079] Reference Figure 1 , the embodiment of the present application provides a method for generating an electroencephalogram image based on autoregressive pre-training and feature alignment, comprising the following steps:

[0080] S1. Collect a large number of unlabeled EEG signals, preprocess them and input them into the autoregressive model, convert the preprocessed EEG signals into EEG codes, then calculate the position codes of the EEG codes, and transmit the EEG codes with position codes added to the EEG autoregressive encoder to obtain potential representations;

[0081] S2, input the potential representation into the EEG prediction layer to predict the EEG signal at the next moment, obtain the EEG prediction result, calculate the loss between the EEG prediction result and the preprocessed EEG signal, and train the EEG autoregressive encoder using the autoregressive pre-training method to obtain the trained EEG autoregressive encoder;

[0082] S3, training the CLIP model using the potential representation to obtain a trained CLIP model;

[0083] S4, using the existing EEG-image pair data set to train the denoising U-Net network in the stable diffusion model to obtain a trained denoising U-Net network;

[0084] S5. Test the trained EEG autoregressive encoder, the trained CLIP model, and the trained denoising U-Net network, and generate the final denoised image.

[0085] In some embodiments, the S1 specifically includes the following steps:

[0086] S11, firstly collect a large number of unlabeled EEG signals, and preprocess them to obtain preprocessed EEG signals, wherein the preprocessing includes filtering, noise reduction, segmentation and other operations;

[0087] Specifically, the EEG signal after preprocessing is represented as X∈R C×U , where EEG signals with a time length greater than T are randomly truncated, and EEG signals with a time series length less than U are interpolated to increase sample points so that their length is U, EEG signals with a channel number greater than C are randomly cropped, and EEG signals with a channel number less than C are copied and filled so that their channel number is C.

[0088] S12, using a one-dimensional convolutional layer to convert the preprocessed EEG signal X into EEG code;

[0089] S13, respectively calculating the sine and cosine outputs through sine and cosine functions of different frequencies, and concatenating them to obtain the position code of the EEG code, and adding the position code to the EEG code;

[0090] S14. Input the EEG code with added sine and cosine position coding into the EEG autoregressive encoder to obtain the latent representation Z.

[0091] In some embodiments, the sine and cosine functions of different frequencies in S13 are specifically as follows:

[0092]

[0093] Among them, k represents the position of the object in the input sequence, and 0≤k≤U / 2; d represents the dimension of the output embedding space; P(k, 2i) and P(k, 2i+1) represent two different position functions, which are used to map the position k in the input sequence to the index (k, 2i) and (k, 2i+1) of the position matrix respectively; n represents a custom scalar; in this embodiment, the value of n is preferably 1000; i represents the value used to map the column index to 0≤i≤d / 2, and a single i value is mapped to both the sine and cosine functions.

[0094] In some embodiments, the EEG autoregressive encoder in S14 is composed of six layers of sequentially connected Transformer decoder modules, and each layer of Transformer decoder modules includes a masked multi-head attention mechanism, a first normalization layer, a feedforward module and a second normalization layer connected in sequence.

[0095] In some embodiments, the S14 includes the following steps:

[0096] S141, inputting the EEG code with position code added into the Transformer decoder module of the first layer, and obtaining the query matrix Q, key matrix K and value matrix V through three different linear transformations of the EEG code;

[0097] Q=XW Q , K = XW K , V = XW V

[0098] Among them, W Q , W K , W V There are three different weight matrices that can be learned;

[0099] S142, then calculate the attention weight A. The calculation process of the attention weight A is expressed by the formula, which is as follows:

[0100]

[0101] Among them, d k is the dimension of the key, Mask represents the mask, usually an upper triangular matrix used to mask future sequence parts, and T represents transpose;

[0102] S143, using the attention weight A and the value matrix V to perform weighted summation to obtain the output O of the masked multi-head attention mechanism in the first layer Transformer decoder module;

[0103] O = A·V;

[0104] S144, normalize the output O of the masked multi-head attention mechanism and input it into the feedforward module to obtain the output F of the feedforward module;

[0105] F = ReLU(O·W1+b1)·W2+b2;

[0106] Among them, ReLU(.) represents the activation function; W1 and b1 represent the weight matrix and bias vector of the first layer of the feedforward network respectively. This layer is a linear transformation layer used to map the input O to an intermediate representation; W2 and b2 represent the weight matrix and bias vector of the second layer of the feedforward network respectively. This layer is also a linear transformation layer used to map the intermediate representation after the activation function ReLU to the specified dimension;

[0107] S145, normalizing the output of the feedforward module to obtain the output B of the Transformer decoder module of the first layer;

[0108] B = FFNN(MMHA(X));

[0109] Among them, MMHA(.) represents the masked multi-head attention mechanism, and FFNN(.) represents the feedforward neural network;

[0110] S146, the output B of the previous layer Transformer decoder module is used as the input of the next layer Transformer decoder module to pass the output result to the next layer Transformer decoder module. After multiple calculations, the output of the EEG autoregressive encoder is obtained, that is, the potential representation Z, which is expressed by the formula, as follows:

[0111] Z=Decoder6(Decoder5(...Decoder1(X));

[0112] Among them, Decoder1(.) to Decoder6(.) represent the first-layer Transformer decoder module to the sixth-layer Transformer decoder module respectively.

[0113] In some embodiments, S2 specifically includes the following steps:

[0114] S21, input the potential representation Z into the EEG prediction layer to predict the EEG signal at the next moment, and obtain the EEG prediction result X′;

[0115] S22, calculate the loss L between the EEG prediction result X′ and the preprocessed EEG signal MSE ;

[0116]

[0117] Where C represents the number of channels, U represents the length of the time series; X′(c,u) represents the predicted EEG signal value, c represents the channel index, and u represents the time point index; Label(c,u) represents the actual EEG signal label value, corresponding to the predicted value;

[0118] S23, cycle S1 to S2 multiple times to minimize the loss L MSE , and adjust the weight parameters of the EEG autoregressive encoder to obtain the trained EEG autoregressive encoder.

[0119] The present invention converts the preprocessed EEG signal X into the EEG prediction result X′. This process predicts the time point of the EEG signal at the next moment and improves the prediction accuracy by minimizing the difference between the real EEG data Label and the EEG prediction result X′.

[0120] In some embodiments, S3 specifically includes the following steps:

[0121] S31, input the latent representation Z into two projection layers to transform it into an EEG embedding with the same dimension as the CLIP image encoder in the CLIP model;

[0122] S32, inputting the real image corresponding to the EEG signal into the CLIP image encoder to obtain image embedding;

[0123] S33. Use cosine similarity to calculate the loss L between EEG embedding and image embedding clip ;

[0124]

[0125] Among them, h(τ θ (y)) represents EEG embedding, h is the projection layer, τ θ (·) represents another projection layer, y represents the output of the EEG autoregressive encoder, i.e., the potential representation Z; E I is the CLIP image encoder; I represents the image corresponding to the EEG signal;

[0126] S34, cycle S1 to S3 multiple times to minimize the loss L clip , and in this process adjust the weight parameters of the CLIP image encoder and the two projection layers in the CLIP model to obtain the trained CLIP model.

[0127] Loss L clip The similarity between the EEG feature vector (i.e., latent representation Z) and the image latent vector is measured by calculating the cosine similarity between the two. By optimizing the objective function, i.e., minimizing the similarity loss, a closer alignment between the EEG feature vector and the image latent vector can be achieved, making the EEG features more similar to the text features. In this way, EEG signals, text, and images can be aligned in a unified space.

[0128] In some embodiments, the implementation of S4 is specifically as follows:

[0129] Reference Figure 3 ,The stable diffusion model mainly includes two stages, namely the diffusion process and the reverse diffusion process;

[0130] The diffusion process is the process of continuously adding Gaussian noise ∈ to the real image in the image potential space. Specifically, the vector quantization encoder (VQ) is first used to encode the original image data in the existing EEG-image dataset, and then Gaussian noise is continuously added to the image until it completely becomes a pure noise image z t . This image is random and does not contain any meaningful information.

[0131] The inverse diffusion process is the Gaussian noise z t The process of continuously denoising and finally generating the real image corresponding to the EEG signal. Specifically, the noisy image vector z t Input to the denoising U-Net network, the output is the noise that needs to be removed in this step, and the input noisy image vector z t Subtracting the predicted noise, we can get the image vector z with some noise removed. t-1 , which is what a denoising U-Net network performs. The denoising U-Net network is repeatedly trained to remove all noise and generate realistic images.

[0132] With the help of the stable diffusion loss function L SD The existing EEG-image pair dataset (i.e., ImageNet EEG dataset) is used to adjust the parameters of the cross-attention layer of the denoising U-Net network and the EEG autoregressive encoder in the stable diffusion model, thereby obtaining the trained denoising U-Net network and the EEG autoregressive encoder with adjusted parameters.

[0133] Stable diffusion loss function L SD The details are as follows:

[0134]

[0135] Among them, the noise ∈ follows the standard normal distribution is added during the diffusion process to generate z t The noise, ∈ θ represents the noise prediction function with the denoising U-Net network parameter θ, z t represents the Gaussian noise used in the inverse diffusion process, τ θ (y) represents conditional information; t represents the time step; Indicates that under the condition that the random variables z and ∈ both obey the standard normal distribution, the calculation formula The expected value of random noise ∈ and model prediction noise ∈ θ (z t , t, τ θ The expected value of the square of the Euclidean distance between (y)).

[0136] In some embodiments, the cross attention layer of the denoising U-Net network is as follows:

[0137]

[0138] in, represents the median value of the denoising U-Net network, are different projection matrices with learnable parameters.

[0139] It should be noted that during the training process, the Gaussian noise z used in the inverse diffusion process t The noise is obtained by continuously adding noise to the image encoding during the diffusion process, and the Gaussian noise z used in the inverse diffusion process during the sampling process t It is randomly generated.

[0140] In this embodiment, the stable diffusion model (SD) uses the pre-trained stable diffusion model 1.5. At the same time, the multimodal learning model CLIP model (Contrastive Language-Image Pre-training) is used to achieve the alignment of EEG features and image features.

[0141] In some embodiments, S5 specifically includes the following steps:

[0142] S51, firstly, the unlabeled EEG signal is preprocessed and input into the trained EEG autoregressive encoder to obtain a latent representation;

[0143] S52, input the latent representation into the trained CLIP model to obtain the aligned EEG features;

[0144] S53. The aligned EEG features are introduced into the cross attention layer of the denoising U-Net network in the stable diffusion model to control the generation of the image and finally generate the denoised image data.

[0145] For ease of understanding, the following examples are given below:

[0146] More than 30,000 unlabeled EEG data were obtained. The input dimension of each sample was [128, 512], indicating that there were 128 EEG channels, each channel had 512 sampling points, and the sampling frequency was 5-95 Hz.

[0147] The EEG data is input into the autoregressive model for training, the EEG signal at the next moment is predicted, and the network is continuously optimized through the loss function MSE.

[0148] After completing the training of the autoregressive model using a large number of EEG signals, that is, completing the EEG pre-training, an EEG autoregressive encoder that can be used to extract EEG features is obtained. This encoder has the ability to predict the EEG signal at the next moment. Next, the pre-trained EEG autoregressive encoder will be used for a specific downstream task, that is, the EEG image generation task.

[0149] The dataset selected in the image generation task is the EEG-image pair dataset, namely the ImageNet EEG dataset, which is a collection of EEG recordings obtained from 6 subjects. During the experiment, 2000 images of objects belonging to 40 different categories in the ImageNet dataset were shown to the subjects at the same time. Each category consists of 50 images, each image is displayed for 0.5 seconds, and then a 10-second pause is made after every 50 images. The EEG data was recorded using a 128-channel Brainvision EEG system, generating a total of 12,000 128-channel EEG sequences. The dataset includes images of various objects, such as animals, vehicles, and daily necessities.

[0150] The EEG data in the ImageNet EEG dataset is input into the pre-trained EEG autoregressive encoder to extract its EEG features (i.e., potential representation Z). Then, the CLIP model is used to align the EEG feature vector with its corresponding image potential vector. The aligned EEG features are input into the image generation model SD (i.e., stable diffusion model) to generate the target image. Some of the image generation results are shown in the attached figure. Figure 2 As shown in the figure, it can be seen that the present invention can effectively capture the semantic content of the EEG signal, thereby generating a picture that is as realistic as possible.

[0151] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered within the protection scope of the present invention. In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for generating electroencephalogram images based on autoregressive pre-training and feature alignment, characterized in that: The steps include: S1. Collect a large number of unlabeled EEG signals, preprocess them and input them into the autoregressive model, convert the preprocessed EEG signals into EEG codes, then calculate the position codes of the EEG codes, and transmit the EEG codes with position codes added to the EEG autoregressive encoder to obtain potential representations; S2, input the potential representation into the EEG prediction layer to predict the EEG signal at the next moment, obtain the EEG prediction result, calculate the loss between the EEG prediction result and the preprocessed EEG signal, and train the EEG autoregressive encoder using the autoregressive pre-training method to obtain the trained EEG autoregressive encoder; S3, training the CLIP model using the potential representation to obtain a trained CLIP model; S4, using the existing EEG-image pair data set to train the denoising U-Net network in the stable diffusion model to obtain a trained denoising U-Net network; S5. Test the trained EEG autoregressive encoder, the trained CLIP model, and the trained denoising U-Net network, and generate the final denoised image.

2. The method for generating an electroencephalogram image according to claim 1, characterized in that: The S1 specifically includes the following steps: S11, firstly collect a large number of unlabeled EEG signals, and preprocess them to obtain preprocessed EEG signals, wherein the preprocessing includes filtering, noise reduction, and segmentation operations; S12, using a one-dimensional convolutional layer to convert the preprocessed EEG signal X into EEG code; S13, respectively calculating the sine and cosine outputs through sine and cosine functions of different frequencies, and concatenating them to obtain the position code of the EEG code, and adding the position code to the EEG code; S14. Input the EEG code with position coding added into the EEG autoregressive encoder to obtain the latent representation Z.

3. The method for generating an electroencephalogram image according to claim 2, characterized in that: The sine and cosine functions of different frequencies in S13 are specifically as follows: Among them, k represents the position of the object in the input sequence; d represents the dimension of the output embedding space; P(k, 2i) and P(k, 2i+1) represent two different position functions, which are used to map the position k in the input sequence to the index (k, 2i) and (k, 2i+1) of the position matrix respectively; n represents a custom scalar; i represents the value used to map the column index to 0≤i≤d / 2.

4. The method for generating an electroencephalogram image according to claim 2, characterized in that: The EEG autoregressive encoder in S14 is composed of six layers of sequentially connected Transformer decoder modules, and each layer of Transformer decoder modules includes a sequentially connected masked multi-head attention mechanism, a first normalization layer, a feedforward module and a second normalization layer.

5. The method for generating an electroencephalogram image according to claim 4, characterized in that: The S14 comprises the following steps: S141, inputting the EEG code with position code added into the Transformer decoder module of the first layer, and obtaining the query matrix Q, key matrix K and value matrix V through three different linear transformations of the EEG code; Q=XW Q ,K=XW K ,V=XW V Among them, W Q , W K , W V There are three different weight matrices that can be learned; S142, then calculate the attention weight A, the calculation process of the attention weight A is expressed by the formula, as follows: Among them, d k is the dimension of the key, Mask represents mask, and T represents transpose; S143, using the attention weight A and the value matrix V to perform weighted summation to obtain the output O of the masked multi-head attention mechanism in the first layer Transformer decoder module; O = A·V; S144, normalize the output O of the masked multi-head attention mechanism and input it into the feedforward module to obtain the output F of the feedforward module; F = ReLU(O·W1+b1)·W2+b2; Wherein, ReLU(.) represents the activation function; W1 and b1 represent the weight matrix and bias vector of the first layer of the feedforward network respectively; W2 and b2 represent the weight matrix and bias vector of the second layer of the feedforward network respectively; S145, normalizing the output of the feedforward module to obtain the output B of the Transformer decoder module of the first layer; B = FFNN(MMHA(X)); Among them, MMHA(.) represents the masked multi-head attention mechanism, and FFNN(.) represents the feedforward neural network; S146, the output B of the previous layer Transformer decoder module is used as the input of the next layer Transformer decoder module to pass the output result to the next layer Transformer decoder module. After multiple calculations, the output of the EEG autoregressive encoder is obtained, that is, the potential representation Z, which is expressed by the formula, as follows: Z=Decoder6(Decoder5(...Decoder1(X)); Among them, Decoder1(.) to Decoder6(.) represent the first-layer Transformer decoder module to the sixth-layer Transformer decoder module respectively.

6. The method for generating an electroencephalogram image according to claim 1, characterized in that: The S2 specifically includes the following steps: S21, input the potential representation Z into the EEG prediction layer to predict the EEG signal at the next moment, and obtain the EEG prediction result X′; S22, calculate the loss L between the EEG prediction result X′ and the preprocessed EEG signal MSE ; Where C represents the number of channels, U represents the length of the time series; X′(c,u) represents the predicted EEG signal value, c represents the channel index, and u represents the time point index; Label(c,u) represents the actual EEG signal label value, corresponding to the predicted value; S23, cycle S1 to S2 multiple times to minimize the loss L MSE , and adjust the weight parameters of the EEG autoregressive encoder to obtain the trained EEG autoregressive encoder.

7. The method for generating an electroencephalogram image according to claim 1, characterized in that: The S3 specifically includes the following steps: S31, input the latent representation Z into two projection layers to transform it into an EEG embedding with the same dimension as the CLIP image encoder in the CLIP model; S32, inputting the real image corresponding to the EEG signal into the CLIP image encoder to obtain image embedding; S33. Use cosine similarity to calculate the loss L between EEG embedding and image embedding clip ; Among them, h(τ θ (y)) represents EEG embedding, h is the projection layer, τ θ (·) represents another projection layer, y represents the output of the EEG autoregressive encoder, i.e., the potential representation Z; E I is the CLIP image encoder; I represents the image corresponding to the EEG signal; S34, cycle S1 to S3 multiple times to minimize the loss L clip , and in this process adjust the weight parameters of the CLIP image encoder and the two projection layers in the CLIP model to obtain the trained CLIP model.

8. The method for generating an electroencephalogram image according to claim 7, characterized in that: The implementation of S4 is specifically as follows: With the help of the stable diffusion loss function L SD The existing EEG-image pair dataset is used to adjust the parameters of the cross-attention layer of the denoising U-Net network and the EEG autoregressive encoder in the stable diffusion model, and the trained denoising U-Net network and the EEG autoregressive encoder with adjusted parameters are obtained; the stable diffusion model mainly includes two stages, namely the diffusion process and the reverse diffusion process; Stable diffusion loss function L SD The details are as follows: Among them, the noise ∈ follows the standard normal distribution is added during the diffusion process to generate z t The noise, ∈ θ represents the noise prediction function with the denoising U-Net network parameter θ, z t represents the Gaussian noise used in the inverse diffusion process; τ θ (y) represents conditional information, and t represents the time step; Indicates that under the condition that the random variables z and ∈ both obey the standard normal distribution, the calculation formula The expected value of random noise ∈ and model prediction noise ∈ θ (z t , t, τ θ The expected value of the square of the Euclidean distance between (y)).

9. The method for generating an electroencephalogram image according to claim 8, characterized in that: The cross attention layer of the denoising U-Net network is as follows: in, represents the median value of the denoising U-Net network, are different projection matrices with learnable parameters.

10. The method for generating an electroencephalogram image according to any one of claims 1 to 9, characterized in that: The S5 specifically includes the following steps: S51, firstly, the unlabeled EEG signal is preprocessed and input into the trained EEG autoregressive encoder to obtain a latent representation; S52, input the latent representation into the trained CLIP model to obtain the aligned EEG features; S53. The aligned EEG features are introduced into the cross attention layer of the denoising U-Net network in the stable diffusion model to control the generation of the image and finally generate the denoised image data.

Images