EEG visual stimulation mixed mode decoding method and system

Abstract

The invention discloses an EEG visual stimulation mixed mode decoding method and system, and the method comprises the following steps: obtaining EEG data, and constructing an image-category-semantic three-state EEG data set; the mixed mode encoder learns from the EEG data to obtain corresponding image, category and semantic mode information; mapping the three-state EEG data set to a corresponding CLIP space to obtain a target mode; performing comparative learning on modal information output by the hybrid modal encoder and a target modal, and performing image retrieval and classification tasks based on the image modal information; and constructing a diffusion prior model, aligning the image, category and semantic modal information output by the hybrid modal encoder with a target modal in a CLIP space, outputting the image, category and semantic modal information subjected to diffusion prior processing, and inputting the image, category and semantic modal information subjected to diffusion prior processing into a pre-trained image generation model to perform a generation task. According to the method, multiple modes of the brain data can be effectively decoded, and the retrieval and reconstruction task effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of EEG data decoding, and in particular to an EEG visual stimulation mixed modality decoding method and system. Background Art

[0002] Early studies used long short-term memory networks and generative adversarial networks to extract temporal features from EEG data and reconstruct natural images. Long short-term memory networks are good at processing temporal dependencies, but may ignore the spatial correlation of multi-channel EEG data (such as the topological relationship between electrodes), and the scarcity of training samples affects the feature representation ability. On the other hand, the adversarial training of generative adversarial networks requires balancing the generator and the discriminator, which is prone to mode collapse or training oscillation. The high noise contained in EEG data will aggravate this problem, resulting in generally unsatisfactory generated image effects. Recent studies have used window slicing and random masking strategies to divide EEG data into different semantic units, focusing on reconstructing the embedding of a single semantic unit. Although this reduces the difficulty of reconstruction, masking the EEG data, which is already noisy and small in size, will increase the difficulty of extracting useful features from the EEG data, thereby impairing the reconstruction effect. Other studies have sought to use image modalities for cross-modal alignment and decode multimodal information from the brain. For modal learning, they currently seek to use image modalities as a reference for alignment to achieve the decoding of meaningful visual stimuli. However, using only a single modality to decode visual stimuli often tends to ignore the correlation between different modalities when the actual brain is visually stimulated, and it is also difficult to capture the actual activity changes in brain areas.

[0003] Common limitations of existing technologies include: (1) Although the Transformer model can fully model the time series model, it tends to focus on the global modeling relationship and thus ignore the expression effect of EEG data at a fine-grained level; (2) Training different modalities separately ignores the fact that the model can learn the correlation between different modalities; (3) Because different parts of the brain will produce different modal electrical data characteristics when receiving natural visual stimulation, only single-modal training of the model will make it difficult for the model to capture the actual activity changes in the brain area.

[0004] Therefore, there is an urgent need for a mixed-modality decoding technology that can capture the commonalities between different modalities and the actual activities of brain areas when learning visual stimulation. Summary of the invention

[0005] In order to overcome the defects and shortcomings of the prior art, the present invention provides an EEG visual stimulation mixed modality decoding method and system, which can better understand the generation and distribution of different modalities in the human brain, effectively decode multiple modalities from brain data, and improve the retrieval and reconstruction task effects.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: The present invention provides an EEG visual stimulation mixed modality decoding method, comprising the following steps: Obtain EEG data and construct a three-state EEG dataset of image, category and semantics; Construct a hybrid modality encoder that learns the corresponding image, category, and semantic modality information from EEG data; Map the image-category-semantic data in the image-category-semantic three-state EEG dataset to the corresponding CLIP space to obtain the target modality; Compare the image, category, and semantic modality information output by the hybrid modality encoder with the target modality, and perform image retrieval and classification tasks based on the image modality information; Construct a diffusion prior model to align the image, category, and semantic modality information output by the mixed modality encoder with the target modality in the CLIP space, and output the image, category, and semantic modality information processed by the diffusion prior. The images, categories, and semantic modal information that have been processed with diffusion priors are input into the pre-trained image generation model to perform the generation task. The modal information that has been processed with diffusion priors is used to provide the degree of representation in the coarse-grained direction during image generation, the category during image generation, and the degree of representation in the fine-grained direction during image generation to complete the reconstruction of visual stimulation.

[0007] As a preferred technical solution, the hybrid modality encoder includes a global feature mapping network and a hybrid mapping network; The global feature mapping network includes a lightweight Transformer network, a multi-time scale hybrid convolutional network, a convolutional network with multiple feature convolutional layers, and a dual-stream spatiotemporal convolutional network in the time-frequency domain; The lightweight Transformer network extracts the temporal features of the EEG data; The multi-time-scale hybrid convolutional network enhances the details of the temporal features based on the local receptive field and outputs the global temporal features; The convolution network of the multi-feature convolution layer extracts spatial dimension features from the multimodal characteristics of the EEG data, analyzes the cortical spatial topological information of the EEG data, and outputs temporal features; The dual-stream spatiotemporal convolutional network in the time-frequency domain fuses the time-domain and frequency-domain information features of the EEG data and outputs global features; The hybrid mapping network decodes the global features to obtain the category modality and the semantic modality.

[0008] As a preferred technical solution, the lightweight Transformer network includes a downsampling layer, an attention layer and a convolution layer, and uses average pooling layers of different sizes and different step lengths to form downsampling layers of different scales. The EEG data is decomposed into time pattern information of different scales through downsampling layers of different scales, and the time pattern information of different scales is reshaped into the same shape through MLP and added to integrate into new pattern information. , based on the Embedding module, the new mode information Mapping to variables , learning variables through attention layers The correlation between different tokens is obtained by outputting the time series features extracted by the lightweight Transformer network through the convolutional layer .

[0009] As a preferred technical solution, the multi-time-scale hybrid convolutional network performs convolution operations on the time series features output by the lightweight Transformer network, extracts time series features of different time scales, reshapes them into time series feature vectors with the same dimension through flattening operations and fully connected layers, and connects them into vector groups. ; For vector group Each vector of the channel is generated through convolution operation and sigmoid activation function, and the descriptors are combined into a new vector group , expressed as: ; Group vectors The vectors in the concatenation are Vector of dimensions ; vector exist The weight matrix is ​​obtained by the softmax function in the dimension , based on the weight matrix and the corresponding input vector, the global temporal features of the multi-time scale hybrid convolutional network output are obtained , expressed as: ; ; in, Indicates that the softmax function operates on the vector The operation is performed on the 0th dimension of represents the dot product operation, Representatives in order from 0 dimension to M -1 dimension for extraction operations.

[0010] As a preferred technical solution, the convolutional network of the multi-feature convolutional layer includes a multi-scale spatial convolution group and a dynamic feature fusion layer. The multi-scale spatial convolution group adopts a parallel deep convolution layer with a different receptive field to extract the local micro-topological association and the global macro-distribution pattern of the electrode array respectively. The dynamic feature fusion layer performs adaptive weighted fusion of the multi-scale spatial features through learnable channel attention weights to output temporal features.

[0011] As a preferred technical solution, in the multi-scale spatial convolution group, feature information of different dimensions of global temporal features are extracted through multiple convolution layers with different convolution kernel sizes, feature information of different dimensions are added, multi-scale information is fused, and channel descriptor representation is obtained through global pooling. The correlation between channels is modeled based on two fully connected layers, and the second fully connected layer is divided into multiple parts after dimensionality is increased. In the dynamic feature fusion layer, the sigmoid function is used to generate their own weighted representations, and the information of different scales is weightedly added to output the temporal features.

[0012] As an optimal technical solution, the dual-stream spatiotemporal convolutional network in the time-frequency domain obtains frequency domain features through fast Fourier transform and ResMLP network modeling, and the time features and frequency domain features output by the convolutional network of the multi-feature convolutional layer are obtained by sigmoid function to obtain time domain weights and frequency domain weights, and the corresponding output features are weighted, and matrix multiplication operations are performed after splicing to obtain mixed time-frequency domain features. The global features are obtained based on the mixed time-frequency domain features, which are specifically expressed as: ; ; ; ; ; ; in, Represents time characteristics, represents the frequency domain characteristics, represents a linear operation, represents the sigmoid function, represents the average pooling operation, represents the feature concatenation operation, represents the mixed time-frequency domain features, Representation characteristics The transpose of represents the convolution operation, represents the convolution kernel size, Representative The dilation rate of the convolution operation, Represents a max pooling operation.

[0013] As a preferred technical solution, the hybrid mapping network includes a brain modality regression layer, a category embedder, a brain modality embedder, a semantic embedder and a multimodal decoder; After passing the global features through the brain modality regression layer, they are input into the category embedder, brain modality embedder, and semantic embedder respectively to obtain the corresponding preliminary modality feature representation; Each preliminary modal feature representation is concatenated in the first dimension and input into the multimodal decoder for decoding to obtain the category modal feature and semantic modal feature.

[0014] As a preferred technical solution, the diffusion prior model is trained using mean square error loss.

[0015] The present invention also provides an EEG visual stimulation mixed modality decoding system for implementing the above-mentioned EEG visual stimulation mixed modality decoding method, the system comprising: an EEG data acquisition module, a three-state EEG data set construction module, a mixed modality encoder construction module, a target modality construction module, a contrast learning module, an image retrieval classification module, a diffusion prior model construction module, and an image generation module; The EEG data acquisition module is used to acquire EEG data; The three-state EEG data set construction module is used to construct an image-category-semantic three-state EEG data set; The hybrid modality encoder construction module is used to construct a hybrid modality encoder, and the hybrid modality encoder learns corresponding image, category, and semantic modality information from EEG data; The target modality construction module is used to map the image-category-semantic data in the image-category-semantic three-state EEG data set to the corresponding CLIP space to obtain the target modality; The contrastive learning module is used to compare the image, category, and semantic modality information output by the hybrid modality encoder with the target modality; The image retrieval and classification module is used to perform image retrieval and classification tasks based on image modality information; The diffusion prior model construction module is used to construct a diffusion prior model, align the image, category, and semantic modality information output by the hybrid modality encoder with the target modality in the CLIP space, and output the image, category, and semantic modality information processed by the diffusion prior; The image generation module is used to input the image, category, and semantic modal information that have been processed by diffusion prior into a pre-trained image generation model to perform a generation task. The modal information that has been processed by diffusion prior is used to provide the degree of representation in the coarse-grained direction during image generation, the category during image generation, and the degree of representation in the fine-grained direction during image generation, so as to complete the reconstruction of visual stimulation.

[0016] Compared with the prior art, the present invention has the following advantages and beneficial effects: (1) In view of the shortcomings of existing EEG encoders in image reconstruction, the present invention is based on the corresponding image, category, and semantic modal information learned from EEG data by the hybrid modality encoder, which can better understand the generation and distribution of different modalities in the human brain, allowing the model to effectively decode multiple modalities from brain data.

[0017] (2) In order to solve the problems of high computational complexity and easy overfitting brought by the Transformer network, the present invention adopts a lightweight Transformer network to suppress the problem of easy overfitting in the ordinary Transformer network. While retaining the powerful ability of the Transformer network to extract time series features, it also avoids the information bottleneck caused by the simplicity of linear mapping of MLP-like methods, making it difficult to capture the dependencies and complex time patterns of the contextual relationships of the time series.

[0018] (3) In order to solve the problem that the Transformer network usually weakens the temporal relationship due to over-emphasis on mutation points, the EEG data is divided into multiple temporal modes and they are residually mixed to obtain enhanced time series data. This enables the Transformer network to comprehensively capture the information of different temporal modes and thus better extract time series features.

[0019] (4) In the past, when designing models, the intrinsic connection between different brain regions in decoding and reconstructing visual stimuli was not taken into account. The feature extraction of EEG data caused by visual stimuli mostly remained in the time domain feature extraction, while ignoring the frequency domain information of EEG data. The present invention constructs a dual-stream spatiotemporal convolutional network in the time and frequency domains to better model brain activities, further improving the model's performance in retrieval and reconstruction tasks.

[0020] (5) In view of the fact that previous EEG encoders all learned for a certain modality and ignored the expression relationship between multiple modalities, the present invention can autonomously learn the weight relationship between multiple modalities to maximize the ability to extract effective information from EEG data, and map the image modality, category modality and semantic modality output by the mixed modality encoder to the target CLIP space to align the distribution of the three EEG embedding information with the target modality in the embedding space, thereby realizing the task of reliably reconstructing the image. The image modality information will take into account the category modality information and semantic information during the training process, and can achieve the best results in both retrieval and generation tasks at the same time without the need for targeted training of a single task. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Schematic diagram of the process of the EEG visual stimulation mixed modality decoding method of the present invention; Figure 2 Schematic diagram of the overall architecture of the hybrid modal encoder of the present invention; Figure 3 Schematic diagram of the overall architecture of the lightweight Transformer network of the present invention; Figure 4 Schematic diagram of the network structure of the multi-time scale hybrid convolutional network of the present invention; Figure 5 A schematic diagram of the network structure of a convolutional network with multiple feature convolutional layers of the present invention Figure 6 Schematic diagram of the network structure of the dual-stream spatiotemporal convolutional network in the time-frequency domain of the present invention; Figure 7 A schematic diagram of the network structure of the hybrid mapping network of the present invention; Figure 8 It is a schematic diagram of the network structure of the embedder of the present invention; Fig. 9 Schematic diagram of the network structure of the multimodal decoder of the present invention; Fig.10 Schematic diagram of the overall implementation framework of the visual stimulus reconstruction of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0023] Example 1 like Figure 1 As shown, this embodiment provides an EEG visual stimulation mixed modality decoding method, comprising the following steps: S1: Construct training set and test set based on EEG data and preprocess EEG data; In this embodiment, the original image-EEG dataset adopts the THINGS-EEG dataset, which contains EEG data collected from 10 subjects under the RSVP paradigm; The training set EEG data and the test set EEG data were collected separately. When collecting EEG data, the images were presented to the subjects in a pseudo-random order. Each image was displayed for 100 milliseconds, followed by a 100-millisecond blank screen, in order to reduce the interference caused by other artifacts. The training set includes 1654 different natural image categories. Each image category in the training set contains 10 images of that category. When the subjects viewed the training set images and collected EEG data, each image appeared four times. The subjects viewed a total of 1654×10×4=66160 images, and then collected 66160 EEG data corresponding to the viewed images. For the construction of the training set, each image and the EEG data collected by the subjects when they were visually stimulated were combined into a (target-training data) group of (image, EEG), and a total of 66160 groups of (image, EEG) were obtained. The test set contains 200 other categories that have not appeared in the training set. Each category in the test set is represented by an image. When viewing the test set image category, each image is repeated 80 times, so the subjects viewed a total of 200×80=16,000 test set images. EEG data were collected while viewing the test set images, and a total of 16,000 EEG data were collected. In the construction of the test set, the 16,000 collected EEG data were averaged and pooled to obtain 200 EEG data corresponding to the 200 test images, that is, 200 groups (image, EEG); The EEG data in the original training set of 66160 groups (images, EEG) and the original test set of 200 groups (images, EEG) were divided into time periods from 0 to 1000 milliseconds after the start of the stimulus, and the average value of 200 milliseconds before the stimulus was used for baseline correction to ensure that the collected EEG data was not affected by the environment and varied greatly. The EEG data was downsampled to 250 Hz and multivariate noise normalization was applied to the EEG data to better allow the model to converge during the training process; For each image in the training set and test set, the BLIP2 image-to-text model is used to generate the semantic description information corresponding to each image, and the semantic description information is added to the original training set and the original test set to form the image-category-semantic three-state EEG data training set and test set; S2: Building a mixed-modality encoder; like Figure 2As shown, the EEG data is input into the hybrid modality encoder, and the hybrid modality encoder learns the corresponding image, category, and semantic modality information from the EEG data. The hybrid modality encoder includes a global feature mapping network and a hybrid mapping network; The global feature mapping network includes a lightweight Transformer network, a multi-time scale hybrid convolutional network, a convolutional network with multiple feature convolutional layers, and a dual-stream spatiotemporal convolutional network in the time-frequency domain; like Figure 3 As shown, the lightweight Transformer network includes: downsampling layer, attention layer and convolution layer; In this embodiment, average pooling layers of different sizes and different step lengths are used to form downsampling layers of different scales, and the EEG data is decomposed into time pattern information of different scales through downsampling layers of different scales. The input EEG data is expressed as: , is the batch dimension of the input, is the number of channels that the EEG data consists of, is the timing information of EEG data, Figure 3 In Represents EEG data after K After the downsampling layer K The temporal pattern information of different scales is reshaped into the same shape through MLP and added to integrate into new pattern information , enhance the expressive power of time series synthesis; Use the Embedding module to embed new mode information Mapping to variables , and embed the position information output, using the attention layer to learn the variables The correlation between different tokens is combined to update the variables of , and then a convolutional layer is used to model the time series features inside the variable to obtain efficient time series representation. Finally, the Transformer network outputs the features , specifically expressed as: ; ; ; The average pooling operation of the lightweight Transformer network achieves feature dimensionality reduction by taking the average of the local receptive field, and has dual advantages in terms of time series feature processing: first, the smoothing property of average pooling can effectively suppress the interference of high-frequency noise on time series modeling; second, the layered time representation formed by multi-scale downsampling can provide complementary time series context information for the Transformer network. The self-attention mechanism of the Transformer network dynamically allocates weights to multi-scale time series features to achieve global dependency modeling across time steps, thereby improving the model's ability to parse complex time series patterns.

[0024] After completing the temporal feature modeling, this embodiment introduces a multi-time scale hybrid convolutional network to perform multi-granularity feature refinement. The multi-time scale hybrid convolutional network is based on the local perception characteristics of the convolution operation. The high-level features output by the Transformer are enhanced in detail through a controllable local receptive field, which effectively compensates for the potential information loss in the pooling process, and finally achieves the coordinated optimization of global pattern perception and local feature retention, which helps to enhance the model's learning of the intrinsic temporal features of EEG data. like Figure 4 As shown, in the multi-time scale mixed convolutional network, the output of the lightweight Transformer network is Perform convolution operations to extract different time scales time series features, through flattening and fully connected layers Reshape the time series features into time series feature vectors with the same dimensions , , and connected into a vector group , for the vector group , for each vector , , the correlation between channels is modeled through a 1×1 convolution operation, and the channel descriptor is generated through a sigmoid activation function , , forming a new vector group , specifically expressed as: ; In getting a new vector group Afterwards, the vector group The vectors in the 0th dimension are concatenated to obtain Vector of dimensions , expressed as: ; exist The weight matrix is ​​obtained by the softmax function in the dimension , the generated weight matrix The corresponding input vector , , , perform element-by-element multiplication and add after passing through the global average pooling layer (GAP), and finally output As the output of the multi-time scale mixed convolutional network, it is specifically expressed as: ; ; in, Indicates that the softmax function operates on the vector The operation is performed on the 0th dimension of represents the dot product operation, Representatives in order from 0 dimension to M -1 dimension for extraction operation, is the final output result of the multi-time scale mixed convolutional network, and B represents the batch size; In this embodiment, a convolutional network based on a multi-feature convolutional layer extracts spatial dimension features of the multimodal characteristics of the EEG data, and analyzes the cortical spatial topological information implicit in the EEG data (specifically, the spatial distribution correlation of multi-channel electrodes); The convolution network of the multi-feature convolution layer includes a multi-scale spatial convolution group and a dynamic feature fusion layer. The multi-scale spatial convolution group uses parallel deep convolution layers with different receptive fields to capture the local microtopological associations and global macroscopic distribution patterns of the electrode array respectively. The deep convolution kernel parameters are initialized according to the international 10-20 system electrode coordinates to conform to the neural electrophysiological characteristics; the dynamic feature fusion layer adaptively weights and fuses the multi-scale spatial features through learnable channel attention weights, and finally outputs a cortical space representation tensor with regular dimensions. like Figure 5 As shown, B represents the characteristic The number of batches, T represents the feature Time series length, C represents the feature The number of channels is 1, and three convolutional layers with different convolution kernel sizes are used to extract feature information of different dimensions contained in the global temporal features. Compared with directly inputting three identical unprocessed features into the network, the former represents different expressions of the same information for processing, so it is meaningful to perform feature fusion or feature interaction on them. After constructing the three inputs in the above way, add them together, fuse the multi-scale information, and obtain the channel descriptor representation through global pooling, which corresponds to the dimension increase operation. In the excitation stage, two fully connected layers are also used to model the correlation between channels. The dimension is increased to 3C in the second layer of the full connection, and then divided into three parts on average. The sigmoid function is used to generate their own weight representations. Finally, the information of the three scales is weighted and then added to obtain the output time feature. ; like Figure 6 As shown, this embodiment is based on a dual-stream spatiotemporal convolutional network in the time-frequency domain to fuse the two different information features of EEG data in the time domain and frequency domain, perform a fast Fourier transform on the EEG data to obtain its expression in the complex domain, and obtain the frequency domain features through a ResMLP network modeling , the temporal features of the convolutional network output of the multi-feature convolutional layer and frequency domain characteristics The time domain weight and frequency domain weight are obtained through the sigmoid function, and the time feature Multiply the time weight information to get the weighted output feature , frequency domain characteristics Multiply the frequency domain weight information to get the output feature ,Will and After concatenation, matrix multiplication is performed. Matrix multiplication can effectively mix the features of two different dimensions to obtain mixed time-frequency domain features. .Will Through the convolution kernel of the same scale but different dilation rates After extracting information of the same scale through 1D convolution, The result of merging all branches is , and finally pass Output , expressed as: ; ; ; ; ; ; in, represent The transpose of In this embodiment, subspace features of two different scales, time domain and frequency domain, are fused through matrix multiplication operations. Under the same network width, matrix multiplication realizes two-dimensional nonlinear feature fusion through cross-dimensional paired mapping multiplication operations, so as to achieve efficient hybrid learning of common features in time and frequency domains with less memory overhead, and then resolve the conflict between local features and global features through convolution operations of different scales. Specifically, the size of the convolution kernel represents the size of the features that can capture the receptive field. The convolution operation with a small convolution kernel focuses more on the learning of local features of EEG data and tends to ignore the general contextual expression relationship, while the convolution operation with a large convolution kernel can effectively capture the contextual expression relationship and tends to ignore the details. Therefore, convolution kernels of different sizes are used to form a convolutional network to capture mixed information of different scales. These convolutions can be expanded to increase their receptive field without increasing the convolution kernel size. As for this, the output of the time-frequency dual-domain feature hybrid learning network Reset to Global Feature , because it comprehensively considers the temporal features. Spatial features, frequency domain features and other important features are learned. It is worth noting that the global features are As the image modality information decoded from the EEG data by the encoder, this image modality information is used for the retrieval task and also as the input data for training the diffusion prior model.

[0025] like Figure 7 As shown, based on the hybrid mapping network in the global feature Decode more detailed category modes and semantic modes, global features After passing through a brain modality regression layer (brain regression ridge), it is divided into three paths and simultaneously input into the corresponding embedding device to obtain a preliminary modality feature representation, which is called fuzzy modality feature. Then it is concatenated in the first dimension and input into the multimodal decoder to further decode the fuzzy feature modality to obtain detailed category modality features and semantic modality features, which are used to train the subsequent diffusion prior model; In this embodiment, a category embedder, a brain modality embedder, and a semantic embedder are constructed respectively, and each global feature is calculated for the three required modal information. The distance to the key feature information is calculated, and the clustering is performed in the vicinity of the specific key voxel according to the calculated distance. Token information. When these tokens are aggregated as a group, the shape is B × G × K Key modal features of grouping , indicating that each global feature is divided into groups, each containing tokens, the modal features within the group may show similar characteristics. Based on this, the three required rough modal information can be obtained, namely brain modal information as the main modality, category modal information as the auxiliary modality, and semantic description modal information. Based on these three rough modal information, more accurate category modal information and semantic description modal information will be decoded in the multimodal decoder; In this embodiment, the structure of each embedder is the same, such as Figure 8 As shown, taking the category embedder as an example, the global feature The local patterns of each group are aggregated to obtain the corresponding modal information features. The embedder consists of two main modules: the Token mapping network and the cross-attention layer. Dimensional operation to obtain , u is input into the Token mapping network for feature learning , , Indicates the number of groups, represents the embedding dimension within the group, and then Input cross-attention layer module to decode the coarse category modality information, where Projection as value and key , and randomly initialize a set of learnable tags as queries , the updated query is used as the output embedding. Similarly, the other two rough modal information can be processed in the same way to obtain more refined modal embedding information; In this embodiment, the three rough modal information obtained after the cross-attention layer processing is The dimensions are concatenated to obtain new global rough modal features , it passes it through a cross-attention layer to learn the representation relationship between the three modalities, and finally reshapes it into category modality information and semantic description modality for output.

[0026] like Fig. 9 As shown, Represents three kinds of coarse modal information in the group The concatenated features in the dimension are decoded into detailed category modes through a cross-attention layer and semantic description modality . Then it is sent to the pooling layer for dimensional integration to obtain and , and then pass it through two MLP layers to output the decoded detailed category features and semantic features, which serve as input data for training the diffusion prior model; S3: Construct a diffusion prior model to map the three modal information output by the hybrid modal encoder into the target CLIP space, so that the three modal information output by the hybrid modal encoder can be recognized by the pre-trained generative model; In this embodiment, the diffusion prior model includes a simplified U-Net model. During training, the image modality, category modality, and semantic modality output by the hybrid modality encoder are trained one-to-one with the corresponding target modality to align the modality information output by the hybrid modality encoder with the target information in the CLIP space, so that the modality information output by the model can be directly input into the pre-trained generative model for image generation. The diffusion prior model is trained using a classifier-free guidance method during the training process, which effectively balances the fidelity of the conditional data and the diversity of the generated output. The diffusion prior model is trained from scratch using the mean square error loss, which is specifically expressed as: ; in, represents the mean square error loss, represents the CLIP embedding of the perturbation after a given diffusion time step t, while Represents the diffusion prior model network.

[0027] In this embodiment, after the hybrid multimodal features are processed by diffusion priori, they can be directly used for image reconstruction like the original CLIP embedding vector. Specifically, in order to ensure the reconstruction of high-quality visual stimulation, three modal information are used, namely, the image modality, category modality and semantic modality output by the hybrid modality encoder. The image modality is used as the main modality of the reconstructed image, and IP-Adapters and SDXL-Turbo are used to simultaneously use different modalities to reconstruct visual stimulation. In the generation stage, for the image embedding information of the main modality, the entire IP-Adapter is used to process the image embedding. For the category and semantic description modalities, according to the process of the brain understanding natural things, that is, extracting the category information and then giving further detailed semantic descriptions, the modified IP-Adapters are used, that is, IP-Adapter-Style and IP-Adapter-Layout are used to process the relationship between the two. The category embedding information acts on IP-Adapter-Style to guide the initial generation category direction of the image, and the semantic information embedding acts on IP-Adapter-Layout. When generating, the generated image is fine-tuned according to the semantic description information.

[0028] like Fig.10As shown, the image modality, category modality and semantic modality output by the hybrid modality encoder correspond to the EEG2 image, EEG2 class, and EEG2caption in the figure. The three modal information are contrasted and learned with the target modality. As the target modality of the hybrid modality encoder contrast learning, the target modality is obtained by passing the specific image, category, and semantic description in the image-category-semantic three-state EEG dataset through the Open CLIP model, and the image modality (EEG2 Image in the figure) trained by contrastive learning is used to perform image retrieval and classification tasks. Subsequently, in the generation stage, the three EEG embedding information output by the mixed modality encoder (EEG2 class, EEG2image, EEG2 caption in the figure) is mapped to the corresponding CLIP space through the diffusion prior model to align the distribution of the three EEG embedding information and the target modality in the embedding space, and the image, category, and semantic modal information processed by the diffusion prior are output. The image, category, and semantic modal information processed by the diffusion prior are input into the pre-trained image generation model (SDXL-Turbo and IP-Adapters) for the generation task. In the generation task, the main embedding information of the image modality guides the entire generation direction, and the category information is used to provide the category of image generation during the generation process to ensure that the generated image does not deviate from the real category, while the semantic description information is used to guide the degree of representation in the fine-grained direction during image generation. Based on the above operations, visual stimuli can be decoded with higher accuracy.

[0029] In the process of training the image modal information in this embodiment, the category modal information and semantic information are taken into consideration, so that the output image modal information can fully consider the interoperability with other modalities, and high-quality decoding of the three modal information can be achieved using only one encoder.

[0030] Example 2 This embodiment provides an EEG visual stimulation mixed modality decoding system, which is used to implement the EEG visual stimulation mixed modality decoding method of the above-mentioned embodiment 1, and the system includes: an EEG data acquisition module, a three-state EEG data set construction module, a mixed modality encoder construction module, a target modality construction module, a contrast learning module, an image retrieval classification module, a diffusion prior model construction module, and an image generation module; In this embodiment, the EEG data acquisition module is used to acquire EEG data; In this embodiment, the three-state EEG data set construction module is used to construct the image-category-semantic three-state EEG data set; In this embodiment, the hybrid modality encoder construction module is used to construct a hybrid modality encoder, and the hybrid modality encoder learns corresponding image, category, and semantic modality information from EEG data; In this embodiment, the target modality construction module is used to map the image-category-semantic data in the image-category-semantic three-state EEG data set to the corresponding CLIP space to obtain the target modality; In this embodiment, the contrastive learning module is used to perform contrastive learning between the image, category, and semantic modality information output by the hybrid modality encoder and the target modality; In this embodiment, the image retrieval and classification module is used to perform image retrieval and classification tasks based on image modality information; In this embodiment, the diffusion prior model construction module is used to construct a diffusion prior model, align the image, category, and semantic modality information output by the mixed modality encoder with the target modality in the CLIP space, and output the image, category, and semantic modality information processed by the diffusion prior; In this embodiment, the image generation module is used to input the image, category, and semantic modal information that have been processed by diffusion prior into a pre-trained image generation model to perform a generation task. The modal information that has been processed by diffusion prior is used to provide the degree of representation in the coarse-grained direction during image generation, the category during image generation, and the degree of representation in the fine-grained direction during image generation, so as to complete the reconstruction of visual stimulation.

[0031] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A mixed modality decoding method for EEG visual stimulation, characterized in that: The steps include: Obtain EEG data and construct a three-state EEG dataset of image, category and semantics; Construct a hybrid modality encoder that learns the corresponding image, category, and semantic modality information from EEG data; Map the image-category-semantic data in the image-category-semantic three-state EEG dataset to the corresponding CLIP space to obtain the target modality; Compare the image, category, and semantic modality information output by the hybrid modality encoder with the target modality, and perform image retrieval and classification tasks based on the image modality information; Construct a diffusion prior model to align the image, category, and semantic modality information output by the mixed modality encoder with the target modality in the CLIP space, and output the image, category, and semantic modality information processed by the diffusion prior. The images, categories, and semantic modal information that have been processed with diffusion priors are input into the pre-trained image generation model to perform the generation task. The modal information that has been processed with diffusion priors is used to provide the degree of representation in the coarse-grained direction during image generation, the category during image generation, and the degree of representation in the fine-grained direction during image generation to complete the reconstruction of visual stimulation.

2. The EEG visual stimulation mixed modality decoding method according to claim 1, characterized in that: The hybrid modality encoder includes a global feature mapping network and a hybrid mapping network; The global feature mapping network includes a lightweight Transformer network, a multi-time scale hybrid convolutional network, a convolutional network with multiple feature convolutional layers, and a dual-stream spatiotemporal convolutional network in the time-frequency domain; The lightweight Transformer network extracts the temporal features of the EEG data; The multi-time-scale hybrid convolutional network enhances the details of the temporal features based on the local receptive field and outputs the global temporal features; The convolution network of the multi-feature convolution layer extracts spatial dimension features from the multimodal characteristics of the EEG data, analyzes the cortical spatial topological information of the EEG data, and outputs temporal features; The dual-stream spatiotemporal convolutional network in the time-frequency domain fuses the time-domain and frequency-domain information features of the EEG data and outputs global features; The hybrid mapping network decodes the global features to obtain the category modality and the semantic modality.

3. The EEG visual stimulation mixed modality decoding method according to claim 2, characterized in that: The lightweight Transformer network includes a downsampling layer, an attention layer, and a convolution layer. Downsampling layers of different scales are formed by using average pooling layers of different sizes and different step lengths. The EEG data is decomposed into time pattern information of different scales through downsampling layers of different scales. The time pattern information of different scales is reshaped into the same shape through MLP and added to integrate into new pattern information. , based on the Embedding module, the new mode information Mapping to variables , learning variables through attention layers The correlation between different tokens is obtained by outputting the time series features extracted by the lightweight Transformer network through the convolutional layer .

4. The EEG visual stimulation mixed modality decoding method according to claim 2, characterized in that: The multi-time-scale hybrid convolutional network performs convolution operations on the time series features output by the lightweight Transformer network, extracts time series features of different time scales, reshapes them into time series feature vectors with the same dimensions through flattening operations and fully connected layers, and connects them into vector groups. ; For vector group Each vector of the channel is generated through convolution operation and sigmoid activation function, and the descriptors are combined into a new vector group , expressed as: ； Group vectors The vectors in the concatenation are Vector of dimensions ; vector exist The weight matrix is ​​obtained by the softmax function in the dimension , based on the weight matrix and the corresponding input vector, the global temporal features of the multi-time scale hybrid convolutional network output are obtained , expressed as: ； ； in, Indicates that the softmax function operates on the vector The operation is performed on the 0th dimension of represents the dot product operation, Representatives in order from 0 dimension to M -1 dimension for extraction operations.

5. The EEG visual stimulation mixed modality decoding method according to claim 2, characterized in that: The convolution network of the multi-feature convolution layer includes a multi-scale spatial convolution group and a dynamic feature fusion layer. The multi-scale spatial convolution group uses parallel deep convolution layers with different receptive fields to extract local microtopological associations and global macroscopic distribution patterns of the electrode array respectively. The dynamic feature fusion layer adaptively weights and fuses multi-scale spatial features through learnable channel attention weights to output temporal features.

6. The EEG visual stimulation mixed modality decoding method according to claim 5, characterized in that: In the multi-scale spatial convolution group, feature information of different dimensions of global temporal features are extracted through multiple convolution layers with different convolution kernel sizes, and the feature information of different dimensions is added, multi-scale information is fused, and channel descriptor representation is obtained through global pooling. The correlation between channels is modeled based on two fully connected layers. After the second fully connected layer is upgraded in dimension, it is evenly divided into multiple parts. In the dynamic feature fusion layer, the sigmoid function is used to generate their own weighted representations, and the information of different scales is weightedly added to output the temporal features.

7. The EEG visual stimulation mixed modality decoding method according to claim 2, characterized in that: The dual-stream spatiotemporal convolutional network in the time-frequency domain obtains frequency domain features through fast Fourier transform and ResMLP network modeling, and the time features and frequency domain features output by the convolutional network of the multi-feature convolutional layer are obtained through the sigmoid function to obtain the time domain weights and frequency domain weights, and the corresponding output features are weighted. After splicing, matrix multiplication operation is performed to obtain mixed time-frequency domain features, and global features are obtained based on the mixed time-frequency domain features, which are specifically expressed as: ； ； ； ； ； ； in, Represents time characteristics, represents the frequency domain characteristics, represents a linear operation, represents the sigmoid function, represents the average pooling operation, represents the feature concatenation operation, represents the mixed time-frequency domain features, Representation characteristics The transpose of represents the convolution operation, represents the convolution kernel size, Representative The dilation rate of the convolution operation, Represents a max pooling operation.

8. The EEG visual stimulation mixed modality decoding method according to claim 2, characterized in that: The hybrid mapping network includes a brain modality regression layer, a category embedder, a brain modality embedder, a semantic embedder, and a multimodal decoder; After passing the global features through the brain modality regression layer, they are input into the category embedder, brain modality embedder, and semantic embedder respectively to obtain the corresponding preliminary modality feature representation; Each preliminary modal feature representation is concatenated in the first dimension and input into the multimodal decoder for decoding to obtain the category modal feature and semantic modal feature.

9. The EEG visual stimulation mixed modality decoding method according to claim 1, characterized in that: The diffusion prior model is trained using mean squared error loss.

10. An EEG visual stimulus mixed modality decoding system, characterized in that: Used to implement the EEG visual stimulus mixed modality decoding method according to any one of claims 1 to 9, the system comprises: an EEG data acquisition module, a three-state EEG data set construction module, a mixed modality encoder construction module, a target modality construction module, a contrast learning module, an image retrieval classification module, a diffusion prior model construction module, and an image generation module; The EEG data acquisition module is used to acquire EEG data; The three-state EEG data set construction module is used to construct an image-category-semantic three-state EEG data set; The hybrid modality encoder construction module is used to construct a hybrid modality encoder, and the hybrid modality encoder learns corresponding image, category, and semantic modality information from EEG data; The target modality construction module is used to map the image-category-semantic data in the image-category-semantic three-state EEG data set to the corresponding CLIP space to obtain the target modality; The contrastive learning module is used to compare the image, category, and semantic modality information output by the hybrid modality encoder with the target modality; The image retrieval and classification module is used to perform image retrieval and classification tasks based on image modality information; The diffusion prior model construction module is used to construct a diffusion prior model, align the image, category, and semantic modality information output by the hybrid modality encoder with the target modality in the CLIP space, and output the image, category, and semantic modality information processed by the diffusion prior; The image generation module is used to input the image, category, and semantic modal information that have been processed by diffusion prior into a pre-trained image generation model to perform a generation task. The modal information that has been processed by diffusion prior is used to provide the degree of representation in the coarse-grained direction during image generation, the category during image generation, and the degree of representation in the fine-grained direction during image generation, so as to complete the reconstruction of visual stimulation.

Images