Visual content generation method based on electroencephalogram signals, computer readable medium

Abstract

The application relates to the cross field of brain-computer interface and computer vision, and provides a visual content generation method based on electroencephalogram signals, and a computer readable medium, which comprises the following steps: acquiring electroencephalogram signals; performing decoupling feature extraction on the electroencephalogram signals to obtain first features for representing high-level semantic categories and second features for representing local visual details; and injecting the first features into a generation model through a first injection mechanism and injecting the second features into the generation model through a second injection mechanism to guide the generation model to generate visual images corresponding to the electroencephalogram signals; wherein the first injection mechanism is different from the second injection mechanism. According to the application, the global semantic information and the local visual detail information in the electroencephalogram signals are separately processed through decoupling feature extraction and a double-branch conditional alignment architecture, and are respectively used in the generation model in the form of textual guidance and continuous feature injection, so that the semantic accuracy and visual fidelity of the reconstructed images are significantly improved.

Description

Technical Field

[0001] This application relates to the intersection of brain-computer interfaces and computer vision, and in particular to methods for generating visual content based on electroencephalogram (EEG) signals and computer-readable media. Background Technology

[0002] Visual perception reconstruction, which aims to deduce the original visual stimuli from brain neural activity, is a key topic in the field of brain-computer interfaces. Electroencephalography (EEG), due to its advantages of being non-invasive, portable, and having high temporal resolution, has become an important signal source for visual reconstruction.

[0003] In recent years, deep learning-based EEG visual reconstruction methods have made some progress. For example, generative models such as generative adversarial networks (GANs) or variational autoencoders (VAEs) can be used to directly map EEG signals to visual images.

[0004] Because EEG signals contain rich, multi-dimensional information, including global semantic information related to the category of visual stimuli (e.g., neural representations that identify whether the target is "airplane" or "panda"), as well as fine-grained visual information related to local image details (e.g., neural responses to specific textures, edges, or color distributions); however, existing EEG-based visual reconstruction methods typically employ a single path to extract features or use only a single type of conditionally guided generative model, resulting in generated images that struggle to simultaneously maintain the correct high-level semantic category and rich local visual details, leading to poor reconstruction fidelity and semantic consistency. Summary of the Invention

[0005] This application provides a method for generating visual content based on electroencephalogram (EEG) signals and a computer-readable medium, which solves the technical problem of contradiction between image semantics and detail reconstruction caused by single-condition guidance in the prior art.

[0006] To achieve the above objectives, this application adopts the following technical solution: Firstly, a method for generating visual content based on electroencephalogram (EEG) signals is provided, including: acquiring EEG signals; The EEG signal is decoupled and feature extracted using a dual-branch encoder to obtain a first feature and a second feature. The first feature is obtained by extracting long-sequence features from the EEG signal using a sequence modeling network and is used to represent high-level semantic categories. The second feature is obtained by extracting local features from the EEG signal using a multi-scale convolutional network and is used to represent local visual details. The first feature is converted into a text prompt and then injected into the text encoder of the generative model to obtain the text condition; The second feature is used as an image condition and injected into the U-Net network of the generative model through the condition injection module to calculate the potential diffusion loss of the generative model. The generation model is used to generate a visual image corresponding to the EEG signal based on the text conditions and the image conditions. Using the gradient generated by the potential diffusion loss, the low-rank adaptation parameters in the multi-scale convolutional network that extracts the second feature and the generative model are jointly updated end-to-end.

[0007] Based on the above technical solution, in the visual content generation method based on EEG signals provided in this application, the EEG signals are decoupled into semantic features and visual features by a dual-branch encoder, and the latent diffusion model is guided to generate visual images by textual injection and image conditional injection, respectively. At the same time, the visual feature extraction network can actively adapt to the latent space distribution of the generation model through end-to-end joint optimization, thereby significantly improving the fidelity and semantic consistency of the image reconstructed from the EEG signals.

[0008] In conjunction with the first aspect above, in one possible implementation, the sequence modeling network is a state-space model network; Long-sequence feature extraction of the EEG signals is performed using a sequence modeling network, including: The EEG signal is subjected to time-domain feature extraction and frequency-domain feature extraction to obtain time-domain features and frequency-domain features; The time-domain features and the frequency-domain features are fused and input into the state-space model network for long sequence modeling.

[0009] By fusing time-frequency dual-domain features, we can more comprehensively deconstruct the complex physiological features of EEG signals, make up for the information loss of single time-domain modeling, and utilize the linear complexity advantage of state-space model to efficiently process long sequences.

[0010] In conjunction with the first aspect described above, in one possible implementation, the time-domain features and the frequency-domain features are fused, including: The time-domain features and the frequency-domain features are added element-wise to obtain the fused features; Learnable positional codes are superimposed on the fused features.

[0011] In conjunction with the first aspect above, in one possible implementation, the step of extracting long-sequence features from the EEG signal using a sequence modeling network further includes: Spatial attention pooling is performed on the features output by the state space model network to weight and aggregate features from different spatial locations, thus obtaining the first feature.

[0012] In conjunction with the first aspect above, in one possible implementation, the multi-scale convolutional network includes at least two levels of one-dimensional convolutional layers, and the kernel size of each convolutional layer is different; Each level of the one-dimensional convolutional layer is followed by a batch normalization layer and a max pooling layer in sequence; The features output from the last convolutional layer are flattened and projected through a fully connected layer to form a fixed-dimensional visual feature vector, which is then used as the second feature.

[0013] In conjunction with the first aspect described above, in one possible implementation, the method further includes: Alignment optimization is performed on the second feature using the target image features; The alignment optimization of the second feature using the target image features includes: Image features of the real image corresponding to the EEG signal are extracted using a pre-trained image encoder. Calculate the cosine embedding loss between the second feature and the image feature; By minimizing the cosine embedding loss, the second feature gradually approaches the image feature.

[0014] Alignment optimization gradually brings the extracted visual features closer to the target image features, enhancing their ability to represent visual content and providing more effective conditions for subsequent diffusion generation.

[0015] In conjunction with the first aspect above, in one possible implementation, the generative model is a potential diffusion model; The text encoder that converts the first feature into a text prompt and then injects it into the generative model includes: Construct a text prototype retrieval table, which stores text prototype features of multiple target categories; Calculate the cosine similarity between the first feature and each text prototype feature in the text prototype retrieval table, and select the category label with the highest similarity; The category labels are constructed into standardized natural language prompts, which are then input into a frozen text encoder to obtain the text conditions. The step of injecting the second feature as an image condition into the U-Net network of the generative model through a condition injection module includes: The second feature is input into the IP-Adapter module, which converts it into a key-value pair compatible with the cross-attention layer in the U-Net network and injects it into the U-Net network.

[0016] In conjunction with the first aspect above, in one possible implementation, the low-rank adaptation parameters include a first weight matrix and a second weight matrix; The dimension of the first weight matrix is The dimension of the second weight matrix is , in, and These are the output and input dimensions of the pre-trained weight matrix, respectively, and r is a preset rank value, which ranges from 4 to 16. The end-to-end joint update includes: Freeze the basic backbone parameters of the variational autoencoder, text encoder, and U-Net network in the generative model; Only update the low-rank adaptation parameters and the parameters of the multi-scale convolutional network that extracts the second feature; During backpropagation, the gradient generated by the potential diffusion loss passes sequentially through the U-Net network and the conditional injection module, and is finally propagated back to the multi-scale convolutional network.

[0017] Secondly, this application provides a visual content generation system based on electroencephalogram (EEG) signals, including: Signal acquisition module, used to acquire electroencephalogram (EEG) signals; The dual-branch encoder includes a semantic branch encoder and a visual branch encoder. The semantic branch encoder includes a sequence modeling network for extracting long-sequence features from the EEG signal to obtain a first feature for representing a high-level semantic category. The visual branch encoder includes a multi-scale convolutional network for extracting local features from the EEG signal to obtain a second feature for representing local visual details. The text condition injection module is used to convert the first feature into a text prompt and then inject it into the text encoder of the generative model to obtain the text condition; A visual conditional injection module is used to inject the second feature as an image condition into the U-Net network of the generative model; The image generation module is used to generate a visual image corresponding to the EEG signal based on the text conditions and the image conditions; The joint optimization module is used to calculate the potential diffusion loss of the generative model and use the gradient generated by the potential diffusion loss to perform end-to-end joint updates of the low-rank adaptation parameters in the visual branch encoder and the generative model.

[0018] Thirdly, this application provides a computer-readable storage medium storing instructions that, when executed on a visual content generation system based on electroencephalogram (EEG) signals, cause the system to perform the methods described in the first aspect and any possible implementation thereof.

[0019] This application provides a visual content generation method based on electroencephalogram (EEG) signals and a computer-readable medium. A dual-branch encoder decouples EEG signals into semantic and visual features, which are then synergistically applied to a latent diffusion model via textual guidance and image conditional injection, respectively. The semantic branch performs long-sequence modeling through time-frequency dual-domain feature fusion and a state-space model, and extracts stable global category priors using spatial attention pooling. The visual branch extracts fine-grained visual features through a multi-scale convolutional network and enhances its representational ability of visual content through alignment optimization. Both branches are injected into the U-Net network of the latent diffusion model through different injection mechanisms, jointly constraining the generation process at both macro and micro levels. Simultaneously, an end-to-end joint optimization strategy utilizes a gradient-through conditional injection module based on the latent diffusion loss, which is fed back to the visual branch encoder. Combined with low-rank adaptation parameters, this enables efficient model fine-tuning, allowing the visual feature extraction network to actively adapt to the latent space distribution of the generation model, thereby significantly improving the semantic accuracy and visual fidelity of the reconstructed images. Attached Figure Description

[0020] Figure 1 A schematic diagram of an image generation framework based on decoupled feature extraction and dual-branch conditional alignment provided in an embodiment of this application; Figure 2 A flowchart illustrating a visual content generation method based on electroencephalogram (EEG) signals provided in this application embodiment; Figure 3 A semantic branching architecture diagram provided for embodiments of this application; Figure 4 A visual branch architecture diagram provided for embodiments of this application; Figure 5 Comparison images of passenger planes, pumpkin lanterns, and giant pandas provided for embodiments of this application; Figure 6 Comparison images of the piano, daisy, tent, golf ball, camera, and tropical fish provided in the embodiments of this application; Figure 7 A visualization comparison chart of semantic branch feature t-SNE provided for embodiments of this application. Detailed Implementation In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0021] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0022] Example 1: like Figure 1 and Figure 2 As shown, this embodiment provides a method for generating visual content based on electroencephalogram (EEG) signals. The overall process of this method aims to solve the mapping problem from EEG signals to visual images, and its core lies in decoupling and independently injecting the conditional information generated under guidance. The method includes the following steps.

[0023] Step S100: Acquire EEG signals.

[0024] Among them, electroencephalogram (EEG) signals are electrophysiological signals collected from the subject's scalp using an EEG device, which record the electrical activity of neuronal populations in the cerebral cortex. In visual perception tasks, when a subject views a specific visual stimulus (such as an object or scene image), their brain produces a specific EEG response pattern associated with that stimulus.

[0025] It should be noted that the acquired EEG signals can be raw multi-channel time series data or data that has undergone preliminary preprocessing (such as filtering and noise reduction).

[0026] Step S200: Decouple the EEG signal and extract features to obtain a first feature for representing high-level semantic categories and a second feature for representing local visual details.

[0027] The first feature is obtained by extracting long-sequence features from EEG signals through a sequence modeling network, and is used to represent high-level semantic categories; the second feature is obtained by extracting local features from EEG signals through a multi-scale convolutional network, and is used to represent local visual details.

[0028] Decoupled feature extraction refers to extracting two functionally complementary feature representations from the same EEG signal through different processing paths.

[0029] The first feature aims to capture stable patterns in EEG signals that are associated with macroscopic category information of visual stimuli, such as neural representations that identify whether a target is an airplane or a panda.

[0030] The second feature aims to capture fine patterns in EEG signals that relate to local, transient details of visual stimuli, such as neural responses to specific textures, edges, or color distributions in a stimulus image.

[0031] This decoupled design allows the subsequent generative model to simultaneously obtain macroscopic semantic constraints and microscopic detail guidance, thereby overcoming the limitations of single feature guidance.

[0032] Step S300: After converting the first feature into a text prompt, inject it into the text encoder of the generative model to obtain the text condition.

[0033] It should be noted that this step converts the high-level semantic category information represented by the first feature into textual conditions that the generative model can understand.

[0034] For example, the most matching category label is first retrieved from the first feature. Then, this category label is constructed into a standardized natural language cue word (such as "A realistic photo of a [class]") and input into a frozen text encoder to obtain a text conditional embedding. This method of textualizing and re-injecting semantic features can provide stable and explicit macro-level category constraints for the generative model.

[0035] In step S400, the second feature is used as an image condition and injected into the U-Net network of the generative model through the condition injection module.

[0036] This step injects the local visual detail information represented by the second feature into the generative model in the form of a continuous vector.

[0037] For example, the second feature is directly input into a conditional injection module (such as an IP-Adapter module), which converts the visual features into key-value pairs compatible with the cross-attention layers in the U-Net network, thereby injecting fine-grained visual detail information into the generation process.

[0038] Step S500: The generation model generates a visual image corresponding to the EEG signal based on text conditions and image conditions.

[0039] It should be noted that during the generation stage, the latent diffusion model, guided by dual conditions, iteratively denoises starting from pure Gaussian noise. The text condition macroscopically constrains the semantic category of the generated image, while the image condition microscopically supplements the local visual texture and color details derived from specific EEG signals. Both are processed in parallel within the attention layer of U-Net, ultimately generating semantically correct and visually faithful images.

[0040] Step S600: Calculate the potential diffusion loss of the generative model, and use the gradient generated by the potential diffusion loss to perform end-to-end joint updates on the low-rank adaptation parameters in the multi-scale convolutional network for extracting the second feature and the generative model.

[0041] It should be noted that the latent diffusion loss measures the accuracy of the model in predicting added noise in the latent space. During the backpropagation phase, the gradient generated by this loss is not only used to update the low-rank fitting parameters in U-Net, but also penetrates the conditional injection module and is directly propagated back to the multi-scale convolutional network that extracts the second feature, forming an end-to-end joint optimization loop.

[0042] Through the steps described above, the method provided in this embodiment achieves end-to-end generation from EEG signals to visual images. Its technical advantages are reflected in the following aspects: by using a dual-branch encoder, the complex EEG-image mapping task is decomposed into two relatively independent sub-tasks: semantic parsing and visual detail perception, reducing the learning difficulty; through independent dual-path injection, the generative model can be subject to accurate category constraints at the macroscopic level and obtain specific detail supplementation at the microscopic level; through end-to-end joint optimization, the visual feature extraction network can actively adapt to the latent space distribution of the generative model, thereby significantly improving the semantic accuracy and visual fidelity of the reconstructed image.

[0043] Example 2: Please see Figure 3 As shown, based on Example 1, this example will elaborate on the extraction process of the first feature. Specifically, the first feature is obtained by extracting long-sequence features from EEG signals using a sequence modeling network. Here, the sequence modeling network refers to a neural network capable of processing signal sequences with long-range time dependencies. Its function is to capture global, stable pattern information spanning a long time window from continuous, high-dimensional EEG time sequences. This pattern information is usually associated with high-level semantic categories of visual stimuli (such as airplanes or pandas). For example, this network can be a variant of a recurrent neural network (RNN), such as a long short-term memory network (LSTM) or a gated recurrent unit (GRU), or it can be a state-space model (SSM) network that has performed well in long-sequence modeling tasks in recent years.

[0044] In this embodiment, a state-space model network is preferably used as the sequence modeling network because it has the advantage of linear time complexity and can process ultra-long sequence data such as EEG signals more efficiently.

[0045] In some implementations, long-sequence feature extraction is performed on EEG signals, including: The EEG signals are subjected to time-domain and frequency-domain feature extraction, and the obtained time-domain and frequency-domain features are fused and then input into the state-space model network.

[0046] This step aims to construct a richer representation of the input from the raw EEG signals. EEG signals are time-domain signals, recording voltage changes over time. However, when the brain processes visual information, neural oscillations at different frequencies (such as alpha, beta, and gamma waves) carry different functional information. Therefore, analyzing only in the time domain may result in the loss of important frequency-domain information.

[0047] This embodiment uses a parallel two-branch structure for feature extraction: one branch directly performs a linear projection of the EEG signal time series to extract its temporal morphological features, thus obtaining the temporal features. Another branch performs a Fast Fourier Transform (FFT) on the same signal segment, transforming it to the frequency domain and extracting the energy distribution characteristics of a specific frequency band to obtain the frequency domain features. Subsequently, the feature vectors obtained from these two branches are added element-wise to achieve deep fusion of time-domain and frequency-domain information. To preserve the absolute temporal position information of the signal in subsequent long-sequence modeling, learnable positional codes are also superimposed on the fused features. This process can be represented as ; Through a customized embedding mechanism, the input EEG data is re-encoded into a high-dimensional token sequence, thereby effectively compensating for the information loss caused by single temporal modeling at the underlying physical level.

[0048] After the fused feature sequences are input into the state-space model network, the network models the features within the entire time window through its internal state transition mechanism, extracting global long-range temporal dependencies. However, considering that the signal activity recorded by different scalp electrodes (i.e., spatial locations) varies significantly when the human brain processes visual stimuli, signals from certain brain regions (such as the occipital visual cortex) may be more sensitive to visual information, while signals from other regions may mainly be background noise.

[0049] In order to adaptively focus on the core brain region features most relevant to visual perception and suppress irrelevant background noise, this embodiment further includes: performing spatial attention pooling on the features output by the state space model network to weighted aggregate features from different spatial locations.

[0050] The specific working principle of the spatial attention pooling mechanism is as follows: the feature map output by the state space model network usually contains multiple spatial locations (corresponding to different electrode channels) and feature vectors at each location.

[0051] This mechanism calculates an attention weight for each spatial location, reflecting the importance of that location's features to the final semantic representation. The calculation typically involves a small neural network that takes the feature vector at each location as input and outputs a scalar weight value. Then, the feature vectors at all locations are weighted and summed according to their corresponding attention weights, ultimately resulting in an aggregated, fixed-dimensional global semantic feature vector.

[0052] This process can be understood as the model learning to focus on which brain regions' signals and assign them higher weights, while ignoring brain regions with weak signals or those irrelevant to the task.

[0053] For example, when identifying aircraft categories, the model might focus more on brain regions associated with shape and motion perception, while suppressing brain regions associated with emotion or language processing. Through this weighted aggregation, the resulting first feature (i.e., the global semantic feature) has a higher signal-to-noise ratio and stronger class discriminative power.

[0054] To verify the effectiveness of the spatial attention pooling mechanism, a comparison scale can be designed. For example, a comparison scale can use simple global average pooling, that is, directly averaging the feature vectors of all spatial locations output by the state space model network without weighting.

[0055] Experiments are expected to show that spatial attention pooling outperforms simple average pooling in subsequent semantic classification tasks (such as Top-1 accuracy) and final image generation quality (such as generation accuracy GA). This is because average pooling treats all brain regions equally and cannot suppress noise, while attention pooling can adaptively focus on key information, thereby extracting purer and more discriminative semantic features.

[0056] This embodiment constructs a complete semantic feature extraction chain by combining time-frequency dual-domain input, long sequence modeling in a state-space model, and spatial attention pooling, providing a stable and accurate high-level semantic category prior for the generative model. After the aforementioned spatial attention pooling, 768-dimensional global semantic features are obtained. Further optimization and supervision are achieved through a contrastive learning paradigm. Specifically, a frozen pre-trained CLIP text encoder is used to extract constructed cue words (such as "A realistic photo of a [class]") as the real text embedding Ti. During the training phase, the model is trained using a classification function. Calculating EEG semantic features With text embedding space The similarity between the values ​​is used to output the predicted classification distribution. This process can be represented as follows: ; Subsequently, the model employs a simplified cross-entropy loss as the optimization objective for contrastive learning. Backpropagation can be performed, and this process can be represented as follows: Where CrossEntropy is the cross-entropy calculation function, Y_s is the predicted probability distribution vector of the model output, and Y_s' is the true label distribution vector; This cross-modal contrastive learning mechanism not only enhances the class discriminative power of EEG semantic features, but also ensures that EEG representations remain consistent with the corresponding text semantics in the shared representation space, thereby providing more reliable global semantic guidance for subsequent category retrieval and conditional injection.

[0057] Example 3: like Figure 4 As shown, based on Example 1, this example will elaborate on the extraction process of the second feature.

[0058] Specifically, the second feature is obtained by extracting local features from EEG signals using a multi-scale convolutional network. Here, a multi-scale convolutional network refers to a neural network composed of convolutional layers with kernels of different sizes. Its core function is to capture fine-grained pattern information related to local, instantaneous details of visual stimuli from continuous EEG time series, such as specific waveform morphology, edge responses, or short-term energy fluctuations. This information typically corresponds to specific textures, color distributions, or neural responses at object edges in the stimulus image. Unlike the sequence modeling network used to extract global semantic features in Example 2, the multi-scale convolutional network excels at extracting local features of the signal, and the two complement each other functionally.

[0059] In this embodiment, a one-dimensional convolutional network (1D-CNN) is preferably used as a multi-scale convolutional network because it can directly process time series data and effectively extract local features at different time scales through convolutional kernels of different sizes.

[0060] The multi-scale convolutional network comprises at least two levels of convolutional layers, with each layer having a different kernel size. By utilizing receptive fields of varying sizes, it extracts local features of EEG signals from multiple scales.

[0061] In a preferred implementation, the network can consist of three convolutional layers with kernel sizes of 15, 7, and 3, respectively.

[0062] The first stage uses a larger convolutional kernel (such as 15), which can capture the overall shape and trend of the waveform within a relatively long window. The second stage uses medium-sized convolutional kernels (such as 7) to extract local patterns at medium time scales; The third stage, using smaller convolutional kernels (such as 3), focuses on extracting very fine, instantaneous waveform details.

[0063] This multi-scale design, from large to small, can systematically filter background EEG noise and accurately extract multi-level local features related to visual perception. Each convolutional layer is typically followed by batch normalization and max pooling operations. Batch normalization helps stabilize the training process and accelerate convergence; max pooling, on the other hand, reduces the dimensionality of the feature map while preserving the most salient features, thus reducing computational cost.

[0064] After multiple convolutional and pooling processes, the resulting feature map is flattened and projected through a fully connected layer into a fixed-dimensional visual feature vector, such as 1024-dimensional. This vector is the second feature extracted in this embodiment, which condenses key information related to local visual details in the EEG signal.

[0065] To ensure that the extracted second feature can accurately represent the visual details of the image, this embodiment further includes: aligning and optimizing the second feature using the target image features.

[0066] Here, target image features refer to the feature representation of the original visual stimulus image (i.e., the real image) corresponding to the EEG signal. The purpose of alignment optimization is to guide the visual feature vector extracted from the EEG signal to gradually approach the representation of the real image in the feature space through supervised learning, thereby enhancing its ability to represent visual content.

[0067] For example, a pre-trained, powerful image feature extractor (such as the CLIP image encoder) can be used to extract feature vectors from real images. This is applicable to images containing real visual stimuli. Extracting the true image embedding representation using the frozen CLIP image encoder. This process can be represented as In the formula, Image features extracted by the CLIP encoder. This indicates the CLIP image encoder. The input image.

[0068] During training, the second feature vector extracted from the EEG signal is calculated. Compared with real image feature vectors Cosine embedding loss between This process can be represented as Where N represents the number of training samples; and They represent the first Visual feature vectors of each sample and feature vectors of the real image; For matching labels (when both are matching positive sample pairs) Otherwise, it is -1.

[0069] By minimizing this loss, multi-scale convolutional networks can be forced to learn EEG feature representations that can map to a space similar to real image features. This alignment optimization mechanism ensures that the second feature is no longer merely an abstract encoding of EEG signals, but is "anchored" to specific visual content, thus providing more accurate and effective visual conditional guidance for subsequent image generation models.

[0070] To verify the effectiveness of the above technical solutions, comparative examples can be designed. For example, one comparative example can use a single-scale convolutional network, where all convolutional layers use convolutional kernels of the same size (e.g., all kernels with a size of 3). Experiments are expected to show that the method using a multi-scale convolutional network is superior to the method using a single-scale convolutional network in terms of the diversity of extracted features and the fidelity of local details in the final generated image. This is because a single-scale network can only extract features from one time scale, which may miss important information at other scales. Another comparative example can retain the multi-scale convolutional network but without feature alignment optimization, i.e., without using target image features for supervision. Experiments are expected to show that the method without alignment optimization may not be able to effectively represent the visual content of the extracted second features, resulting in a weakened image conditional guidance effect in the subsequent generation stage, and the generated image may be blurry in local details or mismatched with semantics. This embodiment, by combining multi-scale convolutional networks and feature alignment optimization, constructs a complete visual feature extraction and optimization chain, providing fine-grained, high-fidelity guidance for local visual details in the generative model.

[0071] Example 4: Based on Example 1, this example will elaborate on the condition injection and optimization process of the generated model.

[0072] Specifically, the generative model is a latent diffusion model; the first injection mechanism is to convert the first feature into a text prompt and then inject it into the text encoder of the latent diffusion model; the second injection mechanism is to use the second feature as an image condition and inject it into the U-Net network of the latent diffusion model through the conditional injection module.

[0073] Among them, the latent diffusion model refers to a type of generative model based on the diffusion process, which mainly generates data by gradually adding noise to the data and then learning to reverse this process.

[0074] This embodiment preferably uses Stable Diffusion 2.1 as the latent diffusion model because it performs well in the field of natural image generation. A text encoder is a model that can convert text descriptions into high-dimensional feature vectors, such as the CLIP text encoder. A conditional injection module is a module that can inject external conditional features into the generation model, such as the IP-Adapter module. The U-Net network is the core network structure in the latent diffusion model responsible for denoising and prediction.

[0075] For example, the implementation process of the first injection mechanism is as follows: First, the first feature (i.e., the global semantic feature) extracted from Example 2 is fed into a pre-built retrieval table; this retrieval table stores text prototype features for all target categories, which are obtained by encoding standard text prompts (such as “A realistic photo of a [class]”) using a frozen CLIP text encoder.

[0076] Then, by calculating the cosine similarity between the first feature and the prototype feature of each category in the retrieval table, the category label with the highest similarity can be retrieved.

[0077] Finally, the category label is automatically constructed into standardized natural language prompts (e.g., "A realistic photo of a panda") and input into the frozen CLIP text encoder to obtain text conditional embeddings. This method of textualizing and re-injecting semantic features provides stable and explicit macro-category constraints for the generative model.

[0078] The implementation process of the second injection mechanism is as follows: The second feature extracted from Example 3 (i.e., the visual feature) is a fixed-dimensional continuous vector (e.g., 1024-dimensional). This vector is directly input into the IP-Adapter module. The IP-Adapter module converts the visual feature into key-value pairs compatible with the cross-attention layers in the U-Net network through a lightweight adapter network. During the forward propagation of U-Net, these visual key-value pairs are processed in parallel with the text key-value pairs, jointly participating in attention calculation, thereby injecting local visual detail information into the generation process.

[0079] This approach, which directly injects visual features as continuous conditions, can provide fine-grained guidance on color, texture, and structure for generative models.

[0080] Through the aforementioned dual-path independent injection mechanism, semantic information and visual detail information can be applied to the generation process in a manner best suited to their characteristics.

[0081] Semantic information is injected through textualization, achieving stability aligned with the pre-trained text conditions; visual information is injected through continuous features, preserving its original fine-grained information. Both are processed in parallel within the attention layer of U-Net, avoiding mutual interference or information confusion during the injection stage, thus achieving effective synergy between macroscopic semantic constraints and microscopic detail guidance.

[0082] For example, given the latent variables, the generated query matrix Q, text key-value pairs ( ) and visual key-value pairs ( The attention output guided by dual conditions can be represented as: Where Output is the multimodal attention output vector. and These are the text key matrix and the visual key matrix, respectively. and Let be the text value matrix and the visual value matrix, respectively; d be the embedding dimension; λ be the modality fusion weight coefficient; and Softmax be the normalization function. Experimental results show that this dual-branch injection mechanism outperforms methods using only single-condition injection in terms of generation accuracy (GA) and Fraser inception distance (FID).

[0083] During the bootstrapping process, this embodiment further includes: Calculate the potential diffusion loss of the generative model; The gradient generated by the potential diffusion loss is used to jointly update the low-rank adaptation parameters in the network that extracts the second feature and the generative model.

[0084] The latent diffusion loss refers to the core loss function used to train the latent diffusion model, which measures the accuracy of the model in predicting added noise in the latent space. Low-rank adaptation parameters are low-rank matrix parameters, such as LoRA parameters, introduced into the weight matrix of the pre-trained model for fine-tuning. Joint update refers to using the gradient generated by the same loss function to update the parameters of multiple network modules simultaneously.

[0085] The optimization steps for the guided generation process are as follows: During the training phase, for the input EEG signal, the first and second features are first extracted by the dual-branch encoder of Examples 2 and 3, and then injected into the potential diffusion model through the above-mentioned injection mechanism.

[0086] The latent diffusion model performs forward diffusion and backward denoising processes in the latent space and calculates the latent diffusion loss. This loss function is defined as the mean square error between the predicted noise and the actual noise, and can be expressed as: In the formula, Let be the loss function of the potential diffusion model. For the expectation, Let be the initial latent variable, ε be the noise following a standard normal distribution, and t be the time step. For a noise prediction network with parameter θ, Let be the potential state at time step t. For text conditional input, Input for visual conditions.

[0087] Next, the gradient of this loss with respect to the model parameters is calculated using the backpropagation algorithm. Crucially, these gradients are not only used to update the low-rank adaptation parameters (LoRA's A and B matrices) in the U-Net network, but also penetrate the IP-Adapter module, directly feeding back to the multi-scale convolutional network (i.e., the visual branch encoder) that extracts the second feature. Therefore, the parameters of the visual branch encoder are dynamically adjusted based on the feedback from the generation task, allowing its extracted features to better adapt to the latent space distribution of the diffusion model.

[0088] By employing an end-to-end joint optimization strategy, the barrier of traditional multi-stage isolated training is broken down. In traditional methods, the feature extraction network and the generative model are usually trained separately, which may result in the feature extraction network failing to generate the feature representation best suited to the generative model.

[0089] This embodiment enables the visual branch network to actively evolve through gradient backpropagation to match the internal representation requirements of the generative model. This not only enhances the representational ability of the second feature for visual content but also achieves precise alignment of cross-modal mapping from EEG signals to visual images.

[0090] Example 5: To further illustrate the application effect of the technical solution of this application on a real dataset, the following describes a specific application process of this method using the publicly available EEGCVPR40 dataset as an example. This dataset contains EEG signals collected from six subjects while viewing visual stimuli from 40 categories. Each category contains 50 images, with a single image presentation time of 0.5 seconds, for a total of approximately 12,000 trials. The EEG signals were acquired using a 128-channel electrode cap at a sampling frequency of 1000Hz. Before applying this method, the raw EEG signals need to be preprocessed. For example, a notch filter of 49-51Hz can be used to remove power frequency interference, and a bandpass filter of 14-71Hz can be used to retain frequency band information related to visual perception. Simultaneously, the first 20ms of signal after the stimulus begins is removed to reduce the residual effect of the previous stimulus.

[0091] After preprocessing, the EEG signals in the EEGCVPR40 dataset can be processed and visual images generated according to the method flow described in Examples 1 to 4.

[0092] Specifically, for each preprocessed EEG signal sample, the decoupling feature extraction step is performed first.

[0093] On the one hand, through the semantic feature extraction path described in Example 2, namely, using time-frequency dual-domain patch embedding, state space model network for long sequence modeling, and combined with spatial attention pooling, the first feature representing the high-level semantic category is extracted from the EEG signal.

[0094] On the other hand, through the visual feature extraction path described in Example 3, that is, using a multi-scale convolutional network to extract local features and using target image features for alignment optimization, a second feature representing local visual details is extracted from the same EEG signal.

[0095] Subsequently, the extracted first and second features are injected into the generative model, respectively.

[0096] For example, the first feature obtains the most matching category label through cosine similarity retrieval and constructs it as a text prompt word, which is then injected into the text encoder of the latent diffusion model to provide macroscopic semantic category constraints. The second feature, as a continuous image condition, is injected into the U-Net network of the latent diffusion model through the IP-Adapter module to provide fine-grained visual detail guidance.

[0097] During the generation phase, the latent diffusion model, guided by dual conditions, iteratively denoises from pure Gaussian noise, ultimately generating a visual image corresponding to the input EEG signal.

[0098] Specifically, the following evaluation metrics are used to measure model performance.

[0099] For image generation quality assessment, the Inception Score (IS) is used to measure the sharpness and class diversity of the generated images. This metric utilizes a pre-trained Inception-v3 network to obtain the class prediction distribution p(y|x) of the generated images, and calculates its KL divergence with the edge class distribution p(y), which can be expressed as: In the formula, For the estimated mutual information value, It is an exponential function. Indicates the generation distribution The following samples Seeking expectations, For Kullback-Leibler divergence, Let x be the conditional probability distribution of y. Let y be the marginal probability distribution; where a higher IS value indicates that the generated image usually has better clarity and class diversity.

[0100] The Fréchet Inception Distance (FID) is used to measure the difference in distribution between the generated and real images in the deep feature space. Using the Inception-v3 network, features of both the real and generated images are extracted and approximated as multivariate Gaussian distributions. The FID can then be expressed as: In the formula, This is the mean vector of the features of the real image. To generate the mean vector of image features, The covariance matrix of the real image features. To generate the covariance matrix of image features, The trace operation of a matrix is ​​represented by the FID value. The lower the FID value, the closer the generated image distribution is to the real image distribution, and the higher the overall fidelity of the generated result.

[0101] Generation accuracy (GA) is used to evaluate whether the generated image maintains consistency with the original visual stimulus category at a high-level semantic level. A pre-trained classification network is used to predict the category of the generated image, and the proportion of predicted categories that match the true categories is calculated, which can be expressed as: In the formula: The total number of samples, For the first Prediction results generated from EEG samples For the first The true label of each sample For classifiers or mapping functions, This indicates that the index corresponding to the class with the highest probability in the prediction results is selected. This is an indicator function; its value is [value] when the condition within the parentheses is true. Otherwise, it is 0; the higher the GA, the more accurately the generated image reflects the visual category perceived by the subject at the semantic level.

[0102] For feature representation and classification evaluation, Top-K classification accuracy is used to measure the model's ability to identify true visual categories from EEG signals, which can be expressed as: In the formula: The model represents the first The category prediction score output by each EEG sample. This represents the set of the top K categories with the highest predicted scores. For true category labels.

[0103] This paper reports the Top-1, Top-3, and Top-5 accuracies to evaluate the model's semantic decoding capabilities across different candidate ranges.

[0104] Cosine similarity is used to measure the alignment between EEG encoded features and target text or image features in a shared representation space. For EEG features... and target modal features Its calculation method can be expressed as: In the formula, The cosine similarity value is... Let i be the EEG feature vector of the i-th sample. Let be the target feature vector of the i-th sample. The closer the cosine similarity is to 1, the more consistent the orientation of the EEG features with the target semantic or visual features in the representation space, and the better the cross-modal mapping effect.

[0105] This paper reports the best mean cosine similarity on the test set, which can be expressed as: In the formula, e is the index of the number of experiments or iterations, and N is the total number of samples or the maximum number of iterations. This represents the EEG feature vector of the i-th sample in the e-th experiment. Let represent the target (Ground Truth) feature vector of the i-th sample, and 1 / e be the normalization coefficient.

[0106] As shown in Table 1, this method outperforms existing methods such as BrainVis and DreamDiffusion in terms of GA, FID and other metrics.

[0107] Table 1

[0108] Based on the aforementioned evaluation metrics, our method achieved a generation accuracy (GA) of 48.67%, an Inception Score (IS) of 31.1783, and a Fréchet Inception Distance (FID) of 98.6195 on an average of 6 subjects. Compared to the existing technology BrainVis, our method shows significant improvements in GA (48.67% vs 45.46%) and FID (98.6195 vs 126.6576), while maintaining a similar level in IS. This indicates that the images generated by our method are superior in both semantic correctness and overall fidelity.

[0109] Figure 5This paper presents the reconstruction results of our proposed method on three typical sample types: passenger planes, jack-o'-lanterns, and giant pandas, compared to BrainVis, DreamDiffusion, and Brain2Image. It can be seen that early GAN methods like Brain2Image often only recover coarse color blocks and blurry outlines, failing to form stable target structures. While DreamDiffusion generates results closer to natural images, some categories exhibit semantic shifts or structural distortions that are not entirely consistent with the real stimuli. BrainVis performs well in overall category semantics, but remains weak in local texture, edge sharpness, and instance appearance details. In contrast, our proposed method maintains more complete subject structures and richer local details on all three sample types: for example, the fuselage outline and tail color of the passenger plane are closer to the real stimuli, the brightness and shadow levels and flame light and shadow of the jack-o'-lantern are more natural, and the black-and-white texture and posture relationship of the giant panda are clearer. This demonstrates that the bi-branch condition is not a simple superposition, but rather an effective complement between global category constraints and local visual reconstruction.

[0110] To further demonstrate the overall reconstruction performance of our method across a wider range of categories, Figure 6 A comparative example of real images and the results generated by the proposed method is provided. It can be seen that, for various categories such as pianos, daisies, tents, golf balls, cameras, and tropical fish, the generated images maintain a high degree of consistency with the real stimuli in terms of subject category, overall composition, and dominant color distribution. For structurally complex targets such as kayaks, German Shepherds, bicycles, and horses, the generated results can also stably preserve the approximate geometric shape and scene layout of the target. At the same time, these results also show that the reconstruction capability of the proposed method is not limited to a few typical categories, but exhibits good adaptability under different semantic types and visual appearance conditions. Of course, in some samples, the generated images still differ from the real images in local texture, fine edges, and instance-level details, but overall, they can reconstruct the core visual content corresponding to the EEG signals relatively stably.

[0111] In this paper's framework, the semantic branch is responsible for extracting the global category prior from the EEG signal and obtaining the best-matching category label through CLIP text prototype retrieval, thereby constructing the text prompts used for diffusion generation. Therefore, its decoding performance is not an isolated classification metric, but directly relates to the accuracy of semantic constraints in subsequent image generation stages. If the semantic branch cannot consistently recover the correct category, even if the generated result appears visually natural, it is difficult to guarantee semantic consistency with the real stimulus. Based on this consideration, this section systematically evaluates the classification and decoding capabilities of the semantic branch.

[0112] As shown in Table 2, the classification results of the six subjects demonstrate that, in terms of feature representation evaluation, the Top-1 classification accuracy for the semantic branch reached 49.29%, while the Top-3 and Top-5 accuracy reached 78.48% and 90.07%, respectively. This indicates that the method can stably extract discriminative semantic features from EEG signals, providing a reliable category prior for subsequent image generation. Subject4 performed exceptionally well, achieving Top-1, Top-3, and Top-5 accuracy of 65.17%, 89.49%, and 96.70%, respectively. This result is consistent with Subject4's best performance on the image generation task, suggesting that a more accurate category prior provides more reliable semantic constraints for subsequent diffusion generation. Overall, the average Top-5 accuracy reached 90.07%, indicating that the model can compress the true categories into a small range of high-confidence candidates in most cases, demonstrating good stability in semantic information extraction.

[0113]

[0114] The comparison results with existing EEG decoding methods further illustrate this point. Our proposed method achieves 49.29% on the most crucial Top-1 metric, higher than EEG-CN's 31.30%, Brain2Image's 16.48%, S-EEGNet's 25.20%, KD-STFT's 41.20%, and BrainVis's 47.66%. It should be noted that BrainVis achieves 79.23% and 91.20% on the more lenient Top-3 and Top-5 metrics, respectively, slightly higher than our proposed method's 78.48% and 90.07%. However, for our task, Top-1 is more directly significant because the subsequent generation stage ultimately uses the optimal category suggestion retrieved by the semantic branch, rather than the top 3 or top 5 candidate categories. Therefore, the improvement in Top-1 better reflects the actual supporting capability of the semantic branch for downstream image generation.

[0115]

[0116] Figure 7 This point is further elaborated from the perspective of feature space. We selected six representative categories on the Subject4 test set and projected the semantic features extracted by the LSTM baseline and our proposed method into a two-dimensional t-SNE space. It can be seen that the feature clusters corresponding to our proposed method are generally more compact, and the inter-class boundaries are clearer, while the LSTM baseline still exhibits a more pronounced discrete distribution in some categories. This demonstrates that the advantage of our semantic branch is not only reflected in the improved Top-1 accuracy but also in its ability to form a more stable and discriminative category representation structure, thus providing a more reliable global semantic prior for text prompt retrieval and subsequent diffusion generation.

[0117] Example 6: This embodiment provides a computer-readable medium storing a computer program that, when executed by a processor, implements the visual content generation method based on electroencephalogram (EEG) signals as described in any one of Embodiments 1 to 5.

[0118] Specifically, the computer-readable medium can be any physical carrier capable of storing computer-executable instructions, such as read-only memory (ROM), random access memory (RAM), disk, optical disk, flash memory (e.g., USB flash drive, solid-state drive), or temporary or permanent storage media that can be downloaded and stored via a network. The computer program stored thereon, when loaded and executed by a processor (e.g., a central processing unit (CPU), graphics processing unit (GPU), or dedicated neural network processor (NPU), can control the processor to perform all or part of the steps described in the foregoing embodiments. For example, the processor can execute the entire process of acquiring electroencephalogram (EEG) signals, decoupling and extracting features from the EEG signals to obtain a first feature and a second feature, and injecting these two features into the generative model through different injection mechanisms to guide image generation.

[0119] It should be understood that the computer-readable medium described in this embodiment is merely an example of an implementation carrier, intended to provide a form of protection that meets the requirements of patent law for the aforementioned method. The medium itself does not contain any new technical content; its technical contribution lies entirely in the stored computer program capable of implementing the aforementioned method. Therefore, the scope of protection of this computer-readable medium is substantially consistent with the scope of protection of the aforementioned method embodiments, and any computer program storage medium capable of implementing the aforementioned method falls within the scope of protection of this application.

[0120] Example 7: This embodiment provides a visual content generation system based on electroencephalogram (EEG) signals, comprising: a signal acquisition module for acquiring EEG signals; a dual-branch encoder, including a semantic branch encoder and a visual branch encoder, wherein the semantic branch encoder includes a sequence modeling network for extracting long-sequence features from the EEG signals to obtain a first feature representing a high-level semantic category, and the visual branch encoder includes a multi-scale convolutional network for extracting local features from the EEG signals to obtain a second feature representing local visual details; a text conditional injection module for converting the first feature into a text prompt and injecting it into the text encoder of the generation model to obtain a text condition; a visual conditional injection module for injecting the second feature as an image condition into the U-Net network of the generation model; an image generation module for generating a visual image corresponding to the EEG signals based on the text condition and the image condition; and a joint optimization module for calculating the potential diffusion loss of the generation model and using the gradient generated by the potential diffusion loss to perform end-to-end joint updates of the low-rank adaptation parameters in the visual branch encoder and the generation model.

[0121] Specifically, the signal acquisition module corresponds to step S100 in Embodiment 1, and is responsible for acquiring EEG signal data from the EEG device. The semantic branch encoder in the dual-branch encoder corresponds to Embodiment 2, and includes a sequence modeling network (preferably a state-space model network) for extracting long-sequence features from the EEG signals to obtain a first feature representing a high-level semantic category. The semantic branch encoder internally includes a time-frequency dual-domain patch embedding module, a state-space model encoder, and a spatial attention pooling module. The visual branch encoder in the dual-branch encoder corresponds to Embodiment 3, and includes a multi-scale convolutional network (preferably a multi-scale one-dimensional convolutional network) for extracting local features from the EEG signals to obtain a second feature representing local visual details. The visual branch encoder internally includes multi-level convolutional layers, batch normalization layers, max-pooling layers, and fully connected projection layers.

[0122] The text conditional injection module corresponds to the first injection mechanism in Example 4, responsible for constructing a text prototype retrieval table, calculating cosine similarity to select category labels, constructing natural language prompts, and inputting them into the frozen text encoder. The visual conditional injection module corresponds to the second injection mechanism in Example 4, responsible for inputting the second feature into the IP-Adapter module and converting it into key-value pairs compatible with U-Net. The image generation module generates visual images based on text and image conditions through an iterative denoising process of the latent diffusion model. The joint optimization module is responsible for calculating the latent diffusion loss and, through backpropagation, transmitting the gradient through the conditional injection module back to the visual branch encoder, achieving end-to-end joint updates.

[0123] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0124] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A method for generating visual content based on electroencephalogram (EEG) signals, characterized in that, Includes the following steps: Acquiring EEG signals; The EEG signal is decoupled and feature extracted using a dual-branch encoder to obtain a first feature and a second feature. The first feature is obtained by extracting long-sequence features from the EEG signal using a sequence modeling network and is used to represent high-level semantic categories. The second feature is obtained by extracting local features from the EEG signal using a multi-scale convolutional network and is used to represent local visual details. The first feature is converted into a text prompt and then injected into the text encoder of the generative model to obtain the text condition; The second feature is used as an image condition and injected into the U-Net network of the generative model through the condition injection module to calculate the potential diffusion loss of the generative model. The generation model is used to generate a visual image corresponding to the EEG signal based on the text conditions and the image conditions. Using the gradient generated by the potential diffusion loss, the low-rank adaptation parameters in the multi-scale convolutional network that extracts the second feature and the generative model are jointly updated end-to-end.

2. The visual content generation method based on electroencephalogram (EEG) signals according to claim 1, characterized in that, The sequence modeling network is a state-space model network; Long-sequence feature extraction of the EEG signals is performed using a sequence modeling network, including: The EEG signal is subjected to time-domain feature extraction and frequency-domain feature extraction to obtain time-domain features and frequency-domain features; The time-domain features and the frequency-domain features are fused and input into the state-space model network for long sequence modeling.

3. The visual content generation method based on electroencephalogram (EEG) signals according to claim 2, characterized in that, The fusion of the time-domain features and the frequency-domain features includes: The time-domain features and the frequency-domain features are added element-wise to obtain the fused features; Learnable positional codes are superimposed on the fused features.

4. The visual content generation method based on electroencephalogram (EEG) signals according to claim 2, characterized in that, The step of extracting long-sequence features from the EEG signal using a sequence modeling network further includes: Spatial attention pooling is performed on the features output by the state space model network to weight and aggregate features from different spatial locations, thus obtaining the first feature.

5. The visual content generation method based on electroencephalogram (EEG) signals according to claim 1, characterized in that, The multi-scale convolutional network includes at least two levels of one-dimensional convolutional layers, and the kernel size of each convolutional layer is different; Each level of the one-dimensional convolutional layer is followed by a batch normalization layer and a max pooling layer in sequence; The features output from the last convolutional layer are flattened and projected through a fully connected layer to form a fixed-dimensional visual feature vector, which is then used as the second feature.

6. The visual content generation method based on electroencephalogram (EEG) signals according to claim 1, characterized in that, The method further includes: Alignment optimization is performed on the second feature using the target image features; The alignment optimization of the second feature using the target image features includes: Image features of the real image corresponding to the EEG signal are extracted using a pre-trained image encoder. Calculate the cosine embedding loss between the second feature and the image feature; By minimizing the cosine embedding loss, the second feature gradually approaches the image feature.

7. The visual content generation method based on electroencephalogram (EEG) signals according to claim 1, characterized in that, The generative model is a potential diffusion model; The text encoder that converts the first feature into a text prompt and then injects it into the generative model includes: Construct a text prototype retrieval table, which stores text prototype features of multiple target categories; Calculate the cosine similarity between the first feature and each text prototype feature in the text prototype retrieval table, and select the category label with the highest similarity; The category labels are constructed into standardized natural language prompts, which are then input into a frozen text encoder to obtain the text conditions. The step of injecting the second feature as an image condition into the U-Net network of the generative model through a condition injection module includes: The second feature is input into the IP-Adapter module, which converts it into a key-value pair compatible with the cross-attention layer in the U-Net network and injects it into the U-Net network.

8. The visual content generation method based on electroencephalogram (EEG) signals according to claim 1, characterized in that, The low-rank adaptation parameters include a first weight matrix and a second weight matrix; The dimension of the first weight matrix is The dimension of the second weight matrix is , in, and These are the output and input dimensions of the pre-trained weight matrix, respectively, and r is a preset rank value, which ranges from 4 to 16. The end-to-end joint update includes: Freeze the basic backbone parameters of the variational autoencoder, text encoder, and U-Net network in the generative model; Only update the low-rank adaptation parameters and the parameters of the multi-scale convolutional network that extracts the second feature; During backpropagation, the gradient generated by the potential diffusion loss passes sequentially through the U-Net network and the conditional injection module, and is finally propagated back to the multi-scale convolutional network.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

10. A visual content generation system based on electroencephalogram (EEG) signals, applied to the method described in any one of claims 1-9, characterized in that, include: Signal acquisition module, used to acquire electroencephalogram (EEG) signals; A dual-branch encoder includes a semantic branch encoder and a visual branch encoder; wherein, the semantic branch encoder includes a sequence modeling network for extracting long-sequence features from the EEG signal to obtain a first feature for representing a high-level semantic category; the visual branch encoder includes a multi-scale convolutional network for extracting local features from the EEG signal to obtain a second feature for representing local visual details; The text condition injection module is used to convert the first feature into a text prompt and then inject it into the text encoder of the generative model to obtain the text condition; A visual conditional injection module is used to inject the second feature as an image condition into the U-Net network of the generative model; The image generation module is used to generate a visual image corresponding to the EEG signal based on the text conditions and the image conditions; The joint optimization module is used to calculate the potential diffusion loss of the generative model and use the gradient generated by the potential diffusion loss to perform end-to-end joint updates of the low-rank adaptation parameters in the visual branch encoder and the generative model.

Images