Visual content retrieval method based on electroencephalogram signals

Abstract

The invention belongs to the technical field of brain-computer interfaces, multi-modal feature alignment and information retrieval, and discloses a visual content retrieval method based on electroencephalogram signals. Cross-subject standardized electroencephalogram samples are obtained and input into an electroencephalogram encoder to extract low-dimensional electroencephalogram signal feature vectors; an image encoder is adopted to process the corresponding retrieval images to extract visual feature vectors; the low-dimensional electroencephalogram signal feature vector and the visual feature vector are jointly input into a prototype attention enhancement module to form a dynamic prototype pool, and prototype enhanced electroencephalogram signal representation is obtained through processing; the method comprises the following steps: performing classification training according to existing prototype enhanced electroencephalogram signal representation and image data pairs; for a plurality of time slices of the to-be-queried electroencephalogram signal sample, calculating a reconstruction error or a signal-to-noise ratio of each time slice to obtain a confidence coefficient; and adopting a weighted average or voting mechanism to fuse a plurality of time slice results, and outputting a stable and robust final visual retrieval classification result.

Description

Technical Field

[0001] This invention relates to the fields of brain-computer interface, multimodal feature alignment and information retrieval technology, and in particular to a visual content retrieval method based on electroencephalogram (EEG) signals. Background Technology

[0002] In recent years, artificial intelligence (AI) has made significant progress in mapping human brain activity. The application of AI technology in visual stimulation tasks involving human brain signals and various other external stimulus paradigms has significantly improved the interpretability of the biological signals emitted by the brain. This enhanced interpretability is particularly evident in the context of large-scale data pre-training, in which the performance of human brain signal decoding tasks has made substantial progress.

[0003] Currently, functional magnetic resonance imaging (fMRI) decoding tasks have shown remarkable effectiveness in decoding human brain activity. However, the widespread application of this technology is hindered by several factors. The high cost associated with fMRI equipment, coupled with its operational complexity, poses a significant obstacle to its widespread adoption and practical application in various settings.

[0004] In contrast, electroencephalography (EEG) devices offer a compelling alternative. Their inherent advantages, including portability, cost-effectiveness, and ease of operation, make them ideal candidates for decoding human brain activity. Furthermore, the temporal resolution of EEG signals allows for the rapid capture of dynamic changes in brain activity, providing a high-fidelity representation of neural processes in real time.

[0005] Despite these advantages, it's worth noting that EEG signals also present a number of challenges. While EEG shows promise in decoding human brain signals, it has not yet reached the level of accuracy achieved by fMRI. This difference can be attributed to several factors, primarily the significant variations in EEG signals between different subjects and the relatively low spatial resolution of the technique. Meanwhile, some recent algorithms have attempted to mitigate these drawbacks by employing masking techniques to preprocess the input EEG signals. However, this approach is not without its limitations. EEG signals are inherently complex, containing multiple layers of information. While masking techniques may be beneficial in some cases, there is a risk of unintentionally obscuring or eliminating crucial information. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a visual content retrieval method based on electroencephalogram (EEG) signals.

[0007] The technical solution of the present invention is as follows: A visual content retrieval method based on electroencephalogram (EEG) signals, comprising the following steps: Step 1. Input EEG signals and corresponding retrieved images. After alignment preprocessing, EEG signals are used to obtain cross-subject standardized EEG samples. Step 2. Input standardized EEG samples across subjects into an EEG encoder to extract low-dimensional EEG signal feature vectors; Step 3. Cross-modal feature fusion: The image encoder is used to process the corresponding retrieved image to extract visual feature vectors; the low-dimensional EEG signal feature vector and the visual feature vector are input together into the multi-head prototype module, and fusion processing is performed to obtain the prototype-enhanced EEG signal representation; The multi-head prototyping module includes multiple prototyping attention enhancement modules and an inverse mapping network; The low-dimensional EEG signal feature vector and the visual feature vector are fused and then input into the prototype attention enhancement module. The prototype attention enhancement module designs M parallel attention mapping functions based on the multi-head attention mechanism. Based on the low-dimensional EEG signal feature vector, prototype weights are assigned to each pixel position of the corresponding visual feature vector to generate M prototype features, which form a dynamic prototype pool with stable representation. The global low-dimensional EEG signal feature vector is not stored separately. Using the low-dimensional EEG signal feature vector as the query vector, the prototype features are weighted and combined by calculating the correlation score with each prototype feature in the dynamic prototype pool to obtain the prototype-enhanced EEG signal representation.

[0008] In the process of generating the prototype features, prototype compactness constraints and prototype diversity constraints are introduced as prototype attention loss. The prototype compactness constraint minimizes the L2 distance between the low-dimensional EEG signal feature vector and its most relevant prototype feature, so that the prototype feature can compactly represent the core cognitive feature pattern in the EEG signal. The prototype compactness constraint ; For prototype compactness constraints; among which, Indicates the number of query channels, For query vectors, Prototype features; The prototype diversity constraint is imposed by applying a minimum interval constraint between different prototype features as follows: Constraints on prototype diversity; For the first One prototype feature, To and The closest prototype features This is a hyperparameter.

[0009] Step 4: The prototype enhanced EEG signal representation and retrieved images are fed into the classification module for classification training. After training, in the retrieval stage, the EEG signal to be queried is input into the EEG encoder for inference to obtain the corresponding low-dimensional EEG signal feature vector. This vector is then matched with the image feature vectors corresponding to each image in the retrieval database and sent to the classification module for matching retrieval. For multiple time segments of the EEG signal to be queried, the reconstruction error or signal-to-noise ratio of each time segment is calculated to obtain the confidence level. A weighted average mechanism is then used to fuse the results of multiple time segments to output the final visual retrieval classification result.

[0010] The alignment preprocessing specifically includes: filtering and artifact removal of the EEG signal; channel selection and time window segmentation based on electrode topology; and using a learnable channel recalibration matrix. Alignment with the reference domain; the channel recalibration matrix The time-varying linear transformation matrix is ​​constructed and trained based on channel statistical features. At the same time, the weights of each channel are adjusted by combining a lightweight channel attention mechanism to perform dynamic channel recalibration and obtain standardized EEG samples across subjects.

[0011] The EEG encoder includes a time-domain branch and a time-frequency branch. The time-domain branch uses dilated convolution and deep convolution to extract rhythmic information. The time-frequency branch generates a spectrum based on short-time Fourier transform or continuous wavelet transform, extracts power spectral density and phase-locked values, and captures brain region synchronization features. The encoding results of the time-domain branch and the time-frequency branch are spliced ​​or weighted in the fusion layer to obtain a low-dimensional EEG signal feature vector. During EEG encoder training, a random masking denoising strategy is used to randomly mask some time windows or frequency bands and reconstruct the original EEG signal through a decoder.

[0012] The random masking denoising strategy is a multi-masking layered training strategy, which includes multiple sub-tasks with different masking ratios. During the masking reconstruction process, the correlation distribution of the reconstructed EEG signal features is required to be consistent with the correlation distribution of the EEG signal features before masking, so as to ensure that the masking training does not destroy the core cognitive feature structure of the EEG signal, and finally fuse the multi-layer decoding results.

[0013] The classification training of the classification module is specifically as follows: the prototype-enhanced EEG signal representation is mapped to a joint embedding space consistent with the visual feature vector through a learnable projection layer; the distance between paired EEG signals and retrieved image samples in the joint embedding space is reduced and the distance between unpaired samples is increased through contrastive learning loss; at the same time, cross-modal cyclic consistency constraints are introduced, and the EEG signal features are reconstructed from the visual feature vector through the inverse mapping network and the reconstruction error is minimized.

[0014] The matching retrieval in the retrieval stage specifically involves mapping the low-dimensional EEG signal feature vector to the query EEG signal embedding vector, integrating it with the existing visual feature vector into the same joint embedding space, and then performing a quantification operation on the similarity between the two in the joint embedding space by the classification module. The retrieval is accelerated based on the approximate nearest neighbor index structure, and the Top-K most similar images are output as the retrieval results. When the query scenario involves the EEG signals of new subjects, the dynamic prototype pool is first updated through the prototype attention enhancement module, and then feature mapping and retrieval are performed.

[0015] The beneficial effects of this invention are as follows: This method effectively improves the quality of EEG signals and the effect of feature extraction by optimizing the preprocessing and encoding / decoding process of EEG signals. In the preprocessing stage, the channel recalibration matrix of time-varying linear transformation and the lightweight channel attention mechanism are combined to achieve the standardization of EEG signals across subjects, solving the problem of large differences in EEG signals among different subjects; the EEG encoder adopts time-domain and time-frequency dual-branch feature extraction, and is equipped with a multi-masking layered training strategy, which removes noise while retaining the core cognitive features of EEG signals, providing more accurate low-dimensional EEG feature vectors for subsequent retrieval. By leveraging a prototype attention enhancement module and a cross-modal feature fusion mechanism, this method significantly enhances the matching accuracy between EEG signals and visual features. The multi-head prototype module dynamically generates prototype features, eliminating the need for additional storage of global EEG features. This saves storage resources and, through prototype compactness and diversity constraints, ensures that prototype features accurately represent core EEG cognitive patterns while avoiding redundancy. During classification retrieval, cross-domain feature alignment, combined with contrastive learning loss and cross-modal cyclic consistency constraints, maps EEG signals and visual features to a unified embedding space and reduces the distance between paired samples, significantly improving the consistency of cross-modal features and laying the foundation for retrieval accuracy. The design of this method in the retrieval phase further ensures the stability, robustness, and adaptability of the retrieval results. By calculating the confidence scores of multiple time segments of the EEG to be queried and using a weighted average fusion method, the impact of fluctuations in single time segment data on the retrieval is effectively reduced. When faced with EEG signals from new subjects, the dynamic prototype pool can be updated through the prototype attention enhancement module before retrieval, eliminating the need to retrain the entire model and expanding the applicability of the method. At the same time, the combination of an approximate nearest neighbor index structure accelerates the retrieval process, balancing accuracy with retrieval efficiency, making it more suitable for practical applications. Attached Figure Description

[0016] Figure 1 This is a logical diagram of a visual content retrieval method based on electroencephalogram (EEG) signals. Figure 2 This is a diagram showing the overall structure of the module processing corresponding to the method described in this invention. Figure 3 This is a structural diagram of a multi-head prototype module; Figure 4 The results of retrieving EEG signals from different visual images are shown in the figure; (a) is the corresponding real image, (b) is the Top 1 retrieval result, (c) is the Top 2 retrieval result, (d) is the Top 3 retrieval result, (e) is the Top 4 retrieval result, and (f) is the Top 5 retrieval result. Detailed Implementation

[0017] The technical solution of this invention takes "EEG feature enhancement - cross-modal accurate alignment - retrieval adaptation optimization" as its core logic, and focuses on introducing a dynamic prototype pool mechanism to solve the problems of insufficient discriminative power of EEG features, difficulty in cross-subject adaptation, and high memory consumption.

[0018] First, EEG data alignment preprocessing is performed: the EEG signal is filtered and artifact removed, channel selection and time window segmentation are performed based on electrode topology, and alignment is achieved using a time-varying linear transform channel recalibration matrix W constructed and trained based on channel statistical features and a reference domain, obtaining standardized EEG samples across subjects. The preprocessed EEG signal is then input into the EEG encoder. The EEG encoder includes a time-domain branch and a time-frequency branch. The time-domain branch uses dilated convolution and depthwise convolution to extract rhythmic information, while the time-frequency branch generates spectra based on short-time Fourier transform or continuous wavelet transform, extracting power spectral density and phase-locked values ​​to capture brain region synchronization features. During encoding, a multi-masking hierarchical training random masking denoising strategy is used, randomly masking certain time windows or frequency bands. During training, the decoder reconstructs the original input EEG signal from the compressed feature representation, requiring the correlation distribution of the reconstructed EEG signal features to remain consistent with that before masking, ensuring that the core cognitive feature structure is not destroyed. This reconstruction process, as a form of self-supervision, ensures that the encoded features retain sufficient information to reconstruct the original signal with high fidelity, while also enhancing the module's ability to capture salient aspects of the EEG signal. Through multiple iterations of this encoding-decoding process, the system gradually improves its feature extraction capabilities. The encoder focuses on the most informative aspects of the EEG signal, while the decoder provides feedback to guide more efficient feature representation. Finally, the encoding results of the time-domain branch and the time-frequency branch are concatenated or weighted at the fusion layer to obtain a low-dimensional EEG signal feature vector.

[0019] Next, to achieve accurate mapping between EEG and visual features, this invention proposes a multi-head prototyping module for EEG signal and image prototype attention enhancement. A dynamic prototype pool is constructed by generating prototype features through multi-head attention. The specific process is as follows: Drawing on the multi-head attention mechanism, M parallel attention mapping functions (M preferably 10-20) are designed. Based on the time-frequency structure (e.g., rhythmic distribution, power spectrum peaks) and spatial topology of EEG features, and combined with pixel semantic information of visual features (e.g., image target contours, regional brightness distribution), prototype weights are assigned to each pixel-level position of the visual features. This ensures that the weights reflect both EEG cognitive features and match the visual attributes of the corresponding image, generating M independent prototype features that fuse cross-modal information, forming the dynamic prototype pool. This dynamic prototype pool eliminates the need to store global EEG feature templates and complete visual feature templates, significantly reducing memory consumption. Furthermore, the prototype features can be generated in real-time to simultaneously adapt to differences in cognitive features among different subjects and differences in visual features among different images. To constrain the quality of prototype features, prototype compactness constraints and prototype diversity constraints are introduced as loss functions: the compactness constraint minimizes the L2 distance between the fused vector of "EEG features and visual features" and its most relevant prototype. Diversity constraints prevent prototype feature collapse by imposing minimum interval constraints on different prototypes, ensuring comprehensive coverage of cognitive-visual association patterns.

[0020] After completing EEG signal alignment preprocessing, EEG feature encoding, and cross-modal alignment in the dynamic prototype pool, this invention sets up a classification module in the training phase to discriminate the matching relationship between prototype-enhanced EEG signal representations and images, and guide the joint optimization of parameters of various related modules. In the training phase, the prototype-enhanced EEG signal representations extracted by the EEG encoder and subjected to prototype attention enhancement processing in the dynamic prototype pool, along with the features of the corresponding images in the training set, are fed into the classification module for classification training. Internally, the classification module maps the prototype-enhanced EEG signal representations and image features to the same feature representation space, determines whether they constitute paired samples, and outputs the corresponding matching probability or similarity score. The training objective of the classification module is to enhance the cross-modal consistency and class discriminativeness of features in the joint embedding space. To this end, this invention employs a joint optimization strategy of a classification loss function and a cross-modal contrastive learning loss function in the training phase. The classification loss function uses cross-entropy loss to constrain the model to correctly distinguish between paired EEG-image samples and unpaired samples. The cross-modal contrastive learning loss minimizes the distance between paired EEG-image samples in the joint embedding space while maximizing the distance between unpaired samples, thereby expanding the feature margin between different sample categories and improving the discriminative power of the joint embedding space. Through this joint optimization process, EEG features, after being mapped to the joint embedding space, can be inversely constrained by the visual feature matching results, thus forming a consistent closed loop of cross-modal feature mapping and achieving cross-modal cyclic consistency constraint. During training, the parameters of the EEG encoder, dynamic prototype pool module, projection layer, and classification module are jointly updated end-to-end. Training is performed using a mini-batch iterative approach, with 100 training iterations. The validation set is used to verify and adjust the model parameters and retrieval performance, enabling the model to gradually achieve continuous improvement in the similarity of paired samples and effective suppression of the similarity of unpaired samples until the retrieval accuracy on the validation set reaches stable convergence.

[0021] In the retrieval phase, the EEG signal samples to be retrieved are first subjected to alignment preprocessing consistent with that in the training phase. The preprocessed sample is then input into the trained EEG encoder to obtain a low-dimensional EEG feature vector. This vector is processed by the prototype attention enhancement module and mapped by the projection layer to become the query EEG signal embedding vector. This embedding vector is then integrated with the visual features stored in the database into the same joint embedding space. The trained classification module quantifies the similarity between the two features (using cosine similarity to calculate the matching degree). Retrieval is accelerated based on an approximate nearest neighbor index structure, and the Top-K most similar images are output as the result. For multiple time segments of the query EEG signal, the confidence level is calculated by weighting the reconstruction error of the segment's EEG signal with the prototype matching degree. A weighted average mechanism is used to fuse the results of multiple segments, ensuring stable and robust output.

[0022] This invention selects an EEG image pair dataset as the baseline dataset. This dataset samples EEG signal image pairs from six subjects and, during the experiment, presents them with 1000 images belonging to 40 different categories of objects from the ImageNet dataset. Each category consists of 50 images, each displayed for 0.5 seconds, followed by a 10-second pause after every 50 images. EEG data was recorded using a 128-channel Brainvision EEG system, resulting in a total of 128-channel EEG sequences. The dataset includes images of various objects, such as animals (dogs, cats, elephants, etc.), vehicles (airplanes, bicycles, cars, etc.), and everyday items (computers, chairs, cups, etc.).

[0023] This invention framework is implemented using PyTorch, leveraging its dynamic computation graph capabilities for efficient model prototyping and experimentation. A custom CUDA kernel is used for performance-critical operations, ensuring optimal utilization of GPU resources. The codebase is built according to best practices in software engineering, employing a modular design principle to facilitate modification and extension of the framework. All EEG signals are filtered within a frequency range of 5-95Hz. Subsequently, the signals are truncated to a common length of 512.

[0024] The extracted EEG features were compared with visual features in the image database. Cosine similarity was used as the matching metric to determine the most similar images. Finally, Top-k accuracy (k=1,5,10) was used as a retrieval evaluation metric to comprehensively measure retrieval performance.

[0025] To more intuitively demonstrate the effectiveness of this method, in Figure 3 The results of different visual image retrieval tasks are shown in the image. Figure 3 As can be seen, this method focuses more on the color, texture, and shape of the visual image corresponding to the EEG signal.

Claims

1. A visual content retrieval method based on electroencephalogram (EEG) signals, characterized in that, Includes the following steps: Step 1. Input EEG signals and corresponding retrieved images. After alignment preprocessing, EEG signals are used to obtain cross-subject standardized EEG samples. Step 2. Input standardized EEG samples across subjects into an EEG encoder to extract low-dimensional EEG signal feature vectors; Step 3. Cross-modal feature fusion: The image encoder is used to process the corresponding retrieved image to extract visual feature vectors; the low-dimensional EEG signal feature vector and the visual feature vector are input together into the multi-head prototype module, and fusion processing is performed to obtain the prototype-enhanced EEG signal representation; The multi-head prototyping module includes multiple prototyping attention enhancement modules and an inverse mapping network; The low-dimensional EEG signal feature vector and the visual feature vector are fused and then input into the prototype attention enhancement module. The prototype attention enhancement module designs M parallel attention mapping functions based on the multi-head attention mechanism. Based on the low-dimensional EEG signal feature vector, prototype weights are assigned to each pixel position of the corresponding visual feature vector to generate M prototype features, which form a dynamic prototype pool with stable representation. The global low-dimensional EEG signal feature vector is not stored separately. Using the low-dimensional EEG signal EEG feature vector as the query vector, the prototype features are weighted and combined by calculating the correlation score with each prototype feature in the dynamic prototype pool to obtain the prototype-enhanced EEG signal representation. In the process of generating the prototype features, prototype compactness constraints and prototype diversity constraints are introduced as prototype attention loss. The prototype compactness constraint minimizes the L2 distance between the low-dimensional EEG signal feature vector and its most relevant prototype feature, so that the prototype feature can compactly represent the core cognitive feature pattern in the EEG signal. The prototype compactness constraint ; For prototype compactness constraints; among which, Indicates the number of query channels, For query vectors, Prototype features; The prototype diversity constraint is imposed by applying a minimum interval constraint between different prototype features as follows: Constraints on prototype diversity; For the first One prototype feature, To and The closest prototype features For hyperparameters; Step 4: Input the prototype-enhanced EEG signal representation and retrieved images into the classification module for classification training; After training, during the retrieval phase, the EEG signal to be queried is input into the EEG encoder for inference, and the corresponding low-dimensional EEG signal feature vector is obtained. This vector is then sent to the classification module for matching and retrieval, along with the image feature vectors corresponding to each image in the retrieval database. For multiple time segments of the EEG signal to be queried, the reconstruction error or signal-to-noise ratio of each time segment is calculated to obtain the confidence level. A weighted average mechanism is then used to fuse the results of multiple time segments, and the final visual retrieval classification result is output.

2. The visual content retrieval method based on electroencephalogram (EEG) signals according to claim 1, characterized in that, The alignment preprocessing specifically includes: filtering and artifact removal of the EEG signal; channel selection and time window segmentation based on electrode topology; and using a learnable channel recalibration matrix. Alignment with the reference domain; the channel recalibration matrix The time-varying linear transformation matrix is ​​constructed and trained based on channel statistical features. At the same time, the weights of each channel are adjusted by combining a lightweight channel attention mechanism to perform dynamic channel recalibration and obtain standardized EEG samples across subjects.

3. The visual content retrieval method based on electroencephalogram (EEG) signals according to claim 1, characterized in that, The EEG encoder includes a time-domain branch and a time-frequency branch; the time-domain branch uses dilated convolution and deep convolution to extract rhythmic information; the time-frequency branch generates a spectrum based on short-time Fourier transform or continuous wavelet transform, extracts power spectral density and phase-locked value, and captures brain region synchronization features. The encoding results of the time-domain branch and the time-frequency branch are spliced ​​or weighted in the fusion layer to obtain a low-dimensional EEG signal feature vector. During the training of the EEG encoder, a random masking denoising strategy is adopted to randomly mask some time windows or frequency bands and reconstruct the original EEG signal through the decoder.

4. The visual content retrieval method based on electroencephalogram (EEG) signals according to claim 3, characterized in that, The random masking denoising strategy is a multi-masking layered training strategy, which includes multiple sub-tasks with different masking ratios. During the masking reconstruction process, the correlation distribution of the reconstructed EEG signal features is required to be consistent with the correlation distribution of the EEG signal features before masking, so as to ensure that the masking training does not destroy the core cognitive feature structure of the EEG signal, and finally fuse the multi-layer decoding results.

5. The visual content retrieval method based on electroencephalogram (EEG) signals according to claim 1, characterized in that, The classification training of the classification module is specifically as follows: the prototype-enhanced EEG signal representation is mapped to a joint embedding space consistent with the visual feature vector through a learnable projection layer; the distance between paired EEG signals and retrieved image samples in the joint embedding space is reduced and the distance between unpaired samples is increased through contrastive learning loss; at the same time, cross-modal cyclic consistency constraints are introduced, and the EEG signal features are reconstructed from the visual feature vector through the inverse mapping network and the reconstruction error is minimized.

6. The visual content retrieval method based on electroencephalogram (EEG) signals according to claim 5, characterized in that, The matching retrieval in the retrieval stage is specifically as follows: the low-dimensional EEG signal feature vector is mapped to the query EEG signal embedding vector, and integrated with the existing visual feature vector into the same joint embedding space. Then, the classification module performs a quantification operation on the similarity between the two in the joint embedding space, accelerates the retrieval based on the approximate nearest neighbor index structure, and outputs the Top-K most similar image as the retrieval result. When the query scenario involves the EEG signals of new subjects, the dynamic prototype pool is first updated through the prototype attention enhancement module, and then feature mapping and retrieval are performed.

Images