An electroencephalogram signal feature matching method based on low-dimensional shared subspace constraint

Abstract

The application provides an electroencephalogram feature matching method based on a low-dimensional shared subspace constraint and relates to the technical field of electroencephalogram matching. First, electroencephalogram signals generated by visual image stimulation and corresponding visual images are acquired; electroencephalogram signal features and image semantic features are extracted respectively, and the two are mapped to a low-dimensional shared subspace; a cross-modal correlation constraint is constructed based on the low-dimensional representation, the matching semantic dimension is enhanced, and the redundant correlation between non-corresponding dimensions is reduced; the model is trained in combination with a cross-modal retrieval loss and a correlation constraint loss; and the corresponding image is determined according to the similarity between the electroencephalogram signal features to be matched and the candidate image features. The scheme establishes a structural consistency relationship between the electroencephalogram signals and the image semantic features in the low-dimensional shared subspace, reduces the influence of high-dimensional feature redundancy and electroencephalogram signal noise on the matching process, and improves the accuracy and stability of electroencephalogram cross-modal image retrieval.

Description

Technical Field

[0001] This invention relates to the technical field of electroencephalogram (EEG) signal matching, and more specifically, to an EEG signal feature matching method based on low-dimensional shared subspace constraints. Background Technology

[0002] Visual content matching based on neural activity data such as electroencephalogram (EEG) signals has become a key research focus. Visual content matching aims to identify or retrieve perceived visual stimuli based on the EEG signals generated when subjects view images. Compared to functional magnetic resonance imaging (fMRI) signals, EEG signals offer advantages such as high temporal resolution, relatively low acquisition costs, and flexible experimental deployment, making them more suitable for real-time visual perception decoding scenarios. However, due to the inherent characteristics of EEG signals, such as low signal-to-noise ratio, significant individual variability, and substantial inter-trial fluctuations, extracting reliable visual semantic information from complex and unstable neural activity remains a significant technical challenge in this field.

[0003] In recent years, with the development of large-scale visual language pre-trained models, existing techniques for visual content matching based on neural activity data such as EEG signals have gradually shifted from traditional closed-category classification methods to cross-modal retrieval methods based on shared semantic spaces. Typical methods usually employ a pre-trained visual encoder to extract semantic features from images, while simultaneously using a neural encoder to extract EEG signal features. Then, the EEG signal features are mapped to the image semantic feature space through a projection layer or a contrastive learning objective. This type of method leverages the rich category structures and semantic relationships already learned in the pre-trained visual model, enabling EEG signal decoding tasks to move beyond fixed-category recognition and support more complex application scenarios such as open-category retrieval, zero-shot image retrieval, and cross-category semantic matching. For the aforementioned cross-modal EEG signal retrieval framework, existing improvement methods typically enhance the matching ability between EEG signal features and image features by adding projection layers, adapter modules, normalization operations, or contrastive learning losses. Some methods also attempt to introduce feature compression or correlation constraints to alleviate redundant noise and modal differences in the high-dimensional feature space. However, these improved methods are still prone to including invalid feature dimensions in the matching process. If dimensionality reduction is simply performed, effective feature information may be lost. If low-dimensional constraints are not applied, invalid feature dimensions will still interfere with cross-modal alignment. Therefore, existing methods are difficult to balance feature preservation and feature matching, which reduces the accuracy of cross-modal matching. Summary of the Invention

[0004] To address the problems of low matching accuracy and difficulty in removing redundant features in existing methods for matching visual images using EEG signal features, this invention proposes an EEG signal feature matching method based on low-dimensional shared subspace constraints. This method preserves more stable shared semantic components between EEG signal features and image features, while reducing redundant noise in EEG signal features that is irrelevant to visual semantics. This improves the alignment stability and matching accuracy in the process of matching EEG signal features to image features.

[0005] To achieve the above-mentioned technical effects, the technical solution of the present invention is as follows: Firstly, this application proposes a method for matching EEG signal features based on low-dimensional shared subspace constraints, comprising the following steps: S1. Collect the electroencephalogram (EEG) signals generated when the subject views visual images; S2. Encode the acquired EEG signals using a preset EEG signal encoding model to obtain a high-dimensional semantic embedding of the EEG signals, and encode the visual image using a preset image encoding model to obtain a high-dimensional semantic embedding of the image. S3. Using a preset low-dimensional adapter for EEG signals, embed high-dimensional semantics of EEG signals into a low-dimensional bottleneck space and use a preset low-dimensional adapter for images to embed high-dimensional semantics of images into a low-dimensional bottleneck space, thereby obtaining low-dimensional bottleneck representations of EEG signals and images respectively. S4. Calculate the low-dimensional cross-modal correlation constraint loss of EEG signals and images based on low-dimensional bottleneck representation of EEG signals and images; S5. Based on the low-dimensional bottleneck representation of EEG signals and the low-dimensional bottleneck representation of images, EEG signal features and image features are obtained respectively, and a retrieval loss is constructed based on the EEG signal output features and image output features. S6. The overall training objective is constructed by combining the retrieval loss and the low-dimensional cross-modal correlation constraint loss. Based on the overall training objective, the trainable parameters of the EEG signal coding model, the low-dimensional EEG signal adapter, and the low-dimensional image adapter are updated respectively. S7. Using the trained EEG signal encoding model and EEG signal low-dimensional adapter, extract the signal features of the EEG signal to be matched, calculate the similarity between the signal features and the image features of the candidate matching image, and determine the target image corresponding to the EEG signal to be matched based on the ranking of similarity.

[0006] In this technical solution, firstly, the EEG signals generated when the subject views a visual image and the corresponding visual image are acquired; then, high-dimensional semantic embeddings of the EEG signals and images are extracted using an EEG signal encoding model and an image encoding model, respectively; finally, the high-dimensional semantic embeddings of the EEG signals and images are mapped to a low-dimensional bottleneck space using a low-dimensional adapter for the EEG signals and an image, respectively, to obtain low-dimensional bottleneck representations of the EEG signals and images; a cross-modal correlation matrix is ​​constructed based on the low-dimensional bottleneck representations, and the correlation between corresponding semantic dimensions of EEG signals and images is enhanced according to the cross-modal correlation matrix, while reducing redundant correlations between non-corresponding semantic dimensions; further, a joint training objective is constructed based on cross-modal retrieval loss and low-dimensional cross-modal correlation constraint loss to optimize the model parameters; finally, the trained model is used to extract features of the EEG signals to be matched, and the corresponding image is determined based on the similarity between the EEG signal features and the candidate image features. This scheme establishes a structural consistency relationship between EEG signals and image semantic features in a low-dimensional shared subspace, reducing the impact of high-dimensional feature redundancy and EEG signal noise on the matching process, and improving the accuracy and stability of cross-modal image retrieval based on EEG signals.

[0007] Preferably, the step of encoding the acquired EEG signals using a preset EEG signal encoding model to obtain a high-dimensional semantic embedding of the EEG signals includes: The acquired EEG signals are unfolded into EEG signal vectors using a pre-defined residual projection network. The expression is:

[0008] in, This indicates the input EEG signal. , Indicates the number of EEG signal channels. Indicates the number of time sampling points. Indicates batch size, Represents the weights of the linear projection layer. Indicates the linear projection layer bias; Residual feedforward transformation is performed on the EEG signal vector to obtain high-dimensional semantic embedding of the EEG signal. The expression is:

[0009] in, , Represents the semantic embedding dimension. and All of these represent the learnable weights of the residual feedforward layer.

[0010] Preferably, before projecting the high-dimensional semantic embedding of the EEG signal onto the low-dimensional bottleneck space using a preset low-dimensional EEG signal adapter, the method further includes batch-level standardization processing of the high-dimensional semantic embedding of the EEG signal, including: Computing high-dimensional semantic embeddings of EEG signals The mean of each feature dimension and variance The expression is:

[0011]

[0012] in, Indicates the first The first of the EEG signal samples Each feature dimension; Based on the obtained mean and variance, the corresponding feature dimensions of each EEG signal high-dimensional semantic embedding are standardized to obtain the standardized EEG signal high-dimensional semantic embedding. .

[0013] Preferably, both the low-dimensional adapter for EEG signals and the low-dimensional adapter for images include a lower projection layer and an upper projection layer connected in sequence; wherein, the lower projection layer maps the high-dimensional semantic embeddings of the EEG signals and images to a low-dimensional bottleneck space, respectively, to obtain a low-dimensional bottleneck representation of the EEG signals. Low-dimensional bottleneck representation of images The expression is:

[0014]

[0015] in, This represents the standardized high-dimensional semantic embedding of the image. This represents the downprojection weights of the low-dimensional adapter for EEG signals. This indicates the lower projection bias of the low-dimensional adapter for EEG signals. This represents the downprojection weights of the low-dimensional image adapter. This represents the downprojection bias of the low-dimensional image adapter; The upper projection layer represents the low-dimensional bottleneck of the EEG signal modality. Low-dimensional bottleneck representation of image modalities By restoring the original semantic dimension, the EEG signal features were obtained. and image features The expression is:

[0016]

[0017] in, This represents the upprojection weights of the low-dimensional adapter for EEG signals. This indicates the upward projection bias of the low-dimensional adapter for EEG signals. This represents the upprojection weights of the low-dimensional image adapter. This represents the upper projection bias of the low-dimensional image adapter.

[0018] Preferably, step S4 includes: Construct low-dimensional bottleneck matrices for EEG signals respectively and image low-dimensional bottleneck matrix And standardize the process; Calculating the low-dimensional bottleneck matrix of EEG signals and image low-dimensional bottleneck matrix Low-dimensional cross-modal correlation matrix between The expression is:

[0019] Based on low-dimensional bottleneck matrix of EEG signals and image low-dimensional bottleneck matrix Low-dimensional cross-modal correlation matrix between Calculate the low-dimensional cross-modal correlation constraint loss.

[0020]

[0021] in, Represents a low-dimensional cross-modal correlation matrix The diagonal constraint terms in the text, Represents a low-dimensional cross-modal correlation matrix The first in The dimension and the first Correlation between dimensions This indicates the constraint weights for off-diagonal terms.

[0022] Preferably, step S5 includes: Characteristics of EEG signals and image features Perform normalization processing; Calculate the similarity between normalized EEG signal features and image features. The expression is:

[0023] in, Indicates the number of digits after normalization. EEG signal characteristics in multiple dimensions Indicates the number of digits after normalization. Image features in 3 dimensions; Based on the similarity between EEG signal features and image features Calculate the retrieval loss from EEG signal to image direction respectively. Image-to-EEG signal retrieval loss The expression is:

[0024]

[0025] in, This represents the smoothing coefficient that controls the distribution of similarity. Retrieval loss based on EEG signal to image orientation Image-to-EEG signal retrieval loss Calculate retrieval loss The expression is: .

[0026] Preferably, step S6 includes: Based on the projection weights of low-dimensional adapters for EEG signals and low-dimensional adapters for images, an adapter weight loss is constructed. The expression is:

[0027] Loss weights are determined based on pre-defined low-dimensional correlation constraints. and adapter weight regularization coefficients For retrieval loss and adapter weight loss We weight the loss with the low-dimensional cross-modal correlation constraint to obtain the overall training objective function. The expression is: ; Backpropagation is performed based on the overall training objective function to update the trainable parameters of the EEG signal encoding model, the EEG signal low-dimensional adapter, and the image low-dimensional adapter.

[0028] Preferably, step S7 includes: Using the trained EEG signal coding model and low-dimensional adapter for EEG signals Extract the signal features of the EEG signal to be matched to obtain the query EEG signal features. The expression is:

[0029] in, Indicates the EEG signal to be matched; Using image coding models and the trained low-dimensional image adapter Extract candidate matching image features to obtain candidate matching image features.

[0030]

[0031] in, Represents candidate matching images, Indicates the candidate matching image index. ; Calculate query brainwave signal features Matching image features with each candidate similarity between The expression is:

[0032] Based on similarity Sort the candidate matching images; Based on a preset search quantity K, the K candidate matching images with the highest similarity are selected from the candidate matching image ranking results, and used as the target image set corresponding to the EEG signal to be matched.

[0033] Secondly, this application also proposes a computer storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform an EEG signal feature matching method based on low-dimensional shared subspace constraints.

[0034] Thirdly, this application also proposes a computer device, which includes a memory, a processor, and a computer program stored in the memory that can be run by the processor. The processor executes the computer program to implement a method for matching EEG signal features based on low-dimensional shared subspace constraints.

[0035] Compared with the prior art, the beneficial effects of the present invention are: This invention proposes a method for matching EEG signal features based on low-dimensional shared subspace constraints. First, it acquires the EEG signals generated when a subject views a visual image, along with the corresponding visual image. Then, it extracts high-dimensional semantic embeddings of the EEG signals and the image using an EEG signal encoding model and an image encoding model, respectively. These high-dimensional semantic embeddings are then mapped to a low-dimensional bottleneck space using a low-dimensional EEG signal adapter and an image low-dimensional adapter, respectively, to obtain low-dimensional bottleneck representations of the EEG signals and the image. A cross-modal correlation matrix is ​​constructed based on these low-dimensional bottleneck representations, and the correlation between corresponding semantic dimensions of EEG signals and images is enhanced based on this matrix, while reducing redundant correlations between non-corresponding semantic dimensions. Furthermore, a joint training objective is constructed based on cross-modal retrieval loss and low-dimensional cross-modal correlation constraint loss to optimize the model parameters. Finally, the trained model extracts the EEG signal features to be matched, and the corresponding image is determined based on the similarity between the EEG signal features and the candidate image features. This scheme establishes a structural consistency relationship between EEG signals and image semantic features in a low-dimensional shared subspace, reducing the impact of high-dimensional feature redundancy and EEG signal noise on the matching process, and improving the accuracy and stability of cross-modal image retrieval based on EEG signals. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating the EEG signal feature matching method based on low-dimensional shared subspace constraints proposed in Embodiment 1 of the present invention. Figure 2 This is a schematic diagram illustrating the training process based on the total loss function proposed in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the EEG signal feature matching reasoning process proposed in Embodiment 2 of the present invention; Figure 4 This is a schematic diagram of the structure of the computer device proposed in Embodiment 3 of the present invention. Detailed Implementation

[0037] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. To better illustrate this embodiment, some parts of the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions; It is understandable to those skilled in the art that some well-known details may be omitted from the accompanying drawings.

[0038] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0039] The positional relationships depicted in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent. Example 1 This embodiment proposes a method for matching EEG signal features based on low-dimensional shared subspace constraints. A flowchart illustrating this method can be found here. Figure 1 This includes the following steps: S1. Collect the electroencephalogram (EEG) signals generated when the subject views visual images; S2. Encode the acquired EEG signals using a preset EEG signal encoding model to obtain a high-dimensional semantic embedding of the EEG signals, and encode the visual image using a preset image encoding model to obtain a high-dimensional semantic embedding of the image. S3. Using a preset low-dimensional adapter for EEG signals, embed high-dimensional semantics of EEG signals into a low-dimensional bottleneck space and use a preset low-dimensional adapter for images to embed high-dimensional semantics of images into a low-dimensional bottleneck space, thereby obtaining low-dimensional bottleneck representations of EEG signals and images respectively. S4. Calculate the low-dimensional cross-modal correlation constraint loss of EEG signals and images based on low-dimensional bottleneck representation of EEG signals and images; S5. Based on the low-dimensional bottleneck representation of EEG signals and the low-dimensional bottleneck representation of images, EEG signal features and image features are obtained respectively, and a retrieval loss is constructed based on the EEG signal output features and image output features. S6. The overall training objective is constructed by combining the retrieval loss and the low-dimensional cross-modal correlation constraint loss. Based on the overall training objective, the trainable parameters of the EEG signal coding model, the low-dimensional EEG signal adapter, and the low-dimensional image adapter are updated respectively. S7. Using the trained EEG signal encoding model and EEG signal low-dimensional adapter, extract the signal features of the EEG signal to be matched, calculate the similarity between the signal features and the image features of the candidate matching image, and determine the target image corresponding to the EEG signal to be matched based on the ranking of similarity.

[0040] In this embodiment, firstly, the EEG signals generated when the subject views a visual image and the corresponding visual image are acquired; then, high-dimensional semantic embeddings of the EEG signals and images are extracted using an EEG signal encoding model and an image encoding model, respectively; finally, the high-dimensional semantic embeddings of the EEG signals and images are mapped to a low-dimensional bottleneck space using a low-dimensional adapter for the EEG signals and an image, respectively, to obtain low-dimensional bottleneck representations of the EEG signals and images; a cross-modal correlation matrix is ​​constructed based on the low-dimensional bottleneck representations, and the correlation between corresponding semantic dimensions of EEG signals and images is enhanced according to the cross-modal correlation matrix, while reducing redundant correlations between non-corresponding semantic dimensions; further, a joint training objective is constructed based on cross-modal retrieval loss and low-dimensional cross-modal correlation constraint loss to optimize the model parameters; finally, the EEG signal features to be matched are extracted using the trained model, and the corresponding image is determined based on the similarity between the EEG signal features and the candidate image features. This scheme establishes a structural consistency relationship between EEG signals and image semantic features in a low-dimensional shared subspace, reducing the impact of high-dimensional feature redundancy and EEG signal noise on the matching process, and improving the accuracy and stability of cross-modal image retrieval based on EEG signals.

[0041] Example 2 In this embodiment, the step of encoding the acquired EEG signals using a preset EEG signal encoding model to obtain high-dimensional semantic embeddings of the EEG signals includes: The acquired EEG signals are unfolded into EEG signal vectors using a pre-defined residual projection network. The expression is:

[0042] in, This indicates the input EEG signal. , Indicates the number of EEG signal channels. Indicates the number of time sampling points. Indicates batch size, Represents the weights of the linear projection layer. Indicates the linear projection layer bias; Residual feedforward transformation is performed on the EEG signal vector to obtain high-dimensional semantic embedding of the EEG signal. The expression is:

[0043] in, , Represents the semantic embedding dimension. and All of these represent the learnable weights of the residual feedforward layer.

[0044] Specifically, before encoding, the process includes acquiring samples of electroencephalogram (EEG) signals generated by subjects while viewing visual stimulus images, and constructing a pairing training set between EEG signals and images. The expression is:

[0045] in, Indicates the first One EEG signal sample, This represents the visual image corresponding to the EEG signal. This indicates the number of training samples. For EEG or MEG brainwave signals, the expression for a single brainwave signal sample is:

[0046] in, Indicates the number of EEG signal channels. Indicates the number of time sampling points; During training, multiple samples are combined into batches of EEG signals for input. The expression is:

[0047] in, Indicates batch size.

[0048] Specifically, the EEG signal encoding model can be implemented using a residual multilayer perceptron, convolutional neural network, Transformer network, or other EEG signal encoding networks.

[0049] Specifically, the preset image coding model can adopt CLIP, SigLIP, DINO, ViT or other visual semantic coding models.

[0050] Specifically, the image encoding model parameters are kept frozen, and only the EEG signal encoder and low-dimensional adaptation module are trained. Image semantic embedding. Represented as:

[0051] in, Represents the image coding model. If the output dimension of the image encoder differs from the embedding dimension of the EEG signal, then they are unified through the image projection layer, expressed as:

[0052] in, This represents the image feature projection layer.

[0053] In this embodiment, before projecting the high-dimensional semantic embedding of the EEG signal onto the low-dimensional bottleneck space using a preset low-dimensional EEG signal adapter, the method further includes batch-level standardization processing of the high-dimensional semantic embedding of the EEG signal, including: Computing high-dimensional semantic embeddings of EEG signals The mean of each feature dimension and variance The expression is:

[0054]

[0055] in, Indicates the first The first of the EEG signal samples Each feature dimension; Based on the obtained mean and variance, the corresponding feature dimensions of each EEG signal high-dimensional semantic embedding are standardized to obtain the standardized EEG signal high-dimensional semantic embedding. .

[0056] Specifically, the high-dimensional semantic embedding matrix of EEG signals in a batch The expression is:

[0057] in, , Indicates the first High-dimensional semantic embedding of EEG signal samples ; Specifically, Indicates the first The mean of each feature dimension in the current batch. Indicates the first The variance of each feature dimension within the current batch; Specifically, the standardized expression is:

[0058] in, It is a very small constant to prevent division by zero; , This represents the standardized EEG signal embedding. This step is used to reduce the impact of inconsistencies in EEG signal feature scales on cross-modal matching.

[0059] In this embodiment, both the low-dimensional adapter for EEG signals and the low-dimensional adapter for images include a lower projection layer and an upper projection layer connected in sequence; wherein, the lower projection layer maps the high-dimensional semantic embeddings of EEG signals and images to a low-dimensional bottleneck space, respectively, to obtain a low-dimensional bottleneck representation of the EEG signals. Low-dimensional bottleneck representation of images The expression is:

[0060]

[0061] in, This represents the standardized high-dimensional semantic embedding of the image. This represents the downprojection weights of the low-dimensional adapter for EEG signals. This indicates the lower projection bias of the low-dimensional adapter for EEG signals. This represents the downprojection weights of the low-dimensional image adapter. This represents the downprojection bias of the low-dimensional image adapter; The upper projection layer represents the low-dimensional bottleneck of the EEG signal modality. Low-dimensional bottleneck representation of image modalities By restoring the original semantic dimension, the EEG signal features were obtained. and image features The expression is:

[0062]

[0063] in, This represents the upprojection weights of the low-dimensional adapter for EEG signals. This indicates the upward projection bias of the low-dimensional adapter for EEG signals. This represents the upprojection weights of the low-dimensional image adapter. This represents the upper projection bias of the low-dimensional image adapter.

[0064] Specifically, the low-dimensional adapter for EEG signals and the low-dimensional adapter for images further include a random inactivation layer, a residual connection layer, and a normalization layer; The random deactivation layer is positioned between the lower projection layer and the upper projection layer. During training, it is used to randomly mask some feature dimensions in the low-dimensional bottleneck representation according to a preset deactivation probability. This reduces the dependence of the low-dimensional adapter on local feature dimensions during feature learning, reduces the impact of noise features and accidental related features in EEG signals on the model training process, and improves the robustness of the low-dimensional shared semantic representation.

[0065] The residual connection layer is used to fuse the features after upprojection with the high-dimensional semantic embedding input to the low-dimensional adapter. Specifically, the feature increment output by the upprojection layer is added element-wise to the original high-dimensional semantic embedding of the corresponding modality to retain the effective information in the original semantic representation. At the same time, the feature adjustment amount learned from the low-dimensional bottleneck space is introduced so that the low-dimensional adapter can achieve feature compression while avoiding the loss of semantic information caused by the reduction of dimensionality.

[0066] The normalization layer is placed after the residual connection layer and is used to normalize the features after residual fusion, so that the fused EEG signal features and image features are consistent in feature scale, reduce the distribution shift caused by feature amplitude differences between different samples and different modalities, and improve the stability of subsequent cross-modal correlation calculation and feature similarity matching.

[0067] By setting the random deactivation layer, residual connection layer and normalization layer in the low-dimensional adapter, the low-dimensional adapter can not only realize information interaction between high-dimensional semantic embedding and low-dimensional bottleneck space, but also retain effective semantic information while reducing feature redundancy, and improve the consistency between EEG signals and image features in the shared semantic space, providing a stable feature representation for subsequent cross-modal correlation constraint learning.

[0068] In this embodiment, step S4 includes: Construct low-dimensional bottleneck matrices for EEG signals respectively and image low-dimensional bottleneck matrix And standardize the process; Calculating the low-dimensional bottleneck matrix of EEG signals and image low-dimensional bottleneck matrix Low-dimensional cross-modal correlation matrix between The expression is:

[0069] Based on low-dimensional bottleneck matrix of EEG signals and image low-dimensional bottleneck matrix Low-dimensional cross-modal correlation matrix between Calculate the low-dimensional cross-modal correlation constraint loss.

[0070]

[0071] in, Represents a low-dimensional cross-modal correlation matrix The diagonal constraint terms in the text, Represents a low-dimensional cross-modal correlation matrix The first in The dimension and the first Correlation between dimensions This indicates the constraint weights for off-diagonal terms.

[0072] Specifically, the low-dimensional bottleneck matrix of the EEG signal and image low-dimensional bottleneck matrix This is a set of low-dimensional bottleneck representations of EEG signals and low-dimensional bottleneck representations of images within a batch. , The expressions are as follows:

[0073]

[0074] Specifically, the low-dimensional cross-modal correlation constraint loss function The first term is a diagonal constraint term, used to drive... The value is close to 1, ensuring a high correlation between the corresponding dimensions of the low-dimensional representations of EEG signals and images; the second term is an off-diagonal redundancy reduction constraint term, used to suppress... This reduces information redundancy between different dimensions. Through the above constraints, this scheme enables the low-dimensional shared subspace to simultaneously satisfy two conditions: first, corresponding dimensions have strong cross-modal consistency; and second, non-corresponding dimensions are decorrelated as much as possible, thereby obtaining a more compact, stable, and discriminable cross-modal shared representation.

[0075] In this embodiment, step S5 includes: Characteristics of EEG signals and image features Perform normalization processing; Calculate the similarity between normalized EEG signal features and image features. The expression is:

[0076] in, Indicates the number of digits after normalization. EEG signal characteristics in multiple dimensions Indicates the number of digits after normalization. Image features in 3 dimensions; Based on the similarity between EEG signal features and image features Calculate the retrieval loss from EEG signal to image direction respectively. Image-to-EEG signal retrieval loss The expression is:

[0077]

[0078] in, This represents the smoothing coefficient that controls the distribution of similarity. Retrieval loss based on EEG signal to image orientation Image-to-EEG signal retrieval loss Calculate retrieval loss The expression is: .

[0079] Specifically, obtaining the characteristics of EEG signal output. and image output features Furthermore, it also includes the characteristics of EEG signal output. and image output features L2 normalization is performed, and the expression is:

[0080]

[0081] in, , .

[0082] In this embodiment, step S6 includes: Based on the projection weights of low-dimensional adapters for EEG signals and low-dimensional adapters for images, an adapter weight loss is constructed. The expression is:

[0083] Loss weights are determined based on pre-defined low-dimensional correlation constraints. and adapter weight regularization coefficients For retrieval loss and adapter weight loss We weight the loss with the low-dimensional cross-modal correlation constraint to obtain the overall training objective function. The expression is: ; Backpropagation is performed based on the overall training objective function to update the trainable parameters of the EEG signal encoding model, the EEG signal low-dimensional adapter, and the image low-dimensional adapter.

[0084] Specifically, through this joint optimization objective, the model not only learns the overall semantic matching relationship between EEG signals and images, but also learns the structural consistency relationship between the two modalities in a low-dimensional shared subspace.

[0085] Specifically, the flowchart for training based on the total loss function is as follows: Figure 2 As shown, this embodiment uses paired samples that correspond one-to-one with EEG signals and visual images to jointly train the EEG signal encoding model, the EEG signal low-dimensional adapter, and the image low-dimensional adapter.

[0086] Specifically, a predetermined number of paired EEG signal-visual image samples are selected from the training sample set to form the current training batch. For the EEG signals in the current training batch, feature encoding is performed using an EEG signal encoding model to obtain high-dimensional semantic embeddings of the EEG signals. The high-dimensional semantic embeddings of the EEG signals are then batch-level standardized to reduce feature scale fluctuations between different EEG signal samples. For the visual images corresponding to each EEG signal, high-dimensional semantic embeddings of the images are extracted using a pre-trained visual encoding model.

[0087] The high-dimensional semantic embeddings of EEG signals and images are input into the low-dimensional adapters for EEG signals and images, respectively. Downprojection processing is then used to obtain the low-dimensional bottleneck representations of EEG signals and images corresponding to the current training batch. The low-dimensional bottleneck representations of each EEG signal sample within a training batch are combined to form a low-dimensional bottleneck matrix for EEG signals; similarly, the low-dimensional bottleneck representations of the corresponding visual images are combined to form a low-dimensional bottleneck matrix for images.

[0088] The low-dimensional bottleneck matrices of EEG signals and images are standardized according to their feature dimensions, and a cross-modal correlation matrix is ​​calculated based on the standardized low-dimensional bottleneck matrices. A low-dimensional cross-modal correlation constraint loss is calculated based on the cross-modal correlation matrix. Specifically, by constraining the diagonal elements of the cross-modal correlation matrix, the corresponding feature dimensions in the low-dimensional bottleneck representations of EEG signals and images maintain a high correlation; by constraining the off-diagonal elements, redundant correlations between non-corresponding feature dimensions are reduced, thereby enabling EEG signal features and image features to form structurally consistent and redundancy-free cross-modal representations in a low-dimensional shared subspace.

[0089] Furthermore, the low-dimensional bottleneck representations of EEG signals and images are restored to their original semantic dimensions by using the upper projection layers of the low-dimensional adapters for EEG signals and images, respectively. Then, the EEG signal output features and image output features are obtained by combining residual connections and normalization processing.

[0090] Based on the EEG signal output features and image output features in the current training batch, cross-modal similarity between samples is calculated. Then, based on the cross-modal similarity, the contrast retrieval loss from EEG signal to image and the contrast retrieval loss from image to EEG signal are calculated respectively. This increases the feature similarity between paired EEG signals and visual images, and decreases the feature similarity between unpaired EEG signals and visual images. The bidirectional contrast retrieval loss is obtained based on the contrast retrieval losses in the two directions.

[0091] The bidirectional contrastive retrieval loss and the low-dimensional cross-modal correlation constraint loss are combined according to preset weights to obtain the total training loss. In one embodiment, a parameter regularization term for the low-dimensional adapter can also be added to the total training loss to limit the parameter size of the low-dimensional adapter.

[0092] Backpropagation is performed based on the total training loss to calculate the gradient of the total training loss relative to each trainable parameter in the EEG signal encoding model, the EEG signal low-dimensional adapter, and the image low-dimensional adapter, and the corresponding model parameters are updated according to the obtained gradient. In a preferred embodiment, the parameters of the pre-trained visual encoding model are kept fixed during training to maintain the stability of the visual semantic space.

[0093] The above process is followed to iterate through multiple training batches until a preset number of training rounds is reached or the total training loss meets a preset convergence condition, thereby obtaining the trained EEG signal encoding model, EEG signal low-dimensional adapter, and image low-dimensional adapter.

[0094] Through the above training methods, the bidirectional contrastive retrieval loss constrains EEG signals and visual images at the sample matching level, while the low-dimensional cross-modal correlation constraint loss constrains the two modalities at the feature dimension structure level. This enables the trained model to establish semantic matching relationships between paired EEG signals and visual images, and to maintain stable cross-modal structural correspondences in the low-dimensional shared subspace.

[0095] In this embodiment, step S7 includes: Using the trained EEG signal coding model and low-dimensional adapter for EEG signals Extract the signal features of the EEG signal to be matched to obtain the query EEG signal features. The expression is:

[0096] in, Indicates the EEG signal to be matched; Using image coding models and the trained low-dimensional image adapter Extract candidate matching image features to obtain candidate matching image features.

[0097]

[0098] in, Represents candidate matching images, Indicates the candidate matching image index. ; Calculate query brainwave signal features Matching image features with each candidate similarity between The expression is:

[0099] Based on similarity Sort the candidate matching images; Based on a preset search quantity K, the K candidate matching images with the highest similarity are selected from the candidate matching image ranking results, and used as the target image set corresponding to the EEG signal to be matched.

[0100] Specifically, the schematic diagram of the reasoning process for matching EEG signal features is as follows: Figure 3 As shown, after the model training is completed, the trained EEG signal encoding model and EEG signal low-dimensional adapter are used to extract features from the EEG signal to be matched, and the obtained EEG signal features are matched with the candidate image features to determine the target image corresponding to the EEG signal to be matched.

[0101] Specifically, the EEG signal to be matched is acquired, and the EEG signal to be matched is input into a trained EEG signal encoding model to obtain the corresponding high-dimensional semantic embedding of the EEG signal. This high-dimensional semantic embedding of the EEG signal is then processed according to the standardization method used in the training phase. The processed high-dimensional semantic embedding of the EEG signal is input into a trained low-dimensional EEG signal adapter, and through down-projection, up-projection, residual connection, and normalization processing, the query EEG signal features corresponding to the EEG signal to be matched are obtained. For each candidate matching image in the candidate matching image set, a visual coding model is used to extract the corresponding high-dimensional semantic embedding, and a trained low-dimensional image adapter is used to process the high-dimensional semantic embedding to obtain the candidate matching image features corresponding to the candidate matching image. In one implementation, the visual coding model and the trained low-dimensional image adapter can be used in advance to extract the image features corresponding to each candidate matching image, and the obtained image features can be stored in the candidate image feature library. After receiving the EEG signal to be matched, the candidate image features can be directly called for matching, reducing the repeated image feature extraction process.

[0102] Further, the similarity between the query EEG signal features and the features of each candidate matching image is calculated. In one embodiment, the similarity is calculated using cosine similarity: The candidate matching images are sorted according to their similarity, and the K candidate matching images with the highest similarity in the sorted results are determined according to the preset number of searches, so as to obtain the target image set corresponding to the EEG signal to be matched.

[0103] In another implementation, when only one matching result needs to be determined, the candidate matching image with the highest similarity is determined as the target image corresponding to the EEG signal to be matched.

[0104] Since the training phase has already utilized low-dimensional cross-modal correlation constraints to establish a structural relationship between EEG signal features and image features in the shared semantic space with corresponding dimensions related and non-corresponding dimensions de-redundant, and has used bidirectional contrastive retrieval loss to improve the feature similarity between paired EEG signals and images, there is no need to recalculate the low-dimensional cross-modal correlation constraint loss or retrieval loss during the inference phase. Instead, it is only necessary to use the trained model to extract EEG signal features and candidate image features and perform similarity matching to obtain the corresponding image retrieval results.

[0105] Example 3 This embodiment also proposes a computer storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to execute an EEG signal feature matching method based on low-dimensional shared subspace constraints.

[0106] In this embodiment, a computer device 100 is proposed, which includes a memory 101, a processor 102, and a computer program stored in the memory 101 that can be executed by the processor. The processor 102 executes the computer program to implement a brainwave signal feature matching method based on low-dimensional shared subspace constraints. A schematic diagram of the device is shown below. Figure 4 As shown.

[0107] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for matching EEG signal features based on low-dimensional shared subspace constraints, characterized in that, Includes the following steps: S1. Collect the electroencephalogram (EEG) signals generated when the subject views visual images; S2. Encode the acquired EEG signals using a preset EEG signal encoding model to obtain a high-dimensional semantic embedding of the EEG signals, and encode the visual image using a preset image encoding model to obtain a high-dimensional semantic embedding of the image. S3. Using a preset low-dimensional adapter for EEG signals, embed high-dimensional semantics of EEG signals into a low-dimensional bottleneck space and use a preset low-dimensional adapter for images to embed high-dimensional semantics of images into a low-dimensional bottleneck space, thereby obtaining low-dimensional bottleneck representations of EEG signals and images respectively. S4. Calculate the low-dimensional cross-modal correlation constraint loss of EEG signals and images based on low-dimensional bottleneck representation of EEG signals and images; S5. Based on the low-dimensional bottleneck representation of EEG signals and the low-dimensional bottleneck representation of images, EEG signal features and image features are obtained respectively, and a retrieval loss is constructed based on the EEG signal output features and image output features. S6. The overall training objective is constructed by combining the retrieval loss and the low-dimensional cross-modal correlation constraint loss. Based on the overall training objective, the trainable parameters of the EEG signal coding model, the low-dimensional EEG signal adapter, and the low-dimensional image adapter are updated respectively. S7. Using the trained EEG signal encoding model and EEG signal low-dimensional adapter, extract the signal features of the EEG signal to be matched, calculate the similarity between the signal features and the image features of the candidate matching image, and determine the target image corresponding to the EEG signal to be matched based on the ranking of similarity.

2. The EEG signal feature matching method based on low-dimensional shared subspace constraints according to claim 1, characterized in that, The process of encoding the acquired EEG signals using a preset EEG signal encoding model to obtain high-dimensional semantic embeddings of the EEG signals includes: The acquired EEG signals are unfolded into EEG signal vectors using a pre-defined residual projection network. The expression is: in, This indicates the input EEG signal. , Indicates the number of EEG signal channels. Indicates the number of time sampling points. Indicates batch size, Represents the weights of the linear projection layer. Indicates the linear projection layer bias; Residual feedforward transformation is performed on the EEG signal vector to obtain high-dimensional semantic embedding of the EEG signal. The expression is: in, , Represents the semantic embedding dimension. and All of these represent the learnable weights of the residual feedforward layer.

3. The EEG signal feature matching method based on low-dimensional shared subspace constraints according to claim 2, characterized in that, Before projecting the high-dimensional semantic embedding of the EEG signal onto the low-dimensional bottleneck space using a preset low-dimensional EEG signal adapter, the process further includes batch-level standardization of the high-dimensional semantic embedding of the EEG signal, including: Computing high-dimensional semantic embeddings of EEG signals The mean of each feature dimension and variance The expression is: in, Indicates the first The first of the EEG signal samples Each feature dimension; Based on the obtained mean and variance, the corresponding feature dimensions of each EEG signal high-dimensional semantic embedding are standardized to obtain the standardized EEG signal high-dimensional semantic embedding. .

4. The EEG signal feature matching method based on low-dimensional shared subspace constraints according to claim 3, characterized in that, Both the low-dimensional adapter for EEG signals and the low-dimensional adapter for images include a lower projection layer and an upper projection layer connected in sequence; wherein, the lower projection layer maps the high-dimensional semantic embeddings of EEG signals and images to a low-dimensional bottleneck space, respectively, to obtain a low-dimensional bottleneck representation of the EEG signals. Low-dimensional bottleneck representation of images The expression is: in, This represents the standardized high-dimensional semantic embedding of the image. This represents the downprojection weights of the low-dimensional adapter for EEG signals. This indicates the lower projection bias of the low-dimensional adapter for EEG signals. This represents the downprojection weights of the low-dimensional image adapter. This represents the downprojection bias of the low-dimensional image adapter; The upper projection layer represents the low-dimensional bottleneck of the EEG signal modality. Low-dimensional bottleneck representation of image modalities By restoring the original semantic dimension, the EEG signal features were obtained. and image features The expression is: in, This represents the upprojection weights of the low-dimensional adapter for EEG signals. This indicates the upward projection bias of the low-dimensional adapter for EEG signals. This represents the upprojection weights of the low-dimensional image adapter. This represents the upper projection bias of the low-dimensional image adapter.

5. The EEG signal feature matching method based on low-dimensional shared subspace constraints according to claim 4, characterized in that, Step S4 includes: Construct low-dimensional bottleneck matrices for EEG signals respectively and image low-dimensional bottleneck matrix And standardize the process; Calculating the low-dimensional bottleneck matrix of EEG signals and image low-dimensional bottleneck matrix Low-dimensional cross-modal correlation matrix between The expression is: Based on low-dimensional bottleneck matrix of EEG signals and image low-dimensional bottleneck matrix Low-dimensional cross-modal correlation matrix between Calculate the low-dimensional cross-modal correlation constraint loss. in, Represents a low-dimensional cross-modal correlation matrix The diagonal constraint terms in Represents a low-dimensional cross-modal correlation matrix The first in The dimension and the first Correlation between dimensions This represents the constraint weights for off-diagonal terms.

6. The EEG signal feature matching method based on low-dimensional shared subspace constraints according to claim 5, characterized in that, Step S5 includes: Characteristics of EEG signals and image features Perform normalization processing; Calculate the similarity between normalized EEG signal features and image features. The expression is: in, Indicates the number of digits after normalization. EEG signal characteristics in multiple dimensions Indicates the number of digits after normalization. Image features in 3 dimensions; Based on the similarity between EEG signal features and image features Calculate the retrieval loss from EEG signal to image direction respectively. Image-to-EEG signal retrieval loss The expression is: in, This represents the smoothing coefficient that controls the distribution of similarity. Retrieval loss based on EEG signal to image orientation Image-to-EEG signal retrieval loss Calculate retrieval loss The expression is: 。 7. The EEG signal feature matching method based on low-dimensional shared subspace constraints according to claim 6, characterized in that, Step S6 includes: Based on the projection weights of low-dimensional adapters for EEG signals and low-dimensional adapters for images, an adapter weight loss is constructed. The expression is: Loss weights are determined based on pre-defined low-dimensional correlation constraints. and adapter weight regularization coefficients For retrieval loss and adapter weight loss We weight the loss with the low-dimensional cross-modal correlation constraint to obtain the overall training objective function. The expression is: ； Backpropagation is performed based on the overall training objective function to update the trainable parameters of the EEG signal encoding model, the EEG signal low-dimensional adapter, and the image low-dimensional adapter.

8. The EEG signal feature matching method based on low-dimensional shared subspace constraints according to claim 6, characterized in that, Step S7 includes: Using the trained EEG signal coding model and low-dimensional adapter for EEG signals Extract the signal features of the EEG signal to be matched to obtain the query EEG signal features. The expression is: in, Indicates the EEG signal to be matched; Using image coding models and the trained low-dimensional image adapter Extract candidate matching image features to obtain candidate matching image features. in, Represents candidate matching images, Indicates the candidate matching image index. ; Calculate query brainwave signal features Matching image features with each candidate Similarity between The expression is: Based on similarity Sort the candidate matching images; Based on a preset search quantity K, the K candidate matching images with the highest similarity are selected from the candidate matching image ranking results, and used as the target image set corresponding to the EEG signal to be matched.

9. A computer storage medium, characterized in that, It stores a computer program, which includes program instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 8.

10. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory that can be run on the processor, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 8.

Images