An electroencephalogram-image collaborative contrast learning method based on asymmetric feedback

Abstract

The present application relates to a kind of electroencephalogram-image collaborative contrast learning method based on asymmetric feedback, belong to electroencephalogram-image collaborative processing technical field, solve the problem that the stability and self-optimization ability of man-machine collaborative determination system is insufficient due to the difference information of the determination result of existing technology not being effectively utilized electroencephalogram and visual model dynamic feedback learning.The method comprises: the multi-modal data flow of the subject in the process of RSVP experiment is synchronously collected and preprocessed;Each preprocessed electroencephalogram signal, corresponding stimulus image and real label are constructed into aligned electroencephalogram-image data pair;Each group of electroencephalogram-image data is trained using collaborative contrast learning model based on asymmetric feedback electroencephalogram-image collaborative contrast learning, and the parameters of collaborative contrast learning model are optimized;Using the optimized collaborative contrast learning model, the preprocessed electroencephalogram signal or stimulus image of new subject is determined.

Description

Technical Field

[0001] This invention relates to the field of EEG image collaborative processing technology, and in particular to an EEG-image collaborative comparative learning method based on asymmetric feedback. Background Technology

[0002] In the RSVP (Rapid Serial Visual Presentation) target detection task, images are presented continuously at extremely high frequency, and participants must identify the target stimuli within a very short time. Existing research indicates that the EEG signals generated by participants during the process contain target-related feature components (such as P300), thus EEG-based target interpretation methods have been extensively studied. However, due to limitations such as low signal-to-noise ratio of EEG signals, large individual variability, and task fatigue, single EEG models still suffer from insufficient stability in practical applications.

[0003] On the other hand, although deep learning-based visual models have achieved high recognition accuracy in RSVP image target detection, they still face the risk of misjudgment when faced with complex scenes or low-quality images. Existing multimodal methods mostly adopt simple feature-level or decision-level fusion strategies, which neither specifically model the "difference between human and machine judgment results" nor fully explore the complementary relationship between EEG judgment results and image judgment results to promote continuous self-optimization of the model.

[0004] Therefore, how to construct a new learning mechanism that dynamically updates the model parameters of the corresponding modalities, and ultimately achieves the co-evolution of human brain perception and visual artificial intelligence, is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] Based on the above analysis, the embodiments of the present invention aim to provide an EEG-image collaborative contrast learning method based on asymmetric feedback, in order to solve the problem of insufficient stability and self-optimization ability of human-machine collaborative judgment system caused by the failure of existing technologies to effectively utilize the difference information between EEG and visual model judgment results for dynamic feedback learning. This invention provides a co-contrast learning method based on asymmetric feedback between electroencephalography (EEG) and images, the method comprising: The multimodal data streams of the subjects during the RSVP experiment were synchronously acquired and preprocessed; each preprocessed EEG signal, the corresponding stimulus image, and the real label were used to construct an aligned EEG-image data pair. The parameters of the EEG-image collaborative contrast learning model were optimized by using a collaborative contrast learning model to train the EEG-image collaborative contrast learning model based on asymmetric feedback for each group of EEG-image data. An optimized co-contrast learning model was used to identify targets in preprocessed EEG signals or stimulus images of new subjects.

[0006] Based on the above method, the present invention also makes the following improvements: Furthermore, the collaborative contrastive learning model includes an EEG decoder, an image decoder, an EEG mapper, an image mapper, and a unified embedding space; among which, The input ends of the EEG decoder and the image decoder are used to receive preprocessed EEG signals and stimulation images, respectively. The output ends of the EEG decoder and the image decoder are connected to the input ends of the EEG mapper and the image mapper, respectively. The output ends of the EEG mapper and the image mapper are connected to a unified embedding space.

[0007] Furthermore, the EEG-image collaborative contrast learning training based on asymmetric feedback executes: Each set of EEG-image data pairs is preprocessed, and the EEG signal and stimulus image are input into the EEG decoder and image decoder, respectively, to obtain the corresponding EEG feature vector and EEG judgment result, and the image feature vector and image judgment result. Using an EEG mapper and an image mapper, EEG feature vectors and image feature vectors are projected into a cross-modal embedding space, respectively, and EEG embedding vectors and image embedding vectors of the same dimension are output into a unified embedding space. In a unified embedding space, the correctness of the corresponding EEG and image judgment results is verified based on the real labels of the stimulus images. Based on the correct judgment asymmetric feedback correction mechanism, the collaborative contrastive learning model is optimized by combining EEG embedding vectors and image embedding vectors.

[0008] Furthermore, the correct asymmetric feedback correction mechanisms based on the judgment include visually guided EEG correction feedback mechanisms and neurally guided visual correction feedback mechanisms.

[0009] Furthermore, when the EEG interpretation result is incorrect while the image interpretation result is correct, a visually guided EEG correction feedback mechanism is triggered: Using the correctly determined image embedding vector as the reference anchor point and the EEG embedding vector as the sample to be corrected, the parameters of the EEG decoder and EEG mapper are optimized.

[0010] Furthermore, when the image judgment result is incorrect while the EEG judgment result is correct, a neurally guided visual correction feedback mechanism is triggered: Using the correctly determined EEG embedding vector as the reference anchor point and the image embedding vector as the sample to be corrected, the parameters of the image decoder and image mapper are optimized.

[0011] Furthermore, when the EEG determination result is consistent with the image determination result, a weakly constrained loss function is used to jointly update the parameters of the EEG decoder and EEG mapper, as well as the image decoder and image mapper.

[0012] Furthermore, the optimized collaborative contrastive learning model is used to perform target determination in a single modality or across modalities.

[0013] Furthermore, single-modal label prediction includes: When only new subjects' preprocessed EEG signals are received, the target determination result is output through the EEG decoder. When only new subjects receive preprocessed stimulus images, the target determination result is output through the image decoder.

[0014] Furthermore, when a new subject receives preprocessed EEG signals and stimulus images, the EEG embedding vector and EEG determination probability of the EEG signals processed by the EEG decoder, and the image embedding vector and image determination probability of the stimulus images processed by the image decoder are obtained respectively. Based on the EEG embedding vector and the image embedding vector, the fusion weight based on the embedding space is calculated. The EEG determination probability and the image determination probability are weighted and fused using the fusion weight to obtain the target determination result.

[0015] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: The proposed EEG-image collaborative contrast learning method based on asymmetric feedback constructs a cross-modal feedback loop and uses the features of the correctly judged modality as feedback guidance signals to perform asymmetric contrast correction on the incorrectly judged modality within a unified embedding space. This solves the problem of poor robustness of a single modality under complex conditions and achieves a synergistic improvement in target detection accuracy in RSVP tasks.

[0016] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0017] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. Figure 1 A flowchart of an EEG-image collaborative contrast learning method based on asymmetric feedback provided in an embodiment of the present invention; Figure 2 Another flowchart of the EEG-image collaborative contrast learning method based on asymmetric feedback provided in an embodiment of the present invention. Detailed Implementation

[0018] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0019] A specific embodiment of the present invention discloses an EEG-image collaborative contrast learning method based on asymmetric feedback, the flowchart of which is shown below. Figure 1 and Figure 2 As shown.

[0020] Step S1: Synchronously acquire and preprocess the multimodal data streams of the subjects during the RSVP experiment; construct aligned EEG-image data pairs by combining each preprocessed EEG signal, the corresponding stimulus image, and the real label.

[0021] In the specific implementation process, the study adopted the RSVP experimental paradigm, which simultaneously collected the EEG signals of the subjects while they viewed a continuous sequence of images, as well as the corresponding stimulus images and their real labels, thus forming a multimodal data stream.

[0022] The acquired EEG signals and stimulus images were preprocessed separately. Then, each preprocessed EEG signal, its corresponding stimulus image, and its ground truth label were used to construct an aligned EEG-image data pair.

[0023] In the specific implementation process, the following preprocessing operations can be performed on the acquired EEG signals. Firstly, noise reduction: this step can employ filtering methods such as low-pass filtering, high-pass filtering, or band-pass filtering to remove high-frequency noise or low-frequency drift interference from the EEG signals; artifact removal can also be performed, such as removing artifact signals caused by blinking, muscle activity, or other external factors, thereby ensuring the purity of the EEG signals. Secondly, segmentation: the continuous EEG signals are divided into multiple smaller segments according to certain time intervals or event markers for subsequent analysis and processing. Thirdly, normalization: this step helps eliminate differences between different individuals and between different channels. For example, when extracting signals, a sliding window method based on stimulus trigger markers can be used, such as selecting signal segments within the time range of -200ms to 1000ms, and using the intra-channel Z-score normalization method to eliminate individual impedance differences, making the data between different individuals or different channels comparable. For stimulus images, preprocessing is also required. One key preprocessing step is size normalization. Since the original stimulus images may vary in size, size normalization adjusts all images to a uniform size for subsequent processing and analysis. In addition, data augmentation can be performed. Data augmentation increases data diversity and improves the model's generalization ability and robustness by rotating, flipping, scaling, and adjusting brightness of images. Step S2: Use the co-contrast learning model to train the EEG-image data of each group based on asymmetric feedback, and optimize the parameters of the co-contrast learning model.

[0024] In this embodiment, the collaborative contrastive learning model includes an EEG decoder, an image decoder, an EEG mapper, an image mapper, and a unified embedding space. The input terminals of the EEG decoder and the image decoder are used to receive preprocessed EEG signals and stimulus images, respectively. The output terminals of the EEG decoder and the image decoder are connected to the input terminals of the EEG mapper and the image mapper, respectively. The output terminals of the EEG mapper and the image mapper are connected to the unified embedding space. The specific training process is described below.

[0025] Step S21: Input the preprocessed EEG signal and stimulation image of each EEG-image data pair into the EEG decoder and image decoder respectively to obtain the corresponding EEG feature vector and EEG judgment result, and the image feature vector and image judgment result.

[0026] In the specific implementation process, the EEG decoder can decode and perceive the preprocessed EEG signal in the following ways: input the preprocessed EEG signal into the EEG decoder, extract high-dimensional EEG features and output the EEG judgment result and EEG judgment probability; at the same time, extract the corresponding EEG feature vector in the feature layer of the EEG decoder.

[0027] In the specific implementation process, the image decoder can decode and perceive the features of the stimulus image in the following ways: input the preprocessed stimulus image into the image decoder, extract visual features and output the image judgment result and image judgment probability; at the same time, extract the corresponding image feature vector in the feature layer of the image decoder.

[0028] In practical implementation, the decoder architecture is not limited to a specific model. As a preferred option, the EEG decoder can employ EEGNet or Deep4Net; for example, when using the EEGNet architecture, features before the fully connected layers are extracted as high-dimensional feature vectors. The image decoder can use ResNet-50 or Swin Transformer, obtaining high-dimensional feature vectors from the output before their global average pooling layers. The decoder not only outputs decision probabilities but also retains deep semantic features for projection into a unified contrastive embedding space in subsequent steps. This design allows for flexible replacement of the feature extraction module based on the complexity of different tasks. It should be noted that the specific model selections described above are merely examples and do not constitute a limitation on the scope of this invention.

[0029] Step S22: Use the EEG mapper and image mapper to project the EEG feature vector and image feature vector across the modal embedding space, respectively, and output the EEG embedding vector and image embedding vector with the same dimension to the unified embedding space.

[0030] In this embodiment, through the processing of the EEG mapper and the image mapper, the features of two heterogeneous modalities can be mapped to the same high-dimensional contrastive learning embedding space, thereby obtaining the EEG embedding vector and the image embedding vector, laying the foundation for calculating the semantic distance between modalities.

[0031] Step S23: In the unified embedding space, verify the correctness of the corresponding EEG judgment results and image judgment results based on the true labels of the stimulus images; based on the correct judgment asymmetric feedback correction mechanism, combine the EEG embedding vector and the image embedding vector to optimize the collaborative contrastive learning model.

[0032] Preferably, in this embodiment, the asymmetric feedback correction mechanism based on correct judgment includes a visually guided EEG correction feedback mechanism and a neurally guided visual correction feedback mechanism. The specific implementation is described below.

[0033] In practice, when the EEG judgment result is incorrect while the image judgment result is correct, a visual-to-EEG feedback mechanism is triggered. This mechanism optimizes the parameters of the EEG decoder and EEG mapper by using the correctly judged image embedding vector as a reference anchor and the EEG embedding vector as a sample to be corrected. Specifically, the correctly judged image embedding vector is used as a high-confidence reference anchor, and the EEG embedding vector is used as a sample to be corrected. A comparative loss function is used to narrow the distance between the two, leveraging the stability of visual features to "guide" the EEG network in updating its parameters.

[0034] For example, the contrastive loss function of a visually guided EEG-corrected feedback mechanism. It can be represented as: (1) in, Represents the EEG embedding vector. This represents the corresponding image embedding vector (as an anchor). This indicates the first [number] in the training batch. One negative sample, This represents the total number of negative samples in the training batch; Represents the cosine similarity function; This represents the temperature coefficient.

[0035] Under this loss function constraint, backpropagation is used to update only the parameters of the EEG decoder. Parameters of EEG mapper While fixing the image branch parameters (the parameters of the image decoder) and parameters of the image mapper This allows for unidirectional visual guidance of brainwaves.

[0036] When the image recognition result is incorrect while the EEG result is correct (e.g., the image is blurry but the human brain recognizes it accurately), a neurally guided visual correction feedback (EEG-to-Visual Feedback) mechanism is triggered: using the correctly judged EEG embedding vector as a reference anchor point and the image embedding vector as the sample to be corrected, the parameters of the image decoder and image mapper are optimized. In specific implementation, the correctly judged EEG embedding vector is used as a high-confidence reference anchor point, and the image embedding vector is used as the sample to be corrected. By utilizing the specific characteristics of human brain perception, the attentional bias of the visual network is "corrected," and the parameters of the image decoder and image mapper are updated.

[0037] For example, in neurally guided vision, the contrast loss function of the neurally guided visual correction feedback mechanism. It can be represented as: (2) Under this loss function constraint, backpropagation only updates the parameters of the image decoder. With parameters of the image mapper While fixed image branch parameters (parameters of the EEG decoder) And parameters of EEG mapper This enables unidirectional guidance of vision via brainwaves.

[0038] When the EEG and image assessment results are consistent (both correct or both incorrect), a weakly constrained loss function is used to jointly update the parameters of the EEG decoder and EEG mapper, and the image decoder and image mapper. This weakly constrained loss function... Represented as: (3) Where sim(·) is the similarity measurement function, Indicates the similarity threshold. This represents the gradient scaling factor (ranging from 0.01 to 0.1).

[0039] When the similarity exceeds a threshold, gradient updates are not performed to avoid overfitting and invalid gradient propagation. During this stage, both EEG and image branch parameters are updated slightly simultaneously to maintain the stability of model training.

[0040] By repeating the above steps, the EEG decoder and image decoder, and the image decoder and image mapper, continuously iterate under the cross-feedback mechanism, thereby achieving bidirectional collaborative optimization between "human teaching machine" and "machine teaching human".

[0041] Step S3: Using the optimized co-contrast learning model, target identification is performed on the preprocessed EEG signals or stimulus images of new subjects.

[0042] In practical applications, the optimized collaborative contrastive learning model is used for single-modal target determination or cross-modal target determination. The specific implementation method is described below.

[0043] In this embodiment, single-modal label prediction includes the following two cases: (1) When only new subjects receive preprocessed EEG signals, the target determination result is output through the EEG decoder; (2) When only new subjects receive pre-processed stimulus images, the target determination result is output through the image decoder.

[0044] In this embodiment, cross-modal label prediction corresponds to the following cases: (3) When a new subject receives preprocessed EEG signals and stimulation images, the EEG embedding vector and EEG determination probability of the EEG signal processed by the EEG decoder and the image embedding vector and image determination probability of the stimulation image processed by the image decoder are obtained respectively. The fusion weight based on the embedding space is calculated based on the EEG embedding vector and the image embedding vector. The EEG determination probability and the image determination probability are weighted and fused using the fusion weight to obtain the target determination result.

[0045] In this embodiment, joint judgment or confidence fusion is performed based on embedding space similarity, thereby improving the overall recognition accuracy. The specific implementation process is described below.

[0046] First, obtain the EEG embedding vectors respectively. With image embedding vector Calculate the cosine similarity between the two. , represented as: (4) Since the range of values ​​for cosine similarity is Linearly map it to The interval is used to obtain the fusion weights based on the embedding space. : (5) When the similarity between the two modal embedding vectors is high, it indicates that the EEG and image judgments are highly consistent and the fusion weights are close to being balanced. When the similarity is low, the weights shift to the side with more stable judgments, thereby improving the robustness of the overall judgment.

[0047] Subsequently, the probability of the EEG determination results from the EEG decoder was obtained. The probability of image determination results from the image decoder The weighted fusion result was obtained. : (6) Finally, the weighted fusion results will be... The category with the highest probability is used as the target determination result. Specifically, if a certain category has the highest probability value in the weighted fusion result y, then that category is determined as the final target determination result. For example, assuming the probability distribution obtained after fusion is y=[0.2, 0.7, 0.1], corresponding to categories A, B, and C respectively, since category B has the highest probability of 0.7, the target determination result is category B. Through this dynamic weighted fusion method based on embedding space similarity, the complementary information of EEG signals and stimulus images can be fully utilized, effectively improving the accuracy of target determination in complex scenes.

[0048] This method can be applied to scenarios such as brain-computer interface target detection, assisted visual recognition, and human-computer collaborative decision-making.

[0049] Compared with the prior art, the present invention has at least the following beneficial effects: (1) A dynamic error correction feedback loop was constructed: This invention abandons the traditional static feature fusion and innovatively proposes an asymmetric feedback mechanism based on the correctness of the judgment. By allowing the "correctly judged" mode to actively provide supervision signals to the "incorrectly judged" mode, the parameters are directly corrected from the source of the error, which significantly improves the training efficiency.

[0050] (2) Enhanced the robustness and complementarity of the model: The stability of the visual model is used to compensate for the low signal-to-noise ratio of EEG, while the flexibility of human brain perception is used to compensate for the blind spots of the visual model in specific scenarios, thus achieving true cross-modal complementary advantages.

[0051] (3) Improved system adaptability: This feedback mechanism does not depend on a specific network backbone structure and can be used as a general plug-in learning strategy, which is widely applicable to various brain-computer collaboration and human-computer hybrid intelligent systems.

[0052] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0053] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A brainwave-image collaborative contrast learning method based on asymmetric feedback, characterized in that, The method includes: The multimodal data streams of the subjects during the RSVP experiment were synchronously acquired and preprocessed; each preprocessed EEG signal, the corresponding stimulus image, and the real label were used to construct an aligned EEG-image data pair. The parameters of the co-contrast learning model were optimized by using a co-contrast learning model to train the EEG-image data of each group based on asymmetric feedback. The co-contrast learning model includes an EEG decoder, an image decoder, an EEG mapper, an image mapper, and a unified embedding space. Using an optimized co-contrast learning model, target identification was performed on preprocessed EEG signals or stimulus images of new subjects. The EEG-image collaborative contrast learning training based on asymmetric feedback executes: Each set of EEG-image data pairs is preprocessed, and the EEG signal and stimulus image are input into the EEG decoder and image decoder, respectively, to obtain the corresponding EEG feature vector and EEG judgment result, and the image feature vector and image judgment result. Using an EEG mapper and an image mapper, EEG feature vectors and image feature vectors are projected into a cross-modal embedding space, respectively, and EEG embedding vectors and image embedding vectors of the same dimension are output into a unified embedding space. In a unified embedding space, the correctness of the corresponding EEG judgment results and image judgment results is verified based on the real labels of the stimulus images. Based on the correct judgment asymmetric feedback correction mechanism, the collaborative contrastive learning model is optimized by combining EEG embedding vectors and image embedding vectors. Among them, the correct judgment asymmetric feedback correction mechanism includes a visually guided EEG correction feedback mechanism and a neurally guided visual correction feedback mechanism. When the EEG interpretation result is incorrect while the image interpretation result is correct, a visually guided EEG correction feedback mechanism is triggered: Using the correctly determined image embedding vector as the reference anchor point and the EEG embedding vector as the sample to be corrected, the parameters of the EEG decoder and EEG mapper are optimized. When the image interpretation result is incorrect while the EEG interpretation result is correct, a neurally guided visual correction feedback mechanism is triggered: Using the correctly determined EEG embedding vector as the reference anchor point and the image embedding vector as the sample to be corrected, the parameters of the image decoder and image mapper are optimized. When the EEG determination result is consistent with the image determination result, a weakly constrained loss function is used to jointly update the parameters of the EEG decoder and EEG mapper, and the image decoder and image mapper.

2. The EEG-image collaborative contrast learning method based on asymmetric feedback according to claim 1, characterized in that, The input ends of the EEG decoder and the image decoder are used to receive preprocessed EEG signals and stimulation images, respectively. The output ends of the EEG decoder and the image decoder are connected to the input ends of the EEG mapper and the image mapper, respectively. The output ends of the EEG mapper and the image mapper are connected to a unified embedding space.

3. The EEG-image collaborative contrast learning method based on asymmetric feedback according to claim 2, characterized in that, The optimized collaborative contrastive learning model is used to determine targets in a single modality or across modalities.

4. The EEG-image collaborative contrast learning method based on asymmetric feedback according to claim 3, characterized in that, Single-modal label prediction includes: When only new subjects' preprocessed EEG signals are received, the target determination result is output through the EEG decoder. When only new subjects receive preprocessed stimulus images, the target determination result is output through the image decoder.

5. The EEG-image collaborative contrast learning method based on asymmetric feedback according to claim 4, characterized in that, When a new subject receives preprocessed EEG signals and stimulus images, the EEG embedding vector and EEG determination probability of the EEG signals processed by the EEG decoder and the image embedding vector and image determination probability of the stimulus images processed by the image decoder are obtained respectively. Based on the EEG embedding vector and the image embedding vector, the fusion weight based on the embedding space is calculated. The EEG and image-based determination probabilities are weighted and fused using fusion weights to obtain the target determination result.

Images