Semantic interference resistant training method and semantic communication method based on electroencephalogram joint learning

Abstract

The invention provides an anti-semantic interference training method and a semantic communication method based on electroencephalogram joint learning. The method comprises the following steps: applying semantic interference to an original image to generate a disturbance image; respectively inputting the original image and the disturbance image into an image encoder to extract image features, and inputting corresponding electroencephalogram signals into an electroencephalogram encoder to extract electroencephalogram features; and based on the original image features, the disturbance image features and the electroencephalogram features, performing comparative learning through a bidirectional InfoNCE loss function, and updating parameters of an image encoder and an electroencephalogram encoder. Joint comparative learning is carried out through the electroencephalogram signals and the image data, the attention of an image encoder to key information is enhanced by utilizing the characteristics of the electroencephalogram signals, and therefore the robustness of a semantic communication system when the semantic communication system is attacked by semantic noise is improved. According to the method, the semantic interference resistance of a semantic communication system is enhanced based on the natural robust feature of the electroencephalogram signal, and the method is suitable for the complex situation that diversified and multi-type semantic interference exists.

Description

EEG-based anti-semantic interference training and semantic communication methods Technical Field

[0001] This invention relates to the field of semantic communication technology, and in particular to an anti-semantic interference training method and a semantic communication method based on EEG co-learning. Background Technology

[0002] With the development of deep neural networks and end-to-end learning technologies, and the increasing demands for intelligent communication systems in 6G networks, semantic communication, oriented towards the semantic layer, has gradually become an emerging and popular technology. As a novel communication architecture, semantic communication systems focus on completing information transmission at the semantic level. By extracting task-related semantic information from the data and sending it to the receiving end, it effectively reduces data transmission volume and bandwidth requirements. A semantic communication system generally includes a knowledge base sharing mechanism, a semantic encoder / decoder, and a channel encoder / decoder. The semantic encoder, leveraging the shared knowledge base, extracts semantic information for specific tasks, while the semantic decoder is responsible for data recovery or task processing based on semantics. Research on semantic communication has introduced important concepts such as semantic entropy, semantic rate distortion, and semantic channel capacity, and has constructed a complete theoretical system for semantic communication.

[0003] However, while semantic communication is closely linked to the characteristics of semantic information, it also faces security challenges. In semantic communication systems, there exists a special type of interference: "semantic interference." Unlike interference in traditional communication systems, semantic interference can modify high-dimensional features while maintaining visual invariance, leading to misunderstandings and decoding errors at the receiving end, resulting in a deviation between the expected and reconstructed semantics. Due to the open nature of communication systems, attackers often exploit semantic perturbations as a means of interference to disrupt or deceive semantic communication systems, thus creating significant security vulnerabilities. Especially in the image domain, semantic interference often possesses strong concealment due to its subtle perturbations; therefore, research on combating semantic interference in the image domain is particularly important.

[0004] Current research has focused on addressing the impact of semantic interference on semantic communication systems, with most studies concentrating on solutions from the field of artificial intelligence. However, overall, existing methods are largely limited to the transfer of AI approaches, such as enhancing the model's ability to extract non-perturbative features. Their focus is mostly confined to single-modal data, and they haven't proposed feasible methods to combat interference. Furthermore, current methods generally treat the perturbation generation method as a known factor to guide adversarial learning, which doesn't reflect typical interference scenarios. Summary of the Invention

[0005] This invention provides a semantic interference resistance training method and a semantic communication method based on EEG co-learning to address the shortcomings of existing technologies where anti-interference capabilities rely on known perturbation patterns and lack generalization ability to unknown interference. The technical solution proposed by this invention is as follows: Firstly, this invention provides a semantic interference resistance training method based on EEG co-learning, comprising: acquiring an original image and its corresponding EEG signal; applying semantic interference to the original image to generate a perturbation image; inputting the original image and the perturbation image into an image encoder respectively to extract corresponding original image features and perturbation image features; inputting the EEG signal into an EEG encoder to extract EEG features; based on the original image features, the perturbation image features, and the EEG features, performing comparative learning through a bidirectional InfoNCE loss function to update the parameters of the image encoder and the EEG encoder; and acquiring the trained image encoder.

[0006] Optionally, the bidirectional InfoNCE loss function implements bidirectional constraints between image features and EEG features, including: image-to-EEG contrast constraints, used to increase the similarity between image features and corresponding EEG features, and decrease the similarity with non-corresponding EEG features; and EEG-to-image contrast constraints, used to increase the similarity between EEG features and corresponding image features, and decrease the similarity with non-corresponding image features.

[0007] Optionally, the calculation of the bidirectional InfoNCE loss function includes: calculating the image-to-EEG direction contrast loss based on the similarity between each image feature and its corresponding EEG feature, and the similarity between the image feature and other EEG features within the batch; calculating the EEG-to-image direction contrast loss based on the similarity between each EEG feature and its corresponding image feature, and the similarity between the EEG feature and other image features within the batch; and determining the bidirectional InfoNCE loss function value based on the two contrast losses.

[0008] Optionally, the step of updating the parameters of the image encoder and the EEG encoder by performing contrastive learning based on the original image features, the perturbed image features, and the EEG features using a bidirectional InfoNCE loss function includes: in the perturbed link, calculating a first contrastive loss based on the perturbed image features and the EEG features; in the original image link, calculating a second contrastive loss based on the original image features and the same EEG feature; calculating a total training loss based on the first and second contrastive losses, and using the total training loss to update the parameters of the image encoder and the EEG encoder through backpropagation.

[0009] Optionally, the image encoder uses a pre-trained model based on a visual Transformer, and the EEG encoder uses a pre-trained model based on a convolutional neural network; the parameter update process is a fine-tuning based on the pre-trained model.

[0010] In a second aspect, the present invention also provides a semantic communication method, comprising: inputting an image to be transmitted into a semantic encoder to extract semantic features; performing channel coding and modulation on the semantic features to generate a transmission signal; and transmitting the transmission signal through a communication channel; wherein the semantic encoder is an image encoder obtained by using the anti-semantic interference training method based on EEG co-learning as described in the first aspect.

[0011] Thirdly, the present invention also provides a semantic interference-resistant training device based on EEG joint learning, comprising: a data acquisition module for acquiring an original image and its corresponding EEG signal; a perturbation generation module for applying semantic interference to the original image to generate a perturbation image; a feature extraction module for inputting the original image and the perturbation image into an image encoder respectively to extract corresponding original image features and perturbation image features, and inputting the EEG signal into an EEG encoder to extract EEG features; a contrastive learning module for performing contrastive learning based on the original image features, the perturbation image features, and the EEG features using a bidirectional InfoNCE loss function to update the parameters of the image encoder and the EEG encoder; and a model output module for acquiring the trained image encoder.

[0012] Fourthly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the anti-semantic interference training method based on EEG co-learning as described in the first aspect above, or the semantic communication method as described in the second aspect above.

[0013] Fifthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the anti-semantic interference training method based on EEG co-learning as described in the first aspect above, or the semantic communication method as described in the second aspect above.

[0014] In a sixth aspect, the present invention also provides a computer program product, comprising a computer program that, when executed by a processor, implements the anti-semantic interference training method based on EEG co-learning as described in the first aspect above, or the semantic communication method as described in the second aspect above.

[0015] Based on the above technical solution, the beneficial effects of this invention compared to existing technologies are as follows: The semantic interference resistance training method and semantic communication method based on EEG joint learning provided by this invention, by acquiring the original image and its corresponding EEG signal, introduces the EEG signal into the interference resistance training process, providing the model with additional and stable semantically relevant information. This multimodal data fusion method breaks the limitations of single-modal data, enabling the model to learn semantic features from a more comprehensive perspective, thereby enhancing its ability to identify and resist unknown interference. Semantic interference is applied to the original image to generate a perturbed image. Then, the original image and the perturbed image are respectively input into an image encoder to extract the corresponding original image features and perturbed image features. Simultaneously, the EEG signal is input into an EEG encoder to extract EEG features. This process not only focuses on the features of the image itself but also incorporates the cognitive features of the human brain regarding images, enabling the model to understand the essence of semantic information more deeply. Based on the original image features, perturbed image features, and EEG features, joint comparative learning is performed using a bidirectional InfoNCE loss function. The features of the EEG signal are used to enhance the image encoder's focus on key information, guiding the model to better distinguish between normal semantic information and interfered semantic information in the feature space. By updating the parameters of the image encoder and EEG encoder in this way, the model can learn more robust feature representations, thereby improving the robustness of the semantic communication system when subjected to semantic noise attacks.

[0016] Because this invention does not pre-assume a perturbation generation method, but rather bases its approach on the unique relationship between semantic perturbations and EEG signals, it enables the model to autonomously learn the relationship between perturbation patterns and semantic features through contrastive learning and multimodal feature fusion. Therefore, it can adapt to various unknown semantic perturbations and is no longer limited to known perturbation patterns. By introducing EEG signals as multimodal information and combining it with a contrastive learning strategy, the model can learn semantic features from a richer perspective. The inherent robustness of EEG signals further enhances the model's ability to grasp semantic information. This allows the model to more accurately identify and resist unknown types of semantic perturbations by leveraging the learned robust feature representations, improving the model's generalization performance. It is particularly suitable for complex situations with diverse and multi-type semantic perturbations, effectively solving the problem of insufficient generalization to unknown perturbations in existing technologies.

[0017] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained through the structures particularly pointed out in the description and the drawings.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 is a flowchart illustrating the anti-semantic interference training method based on EEG co-learning provided by the present invention.

[0021] Figure 2 is a schematic diagram of the comparative learning process provided by the present invention.

[0022] Figure 3 shows the performance verification results of different models provided by this invention under FGSM attack.

[0023] Figure 4 shows the performance verification results of different models provided by this invention under FGSM black-box and white-box attacks.

[0024] Figure 5 is a schematic diagram of the structure of the anti-semantic interference training device based on EEG co-learning provided by the present invention.

[0025] Figure 6 is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] The terms involved in this invention are as follows: (1) Semantic communication: a method that solves the problem of information meaning expression and transmission at the semantic level, and moves part or all of the information meaning understanding link to the sending end, thereby effectively reducing the amount of data transmission and reducing bandwidth requirements. A semantic communication system generally includes knowledge base sharing, semantic codec and channel codec, etc. The semantic encoder extracts semantic information for specific tasks with the help of the shared knowledge base, while the semantic decoder is responsible for restoring the data or performing task processing based on semantics.

[0028] (2) Semantic interference: A special form of interference designed to mislead the receiver's semantic understanding of information, usually achieved by making small perturbations to the signal. Unlike traditional noise interference, semantic interference is characterized by the concealment and low power of its perturbations, maintaining visual invariance. It is usually impossible to detect by conventional methods, but it can have a significant impact on the semantic decoding process. The goal of semantic interference is to change the receiver's understanding of information, leading to incorrect semantic decoding.

[0029] (3) Contrastive learning: A self-supervised learning method that aims to learn features by comparing the similarity between pairs of positive and negative samples. In contrastive learning, the model learns effective feature representations of the data by bringing similar samples closer together and pushing different samples further apart. This method does not rely on label information but trains the model by comparing the relationships between samples. It is often used in unsupervised learning scenarios, especially in feature extraction and representation learning.

[0030] (4) Adversarial training: a defense technique that improves the robustness of a model by actively generating and learning adversarial samples. It exposes the model to carefully constructed malicious samples during training, forcing the model to learn robust features that are not sensitive to input perturbations, thereby enhancing the model's stability when facing attacks such as semantic interference.

[0031] This invention proposes a semantic interference-resistant training method based on joint EEG learning, which utilizes joint comparative learning between EEG signals and images. Based on the visual invariance of semantic perturbations, this method ensures that EEG signals are not affected by visual stimuli. By designing joint comparative learning between EEG signals and image data, it leverages the characteristics of EEG signals to enhance the image encoder's focus on key information, thereby improving the robustness of the semantic communication system against semantic noise attacks. This method, based on the inherent robustness of EEG signals, enhances the semantic communication system's resistance to semantic interference and is applicable to complex situations involving diverse and multi-faceted semantic interference.

[0032] Referring to Figure 1, the anti-semantic interference training method based on EEG co-learning includes the following: S110, acquiring the original image and its corresponding EEG signal.

[0033] Collect raw images covering multiple scenes and categories to ensure the diversity and representativeness of the dataset. Image data should include different themes such as natural landscapes, people, animals, and architecture, and be processed using a uniform resolution and color mode. Data sources include public datasets (such as ImageNet and COCO) or self-built datasets with standardized annotations.

[0034] The EEG signals generated by subjects viewing original images are recorded synchronously using an EEG acquisition device. The experiment must be conducted in a standard environment to control environmental interference. The acquired EEG signals must undergo preprocessing procedures such as filtering and noise reduction to eliminate physiological artifacts and equipment noise, ensuring signal quality and the reliability of subsequent processing.

[0035] S120. Apply semantic interference to the original image to generate a perturbed image.

[0036] The perturbation generator employs a U-Net-based encoder-decoder architecture. The encoder extracts multi-scale deep features from the input image through a series of convolutions and downsampling operations, forming a feature pyramid. The decoder fuses the deep semantic features extracted by the encoder with corresponding shallow features at different scales through upsampling and skip connections, generating a perturbation map with the same size as the input image. The specific structure of the perturbation generator can be found in existing descriptions and will not be elaborated upon here.

[0037] A perturbation generator is used to add semantic perturbation to the original image. The specific process is as follows: the original image is input into the perturbation generator; the encoder extracts hierarchical features from the image; and the decoder generates a corresponding perturbation map based on these features. The generated perturbation map is then added pixel-by-pixel to the original image to obtain the final perturbation image. By adjusting the generator's control parameters, semantic perturbation of different intensities and types can be generated, constructing a training sample set with multi-level perturbation intensities for model training and evaluation in the subsequent contrastive learning phase.

[0038] S130. Input the original image and the perturbed image into the image encoder respectively to extract the corresponding original image features and perturbed image features, and input the EEG signal into the EEG encoder to extract the EEG features.

[0039] The original image and the generated perturbed image are input into the image encoder, respectively. Hierarchical features are extracted through operations such as convolution and pooling, and a fixed-dimensional feature vector is output. This process yields the feature representations of the original image and the perturbed image, namely the original image features and perturbed image features mentioned above. The image encoder can adopt a structure based on a Convolutional Neural Network (CNN), such as a Residual Network (ResNet); it can also adopt a structure based on a Vision Transformer (ViT), such as the image encoder part of the CLIP model (based on the ViT-B / 32 architecture) or its variants.

[0040] The preprocessed EEG signal is input into the EEG encoder, temporal features are extracted, and a feature vector matching the feature dimension of the image is output, thus obtaining the aforementioned EEG features. To address the temporal characteristics of EEG signals, the EEG encoder can employ a structure based on a Temporal Convolutional Network (TCN), such as EEGNet-V4; or a structure based on a Recurrent Neural Network (RNN), such as a Long Short-Term Memory (LSTM) network.

[0041] S140. Based on the original image features, the perturbed image features, and the EEG features, the parameters of the image encoder and the EEG encoder are updated by comparative learning through a bidirectional InfoNCE loss function.

[0042] A cross-modal contrastive learning framework is constructed based on the bidirectional InfoNCE loss function. The core mechanism of this framework is as follows: positive sample pairs consist of cross-modal data from the same semantic source, specifically including two combinations: original image features and corresponding EEG features, and perturbed image features and corresponding EEG features. This is because original images, perturbed images, and EEG signals with the same semantic content share the same semantic core. Negative sample pairs consist of cross-modal data from different semantic sources. Within a training batch, for a given image feature, it forms a negative sample pair with all non-corresponding EEG features within the batch; conversely, for a given EEG feature, it also forms a negative sample pair with all non-corresponding image features within the batch. Through this positive and negative sample pair design, the bidirectional InfoNCE loss function can bring cross-modal feature representations with the same semantic content closer together in the feature space, while simultaneously pushing away feature representations with different semantic contents, thereby achieving stable feature alignment between image modality and EEG modality. During the parameter update process, the gradient is calculated based on the calculated loss function value. The network parameters of the image encoder and the EEG encoder are updated synchronously using the optimization algorithm. Through iterative training, the image encoder gradually acquires robust feature extraction capabilities that are insensitive to semantic interference.

[0043] S150, Obtain the trained image encoder.

[0044] Training terminates when the model's performance metrics on the validation set reach a preset threshold, and the trained image encoder parameters are saved. This encoder has the ability to extract robust features resistant to semantic interference and can be deployed in semantic communication systems as a semantic encoder.

[0045] This invention is based on the visual invariance of semantic perturbations. The key idea is that semantic interference can lead to misunderstandings and decoding errors of semantic information at the receiving end while maintaining visual invariance, and this visual invariance does not affect the EEG signals evoked by visual stimuli. EEG signals contain information about the human brain's cognition and understanding of images, are intrinsically linked to the semantic information of images, and are naturally robust. By acquiring the original image and its corresponding EEG signal, the EEG signal is introduced into the anti-interference training process, providing the model with additional, stable semantically relevant information. This multimodal data fusion method breaks the limitations of single-modal data, enabling the model to learn semantic features from a more comprehensive perspective, thereby enhancing its ability to identify and resist unknown interference. Semantic perturbations are applied to the original image to generate a perturbed image. Then, the original image and the perturbed image are input into an image encoder to extract the corresponding original image features and perturbed image features, respectively. Simultaneously, the EEG signal is input into an EEG encoder to extract EEG features. This process not only focuses on the features of the image itself but also incorporates the cognitive features of the human brain regarding images, enabling the model to understand the essence of semantic information more deeply. Based on original image features, perturbed image features, and EEG features, joint contrastive learning is performed using a bidirectional InfoNCE loss function. The features of the EEG signals are utilized to enhance the image encoder's focus on key information, guiding the model to better distinguish between normal and perturbed semantic information in the feature space. This approach updates the parameters of both the image encoder and the EEG encoder, enabling the model to learn more robust feature representations, thereby improving the robustness of the semantic communication system against semantic noise attacks.

[0046] Because this invention does not pre-assume a perturbation generation method, but rather bases its approach on the unique relationship between semantic perturbations and EEG signals, it enables the model to autonomously learn the relationship between perturbation patterns and semantic features through contrastive learning and multimodal feature fusion. Therefore, it can adapt to various unknown semantic perturbations and is no longer limited to known perturbation patterns. By introducing EEG signals as multimodal information and combining it with a contrastive learning strategy, the model can learn semantic features from a richer perspective. The inherent robustness of EEG signals further enhances the model's ability to grasp semantic information. This allows the model to more accurately identify and resist unknown types of semantic perturbations by leveraging the learned robust feature representations, improving the model's generalization performance. It is particularly suitable for complex situations with diverse and multi-type semantic perturbations, effectively solving the problem of insufficient generalization to unknown perturbations in existing technologies.

[0047] In an optional embodiment, the bidirectional InfoNCE loss function implements bidirectional constraints between image features and EEG features, including: image-to-EEG contrast constraints, used to increase the similarity between image features and corresponding EEG features, and decrease the similarity with non-corresponding EEG features; and EEG-to-image contrast constraints, used to increase the similarity between EEG features and corresponding image features, and decrease the similarity with non-corresponding image features.

[0048] Specifically, during training, each set of input data includes an original image, its corresponding perturbation image, and the corresponding EEG signal. The original image is then processed by an image encoder. The extracted image features and the corresponding EEG signals are processed by an EEG encoder. The extracted EEG features form positive sample pairs. For example, there are images. and their corresponding EEG signals The image features extracted by the image encoder are The brainwave features extracted by the EEG encoder are ,So( , A positive sample pair is formed by combining the original image features with other EEG features besides the corresponding EEG features. A negative sample pair is formed by combining these original image features with other EEG features. Assume the dataset also contains EEG signals. Its corresponding EEG characteristics ,So( , A negative sample pair is formed by randomly selecting multiple non-corresponding EEG features from the batch data and combining them with the current image feature to create a negative sample pair, thereby increasing the diversity and stability of the training. Cosine similarity is used to measure the degree of similarity between the image feature and the EEG feature.

[0049] Similar to the image-to-EEG contrast constraint, EEG features are paired with their corresponding image features to form positive sample pairs. For example, EEG features... Corresponding image features Forming positive sample pairs ( , This EEG feature is paired with other image features besides the corresponding image feature to form a negative sample pair. Assume there are also images... Its corresponding image features are ,So( , A negative sample pair is formed by randomly selecting multiple non-corresponding image features from the batch data and combining them with the current EEG feature. Cosine similarity is also used to calculate the similarity between the EEG feature and the image feature. The loss function for the image-to-EEG contrast constraint is... Loss function for the contrast constraint part of EEG to image All are constructed based on the idea of ​​the InfoNCE loss function.

[0050] This invention employs a bidirectional contrast constraint between image-to-EEG and EEG-to-image models, enabling the model to learn a closer correspondence between image features and EEG features. This correspondence ensures that image features not only contain the visual information of the image itself but also incorporate semantic information related to human brain cognition (reflected through EEG signals). When an image suffers semantic interference, due to the close connection between image features and EEG features, the model can use the stable semantic information in the EEG features to correct the interfered image features, thereby enhancing the robustness of the image feature representation and improving the model's resistance to semantic interference. The bidirectional constraint prompts both the image encoder and the EEG encoder to focus more on key information related to the corresponding modality when extracting features. When extracting image features, the image encoder tends to extract features that have a high similarity to the corresponding EEG features, thus filtering out some irrelevant or easily interfered information; similarly, when extracting EEG features, the EEG encoder also focuses more on information related to the corresponding image features. This mutually reinforcing feature extraction method makes the extracted image features and EEG features more accurate and effective, better representing the semantic content of images and EEG signals. Because the bidirectional InfoNCE loss function incorporates multimodal (image and EEG) information and enables the model to learn more general feature representations through bidirectional constraints, this general feature representation not only performs well on training data but also maintains good performance when faced with unseen data or data from different scenarios. Therefore, the model's generalization ability is significantly improved, enabling it to adapt to a wider range of application scenarios and complex environmental conditions.

[0051] In semantic communication systems, accurate transmission of semantic information is crucial. Image encoders trained with a bidirectional InfoNCE loss function can extract more robust and accurate image features. These features are better able to resist semantic noise interference during transmission, ensuring that the receiver can correctly decode and understand the semantic information of the image. Therefore, the application of the bidirectional InfoNCE loss function helps improve the overall performance of semantic communication systems and enhances the accuracy and reliability of information transmission.

[0052] The semantic interference-resistant training method based on EEG co-learning provided by this invention uses EEG signals as a stable cross-modal supervision signal to guide the image feature extraction network (i.e., image encoder) to focus on perturbation-invariant semantic features during training, thereby effectively resisting the performance degradation caused by semantic interference. Compared with existing single-modal anti-interference methods, this invention introduces EEG modality into the semantic communication scenario, breaking through the limitation of relying solely on image modality for robustness enhancement, and forming a novel cross-modal co-training approach. Its key innovation lies in utilizing the perturbation insensitivity of EEG signals, combined with bidirectional contrastive learning and dual-path co-training mechanisms, to improve system robustness while maintaining performance on clean samples, providing a new solution for the security and practicality of semantic communication. The designed structure is shown in Figure 2.

[0053] In its training structure design, this invention fully utilizes the characteristic that EEG signals are insensitive to pixel-level perturbations. Since semantic interference often does not alter the visual perception of an image and is difficult for the human eye to detect, it can severely affect the discrimination results of deep neural networks. Therefore, for the same image, regardless of whether it is subject to semantic interference, the EEG signals generated by the subject remain consistent. This stability provides a natural advantage for cross-modal supervision. This invention perturbs the image... The corresponding EEG signals As positive sample pairs, input image encoder With EEG encoder Contrastive learning in the feature space forces the model to focus on robust features that can align across modalities, rather than fragile features that are easily disrupted by perturbations. This shifts the training objective from traditional classification accuracy to cross-modal consistency, fundamentally changing the approach to robustness enhancement.

[0054] In an optional embodiment, regarding the design of the loss function for contrastive learning, this invention proposes a bidirectional InfoNCE loss function, which, compared to traditional unidirectional contrastive learning, can more fully utilize cross-modal duality. Let the batch size be... The cosine similarity is Temperature parameters are The calculation of the bidirectional InfoNCE loss function includes: calculating the image-to-EEG contrast loss based on the similarity between each image feature and its corresponding EEG feature, and the similarity between the image feature and other EEG features within the batch; calculating the EEG-to-image contrast loss based on the similarity between each EEG feature and its corresponding image feature, and the similarity between the EEG feature and other image features within the batch; and determining the bidirectional InfoNCE loss function value based on the two contrast losses.

[0055] The bidirectional InfoNCE loss function value L is calculated using the following formula: In Formula 1, This represents the i-th image (original image or perturbed image). This represents the i-th EEG signal. Indicates an image encoder. Indicates brain encoder, Represents an exponential function. This represents an exponential function. In Formula 1, the first part represents the contrast loss from the image to the EEG direction, and the second part represents the contrast loss from the EEG to the image direction.

[0056] The first part ensures that image features can be uniquely matched to corresponding EEG features, and the second part ensures that EEG features can also be uniquely mapped to the correct image features. This bidirectional constraint is more rigorous than the traditional unidirectional InfoNCE, and can form a stable and tight cross-modal feature aggregation, enabling the model to maintain robust semantic representations even when faced with complex interference.

[0057] This invention applies constraints simultaneously in two directions—image-to-EEG and EEG-to-image—using a bidirectional InfoNCE loss function. This ensures that image features and EEG features are close to corresponding samples and far from non-corresponding samples in the feature space. This bidirectional constraint mechanism is more stringent than unidirectional constraints, enabling the model to learn a more precise correspondence between image features and EEG features, thereby enhancing feature alignment. For example, in image-EEG semantic alignment tasks, the model can more accurately find the association between objects in the image and corresponding semantic information in the EEG signal, improving the accuracy of semantic alignment. By considering the similarity relationship in two directions, the bidirectional InfoNCE loss function can effectively reduce ambiguity between features. In unidirectional constraints, there may be situations where an image feature has high similarity to multiple EEG features, or vice versa, leading to unclear feature correspondences. Bidirectional constraints, however, can filter and constrain features from two perspectives, ensuring that each image feature has high similarity only to its most corresponding EEG feature, and each EEG feature also has high similarity only to its most corresponding image feature, thus reducing feature ambiguity. Furthermore, the bidirectional InfoNCE loss function incorporates multimodal (image and EEG) information and, through bidirectional contrastive learning, enables the model to learn more general feature representations. These general feature representations not only include visual information from images and physiological information from EEG, but also capture the intrinsic connections between them. When faced with tasks in different scenarios and with different data distributions, this general feature representation can better adapt to new data, thereby improving the model's generalization ability. In practical applications, image and EEG data are often subject to various noise interferences, such as changes in lighting and occlusion in images, and physiological noise in EEG signals. The bidirectional InfoNCE loss function strengthens the correspondence between image features and EEG features, enabling the model to better resist these noise interferences. Even with some noise in the data, the model can still accurately find the matching relationship between images and EEG based on the learned stable feature correspondences, thereby improving the model's generalization ability in noisy environments.

[0058] In an optional embodiment, the step S140 above, which involves performing contrastive learning based on the original image features, the perturbed image features, and the EEG features using a bidirectional InfoNCE loss function to update the parameters of the image encoder and the EEG encoder, includes: S1401, in the perturbed link, calculating a first contrastive loss based on the perturbed image features and the EEG features; S1402, in the original image link, calculating a second contrastive loss based on the original image features and the same EEG feature; S1403, calculating a total training loss based on the first and second contrastive losses, and using the total training loss to update the parameters of the image encoder and the EEG encoder through backpropagation.

[0059] This invention proposes a dual-path joint learning scheme in its training mechanism. While training solely based on perturbation samples can enhance model robustness, it often leads to a significant performance degradation on clean samples. To address this contradiction, this invention designs two parallel training paths: one path is a perturbation path, in which perturbation images are input. With EEG signals This is used to enhance the model's robustness against interference. Image encoders are utilized in this link. and EEG encoder Extracting perturbation image features and corresponding EEG features Calculate the second contrast loss This link utilizes the stability of EEG signals to perturbations to guide the model in learning to extract robust semantic features from perturbed images. The other link is the original image link, which takes the original image as input. With the same EEG signal This is used to maintain the model's performance on normal samples. Image encoders are utilized in this link. and EEG encoder Extracting features from the original image EEG characteristics Calculate the second contrast loss The goal of this link is to maintain the model's basic performance under undisturbed conditions and ensure accurate extraction of semantic features. This involves using Equation 1... Replace with Calculate the first alignment loss. Replace with Calculate the second alignment loss The loss functions of the two links are weighted and combined, defined as: In formula 2, This represents the total training loss. This is a balancing coefficient used to dynamically adjust the relationship between robustness enhancement and performance preservation. Through this dual-path mechanism, the present invention can effectively avoid overfitting interference while improving model robustness, achieving a balance between anti-interference performance and original performance.

[0060] This invention introduces perturbation links, exposing the model to various perturbed image samples during training. These perturbations simulate various changes images might encounter in real-world applications, such as noise interference and geometric deformation. When calculating the second contrast loss, the model needs to learn a stable correlation between perturbed image features and corresponding EEG features. This allows the model to accurately extract features and match them with EEG signals even when faced with noisy or deformed images in real-world scenarios, thus improving the model's robustness to image changes. Real-world image and EEG data often contain various types of noise. The combined use of original image links and perturbation links allows the model to learn feature representations of the data from different perspectives. In the original image links, the model learns the basic correspondence between images and EEG signals; in the perturbation links, the model further learns the correspondence under noise and deformation conditions. This comprehensive learning approach enables the model to better resist the effects of data noise, improving its stability and reliability.

[0061] The original and perturbed images have different feature distributions. By performing contrastive learning on both paths, the image encoder needs to learn to distinguish the features of the original and perturbed images while ensuring their correlation with the corresponding EEG features. This allows the image encoder to extract richer and more discriminative image features. For perturbed images, the encoder needs to extract stable features that are resistant to perturbations, such as structural information of the image. This rich feature representation helps improve the model's performance in various tasks. During the calculation of contrastive loss, the EEG encoder needs to adjust its parameters based on image features (including features of the original and perturbed images) to better match the image features. This allows the EEG encoder to learn EEG feature representations that are more relevant to the image features. Through interactive learning with image features, the EEG encoder can mine potential information in the EEG signals that is related to the semantics of the image, thereby optimizing the extraction of EEG features and improving the quality of EEG features.

[0062] To clearly present the process and key steps of the semantic interference-resistant training method based on EEG co-learning proposed in this invention, the core algorithm of this invention is further described below. This algorithm covers the complete process of data sampling, feature extraction, construction of the dual-path contrastive learning objective, and parameter updating, and can intuitively demonstrate the execution logic and optimization mechanism of this invention during the training phase. The goal of this algorithm is to utilize a joint encoder of image data and EEG signals to improve the robustness of semantic features through dual-path contrastive learning, while maintaining performance on clean samples. The algorithm flow is as follows: For each training batch, the following steps are performed: Sample N pairs of samples from the dataset. The disturbed image is obtained through the perturbation generator: Extracting features from the original image and the perturbed image: In the formula, N is the batch size, which is the number of image-EEG sample pairs processed in one training iteration. , , These are the original image, the perturbed image, and the corresponding EEG signal, respectively. , , These are the original image features, perturbation image features, and EEG features of the i-th sample, respectively.

[0063] In the perturbed link, first calculate the cosine similarity: Next, calculate the bidirectional InfoNCE loss: In the formula, To represent the cosine similarity between the i-th perturbation image feature and the i-th EEG feature, Let cosine similarity be the value between the i-th perturbation image feature and the j-th EEG feature. Let be the cosine similarity between the j-th EEG feature and the i-th perturbation image feature. , These represent the contrast loss from the image to the EEG direction corresponding to the perturbed image, and the contrast loss from the EEG to the image direction, respectively. This refers to the temperature parameter. This represents the first alignment loss corresponding to the perturbed image.

[0064] In the unperturbed link (i.e., the original image link mentioned above), first calculate the cosine similarity: Next, calculate the bidirectional InfoNCE loss: In the formula, To represent the cosine similarity between the i-th original image feature and the i-th EEG feature, Let cosine similarity be the ratio between the i-th original image feature and the j-th EEG feature. Let be the cosine similarity between the j-th EEG feature and the i-th original image feature. , These represent the contrast loss from the original image to the EEG direction and the contrast loss from the EEG to the image direction, respectively. This refers to the temperature parameter. This represents the second alignment loss corresponding to the original image.

[0065] The total training loss is calculated using Formula 2 above. Perform backpropagation to update the image encoder. and EEG encoder Set the parameters to end the batch loop.

[0066] In an optional embodiment, the present invention employs a fine-tuned joint learning mechanism in its training strategy. Considering that semantic communication tasks typically rely on pre-trained base models, using unsupervised contrastive learning entirely could lead to a significant deterioration in the original task performance. Therefore, the present invention does not train directly from scratch, but first utilizes pre-trained models (such as image encoders and EEG encoders) to acquire general features, and then performs small-scale parameter fine-tuning within a dual-path contrastive learning framework. This not only avoids performance collapse in downstream tasks such as classification, but also ensures that the learned features possess stronger generalization ability. Experiments show that the fine-tuned encoder significantly improves accuracy when facing interfering samples, while maintaining essentially unchanged performance on clean samples, effectively resolving the conflict between robustness and accuracy. The image encoder uses a pre-trained model based on a visual Transformer, and the EEG encoder uses a pre-trained model based on a convolutional neural network; the parameter update process is a fine-tuning based on the pre-trained models.

[0067] The Visual Transformer (ViT) model possesses powerful global information modeling capabilities. Through its self-attention mechanism, it captures long-distance dependencies between different regions in an image, having already learned rich semantic features during pre-training. For example, when pre-trained on large-scale image datasets like ImageNet, the ViT model can identify general features such as the shape, texture, and spatial relationships of various objects in an image. When applied to the image encoder of this invention and fine-tuned, it can quickly adapt to the task requirements of contrastive learning with EEG signals. Because the pre-trained model already possesses the ability to understand basic image semantics, the fine-tuning process can focus more on learning the correspondence between image features and EEG features. For example, when faced with images containing semantic interference, the ViT model can leverage its knowledge accumulated during pre-training to better extract key semantic features unaffected by the interference, thereby enhancing the model's ability to accurately capture image semantics.

[0068] Pre-trained models based on convolutional neural networks (CNNs) have unique advantages in processing time-series data such as electroencephalogram (EEG) signals. CNNs can extract local features of EEG signals in both the time and channel dimensions through convolutional kernels, such as EEG wave features at different frequency bands. During pre-training, the CNN model may have already learned basic patterns and feature representations of EEG signals in some EEG signal classification tasks (such as emotion recognition and brain disease diagnosis). In the fine-tuning process of this invention, the EEG encoder can quickly adjust its parameters to better match the features extracted by the image encoder. For example, through fine-tuning, the CNN can more accurately extract EEG features related to image semantics, such as specific EEG patterns generated by the brain when viewing a specific image, thereby enhancing the correlation between EEG signals and image semantics.

[0069] In semantic communication scenarios, semantic interference is a common challenge, designed to mislead the receiver's semantic understanding of information. Since the image encoder uses a ViT-based pre-trained model, it has been exposed to a large amount of image data during pre-training, giving it a certain adaptability to various image variations and noise. During the fine-tuning phase, comparative learning using EEG signals allows the image encoder to learn more stable semantic features. For example, when an image is semantically interfered with, the ViT model can identify key semantic features unaffected by interference by comparing them with EEG signals, thus enhancing the model's resistance to semantic interference. The EEG encoder, during fine-tuning, can also better adapt to EEG signals corresponding to images with interference. Convolutional neural networks can reduce the impact of semantic interference on EEG feature extraction by learning a stable correspondence between images and EEG signals. For example, even when the image is interfered with, the EEG encoder can still extract EEG features semantically relevant to the original image, ensuring the model's robustness in complex interference environments.

[0070] Pre-trained models are typically trained on large-scale, diverse datasets, giving them a certain degree of generalization ability. In this invention, the image encoder and EEG encoder based on pre-trained models can better adapt to image and EEG data from different sources and with different characteristics during fine-tuning. For example, when faced with EEG signals from different experimental environments, different subjects, and images of different styles, the fine-tuned model can quickly adjust its parameters and extract common semantic features, thereby improving the model's robustness under different data distributions. Because a pre-trained model is used, parameter updates do not require training from scratch. The pre-trained model has already learned a large amount of basic features and knowledge; the fine-tuning process only requires minor parameter adjustments based on this existing knowledge, significantly reducing the time and computational resources required for training. The pre-trained model provides a good initial parameter space for the fine-tuning process. Fine-tuning based on this space allows the model to find the optimal parameter combination more quickly, thus accelerating convergence. For example, during training, compared to training from a randomly initialized model, the fine-tuning method based on a pre-trained model can achieve similar performance levels in fewer training epochs. This not only improves training efficiency but also reduces the risk of overfitting, because the model can learn meaningful feature representations in fewer training epochs without overfitting to noise in the training data.

[0071] This invention provides a semantic interference-resistant training method based on EEG joint learning. First, it leverages the insensitivity of EEG signals to pixel-level perturbations, using them as cross-modal supervision signals to guide the image encoder in learning perturbation-invariant robust semantic features during training. Second, a bidirectional InfoNCE loss function is designed to simultaneously constrain features in both image-to-EEG and EEG-to-image directions, ensuring the stability of cross-modal feature alignment. Furthermore, this invention proposes a dual-path joint learning mechanism, introducing contrastive learning in both perturbation-prone and non-perturbation-prone links. A dynamic trade-off between robustness enhancement and original performance preservation is achieved through balancing coefficients, and combined with a fine-tuning training strategy, it avoids the damage to downstream task performance caused by unsupervised training. This method balances interference resistance and task performance, demonstrating significant advantages in complex scenarios such as white-box and black-box attacks, and can provide new solutions and technical paths for the security, stability, and practicality of future semantic communication systems.

[0072] In the simulation, the semantic encoding of image samples and EEG signal samples adopted the image encoder part of CLIP (based on the ViT-B / 32 model) and EEGNet-V4, respectively. The EEG signal and image data used came from the THING-EEG dataset. The structure used contained 27 categories, each category was divided into 80 images for training and 20 images for testing. The EEG data dimension was [B, 17, 100], representing 17 channels, 100 time points of EEG sampling data, and the image size was 224×224, RGB3 channels. The baseline algorithms compared included: (1) Baseline (pre-trained image encoder): the CLIP model was used as the image encoder and pre-trained on a pure image dataset. This scheme only considers single-modal image semantic feature extraction. The training parameters were set to batch size 32, learning rate 1e-4, training epochs 50, and loss function was cross-entropy. This method is used to characterize the performance limit of traditional single-modal semantic feature extraction.

[0073] (2) EEG-Img Joint Training of the Present Invention (EEG-Img_training): Based on the image encoder, EEG signals are introduced as an auxiliary modality to construct a cross-modal contrastive learning framework. The training parameters are consistent with the baseline (batch size 32, learning rate 1e-4, training epochs 50), and the loss function is InfoNCE (temperature parameter 0.07). In addition, an adversarial training mechanism is introduced during the training process, using FGSM attack (perturbation intensity ε=0.003), and the adversarial loss is fused with a weight coefficient k=0.3. This scheme is used to verify the robustness improvement effect of cross-modal joint learning and adversarial mechanism on semantic interference.

[0074] (3) Adversarial Training: Training is based solely on the image modality, without introducing EEG signals. The training parameters are consistent with the baseline (batch size 32, learning rate 1e-4, training epochs 50), and adversarial training is performed using an FGSM attack (perturbation intensity ε=0.003), with the loss function being cross-entropy (CE). This method is used to characterize the effectiveness of traditional single-modal adversarial training in dealing with semantic interference and is used for comparison with EEG-Img joint training.

[0075] In the experiment, by comparing the baseline's single-modal image pre-training scheme, AdvTraining's single-modal adversarial learning scheme, and the cross-modal contrastive learning and adversarial fusion scheme of EEG-Img_training of this invention, the improvement of semantic communication anti-interference ability after the introduction of EEG modality can be systematically verified.

[0076] The first simulation experiment compared the classification accuracy of three models—Baseline, EEG-Img_training, and AdvTraining—under the Fast Gradient Sign Method (FGSM) attack. All three models were trained with the same parameter settings and then tested at different attack intensities (Epsilon). The average classification accuracy was used to measure their robustness to interference.

[0077] Figure 3 shows the results under different attack intensities. It can be seen that the performance of all three models gradually decreases as the attack intensity increases. When the attack intensity is 0, the performance is slightly lower than the comparison scheme. This is because joint learning to enhance robustness affects the performance of clean samples. As the attack intensity increases, the accuracy of EEG-Img_training is higher than the other schemes, indicating that joint training with EEG signals can better capture perturbation-invariant features and shows stronger robustness compared to single-modal pre-training and single-modal adversarial training.

[0078] The second simulation experiment further compared the performance of EEG-Img_training and AdvTraining under white-box and black-box attack scenarios. The experiment was set up so that the attack intensity gradually increased. White-box attacks directly used the gradient of the target model to generate perturbations, while black-box attacks generated perturbations on the substitute model and transferred them to the target model.

[0079] Figure 4 illustrates the experimental results. Except for slightly weaker performance on clean samples, the EEG-Img_training of this invention outperforms AdvTraining in both attack scenarios, with a particularly significant advantage in black-box attacks. This indicates that cross-modal joint training can not only effectively resist white-box attacks on known models but also maintain high robustness in transfer attacks, thus demonstrating stronger practical application value.

[0080] The semantic communication method provided by this invention is described below. The semantic communication method described below can be referred to in correspondence with the semantic interference resistance training method based on EEG co-learning described above.

[0081] The semantic communication method provided by the present invention includes: inputting an image to be transmitted into a semantic encoder to extract semantic features; performing channel coding and modulation on the semantic features to generate a transmission signal; and transmitting the transmission signal through a communication channel; wherein the semantic encoder is an image encoder obtained by using the above-described anti-semantic interference training method based on EEG co-learning.

[0082] Specifically, a trained semantic encoder is deployed at the transmitting end of the semantic communication system. This encoder is an image encoder obtained based on an EEG-based co-learning method to resist semantic interference, specifically employing the ViT-B / 32 architecture of the CLIP model and fine-tuned through dual-path contrastive learning. During system initialization, pre-trained weights are loaded, the input image size is configured as 224×224×3, and the output semantic feature dimension is 512. When the image to be transmitted arrives at the transmitting end, the input image is scaled to 224×224 resolution and normalized to meet the encoder's input requirements. The pre-processed image is then input into the semantic encoder, and semantic feature vectors are extracted from the last hidden state of the network through forward propagation computation using a visual Transformer. The extracted 512-dimensional semantic feature vectors are L2 normalized to ensure that the features are distributed on a unit hypersphere for easier subsequent processing.

[0083] After obtaining semantic features, communication adaptation is performed. Specifically, a low-density parity-check code (LDPC code) is used to channel-encode the semantic feature vector to obtain the encoded bitstream. The coding rate is set to 3 / 4 to enhance transmission reliability. Quadrature amplitude modulation (QAM) is used to group and map the encoded bitstream into complex baseband symbol sequences, with every 4 bits mapped to a complex symbol, forming two baseband signals: in-phase (I) and quadrature (Q). The I and Q baseband signals are multiplied by the in-phase and quadrature carriers respectively using a quadrature upconverter at the RF front-end, and then combined to generate an RF modulated signal. The RF modulated signal is amplified to reach a predetermined transmit power, and then a bandpass filter is used to suppress out-of-band spurious radiation, resulting in a processed RF signal. The processed RF signal is fed to the antenna, and the electromagnetic waves are radiated to the receiver through the wireless channel.

[0084] When semantic interference exists in the transmission environment, this method maintains stable extracted semantic features even when the input image is disturbed, thanks to the semantic encoder's adversarial training supervised by EEG signals. Based on a bidirectional alignment mechanism between EEG signals and image semantics, it ensures a high degree of consistency between the extracted features and the original semantic content. Under unknown interference modes, the encoder automatically focuses on essential semantic features and suppresses interference-related features.

[0085] This method ensures robust communication even in the presence of semantic interference in the transmission environment. This is because: firstly, the semantic encoder, based on adversarial training supervised by EEG signals, maintains stable extracted semantic features even when the input image is disturbed; secondly, the bidirectional alignment mechanism between EEG signals and image semantics inherently constrains the consistency between semantic features and the original semantic content; furthermore, when faced with unknown interference patterns, the encoder can automatically focus on semantic features, effectively suppressing interference-related feature components, thereby ensuring reliable transmission of semantic information in interference environments.

[0086] The following describes the anti-semantic interference training device based on EEG co-learning provided by the present invention. The anti-semantic interference training device based on EEG co-learning described below and the anti-semantic interference training method based on EEG co-learning described above can be referred to and correspond to each other.

[0087] The semantic interference-resistant training device based on EEG co-learning provided by the present invention, as shown in FIG5, includes: a data acquisition module 210 for acquiring an original image and its corresponding EEG signal; a perturbation generation module 220 for applying semantic interference to the original image to generate a perturbation image; a feature extraction module 230 for inputting the original image and the perturbation image into an image encoder respectively to extract corresponding original image features and perturbation image features, and inputting the EEG signal into an EEG encoder to extract EEG features; a contrastive learning module 240 for performing contrastive learning based on the original image features, the perturbation image features, and the EEG features using a bidirectional InfoNCE loss function to update the parameters of the image encoder and the EEG encoder; and a model output module 250 for acquiring the trained image encoder.

[0088] Figure 6 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 6, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communication bus 340. The processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute the aforementioned anti-semantic interference training method based on EEG co-learning, or the aforementioned semantic communication method.

[0089] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0090] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the brainwave-based anti-semantic interference training method or semantic communication method provided by the above methods.

[0091] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the brainwave-based co-learning anti-semantic interference training method or semantic communication method provided by the above methods.

[0092] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0093] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A semantic interference-resistant training method based on EEG co-learning, characterized in that, include: Acquire the original image and its corresponding electroencephalogram (EEG) signal; Semantic interference is applied to the original image to generate a perturbed image; The original image and the perturbed image are respectively input into the image encoder to extract the corresponding original image features and perturbed image features. The EEG signal is input into the EEG encoder to extract EEG features. Based on the original image features, the perturbed image features, and the EEG features, comparative learning is performed using a bidirectional InfoNCE loss function to update the parameters of the image encoder and the EEG encoder. The trained image encoder is then obtained.

2. The method for resisting semantic interference based on EEG co-learning according to claim 1, characterized in that, The bidirectional InfoNCE loss function implements bidirectional constraints between image features and EEG features, including: image-to-EEG contrast constraints, which increase the similarity between image features and corresponding EEG features and decrease the similarity with non-corresponding EEG features; and EEG-to-image contrast constraints, which increase the similarity between EEG features and corresponding image features and decrease the similarity with non-corresponding image features.

3. The method for resisting semantic interference based on EEG co-learning according to claim 1, characterized in that, The calculation of the bidirectional InfoNCE loss function includes: calculating the image-to-EEG contrast loss based on the similarity between each image feature and its corresponding EEG feature, and the similarity between the image feature and other EEG features within the batch; calculating the EEG-to-image contrast loss based on the similarity between each EEG feature and its corresponding image feature, and the similarity between the EEG feature and other image features within the batch; and determining the bidirectional InfoNCE loss function value based on the two contrast losses.

4. The method for resisting semantic interference based on EEG co-learning according to claim 1, characterized in that, The step of updating the parameters of the image encoder and the EEG encoder by performing contrastive learning based on the original image features, the perturbed image features, and the EEG features using a bidirectional InfoNCE loss function includes: in the perturbed link, calculating a first contrastive loss based on the perturbed image features and the EEG features; in the original image link, calculating a second contrastive loss based on the original image features and the same EEG feature; calculating a total training loss based on the first and second contrastive losses, and using the total training loss to update the parameters of the image encoder and the EEG encoder through backpropagation.

5. The method for resisting semantic interference based on EEG co-learning according to claim 1, characterized in that, The image encoder uses a pre-trained model based on the visual Transformer, and the EEG encoder uses a pre-trained model based on the convolutional neural network; the parameter update process is a fine-tuning based on the pre-trained model.

6. A semantic communication method, characterized in that, include: The image to be transmitted is input into the semantic encoder to extract semantic features; the semantic features are then channel-coded and modulated to generate the transmission signal. The transmission signal is sent through a communication channel; wherein the semantic encoder is an image encoder obtained by using the anti-semantic interference training method based on EEG co-learning as described in any one of claims 1 to 5.

7. A semantic interference-resistant training device based on EEG co-learning, characterized in that, include: The data acquisition module is used to acquire the raw images and their corresponding electroencephalogram (EEG) signals; The perturbation generation module is used to apply semantic interference to the original image to generate a perturbation image; The feature extraction module is used to input the original image and the perturbed image into the image encoder respectively, extract the corresponding original image features and perturbed image features, and input the EEG signal into the EEG encoder to extract EEG features; The contrastive learning module is used to perform contrastive learning based on the original image features, the perturbed image features, and the EEG features using a bidirectional InfoNCE loss function, and update the parameters of the image encoder and the EEG encoder; the model output module is used to obtain the trained image encoder.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the semantic interference-resistant training method based on EEG co-learning as described in any one of claims 1 to 6, or the semantic communication method as described in claim 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the anti-semantic interference training method based on EEG co-learning as described in any one of claims 1 to 6, or the semantic communication method as described in claim 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the anti-semantic interference training method based on EEG co-learning as described in any one of claims 1 to 6, or the semantic communication method as described in claim 6.

Images