Cross-sensory emotion transfer learning method and system, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a cross-sensory emotion transfer learning method and system, electronic equipment and a storage medium. The method comprises the following steps: determining a difference entropy electroencephalogram characteristic of characteristic consistency of visual evoked electroencephalogram data and olfactory evoked electroencephalogram data; a feature extractor based on a Transform structure obtains electroencephalogram time sequence feature representation of the visual evoked electroencephalogram data and the olfactory evoked electroencephalogram data; and emotion classification cross entropy loss and domain adversarial loss are determined by a domain adversarial neural network constructed by an emotion classifier and a domain discriminator to train the domain adversarial neural network, so that the distribution gap between emotions is reduced, and the features of visual evoked emotions and olfactory evoked emotions are generalized. According to the embodiment of the invention, Transform is used for EEG sequence feature extraction, a multi-head self-attention mechanism is introduced, evolution features of an emotional state in a time dimension are modeled, and feature distribution between source stimulation and target stimulation is effectively aligned by introducing a gradient inversion mechanism and a domain discriminator behind a feature extractor.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition, and in particular to a cross-sensory emotion transfer learning method, system, electronic device and storage medium. Background Art

[0002] Olfactory stimulation has been shown to be effective in evoking emotional responses. However, despite its potential, olfactory stimulation has been less thoroughly studied compared to visual stimulation and auditory stimulation in emotion recognition. When addressing emotion recognition with olfactory stimulation, the following methods are commonly used: DANN (Domain-Adversarial Neural Network): A transfer learning method that aligns source and target domain features through adversarial training. This model consists of three main components: a feature extractor, a label predictor, and a domain classifier.

[0003] 2. Cross-subject emotion recognition model: When building an emotion recognition system that can generalize across different individuals, the cross-subject emotion recognition model mainly focuses on modeling between different individuals under the same stimulus modality (such as vision or speech).

[0004] 3. Cross-modal emotion recognition research: Utilize multiple perceptual modalities (such as EEG, eye movements, facial expressions, speech, etc.) for joint analysis to improve the accuracy and robustness of emotion recognition.

[0005] In the process of implementing the present invention, the inventors discovered that there are at least the following problems in the related art: 1. DANN is a domain adaptation method based on adversarial training. Its feature alignment relies on shallow feature distribution adversarial methods and a simple gradient reversal mechanism, lacking the ability to model the structural characteristics of the data itself. This method assumes that the source and target domains are homomodal or have similar distributions. Therefore, it cannot effectively capture the essential differences in perceptual mechanisms when transferring cross-sensory stimuli (such as vision and smell). Furthermore, DANN's design does not consider the strong temporal dependencies and high noise characteristics of EEG data, resulting in poor robustness in EEG data transfer.

[0006] 2. Most cross-subject emotion recognition models focus on modeling inter-individual feature differences, overlooking the inherently different processing pathways of different emotion elicitation modalities (e.g., visual and olfactory). Because training data is often limited to a single stimulus modality, existing methods can only generalize to new individuals but struggle to generalize to new sensory domains. Consequently, these methods have poor adaptability when handling cross-stimulus emotion transfer tasks, such as "visually evoked → olfactory evoked," and are unable to effectively extract common emotional feature representations across stimuli.

[0007] 3. Current cross-modal emotion recognition efforts are mostly based on "multimodal fusion" scenarios, relying on simultaneously collected multi-channel data. Their goal is "multi-source complementation and mutual enhancement," rather than transfer learning. Because their core methods rely on feature fusion or joint representation learning, they lack transfer capabilities and are unable to adapt when the source and target domains are completely different modalities (e.g., vision to olfaction). Furthermore, this type of research typically relies on large amounts of labeled data, making generalization difficult in actual cross-modal transfer. Summary of the Invention

[0008] In order to at least solve the problem that existing technologies cannot effectively deal with emotion recognition transfer between different stimulation modalities (such as vision and smell).

[0009] In a first aspect, an embodiment of the present invention provides a cross-sensory emotion transfer learning method, comprising: Processing the dynamic feature differences of the visually evoked EEG data and the olfactory evoked EEG data due to different sensory emotional stimuli based on a time sliding window, and determining the differential entropy EEG features of the feature consistency between the visually evoked EEG data and the olfactory evoked EEG data; Inputting the differential entropy EEG features into a feature extractor based on a Transformer structure, adding position encoding for enhancing time series information to the differential entropy EEG features in the feature extractor, modeling the evolution of visually evoked emotions and olfactory evoked emotions in the time dimension through a head self-attention module and a feedforward network in the feature extractor, and obtaining EEG time series feature representations of the visually evoked EEG data and the olfactory evoked EEG data; Inputting the EEG time series feature representation into a domain adversarial neural network constructed by an emotion classifier and a domain discriminator, classifying the EEG time series feature representation by the emotion classifier to determine the emotion classification cross entropy loss, and performing domain discrimination on the EEG time series feature representation by the domain discriminator to determine the domain adversarial loss; The domain adversarial neural network is trained based on the emotion classification cross entropy loss and the domain adversarial loss to reduce the distribution gap between the visually induced emotions and the olfactory induced emotions, generalize the features of the visually induced emotions and the olfactory induced emotions, and achieve cross-sensory transfer learning.

[0010] In a second aspect, an embodiment of the present invention provides a cross-sensory emotion transfer learning system, including: A preprocessing module is used to process the dynamic feature differences of the visual evoked EEG data and the olfactory evoked EEG data due to different sensory emotional stimuli based on a time sliding window, and determine the differential entropy EEG features of the consistency of the features of the visual evoked EEG data and the olfactory evoked EEG data; A feature extraction module is used to input the differential entropy EEG features into a feature extractor based on a Transformer structure, add position encoding for strengthening time series information to the differential entropy EEG features in the feature extractor, and model the evolution of visually evoked emotions and olfactory evoked emotions in the time dimension through the head self-attention module and feedforward network in the feature extractor to obtain EEG time series feature representations of the visually evoked EEG data and the olfactory evoked EEG data; a loss determination module, configured to input the EEG time series feature representation into a domain adversarial neural network constructed by an emotion classifier and a domain discriminator, classify the EEG time series feature representation by the emotion classifier to determine the emotion classification cross entropy loss, and perform domain discrimination on the EEG time series feature representation by the domain discriminator to determine the domain adversarial loss; A training learning module is used to train the domain adversarial neural network based on the emotion classification cross entropy loss and the domain adversarial loss, reduce the distribution gap between the visually induced emotions and the olfactory induced emotions, generalize the characteristics of the visually induced emotions and the olfactory induced emotions, and realize cross-sensory transfer learning.

[0011] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the cross-sensory emotion transfer learning method of any embodiment of the present invention.

[0012] In a fourth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the cross-sensory emotion transfer learning method of any embodiment of the present invention are implemented.

[0013] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the steps of the cross-sensory emotion transfer learning method of any embodiment of the present invention are implemented.

[0014] The beneficial effects of the embodiments of the present invention are as follows: this method uses Transformer for EEG sequence feature extraction, introduces a multi-head self-attention mechanism, and models the evolution characteristics of emotional states in the time dimension. At the same time, combined with the domain adversarial neural network (DANN) structure, by introducing a gradient reversal mechanism and a domain discriminator after the feature extractor, the feature distribution between the source stimulus and the target stimulus is effectively aligned. Under the premise of using only source modality labeled data for supervised training and target modality data not relying on labels, this method uses the adversarial mechanism to achieve the generalization transfer of EEG features induced by different senses, and has good cross-stimulus generalization capabilities. It breaks through the bottleneck that existing transfer learning tasks are limited to "same modality" or "slight modality differences", and realizes the transfer of emotion modeling from visual induction to olfactory induction. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 This is a flow chart of a cross-sensory emotion transfer learning method provided by one embodiment of the present invention; Figure 2 This is a schematic diagram of odor classification based on emotional state in a cross-sensory emotion transfer learning method provided by one embodiment of the present invention; Figure 3 Schematic diagram of a Transformer-based DANN feature extraction framework for a cross-sensory emotion transfer learning method provided by one embodiment of the present invention; Figure 4 This is a schematic diagram of the average accuracy (%) and standard deviation (%) of 21 methods using DE features in three categories of cross-sensory emotion recognition using a cross-sensory emotion transfer learning method provided by one embodiment of the present invention; Figure 5 This is a schematic diagram of the average accuracy (%) and standard deviation (%) of DE features in three categories of cross-stimulus transfer learning for emotion recognition using 21 methods provided by an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a cross-sensory emotion transfer learning system provided by one embodiment of the present invention; Figure 7 A schematic structural diagram of an embodiment of an electronic device for cross-sensory emotion transfer learning provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0018] like Figure 1 FIG2 is a flow chart of a cross-sensory emotion transfer learning method provided by an embodiment of the present invention, comprising the following steps: S11: processing the dynamic feature differences of the visual evoked EEG data and the olfactory evoked EEG data due to different sensory emotional stimuli based on a time sliding window, and determining a differential entropy EEG feature of feature consistency between the visual evoked EEG data and the olfactory evoked EEG data; S12: Inputting the differential entropy EEG features into a feature extractor based on a Transformer structure, adding position encoding for strengthening time series information to the differential entropy EEG features in the feature extractor, modeling the evolution of visually evoked emotions and olfactory evoked emotions in the time dimension through the head self-attention module and feedforward network in the feature extractor, and obtaining EEG time series feature representations of the visually evoked EEG data and the olfactory evoked EEG data; S13: Inputting the EEG time series feature representation into a domain adversarial neural network constructed by an emotion classifier and a domain discriminator, classifying the EEG time series feature representation by the emotion classifier to determine the emotion classification cross entropy loss, and performing domain discrimination on the EEG time series feature representation by the domain discriminator to determine the domain adversarial loss; S14: The domain adversarial neural network is trained based on the emotion classification cross entropy loss and the domain adversarial loss to reduce the distribution gap between the visually induced emotions and the olfactory induced emotions, generalize the features of the visually induced emotions and the olfactory induced emotions, and realize cross-sensory transfer learning.

[0019] In this implementation, to address the inability of existing methods to effectively transfer emotion recognition across different stimulus modalities (such as visual and olfactory sensory modalities), we propose a Transformer-based DANN (Domain-Adversarial Neural Network) method for cross-stimulus emotion EEG transfer learning. This method effectively transfers knowledge between the source stimulus modality (referred to as the source modality, such as video-induced emotion) and the target stimulus modality (referred to as the target modality, such as odor-induced emotion), improving the model's generalization ability in cross-sensory emotion recognition.

[0020] This method's model architecture integrates a Transformer with a DANN. This differs from existing DANNs, which lack the ability to model the structural characteristics of the data itself when addressing cross-sensory emotion transfer. When the source and target domains are of the same modality or have similar distributions, they cannot effectively capture the essential differences in perceptual mechanisms when transferring cross-sensory stimuli (visual and olfactory). When processing highly nonlinear and complex EEG signals, DANN's shallow feature alignment approach suffers from insufficient generalization and insufficient feature distribution overlap, making it particularly unsuitable for EEG data, which exhibits strong individual differences and temporal dependencies.

[0021] This method leverages the temporal modeling capabilities of the Transformer to process EEG sequence data and combines it with the adversarial training mechanism of a DANN to learn domain-invariant features. It also introduces a Gradient Reversal Layer (GRL) and a domain discriminator to make the features extracted by the model indistinguishable between the source and target domains, thereby achieving feature alignment. Using video-evoked EEG data as the source domain and olfactory-evoked EEG data as the target domain, a cross-stimulus transfer learning framework is constructed.

[0022] For step S11, in order to achieve cross-sensory (source domain data → target domain data) emotion transfer learning, specifically, the visually evoked EEG data is used as the source domain data, and the olfactory evoked EEG data is used as the target domain data to achieve visual to olfactory cross-sensory transfer learning, which requires visually evoked EEG data and olfactory evoked EEG data.

[0023] Regarding the above data, this method uses two datasets: 1. SEED dataset, 2. EEG dataset obtained during olfactory stimulation. Among them, the SEED dataset contains EEG records of 15 subjects (7 males and 8 females, with an average age of 23.27 years) collected using a 62-channel ESI neural scanning system. The subjects watched 15 movie clips that were designed to induce three different emotional states: positive, neutral, and negative. Each clip was about 4 minutes long to ensure emotional coherence and enhance its emotional impact. Before each clip, the subjects received a 5-second prompt. After the clip was played, the subjects were allocated 45 seconds for self-assessment and 15 seconds for a break. In order to prevent the same emotions from being induced by consecutive clips, the order of clip playback was randomized. Among them, the EEG dataset obtained during olfactory stimulation includes records of 9 subjects (7 males and 2 females, with an average age of 21 years). These data were collected when the subjects were in an environment with five odors. These odors evoke three emotional states, such as Figure 2 As shown, they were classified according to the associated emotional responses. Positive odors include lavender and citrus, neutral odor is γ-nonalactone, and negative odors are malodorous gases, for example, can include isovaleric acid and 3-methylindole. During each test, the subjects were in the odor environment in a random order, each stay lasting 15 seconds, and then returned to the odorless environment to rest for 30 seconds. Each test was conducted five times. The EEG signals of the two data sets were recorded using the ANT Neuro system at a sampling rate of 1000Hz. The experiment was conducted in a strictly controlled environment to reduce external noise and odor interference. Visually evoked EEG data and olfactory evoked EEG data were obtained in the above manner.

[0024] Because visual and olfactory stimuli affect subjects differently and cannot be directly used, alignment and preprocessing are required. This method uses a data processing pipeline to align the SEED dataset and the EEG dataset obtained during olfactory stimulation to maintain data consistency between the two sensory data sets and processes them based on the characteristics of each dataset. Initially, a 50Hz notch filter is used to remove interference, followed by a 1-75Hz bandpass filter to isolate signals within the standard EEG frequency range. The EEG signals are then segmented into five frequency bands (δ, θ, α, β, and γ): Delta (1-4Hz), Theta (4-8Hz), Alpha (8-14Hz), Beta (14-31Hz), and Gamma (31-50Hz). From these segmented signals, 305-dimensional DE (Differential Entropy) features are extracted using sliding windows of varying lengths, depending on the sensory stimulus evoking different emotions. LDS (Linear Dynamic System) smoothing is then applied to reduce artifacts unrelated to emotion.

[0025] As an embodiment, the time sliding window-based method for solving the dynamic feature differences in visually evoked EEG data and olfactory evoked EEG data due to different sensory emotional stimuli includes: Using a sliding window of a first time to match the stimulation duration of the visually evoked EEG data, and using a sliding window of a second time to match the stimulation duration of the olfactory evoked EEG data, wherein the first time is greater than the second time; After using the sliding window matching, feature alignment is performed to obtain differential entropy EEG features of feature consistency between the visually evoked EEG data and the olfactory evoked EEG data.

[0026] In this embodiment, for the two datasets, the main difference in data processing lies in the choice of sliding time window. For the SEED dataset involving video-induced stimulation, a 4-second sliding window is used to match the longer stimulation duration. In contrast, the olfactory stimulation EEG dataset uses a shorter 1-second sliding window to adapt to the shorter odor exposure time. This adjustment ensures that the extracted features accurately capture the temporal dynamics of each stimulus. By aligning the processing pipeline with the specific features of the two datasets, the goal is to maintain consistency while optimizing the feature extraction of visual, auditory and olfactory stimulation. In this way, differential entropy EEG features of the feature consistency of visually evoked EEG data and olfactory evoked EEG data are obtained.

[0027] For step S12, this method constructs a feature extractor based on the Transformer architecture. The Transformer model utilizes a self-attention mechanism to dynamically adjust attention weights to process input and output sequences. Its architecture combines an encoder and a decoder, each consisting of several stacked identical layers. The encoder converts the input sequence into a continuous representation, while the decoder generates the corresponding output.

[0028] On the feature extractor side, after inputting the differential entropy EEG features, position encoding is added to preserve the temporal characteristics of the EEG sequence. An N-layer Transformer encoder is used, with each layer comprising a multi-head self-attention module and a feedforward network. The output is a temporal feature representation of the EEG data for visually evoked and olfactory evoked EEG data, which is used for subsequent classification and discrimination. The number of attention heads in the self-attention layer is selected from [1, 2, 4, 8], and the number of encoder layers is set to [1, 2, 4]. The output dimension of the feedforward layer is fixed to 512. This configuration enables the Transformer to effectively manage continuous data and capture long-range dependencies. Furthermore, its parallel processing capabilities enable efficient processing of large datasets. It should be noted that this application does not utilize the feature extractor, classifier, and discriminator separately. Instead, the feature extractor, classifier, and discriminator are coordinated throughout the entire architecture. The specific effects will be described later in the context of the classifier and discriminator.

[0029] In step S13, the present method inputs the EEG temporal feature representation into a domain adversarial neural network constructed by an emotion classifier and a domain discriminator. The emotion classifier and domain discriminator in the domain adversarial neural network, along with the Transformer-based feature extractor in the above steps, form the Transformer-based domain adaptation architecture for cross-sensory emotion transfer in the present method.

[0030] In general, this method utilizes the Transformer feature extractor architecture. Transformer uses its internal self-attention mechanism to process inputs in parallel, thereby improving computational efficiency. In order to utilize the time dimension of EEG data, this method introduces position embedding to enrich the time series information. The features are then processed by an encoder consisting of N identical layers, each of which contains a multi-head self-attention module and a feedforward network. The overall framework is as follows Figure 3 shown.

[0031] This method uses a DANN for domain adaptation. The core idea of DANN is to reduce the distribution difference between the source and target domains by introducing an adversarial loss function into the neural network. Simply put, the EEG time series features of visually evoked EEG data and olfactory evoked EEG data are input into: 1. An emotion classifier predicts the emotion category (positive, neutral, negative) of the input sample; 2. A domain discriminator distinguishes whether the sample originates from the source domain or the target domain.

[0032] As an embodiment, an emotion classifier is used to determine a predicted emotion category represented by the EEG time series feature, and an emotion classification cross entropy loss is determined based on the predicted emotion category and a preset benchmark emotion category; A domain discriminator is used to determine a prediction domain of the EEG time series feature representation, and a domain adversarial loss is determined based on the prediction domain.

[0033] The emotion classification cross entropy loss and / or the domain adversarial loss are weighted based on preset hyperparameters to determine a multi-task joint loss function.

[0034] In this implementation, the specific implementation of the entire framework is described. In the feature extractor based on the Transformer structure, the multi-head attention mechanism implements the calculation of scaled dot product attention by receiving the query (Q), key (K) and value (V). The output consists of the weighted sum of the values, where the weight is calculated based on the query key interaction. The attention function is described by the following equation: Assume O is the output of the above equation. For the self-attention sublayer with h heads, the query, key, and value are linearly projected h times with learnable projections. The calculation of multi-head attention is as follows: Predictions: and Multi-head attention processes multiple inputs simultaneously and generates outputs, which are then combined and passed to a fully connected feed-forward sublayer. This sublayer consists of two linear transformation layers followed by a Gaussian Error Linear Unit (GELU) activation function, which helps alleviate the vanishing gradient problem.

[0035] The adversarial learning network of this method contains a gradient reversal layer that reverses the gradient during backpropagation to induce identity transformation during forward propagation. This design helps promote the convergence of the distribution of source data (xs) and target data (xt), thereby reducing their distinguishability. During the adversarial training phase, each batch of source data (xs) is paired with similar unlabeled target data (xt) selected by the discriminator from the target domain. After processing by the feature extractor (Gf), both sets of data are fed into the domain classifier (Gd), which aims to determine whether the data is from the source domain or the target domain. For the source data xs, the adversarial loss is determined as follows: Among them, ds is the domain label of xs, which takes the value 1 and comes from the source domain. Similarly, for the target domain data xt, the loss is determined as: In this case, the domain label dt is 0. The transfer loss is obtained by averaging the adversarial losses on the source and target data.

[0036] During the training phase, the classifier (Gy) operates on the source data (xs), paired with the true labels (ys) of the preset baseline emotion categories, and processed through the feature extractor (Gf). The classifier is constructed as a linear network and uses the cross entropy loss as the classification loss metric, which is calculated as follows: The multi-task joint loss function includes classification loss and transfer loss, as shown in the following equation: Where λ represents the weight assigned to the transfer loss, which can be set to 1 in this method. Lclf is the cross entropy loss for sentiment classification, and Ltransfer is the domain adversarial loss (the ability to distinguish between the source domain and the target domain).

[0037] In step S14, the domain adversarial neural network was trained based on the emotion classification cross-entropy loss and the domain adversarial loss. During training, 50 random hyperparameter combinations were evaluated, varying the number of attention heads and encoder layers. The model for each subject underwent 60 training epochs, and the optimal parameters were determined for each iteration. Furthermore, emotion recognition performance was explored in a cross-disciplinary setting. This included analyzing changes in EEG signals before and after olfactory stimulation within the subject's scene. To mitigate performance degradation caused by individual differences in cross-disciplinary tasks, a domain adaptation (DA) technique was employed. Both MLP (Multi-Layer Perceptron) and Transformer-based DA methods were implemented. The Transformer-based DA combines the Transformer framework with a DANN. The feature extractor identifies domain-specific features, the classifier performs emotion recognition, and the discriminator distinguishes source domain features from target domain features. These components work together to generate domain-invariant representations during training. Domain adaptation enables the model to improve performance on target domain tasks after training on the source domain. By embedding a discriminator and gradient reversal layers, our model can learn domain-invariant features, thereby improving its generalization ability. Furthermore, by using the discriminator for adversarial training, our model reduces its reliance on domain-specific information and mitigates the risk of overfitting to target domain data. Therefore, domain adaptation effectively narrows the distribution gap between the source and target domains, enabling feature sharing and generalization, thereby enhancing the model's adaptability and performance in new domains.

[0038] Furthermore, a gradient reversal layer is introduced in the training, and the gradient returned by the discriminator is reversed by the gradient reversal layer, so that the feature extractor learns domain-invariant features.

[0039] In this implementation, the gradient reversal layer reverses the gradient during backpropagation, encouraging the network to learn domain-invariant feature representations, thereby achieving domain adaptation. DANN is applied in domain adaptation to align the distributions between the source and target domains. In contrast, in the context of domain generalization, it helps extract domain-invariant features from the source domain, enabling generalization to unseen domains.

[0040] From this implementation, it can be seen that this method uses Transformer for EEG sequence feature extraction, introduces a multi-head self-attention mechanism, and models the evolutionary characteristics of emotional states in the time dimension. At the same time, combined with the domain adversarial neural network (DANN) structure, by introducing a gradient reversal mechanism and a domain discriminator after the feature extractor, the feature distribution between the source stimulus (video) and the target stimulus (olfaction) is adversarially aligned. This method relies only on labeled data of the source modality in supervised training, while the target modality data participates in the training in an unlabeled form, thereby significantly improving the model's cross-stimulus generalization ability without the need for target domain labels. This combined model breaks through the bottleneck of existing transfer learning tasks that are limited to "same modality" or "slight modality differences", and realizes the transfer of emotion modeling from visual induction to olfactory induction.

[0041] The method is experimentally illustrated as follows: Figure 4 Results are presented for cross-subject emotion recognition using an olfactory-induced EEG dataset, employing a cross-validation approach. In this approach, data from eight participants were selected as the source domain, while the data from the remaining participants served as the target domain for domain adaptation (DA) and domain generalization (DG) techniques. Multiple models were evaluated, including those based on MLP, CNN, and Transformer architectures. The Transformer-based DANN was compared to other DA and DG methods, including DANN, Deep Subspace Alignment Network (DSAN), Dynamic Adversarial Adaptation Network (DAAN), Correlation Aligned Domain Adaptation (CORAL DA), and Correlation Aligned Domain Generalization (CORALDG), as well as baseline models.

[0042] Using a Transformer-based DANN for both DA and DG, our approach achieved state-of-the-art emotion classification results. Specifically, using DE features from full-band EEG signals, the Transformer-based DANN approach achieved an average accuracy of 51.2% and a standard deviation of 7.8% for DA, and an average accuracy of 50.3% and a standard deviation of 7.7% for DG. These results highlight the stability of Transformer neural networks and validate the effectiveness of the DANN module in both DA and DG settings.

[0043] like Figure 5We summarize the results of a cross-stimulus transfer learning task conducted after conducting cross-domain emotion classification experiments on an olfactory dataset. For consistency and comparability, the transfer learning process follows the same methodology as for cross-disciplinary emotion classification. In this setup, data from 15 subjects in the SEED dataset was used as the source domain, while data from nine subjects in the olfactory induction dataset was systematically selected as the target domain. This approach achieves cross-stimulus emotion transfer learning, validates the model's generalization ability across diverse emotion induction datasets, and demonstrates its potential for cross-stimulus emotion classification. Furthermore, transfer learning improves emotion recognition performance across diverse sensory conditions, enhancing the model's generalization and robustness. Furthermore, transfer learning significantly reduces the reliance on large amounts of labeled data. By leveraging emotion-related data from other stimuli, the model maintains strong performance even on novel emotion recognition tasks. These experiments not only deepen our understanding of emotion recognition but also provide essential empirical evidence for developing robust models that can handle diverse emotion induction conditions. These results are particularly relevant to the field of affective computing, advancing research in cross-stimulus emotion recognition and establishing strong experimental support for practical applications.

[0044] In a cross-stimulus emotion transfer learning task targeting positive, neutral, and negative emotions, our method transferred emotions from the SEED dataset induced by video stimuli to the olfactory stimuli-induced emotion dataset used in this paper. Notably, the application of the Transformer-based DANN to DA and DG achieved excellent results, reaching an average accuracy of 42.9% and a standard deviation of 4.7%, and a standard deviation of 44.0% and a standard deviation of 6.0%, respectively.

[0045] These results not only demonstrate the feasibility of cross-stimulus emotion transfer learning but also highlight its potential for practical applications. By transferring emotion data from visual and auditory stimuli to olfactory stimuli, this method can develop more general and powerful emotion recognition systems applicable to multiple sensory stimuli. This has important implications for the fields of psychology, neuroscience, and human-computer interaction, providing new perspectives and methods for multimodal emotion recognition technology. These advances may lead to the development of more natural and intuitive human-computer interaction interfaces. In addition, this method has the potential to use less labeled data for emotion recognition, reducing the cost of data collection and annotation, and paving the way for further research and application in affective computing.

[0046] like Figure 6 The figure shows a schematic structural diagram of a cross-sensory emotion transfer learning system provided by one embodiment of the present invention. The system can execute the cross-sensory emotion transfer learning method described in any of the above embodiments and be configured in a terminal.

[0047] A cross-sensory emotion transfer learning system 10 provided in this embodiment includes: a preprocessing module 11, a feature extraction module 12, a loss determination module 13 and a training learning module 14.

[0048] Among them, the preprocessing module 11 is used to process the dynamic feature differences of visual evoked EEG data and olfactory evoked EEG data due to different sensory emotional stimulation based on the time sliding window, and determine the differential entropy EEG features of the feature consistency between the visual evoked EEG data and the olfactory evoked EEG data; the feature extraction module 12 is used to input the differential entropy EEG features into a feature extractor based on the Transformer structure, and add position coding for strengthening time series information to the differential entropy EEG features in the feature extractor, and model the evolution of visual evoked emotions and olfactory evoked emotions in the time dimension through the head self-attention module and feedforward network in the feature extractor, so as to obtain the visual evoked EEG data and olfactory evoked emotions. The EEG time series feature representation of the EEG data is generated; the loss determination module 13 is used to input the EEG time series feature representation into the domain adversarial neural network constructed by the emotion classifier and the domain discriminator, classify the EEG time series feature representation by the emotion classifier, determine the emotion classification cross entropy loss, and perform domain discrimination on the EEG time series feature representation by the domain discriminator to determine the domain adversarial loss; the training learning module 14 is used to train the domain adversarial neural network based on the emotion classification cross entropy loss and the domain adversarial loss, reduce the distribution gap between the visually induced emotions and the olfactory induced emotions, generalize the characteristics of the visually induced emotions and the olfactory induced emotions, and realize cross-sensory transfer learning.

[0049] An embodiment of the present invention further provides a non-volatile computer storage medium storing computer-executable instructions, which can execute the cross-sensory emotion transfer learning method in any of the above method embodiments; As an embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, and the computer-executable instructions are configured as follows: Processing the dynamic feature differences of the visually evoked EEG data and the olfactory evoked EEG data due to different sensory emotional stimuli based on a time sliding window, and determining the differential entropy EEG features of the feature consistency between the visually evoked EEG data and the olfactory evoked EEG data; Inputting the differential entropy EEG features into a feature extractor based on a Transformer structure, adding position encoding for enhancing time series information to the differential entropy EEG features in the feature extractor, modeling the evolution of visually evoked emotions and olfactory evoked emotions in the time dimension through a head self-attention module and a feedforward network in the feature extractor, and obtaining EEG time series feature representations of the visually evoked EEG data and the olfactory evoked EEG data; Inputting the EEG time series feature representation into a domain adversarial neural network constructed by an emotion classifier and a domain discriminator, classifying the EEG time series feature representation by the emotion classifier to determine the emotion classification cross entropy loss, and performing domain discrimination on the EEG time series feature representation by the domain discriminator to determine the domain adversarial loss; The domain adversarial neural network is trained based on the emotion classification cross entropy loss and the domain adversarial loss to reduce the distribution gap between the visually induced emotions and the olfactory induced emotions, generalize the features of the visually induced emotions and the olfactory induced emotions, and achieve cross-sensory transfer learning.

[0050] A non-volatile computer-readable storage medium can be used to store non-volatile software programs, non-volatile computer executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of the present invention. One or more program instructions stored in the non-volatile computer-readable storage medium, when executed by a processor, perform the cross-sensory emotion transfer learning method in any of the above-described method embodiments.

[0051] Figure 7 This is a hardware structure diagram of an electronic device for a cross-sensory emotion transfer learning method provided by another embodiment of the present application, such as Figure 7 As shown, the device includes: One or more processors 710 and memory 720, Figure 7 A processor 710 is used as an example. The device for the cross-sensory emotion transfer learning method may further include: an input device 730 and an output device 740.

[0052] The processor 710, the memory 720, the input device 730 and the output device 740 may be connected via a bus or other means. Figure 7 The bus connection is taken as an example.

[0053] Memory 720, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the cross-sensory emotion transfer learning method in the embodiments of the present application. Processor 710 executes the non-volatile software programs, instructions, and modules stored in memory 720 to execute various server functional applications and data processing, thereby implementing the cross-sensory emotion transfer learning method in the above-mentioned method embodiment.

[0054] The memory 720 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data, etc. In addition, the memory 720 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 720 may optionally include a memory remotely located relative to the processor 710, and these remote memories may be connected to the mobile device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0055] The input device 730 can receive input digital or character information. The output device 740 can include a display device such as a display screen.

[0056] The one or more modules are stored in the memory 720 , and when executed by the one or more processors 710 , perform the cross-sensory emotion transfer learning method in any of the above method embodiments.

[0057] The above-mentioned product can execute the method provided in the embodiment of this application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of this application.

[0058] The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the device, etc. In addition, the non-volatile computer-readable storage medium may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0059] An embodiment of the present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the cross-sensory emotion transfer learning method of any embodiment of the present invention.

[0060] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and their primary purpose is to provide voice and data communications. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0061] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers and have computing and processing capabilities, and generally also have mobile Internet access. These terminals include PDAs, MIDs, and UMPC devices, such as tablet computers.

[0062] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0063] (4) Other electronic devices with data processing functions.

[0064] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include" and "comprise" include not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article or device. In the absence of further limitations, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements.

[0065] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0066] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A cross-sensory emotion transfer learning method, comprising: Processing the dynamic feature differences of the visually evoked EEG data and the olfactory evoked EEG data due to different sensory emotional stimuli based on a time sliding window, and determining the differential entropy EEG features of the feature consistency between the visually evoked EEG data and the olfactory evoked EEG data; Inputting the differential entropy EEG features into a feature extractor based on a Transformer structure, adding position encoding for enhancing time series information to the differential entropy EEG features in the feature extractor, modeling the evolution of visually evoked emotions and olfactory evoked emotions in the time dimension through a head self-attention module and a feedforward network in the feature extractor, and obtaining EEG time series feature representations of the visually evoked EEG data and the olfactory evoked EEG data; Inputting the EEG time series feature representation into a domain adversarial neural network constructed by an emotion classifier and a domain discriminator, classifying the EEG time series feature representation by the emotion classifier to determine the emotion classification cross entropy loss, and performing domain discrimination on the EEG time series feature representation by the domain discriminator to determine the domain adversarial loss; The domain adversarial neural network is trained based on the emotion classification cross entropy loss and the domain adversarial loss to reduce the distribution gap between the visually induced emotions and the olfactory induced emotions, generalize the features of the visually induced emotions and the olfactory induced emotions, and achieve cross-sensory transfer learning.

2. The method according to claim 1, wherein The training of the domain adversarial neural network based on the emotion classification cross entropy loss and the domain adversarial loss further includes: A gradient reversal layer is introduced into the training, and the gradient returned by the discriminator is reversed by the gradient reversal layer, so that the feature extractor learns domain-invariant features.

3. The method according to claim 1, wherein The classifying the EEG time series feature representation by the emotion classifier to determine the emotion classification cross entropy loss, and performing domain discrimination on the EEG time series feature representation by the domain discriminator to determine the domain adversarial loss includes: Determining a predicted emotion category represented by the EEG time series feature using an emotion classifier, and determining an emotion classification cross entropy loss based on the predicted emotion category and a preset benchmark emotion category; A domain discriminator is used to determine a prediction domain of the EEG time series feature representation, and a domain adversarial loss is determined based on the prediction domain.

4. The method according to claim 3, wherein: The training of the domain adversarial neural network based on the emotion classification cross entropy loss and the domain adversarial loss includes: The emotion classification cross entropy loss and / or the domain adversarial loss are weighted based on preset hyperparameters to determine a multi-task joint loss function, and the domain adversarial neural network is trained using the multi-task joint loss function until a preset target is reached.

5. The method according to claim 1, wherein The time sliding window-based solution to the dynamic feature differences in visually evoked EEG data and olfactory evoked EEG data due to different sensory emotional stimuli includes: Using a sliding window of a first time to match the stimulation duration of the visually evoked EEG data, and using a sliding window of a second time to match the stimulation duration of the olfactory evoked EEG data, wherein the first time is greater than the second time; After using the sliding window matching, feature alignment is performed to obtain differential entropy EEG features of feature consistency between the visually evoked EEG data and the olfactory evoked EEG data.

6. The method according to claim 1, wherein The visually evoked EEG data is used as source domain data, and the olfactory evoked EEG data is used as target domain data to achieve cross-sensory transfer learning from vision to olfaction.

7. A cross-sensory emotion transfer learning system, comprising: A preprocessing module is used to process the dynamic feature differences of the visual evoked EEG data and the olfactory evoked EEG data due to different sensory emotional stimuli based on a time sliding window, and determine the differential entropy EEG features of the consistency of the features of the visual evoked EEG data and the olfactory evoked EEG data; A feature extraction module is used to input the differential entropy EEG features into a feature extractor based on a Transformer structure, add position encoding for strengthening time series information to the differential entropy EEG features in the feature extractor, and model the evolution of visually evoked emotions and olfactory evoked emotions in the time dimension through the head self-attention module and feedforward network in the feature extractor to obtain EEG time series feature representations of the visually evoked EEG data and the olfactory evoked EEG data; a loss determination module, configured to input the EEG time series feature representation into a domain adversarial neural network constructed by an emotion classifier and a domain discriminator, classify the EEG time series feature representation by the emotion classifier to determine the emotion classification cross entropy loss, and perform domain discrimination on the EEG time series feature representation by the domain discriminator to determine the domain adversarial loss; A training learning module is used to train the domain adversarial neural network based on the emotion classification cross entropy loss and the domain adversarial loss, reduce the distribution gap between the visually induced emotions and the olfactory induced emotions, generalize the characteristics of the visually induced emotions and the olfactory induced emotions, and realize cross-sensory transfer learning.

8. A storage medium having a computer program product stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer program product having instructions embedded on a storage medium, wherein the instructions implement the steps of the method according to any one of claims 1 to 6.

10. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 6.

Images