Method for constructing connection between electroencephalogram data and emotional state through hierarchical feature fusion

Abstract

The present application relates to electroencephalogram image processing and deep learning technical field, disclose the method for constructing the connection between electroencephalogram data and emotional state by hierarchical feature fusion, HSFS-Transformer model is designed, aims at through layered extraction global and local features fully excavates the space-time characteristics of EEG signal, improves the performance of emotional classification identification task, then collects the EEG data of sensory member to different citrus flavor for the offline training of HSFS-Transformer model, so as to obtain the emotional value binary classification model and four classification model that can be carried out to different citrus flavor.The present application models the overall spatial relationship of electroencephalogram signal and the fine-grained feature in specific brain area respectively, realizes the multi-granularity hierarchical feature representation, overcomes the shortcomings such as strong subjectivity, weak practicability and poor repeatability of conventional sensory analysis, and simultaneously introduces the emotional value evaluation concept in sensory evaluation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of electroencephalogram image processing and deep learning, and particularly relates to a method for emotion state analysis by constructing electroencephalogram data through hierarchical feature fusion. BACKGROUND

[0002] With the development of society and the development of living standards, consumers and food companies show an increasingly strong interest in emotion-driven foods. So-called emotion-driven foods refer to foods that can evoke emotional value (usually positive emotions) during consumption. However, current conventional food sensory evaluation mainly relies on subjective and lagging self-reporting methods, usually through questionnaires or scales to record consumers' acceptance, preference and other explicit reactions during the tasting process. Food consumption is a complex experience, not only involving the intrinsic properties of food (such as sensory characteristics and physical properties), but also including extrinsic attributes (such as brand, price and packaging design). Consumers' emotional response to a product (such as liking or disliking) largely determines their acceptance and preference, and in turn affects their purchase decisions. However, most self-reports reflect perceptions and hedonic responses, which often fail to fully capture consumers' true experiences of a food product, as many responses are physiological and unconscious, making it difficult to accurately express them through self-perception. Therefore, capturing consumers' implicit physiological responses is particularly important for a deep understanding of their degree of liking and preference for a product. This not only helps to improve the sensory evaluation system, but also provides a new research perspective for food innovation and consumer experience optimization.

[0003] Modern sensory analysis techniques expand the connotation of traditional sensory analysis, not only relying on people to conduct sensory evaluation, but also using analytical instruments and intelligent sensory instruments as tools to assist sensory evaluation, making sensory analysis more deterministic and accurate. Intelligent sensory equipment has the advantages of stability and superior reproducibility, can quickly determine some ambiguous samples, shorten the time cycle of product quality determination, greatly reduce the workload of artificial sensory panel, and has the advantages of stable data and quantization. They are indirect and not subjective descriptions, so they can be used to record consumers' feelings when tasting, smelling or seeing different types of food, without the need for cognitive conversion after the experience. Modern sensory analysis techniques include four categories: electroencephalogram (EEG) for capturing brain activity, near-infrared spectroscopy (fNIRs), functional magnetic resonance imaging (fMRI); electrocardiogram (ECG), electromyogram (EMG) for capturing autonomic nervous system activity; eye tracking, facial recognition analysis software for capturing facial expression response, and respiratory detection band for capturing behavioral pattern response. Among them, EEG is considered an important brain function imaging tool for understanding the brain activity of consumers during food consumption due to its high time resolution, non-invasiveness, ease of operation and scalability. Although EEG technology has achieved encouraging results in capturing consumer perception, due to the large differences in individual experiences and emotional states expressed, and the difficulty in associating EEG features with different emotional states between different individuals. However, due to the influence of EEG signals by time, potential psychological conditions, language and cultural differences and other factors, it is difficult to find a suitable emotional classification method, which makes EEG signal analysis face many challenges.

[0004] The development of artificial intelligence and deep learning opens up new directions for EEG data mining and feature extraction. Although traditional machine learning methods, such as support vector machine (SVM), random forest (RF) and K nearest neighbor (KNN), perform well in handling simple, low-dimensional small samples, they often fail to achieve ideal results when faced with complex, high-dimensional large-scale data. With the continuous development of deep learning models, such as convolutional neural network (CNN), recurrent neural network (RNN) and Transformer model based on attention mechanism, they have shown excellent results in computer vision and natural language processing. In the task of EEG-based emotion recognition, deep learning methods usually mine the time and frequency domain features of EEG signals, build efficient classification architecture, and achieve more accurate and detailed emotion recognition. Among them, the Transformer model is particularly good at capturing long-range dependencies in sequence data due to its multi-head self-attention mechanism and parallel processing capability. Compared with other neural network models (such as CNN and RNN), Transformer has faster convergence speed, stronger feature extraction ability and higher computational efficiency. In addition, Transformer preserves the time structure of the sequence by introducing position embedding and class labels while integrating global context information, which makes it show unique advantages in EEG-based emotion recognition. Based on the image segmentation strategy proposed by Vit-Transformer, EEG signals can be divided into small blocks in the time or spatial dimension, so as to extract more detailed feature information. This strategy has achieved excellent performance in emotion recognition caused by visual stimulation. Although olfaction plays a crucial role in emotion and sensory experience, research on emotion recognition triggered by olfactory stimuli is still significantly insufficient. In-depth exploration of EEG emotion recognition under olfactory induction is expected to provide new research perspectives and technical support for emotion-driven food innovation. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a method for constructing the relationship between EEG data and emotional state through hierarchical feature fusion, which introduces global and local classification labels to extract global and local features from EEG data, and uses the HSFS-Transformer model to improve the performance of emotion binary classification or four-class classification tasks, and to construct the relationship between implicit EEG data and explicit emotional response.

[0006] To solve the above technical problems, the present application provides a method for constructing the relationship between EEG data and emotional state through hierarchical feature fusion, comprising the following steps:

[0007] S1, construct an EEG (electroencephalogram) spatial feature information extraction model, including sequentially connected data input and normalization module, position coding and classification label module, local and global attention extraction module, Transformer coding module and emotion classification module;

[0008] S2, gather sensory members to smell the citrus flavor, and capture their EEG data under the intervention of olfaction, record the sensory scores and emotional tendencies, then perform preprocessing operation on the EEG data, construct a binary classification training data set with the preprocessed EEG data and sensory scores, and construct a four-class classification training data set with the preprocessed EEG data and emotional tendencies;

[0009] S3, train the EEG spatial feature information extraction model based on the binary classification training data set to obtain an EEG spatial feature information extraction binary classification model, and train the EEG spatial feature information extraction model based on the four-class classification training data set to obtain an EEG spatial feature information extraction four-class classification model.

[0010] As an improvement of the method for constructing the relationship between EEG data and emotional state by hierarchical feature fusion of the application:

[0011] The operation of the data input and processing module is to normalize the input data (Input) by LayerNorm (layer normalization) to adjust the standard normal distribution with a mean of 0 and a variance of 1, and the Input is the preprocessed EEG data.

[0012] As a further improvement of the method for constructing the relationship between EEG data and emotional state by hierarchical feature fusion of the application:

[0013] The operation of the position coding and classification label module is:

[0014] First, different linear transformations are applied to the normalized EEG data to generate local position coding layers and global position coding layers;

[0015] Then, independent and learnable classification labels are introduced to the position coded EEG data in the electrode channel dimension to generate global EEG data with global classification labels, and local EEG data with local classification labels based on brain regions.

[0016] As a further improvement of the method for constructing the relationship between EEG data and emotional state by hierarchical feature fusion of the application:

[0017] The local and global attention extraction module includes one global attention module and seven local attention modules;

[0018] The global EEG data with global classification labels obtains global features through a global attention module:

[0019]

[0020] wherein , and are a global query tensor, a global key tensor and a global value tensor respectively; H is the number of multi-heads; is the feature dimension of each head; Softmax(·) represents weight calculation; Concat(·) represents concatenating multi-head results; Linear(·) represents full connection linear transformation;

[0021] The local EEG data with local classification labels is input into a local attention module according to the classification of local brain regions to obtain classified local brain region features:

[0022]

[0023] wherein , and ∈ are a local query tensor, a local key tensor and a local value tensor of a local brain region respectively; n r is the total number of electrodes of the local brain region; R is 7.

[0024] As a further improvement of the method for constructing the relationship between electroencephalogram data and emotional state through hierarchical feature fusion of the application:

[0025] The operation of the Transformer encoding module is as follows:

[0026] (1) Fusion features:

[0027]

[0028] (2) The fusion feature matrix Output_1 is subjected to a LayerNorm regularization operation, and then subjected to feature extraction through an MLP module.

[0029] As a further improvement of the method for constructing the relationship between electroencephalogram data and emotional state through hierarchical feature fusion of the application:

[0030] The operation of the emotion classification module is as follows: the output of the MLP module is added to the fusion feature matrix Output_1 in a residual connection manner, and then a linear transformation is performed through a layer combined with a ReLU activation function to generate a Final output (final output) for representing the predicted emotional state classification.

[0031] As a further improvement of the method for constructing the connection between electroencephalogram data and emotional state through hierarchical feature fusion of the application:

[0032] The citrus flavors include D-limonene, four kinds of citrus essential oils, and two-fold concentrated essential oils of the four kinds of citrus essences, and the citrus essential oils are sweet orange essential oil, lemon essential oil, bergamot essential oil, and grapefruit essential oil respectively.

[0033] The collection process of the EEG data, the sensory score, and the emotional tendency is as follows:

[0034] The sensory members smell the citrus flavors in turn, and when smelling each citrus flavor, the EEG data of the sensory member under the olfactory intervention are collected, the sensory score is recorded by using the positive and negative scoring system, and the emotional tendency is recorded by using the PAD emotional scale, wherein the EEG data of each citrus flavor are collected for 3 times, and the sensory score and the emotional tendency are collected for 1 time.

[0035] As a further improvement of the method for constructing the connection between electroencephalogram data and emotional state through hierarchical feature fusion of the application:

[0036] The preprocessing of the EEG data is as follows:

[0037] Channel positioning; removing HEO, VEO, CB1, CB2, and TRIGGER electrodes; reducing the sampling rate; applying high-pass filtering, low-pass filtering, and band-stop filtering; extracting the corresponding EEG segment according to the information marked by different citrus flavor stimuli; baseline correction; repairing bad leads by using spherical interpolation; setting the reference electrode as the bilateral mastoid electrode; decomposing the data by independent component analysis; marking and removing artifacts by manual inspection.

[0038] As a further improvement of the method for constructing the connection between electroencephalogram data and emotional state through hierarchical feature fusion of the application:

[0039] The construction process of the binary classification training data set is as follows:

[0040] Firstly, each preprocessed EEG data is manually labeled with emotional binary classification labels 0 and 1 according to the corresponding sensory score, wherein 1 represents a positive report and 0 represents a negative report; then, a time window of 3s-11s is extracted from each preprocessed EEG data and segmented at a length of 1s, and all segmented EEG segments are randomly and equally divided into 5 subsets, of which 4 are used as a binary classification training set and 1 is used as a binary classification test set.

[0041] As a further improvement of the method for constructing the connection between electroencephalogram data and emotional state through hierarchical feature fusion of the application:

[0042] The construction process of the four-class classification training data set is as follows:

[0043] First, each pre-processed EEG data is manually labeled with an emotional four-class label according to the corresponding emotional tendency; then, a 3s~11s time window is extracted for each sample of the pre-processed EEG data and segmented with a 1s length, and all segmented EEG segments are randomly and equally divided into 5 subsets, 4 of which are used as a binary classification training set, and 1 is used as a binary classification test set;

[0044] The emotional four-class label is: projecting the emotional tendency into 14 basic emotions, and further dividing the 14 basic emotions into four categories according to the emotional valence-arousal theory of Russell as the emotional four-class label, including positive high valence emotion, positive low valence emotion, negative high valence emotion and negative low valence emotion.

[0045] The beneficial effects of the present application mainly include:

[0046] (1) Layered space fusion feature extraction architecture: innovatively combines global attention and local attention to model the overall spatial relationship of the electroencephalogram signal and the fine-grained features in the specific brain area.

[0047] (2) Multi-class label mechanism: global classification labels and local classification labels are introduced at the same time: the global label is used to summarize the features of the overall brain area, and the local label represents the feature expression of each local brain area. Through this design, the model can realize multi-granularity, hierarchical feature representation, and take into account global and local emotional features.

[0048] (3) Residual connection and feature integration structure: fuse the original input, global features and local features, improve the stability of the model and improve the gradient propagation through the residual structure and regularization, which helps to capture complex electroencephalogram space-time dependence.

[0049] (4) Rich traditional sensory evaluation mode: based on intelligent sensing technology, it overcomes the shortcomings of strong subjectivity, weak practicality and poor repeatability of conventional sensory analysis, and introduces the concept of emotional value evaluation in sensory evaluation, capturing implicit and objective.

[0050] (5) Suitable for various electroencephalogram emotion recognition scenarios: can be widely used in emotion monitoring, psychological health assessment, brain-computer interface and other fields, and improves the intelligent level of emotional perception. BRIEF DESCRIPTION OF DRAWINGS

[0051] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0052] Figure 1 The structure diagram of the HSFS-Transformer model of the present application is shown in the figure;

[0053] Figure 2 The position division and electrode distribution diagram of the local brain area is shown in the figure;

[0054] Figure 2 The electrode names in the frontal lobe area are as follows:

[0055] The electrode names in the frontal lobe area are as follows:

[0056] The electrode names in the frontal lobe area are as follows:

[0057] The electrode names in the frontal lobe area are as follows:

[0058] The electrode names in the frontal lobe area are as follows:

[0059] The electrode names in the frontal lobe area are as follows:

[0060] The electrode names in the frontal lobe area are as follows:

[0061] The electrode names in the frontal lobe area are as follows:

[0062] Figure 3 The structure diagram of the global attention module of the application is shown in the figure;

[0063] Figure 4 The structure diagram of the local attention module of the application is shown in the figure;

[0064] Figure 5 The emotion two-classification confusion matrix (a) and the emotion four-classification confusion matrix (b) of citrus flavor induction using HSFS-Transformer are shown in the figure. DETAILED DESCRIPTION

[0065] The application will be further described below in conjunction with specific embodiments, but the protection scope of the application is not limited to this:

[0066] Example 1, a method for constructing the connection between electroencephalogram data and emotional state by hierarchical feature fusion.

[0067] This invention proposes a multi-level spatial feature extraction strategy based on electroencephalography (EEG), HSFS (Hierarchical Spatial Feature Strategy). This strategy, based on the Transformer framework, designs the HSFS-Transformer model, aiming to fully exploit the spatiotemporal features of EEG signals by extracting global and local features hierarchically, thereby improving the performance of emotion classification and recognition tasks. Then, EEG data generated by sensory members regarding different citrus flavors are collected for offline training of the HSFS-Transformer model, resulting in binary and quadruple classification models capable of classifying the emotional value of sensory stimuli of different citrus flavors.

[0068] Step 1: Construct a multi-level EEG spatial feature information extraction model (HSFS-Transformer) based on the Transformer model.

[0069] The overall architecture of the HSFS-Transformer model is as follows: Figure 1 As shown, the system mainly includes a data input and normalization module, a location encoding and classification labeling module, a local and global attention extraction module, a Transformer encoding module, and an emotion classification module. After preprocessing, the collected EEG data is transformed into a normal distribution between 0 and 1 by the data input and normalization module. The normalized EEG data is then processed by the location encoding and classification labeling module to obtain global EEG data with global classification labels (Global input) and local EEG data with local classification labels (Local input). The global attention extraction module then extracts global features from the Global input, and the local attention module extracts local brain region features from the Local input. Finally, the Transformer encoding module and the emotion classification module output the emotion classification of the EEG data (binary or quadruple classification results).

[0070] Step 1.1: Data Input and Normalization

[0071] For any time step, the EEG data acquired by a single electrode can be expressed as: , where i represents the i-th electrode, and t represents the index of the time step (the time step of EEG data usually corresponds to a sampling point; for example, a sampling rate of 128 Hz means 128 time steps are collected per second). Therefore, EEG data for a local or global brain region can be represented as . , where n represents the number of electrodes.

[0072] The input EEG data is a two-dimensional array with the shape of (number of electrodes x time steps), denoted as Input. A LayerNorm is applied to normalize the data to a standard normal distribution with a mean of 0 and a variance of 1, to improve the training stability and feature expression ability of the model. The shape of the processed data remains unchanged, still (number of electrodes x time steps).

[0073] Step 1.2 Position encoding and classification label embedding

[0074] To enable the model to understand the spatial position information of the electrodes, the normalized EEG data is processed by position encoding and classification label embedding:

[0075] Based on the Transformer processing architecture, global and local information is projected into a higher-dimensional space through different linear transformation operations to model multi-scale position information. Compared with the basic Transformer network, hierarchical position encoding provides different learnable semantic environments for subsequent hierarchical features. Among them, the EEG data of each electrode at each time step is projected into a high-dimensional space (640 dimensions), effectively capturing its position information. The classification of local brain regions includes frontal lobe, frontal lobe, central region, left temporal lobe, right temporal lobe, parietal lobe, and occipital lobe. The position division of local brain regions and electrode distribution are detailed in Figure 2 .

[0076] Then, different and independent learnable classification labels (class token) are introduced to the position-encoded EEG data in the electrode dimension, with a shape of (1 x 640), which are used to summarize the position information and feature association between electrodes. For the overall EEG data after position encoding, a global classification label (Global cls token) is added along the electrode channel dimension, while for the EEG data of different brain regions, different individual local classification labels (Local cls token) are assigned based on electrode distribution and brain region division (7 local brain regions). Compared with the conventional Transformer model, the introduction of local and global learnable labels helps the model better capture local / overall features, thereby improving the emotion classification ability.

[0077] After the above processing, the data before entering the Transformer attention layer is obtained, which includes global EEG data with global classification labels (Global input) and local EEG data with local classification labels (Local Input), with specific shape representation as: , where represents the feature representation of the local brain region or the global brain region, y cls is the classification label, y1, y2, y3, …, yn These represent the high-dimensional features of each electrode at the current time step.

[0078] Step 1.3 Use local and global attention extraction modules to extract features hierarchically.

[0079] The local and global attention extraction modules consist of the following components: one global attention module and seven local attention modules (corresponding to...). Figure 2 (Seven different brain regions shown).

[0080] (1) Characteristics The text indicates that global EEG data with global classification labels (Global Input) is used to extract macroscopic global features of the entire brain through the global attention module.

[0081] For the global attention module, its input is denoted as Global Input, with a shape of (61, 640), where 61 represents the number of electrodes plus a global classification label, and 640 is the feature dimension. Global Input first undergoes three independent linear transformations (640→640) to generate a global query tensor. ), global key tensor ( ) and global value tensors ( Although they maintain the same shape as the Global Input (61, 640), their positional information in the feature space has been mapped to different subspaces. Next, these tensors are divided into multiple attention heads to capture semantic features in different subspaces, thereby improving the model's ability to model diverse features and its generalization performance. Within each attention head, a scaled dot-product attention mechanism is used to compute... and The attention weights are calculated to obtain an attention matrix of shape (61, 61), representing the level of attention among all electrodes and the global classification label. Subsequently, the attention weights are scaled and normalized using the Softmax function to form a probability distribution that highlights key spatiotemporal features. The output of each attention head is calculated by combining the attention matrix with... After performing multiplication, the outputs of all attention heads are concatenated along the feature dimension and then fed into a linear layer to integrate the multi-head information, ultimately forming a unified feature representation. See details for the global attention module. Figure 3 The corresponding mathematical formula is shown in equation (1).

[0082] (1)

[0083] in , and are global query tensor, global key tensor and global value tensor respectively (the subscript i here is not strictly defined, used to distinguish the local attention in , and ); H is the number of heads; is the feature dimension of each head, represents the hidden layer dimension; Softmax(·) represents weight calculation; Concat(·) represents the operation of splicing multi-head results; Linear(·) represents full connection linear transformation. The final output result Global output has a shape of (61, 640), which is consistent with the input shape of Global Input.

[0084] (2) Feature Local EEG data (Local Input) with local classification labels in each local brain area is independently input into a local attention module to obtain classified local brain area features. This design enables the model to focus on key information within a specific brain area and capture inter-regional dependencies.

[0085] For the local attention module, the input is denoted as Local Input, with a shape of (1+Mun electrode, 640), where 1+Mun electrode represents the number of electrodes in a certain brain area plus a global classification label, and 640 is the feature dimension. Similar to the calculation process of the global attention module, 7 different Local Inputs are input into 7 independent local attention modules. Each module obtains 7 independent local attention outputs by calculating the corresponding attention matrix and performing probability scaling, and the output shape remains (1+Mun electrode, 640). Among them, the class label in each output captures the independent, fine-grained, and specific regional information of each brain area. In these local outputs, the class label captures independent, fine-grained, and specific regional information in each brain area, thereby improving the sensitivity and discrimination ability of the model to regional features. Subsequently, the local outputs of all 7 brain areas are spliced in the electrode dimension, and a fully connected linear layer (7→61) is used to obtain the final output Local output of the local region, which has a shape of (61, 640) and is consistent with the dimension of Global Output. The specific calculation process of the local attention module is shown in Figure 4 , and the corresponding mathematical formula is shown in equation (2).

[0086] (2)

[0087] wherein , and ∈ are the local query tensor, the local key tensor and the local value tensor of the specific local brain region, respectively; n r is the total number of electrodes in the local brain region; R represents the number of local brain regions is 7; H is the number of multi-head; d k is the feature dimension of each head; Softmax(·) represents weight calculation; inner Concat(·) represents concatenating multi-head results in the same brain region; outer Concat(·) represents concatenating classification labels of different brain regions; each Linear(·) represents a fully connected layer for converting multi-head information into a specific dimension representation.

[0088] Step 1.4. Transformer encoding module and feature fusion

[0089] The Transformer encoding module is composed of a multi-layer perceptron (MLP) and a layer normalization (LayerNorm). Compared with the basic Transformer network, HSFS-Transformer further introduces MLP and LayerNorm in the feature fusion process to stabilize the training process and reduce the risk of gradient vanishing or explosion.

[0090] The original input Input, Local output and Global output are added to obtain the fusion feature matrix Output_1, which maintains the shape of (61, 640). The fusion feature matrix Output_1 effectively integrates the original data input, local brain region features and global feature information, forming a whole representation with multi-scale feature representation. The corresponding mathematical formula is shown in equation (3).

[0091] (3)

[0092] The fusion feature matrix Output_1 is then subjected to another LayerNorm regularization operation to improve the stability of the feature distribution, and then input into the MLP module for nonlinear transformation and further feature extraction. This process helps the model to more fully capture and integrate relevant information in multi-source features, improving the overall modeling ability. The above fusion and feature extraction process is performed once in each attention layer iteration.

[0093] Step 1.5, emotion classification module: the last stage adds the classification labels and performs emotion classification through a fully connected linear layer and an activation function.

[0094] ​After multiple iterations through the attention layer, the fusion feature matrix Output_1 effectively captures the key features of the EEG signal. To further enhance the model's feature representation capability and gradient transmission effect, a residual connection is used to add the output of the MLP module to the fusion feature matrix Output_1, obtaining the converted feature Output_2. This residual structure helps stabilize model training and supports a more rich feature learning process by preserving the semantic representation of shallow and deep layers. The specific calculation method is shown in equation (4):

[0095] (4)

[0096] After the above comprehensive and hierarchical feature extraction and integration, the classification label in Output_2 can effectively capture the inherent complex spatio-temporal dependence in the EEG signal. Finally, the model generates Final output through a linear transformation combined with a ReLU activation function, which is used to represent the predicted emotional state.

[0097] Step 2, offline training of the HSFS-Transformer model

[0098] Step 2.1, collection of training data

[0099] Step 2.1.1, collection of citrus flavors

[0100] 9 kinds of citrus flavors, including D-limonene, 4 kinds of citrus essential oils, and 2-fold concentrated essential oils of 4 kinds of citrus essences. The 4 kinds of citrus essential oils are sweet orange essential oil, lemon essential oil, bergamot essential oil, and grapefruit essential oil. Among them, D-limonene and 4 kinds of citrus essential oils are commercially available cold-pressed natural essential oils, and the components of each essential oil are detected by GC-MS without external addition of ingredients. All solvents are stored in a 4°C refrigerator away from light for testing.

[0101] The 2-fold concentrated essential oils of 4 kinds of citrus essences are prepared by a molecular distillation device, controlling the ratio of light components and heavy components to be 1:1. The extraction of the molecular distillation device is as follows: a certain amount of original citrus essential oil is introduced into a 1 L sample inlet, the initial distillation temperature is controlled at 35°C, the initial pressure is 350 Pa, the stirring speed is 250 rpm, and the initial flow rate is 5 mL / min. By observing the changes in the flow rate of light components and heavy components in the collector, the content components in the two collectors are adjusted to 1:1 by rotating the nut.

[0102] Step 2.1.2, recruitment of sensory panelists and subjective scoring

[0103] The recruitment criteria for sensory panelists are as follows: according to the content of GB / T 1629.1-2012, the subjects are screened.

[0104] Sensory scores (positive and negative classification) and emotional four classification based on PAD emotional scale and "arousal-valence" theory were used for subsequent binary classification model training and four classification model training, respectively.

[0105] (1) Sensory scores (positive and negative classification)

[0106] The sensory scores were evaluated using a positive and negative scoring system, with a score range of -5 to 5. Negative scores represent the subjects' negative acceptance of the odor, while positive scores represent the subjects' positive acceptance of the odor. The larger the numerical value, the stronger the preference.

[0107] (2) Emotional four classification

[0108] The PAD emotional scale developed by the Chinese Academy of Sciences was used to record their emotional tendencies after smelling different citrus flavors.

[0109] Step 2.1.3, the detailed process of EEG information capture is as follows:

[0110] Experimental environment and preparation: The experiment was conducted in a sensory laboratory with strict environmental parameters: temperature 20-22℃, humidity 50-55%, soft light and good ventilation. Before the formal experiment, the subjects needed to be trained, including: familiar with the experimental instructions (such as closing eyes, resting, etc.); reduce body movement to reduce motion artifacts; maintain natural breathing and focus on odor perception.

[0111] Experimental design and implementation: The experimental process was divided into 3 rounds, each round including smelling 9 different samples (9 kinds of citrus flavors) in turn. At the beginning of each experiment, the subjects needed to rinse their mouths with ultrapure water three times to minimize the interference of taste on smell. Then, the subjects closed their eyes, remained in a resting state, and started data recording when the experimenter gave the instruction "start collecting resting state". At this time, mark "0" in the Curry 9 signal acquisition system, and record 15s of resting state EEG signal (for baseline correction). Then, the experimenter gave the instruction "please smell sample No. 1", and the experimental assistant sent the smelling paper containing 5 μL of citrus essential oil (or D-limonene) to the subject's nose 2-3 cm below, and gently shook it. The subjects closed their eyes and smelled for 15s while keeping their heads still to reduce motion artifacts. After smelling, the experimenter gave the instruction to remove the stimulus, and the subjects could open their eyes and rest for 20s, and smell coffee beans to clear the residual odor and prevent olfactory fatigue. Due to the similarity and familiarity between the odors, the stimuli were presented in a single-blind, fixed order, ensuring that there were significant differences in odor between the two types of stimuli. After each round of experiment, the subjects had sufficient time to rest and relax, so that their EEG signals could be restored to the baseline level as much as possible.

[0112] EEG signal acquisition and quality control: A 64-channel EEG cap (international 10 / 10 system) was used, and the electrode impedance was controlled below 10 Ω. The reference electrode was Reference. The data were recorded using Curry 9 system with SynAmps2 / RT amplifier, with a sampling rate of 1000 Hz, DC mode operation, equipped with 400 Hz low-pass filter and 24 nV / LSB precision.

[0113] 30 sensory members smelled 9 kinds of citrus flavors, and captured their EEG information under the intervention of smell. Their sensory scores were recorded by using a positive and negative scoring system of -5 to 5, and their emotional tendencies after smelling citrus flavors were recorded by using PAD emotional scale. Each citrus flavor was collected 3 times of EEG data, 1 time of sensory score and 1 time of emotional tendency result. Finally, 30 sensory members collected 710 EEG data, 270 sensory score reports and 270 emotional tendencies for 9 groups of citrus smells. The 270 sensory scores included 197 positive reports (acceptance > 0) and 73 negative reports (acceptance < 0). The 270 emotional tendencies included 8 reports of resentment, 11 reports of fear, 17 reports of hostility, 6 reports of disgust, 7 reports of sadness, 16 reports of anxiety, 8 reports of contempt, 20 reports of boredom, 6 reports of lightness, 51 reports of harmony, 8 reports of dependence, 8 reports of optimism, 77 reports of surprise and 18 reports of joy.

[0114] Step 2.2, EEG data preprocessing

[0115] The 710 EEG data were sequentially subjected to the following data preprocessing operations

[0116] Channel positioning: the spatial positions of all electrodes were determined;

[0117] Delete useless electrodes: remove irrelevant electrodes such as HEO, VEO, CB1, CB2 and TRIGGER;

[0118] Reduce sampling rate: reduce the sampling rate to 500 Hz;

[0119] Filtering: apply high-pass filtering (0.1 Hz), low-pass filtering (30 Hz) and band-stop filtering (48-51 Hz);

[0120] Segment extraction: extract the EEG segment (-2~15s) from 2s before the marker to 15s after the stimulation according to the information marked by different citrus flavor stimuli;

[0121] Baseline correction: remove the baseline of the data by taking the resting state 2s before the stimulation as the reference;

[0122] Bad lead interpolation: repair bad leads using spherical interpolation method;

[0123] Referring back: Set the reference electrode as M1 and M2 (bilateral mastoid electrodes);

[0124] Independent Component Analysis (ICA): ICA decomposition of the data;

[0125] Artifact identification and removal: Artifact labeling with the ICLabel tool and possible artifacts removed by manual inspection.

[0126] Step 2.3, Construction of the dataset for offline training

[0127] (1) Binary classification training dataset

[0128] The 270 pre-processed EEG data and 270 sensory score reports were used as the binary classification training data source. First, each pre-processed EEG data was manually labeled with emotion binary classification labels 0 and 1 according to the corresponding sensory score, where 1 represents a positive report (acceptance > 0) and 0 represents a negative report (acceptance < 0). Each report corresponds to 3 EEG signals.

[0129] Then, to reduce the impact of olfactory adaptability and long-term olfactory fatigue, each sample of the pre-processed EEG data in step 2.2 was extracted for 3s~11s, a total of 8s, time window for subsequent analysis. Next, the 8s time window was divided into 8 segments with a 1s time length, each segment including stable EEG brain signal features. Since the training process of the present application uses 5-fold cross-validation, all EEG segments are randomly and equally divided into 5 subsets, 4 of which are used as the binary classification training set and one as the binary classification test set data.

[0130] After the above processing, the pre-processed EEG signal produces 6160 EEG segments with a 1s time interval, including 4392 positive reports and 1768 negative reports (of which, the data of 3 subjects are excluded due to noise interference or baseline drift). The 4 subsets of the binary classification training set include a total of 4928 EEG segments, and the binary classification test set includes 1232 EEG segments.

[0131] (2) Four-class training dataset

[0132] 270 pieces of pre-processed EEG data and 270 pieces of emotional tendency were used as the data source for four-class training. First, the emotional tendency obtained by the PAD emotion scale was further projected into 14 basic emotions. On this basis, according to the valence-arousal theory of Russell's emotion, the 14 basic emotions were further divided into four categories as labels, i.e. the four-class emotion labels included: positive high valence emotion (HP): joy, optimism and surprise; positive low valence emotion (LP): relaxation, harmony and dependence; negative high valence emotion (HN): fear, anxiety, hatred and hostility; negative low valence emotion (LN): boredom, sadness and contempt. Each pre-processed EEG data was manually labeled with four-class emotion labels according to the corresponding emotional tendency.

[0133] Then, each sample of the pre-processed EEG data was extracted for a time window of 3s~11s, a total of 8s, and divided into 8 parts with a length of 1s, resulting in a total of 6160 EEG segments, including 2296 EEG segments of positive low valence emotion (LP), 1616 EEG segments of positive high valence emotion (HP), 1136 EEG segments of negative high valence emotion (HN), and 1112 EEG segments of negative low valence emotion (LN) (of which, the data of 3 subjects were excluded due to noise interference or baseline drift). The four-class training process also used the five-fold cross-validation method, and all EEG segments were randomly and equally divided into 5 subsets, of which 4 subsets were used as the four-class training set and one subset was used as the four-class test set data. The four-class training set included a total of 4928 EEG segments, and the four-class test set included 1232 EEG segments.

[0134] Step 2.4. Model training and evaluation

[0135] Step 2.4.1. Training of two-class model

[0136] The binary classification training set data is used as the input of the model, and the corresponding acceptability (0 represents negative acceptability and 1 represents positive acceptability) is used as the output of the model. The model is trained by using the five-fold cross-validation method, and the specific parameters and hyperparameters are as follows: the training batch size is 64, the hidden size is 640, the Adam optimizer is used, the learning rate and the weight decay are both 0.0001, the model is trained for 200 times, the attention layer is divided into 4 heads, the attention layer is cycled for 4 times, the dropout rate of the attention layer and the MLP layer is set to 0.2 to prevent model overfitting. The cross-entropy loss function is used to calculate the loss value. In each batch training process, the loss function is calculated by forward propagation first, then the gradient is calculated by back propagation, and the training parameters are updated based on the gradient using the Adam optimizer. Then, the binary classification test set is used to evaluate the classification performance of the model by using the five-fold cross-validation method according to the following indicators: accuracy, precision, recall, F1 score, Kappa coefficient (Cohen's Kappa) and Jaccard coefficient (Jaccard Index). If the accuracy of the test set does not improve after 20 consecutive training epochs (epochs), the training is stopped. In terms of accuracy and precision of emotion binary classification, the model reaches 88.84±0.41% and 86.86±0.51% respectively, and the confusion matrix is shown in Figure 5 (a), and other model performance indicators are shown in Table 1:

[0137] Table 1. Results of emotion binary classification

[0138]

[0139] (2) Training of four-classification model

[0140] The four-classification training set and the four-classification test set are used as the input of the model, and the training and testing process is consistent with the binary classification training and testing process. The confusion matrix of emotion four classification on the test set is shown in Figure 5 (b), and other model performance indicators are shown in Table 2,

[0141] Table 2. Results of emotion four classification

[0142]

[0143] 3. Experiment

[0144] The training set constructed in step 2.3 is used for model training, and the test performance indicators are verified on the test set data.

[0145] 3.1 Ablation experiments

[0146] (1) The attention module in the HSFS-Transformer model is changed to only use a global attention module, and the remaining structure remains unchanged. The resulting results on the test set are: the accuracy of the emotion four classification task is 76.12±2.49%, the precision is 76.53±2.65%, the recall is 75.03±3.01, the F1 score is 75.66±2.88, the Kappa index is 66.88±3.53, and the Jaccard index is 60.99±3.66. Compared with the results in Table 2 in step 2.4.1 (2), it can be known that the HSFS-Transformer model using only the global attention module has a lower classification effect on emotions than the HSFS-Transformer model proposed in the present application. Therefore, the importance of the hierarchical feature fusion extraction strategy proposed in the present application is proved.

[0147] (2) The HSFS-Transformer model does not use classification labels, but only uses global average to realize emotion classification (i.e. cancel cls-token, use pool mean to classify emotions), and the rest remains unchanged.

[0148] The resulting results are: the accuracy of the emotion four classification task is 54.32±1.97%, the precision is 53.63±1.84%, the recall is 52.77±1.67, the F1 score is 53.08±1.74, the Kappa index is 36.59±2.63, and the Jaccard index is 36.25±1.64. Compared with the results in Table 2 in step 2.4.1 (2), it can be known that the classification effect of the model on emotions is lower than that of the HSFS-Transformer model proposed in the present application. Therefore, the effectiveness and importance of the multi-classification label mechanism proposed in the present application are proved.

[0149] 3.2 Comparison experiment with other machine learning models

[0150] The HSFS-Transformer model of the present application is compared with other conventional machine learning models, such as support vector machine (SVM), random forest algorithm (RF), XGBoost model, and conventional deep learning network recurrent neural network (RNN), long short-term memory neural network (LSTM), convolutional neural network (CNN), and basic Transformer model. The same training set is used to train each model, and the same test set data is used to evaluate the performance of the model. The results are shown in Table 3.

[0151] Table 3. Performance of different machine learning models in emotion four classification task

[0152]

[0153] The results in Table 3 further illustrate that the HSFS-Transformer proposed in the present application can comprehensively capture useful features in the EEG signal, thereby effectively associating objective EEG data and subjective emotional responses, compared with other existing networks.

[0154] Finally, it should be noted that the above only lists several specific embodiments of the present application. Obviously, the present application is not limited to the above embodiments, but can have many variations. All variations that can be directly derived or inferred from the disclosed content by those of ordinary skill in the art should be considered within the scope of the present application.

Claims

1. A method for constructing the connection between EEG data and emotional states through hierarchical feature fusion, characterized in that... Includes the following steps: S1. Construct a model for extracting spatial features from electroencephalograms, including a data input and normalization module, a location encoding and classification label module, a local and global attention extraction module, a Transformer encoding module, and an emotion classification module connected in sequence. The operation of the location encoding and classification label module is as follows: First, different linear transformations are applied to the normalized EEG data to generate a local location encoding layer and a global location encoding layer; then, independent and learnable classification labels are introduced into the location-encoded EEG data in the electrode channel dimension to generate global EEG data with global classification labels and local EEG data with local classification labels based on brain regions. The local and global attention extraction module includes one global attention module and seven local attention modules. Global EEG data with global classification labels obtains global features through the global attention module, and local EEG data with local classification labels are input into the local attention module according to the classification of local brain regions to obtain the classified local brain region features. The operation of the Transformer encoding module is as follows: (1) Fusion characteristics: in, This is the output of the global attention module. This is the output of the local attention module; (2) The fused feature matrix Output_1 is normalized by layer normalization and then processed by the MLP module for feature extraction; S2. Summarize sensory members to smell citrus flavor and capture their EEG data under olfactory intervention, while recording sensory scores and emotional tendencies; then preprocess the EEG data, construct a binary classification training dataset with the preprocessed EEG data and sensory scores, and construct a four-class classification training dataset with the preprocessed EEG data and emotional tendencies. S3. Train the model based on EEG spatial feature information extraction using a binary classification training dataset to obtain a binary classification model based on EEG spatial feature information extraction, and train the model based on EEG spatial feature information extraction using a four-class classification training dataset to obtain a four-class classification model based on EEG spatial feature information extraction.

2. The method for constructing the connection between EEG data and emotional state through hierarchical feature fusion according to claim 1, characterized in that: The operation of the data input and processing module is to normalize the input data through layer normalization and adjust it to a standard normal distribution with a mean of 0 and a variance of 1. The input data is the preprocessed EEG data.

3. The method for constructing the connection between EEG data and emotional state through hierarchical feature fusion according to claim 2, characterized in that: ； in , and These are the global query tensor, the global key tensor, and the global value tensor, respectively; H is the number of multi-head tensors. These are the feature dimensions of each head; Softmax(·) represents weight calculation; Concat(·) represents concatenating the results of multiple heads; Linear(·) represents a fully connected linear transformation; ； in , and ∈ These are the local query tensor, local key tensor, and local value tensor of a local brain region, respectively; n r The total number of electrodes in the local brain region; R is 7.

4. The method for constructing the connection between EEG data and emotional state through hierarchical feature fusion according to claim 3, characterized in that: The operation of the emotion classification module is as follows: using residual connection, the output of the MLP module is added to the fusion feature matrix Output_1, and then a linear transformation combined with the ReLU activation function is used to generate the final output, which is used to represent the predicted emotion state classification.

5. The method for constructing the connection between EEG data and emotional state through hierarchical feature fusion according to claim 4, characterized in that: The citrus flavor includes D-limonene, four citrus essential oils, and four citrus extracts in double-concentrated essential oils: sweet orange, lemon, bergamot, and grapefruit. The process of collecting EEG data, sensory scores, and emotional tendencies is as follows: Sensory members smelled the citrus flavors in turn. When smelling each citrus flavor, the electroencephalogram (EEG) data of the sensory members under olfactory intervention was collected. At the same time, the sensory scores were recorded using a positive and negative scoring system and the emotional tendency was recorded using the PAD emotion scale. The EEG data of each citrus flavor was collected three times, and the sensory scores and emotional tendency were collected once each.

6. The method for constructing the connection between EEG data and emotional state through hierarchical feature fusion according to claim 5, characterized in that: The preprocessing of the EEG data is as follows: Channel localization; removal of HEO, VEO, CB1, CB2, and TRIGGER electrodes; reduction of sampling rate; application of high-pass, low-pass, and band-stop filtering; extraction of corresponding EEG segments based on information labeled by different citrus flavor stimuli; baseline correction; Bad conduction was repaired using spherical interpolation; the reference electrode was set as a bilateral papillary electrode; the data was decomposed using independent component analysis; artifacts were marked and removed by manual inspection.

7. The method for constructing the connection between EEG data and emotional state through hierarchical feature fusion according to claim 6, characterized in that: The process of constructing the binary classification training dataset is as follows: First, each preprocessed EEG data point is manually labeled with an emotion binary classification label of 0 and 1 based on its corresponding sensory score, where 1 represents a positive report and 0 represents a negative report. Then, a time window of 3 to 11 seconds is extracted from each preprocessed EEG data point and divided into segments of 1 second each. All segments of the EEG data are randomly and equally divided into 5 subsets, of which 4 are used as the binary classification training set and 1 is used as the binary classification test set.

8. The method for constructing the connection between EEG data and emotional state through hierarchical feature fusion according to claim 7, characterized in that: The process of constructing the four-class classification training dataset is as follows: First, each preprocessed EEG data is manually labeled with a four-category emotion classification based on its corresponding emotional tendency. Then, a time window of 3s to 11s is extracted from each sample of the preprocessed EEG data and divided into segments of 1s in length. All segments of the EEG data are randomly and equally divided into 5 subsets, of which 4 are used as a binary classification training set and 1 is used as a binary classification test set. The four-category emotional label is as follows: emotional tendencies are projected into 14 basic emotions. Based on the emotional valence-arousal theory, the 14 basic emotions are further divided into four categories as the four-category emotional label, including positive high-valence emotions, positive low-valence emotions, negative high-valence emotions, and negative low-valence emotions.

Images