A multi-modal sentiment analysis method and system based on pseudo twin network

Abstract

The application provides a multi-modal emotion analysis method and system based on a pseudo twin network. The method uses a pyramid network integrating a deep separable convolution strategy as an electroencephalogram processing module to mine and fuse time and space emotion perception features of electroencephalograms at multiple scales, and extract deeper and more effective intrinsic features of electroencephalograms. Meanwhile, two fully connected network layers are used as an electrooculogram processing module to extract behavior features of electrooculograms. In addition, in order to further learn the relationship between the modes, the application uses a pseudo twin network containing a double-branch subnetwork to transform emotion features of each mode, and uses similarity constraints in the module to coordinate features of different modes into a similar hyperspace. Then, feature fusion is adopted to learn the weight of different modes participating in model classification decision for emotion recognition, so that the consistency and complementarity of multi-modal emotion features are realized, and the performance and generalization ability of the model are improved.

Description

Technical Field

[0001] This invention belongs to the field of physiological signal emotion analysis technology in neuroscience, and is a multimodal emotion analysis method and system based on pseudo-twin networks. Background Technology

[0002] Emotions play a vital role in human life. Recognizing emotions is often intended to facilitate appropriate interaction. However, in individuals with disorders of emotional expression, they often struggle to accurately convey their inner feelings through facial expressions. For example, people with anxiety disorders and behavioral disorders often find it difficult to effectively express their emotions. To accurately assess patients' emotional expression and assist physicians in making informed decisions, researchers have identified emotion recognition as an important research area in artificial intelligence, medicine, and computer science.

[0003] Currently, sentiment analysis methods are mainly divided into two categories: physiological signals and non-physiological signals. Non-physiological signals are the external manifestations of human emotions, typically using vocal features, facial expressions, and body movements for emotion recognition. However, these signals are easily hidden or interfered with, thus affecting the accuracy of the analysis. To improve the reliability of emotion recognition, many researchers have focused on using physiological signals for sentiment analysis. Physiological signals include electroencephalography (EEG), electrooculography (EOG), electromyography (EMG), and electrodermal conductance (EDC), which originate from the body's physiological processes and activities and directly reflect the body's internal physiological state. They are unaffected by cognitive factors and possess objectivity. Therefore, accurately capturing an individual's true emotional state using physiological signals is considered a reliable sentiment analysis method.

[0004] Electroencephalography (EEG), directly controlled by the central nervous system, contains rich emotional information compared to other physiological signals and has been extensively studied in emotion analysis. However, some existing research methods have neglected the full utilization of superficial and deep EEG features. Furthermore, human emotional states are complex and variable, making it challenging to fully express emotions using only one modality. The eyes are considered the windows to the soul, and electrooculography (EOG) signals, including gaze tracking, eye movement detection, and blink frequency, are also closely related to emotions. Related studies have demonstrated a mapping relationship between eye fixation states and user emotional states, finding that eye movements can reflect parts of a person's emotional state. For example, when a person is relaxed and smiling, their eye movements are at eye level; when a person is angry, their eyes widen and become fixed. Other studies have shown that blinking provides relevant information about a user's state and emotional characteristics because the activity of attention requiring thought leads to a decrease in blink frequency. In short, signals related to the eyes, especially those obtained through electrooculography, provide valuable information for emotion research. Therefore, studying multimodal physiological signals for emotion recognition has become particularly important. However, it is a challenge to link the correlations between modes and fully leverage the advantages of both modes. Summary of the Invention

[0005] To address the aforementioned shortcomings, this invention proposes a multimodal sentiment analysis method and system based on pseudo-Twin networks. This method utilizes a pyramid network integrating a deep separable convolution strategy as the EEG processing module to mine and fuse temporal and spatial sentiment perception features from multiple EEG scales, extracting deeper and more effective intrinsic EEG features. Simultaneously, two fully connected network layers are used as the EEG processing module to extract behavioral features from the eye. Furthermore, to further learn the relationships between modalities, this invention uses a pseudo-Twin network with bi-branch subnetworks to transform the sentiment features of each modality, and uses similarity constraints in this module to coordinate features from different modalities into a similar hyperspace. Then, feature fusion is employed to learn the weights of different modalities for sentiment recognition and participate in the model's classification decision, achieving consistency and complementarity of multimodal sentiment features, thereby improving the model's performance and generalization ability.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a multimodal sentiment analysis method based on pseudo-twin networks, the detailed technical solution of which includes the following steps:

[0007] Step 1: Data Collection;

[0008] Multimodal physiological signals induced in subjects while watching video clips were collected, and the physiological signals were labeled with emotional tags; wherein the multimodal physiological signals include electroencephalogram (EEG) signals and electrooculogram (EOG) signals;

[0009] Step 2: Preprocess the collected multimodal physiological signals and extract features to obtain EEG and EEG features;

[0010] Step 3: Generation of multimodal sample pairs;

[0011] Traverse each EEG feature sample x eeg Electrooculography (EOG) feature samples x were selected from the same subject at the same time. eog(m) As a multimodal positive sample pair {x eeg x eog(m)}, and the label Y is denoted as 1;

[0012] Traverse each EEG feature sample x eeg Randomly select electrooculography (EOG) feature samples of different emotions from the same subject x eog(n) As a multimodal negative sample pair {x eeg x eog(n) Let the label Y be denoted as o;

[0013] For ease of description, multimodal samples are used for {x}. eeg x eog Let} represent the value of x. eog =x eog(m) or x eog(n) For the label Y = 1 or 0, 1 represents x eeg and x eog Two modal samples belong to the same sentiment, where 0 represents x. eeg and x eog The two modal samples do not belong to the same sentiment;

[0014] Step 4: Construct the EEG processing module:

[0015] The EEG processing module employs a pyramid network integrating depthwise separable convolutions to process EEG features x in multimodal sample pairs. eeg Extracting multi-scale integrated spatiotemporal intrinsic modal features F eeg ;

[0016] Step 5: Construct the EOG processing module;

[0017] The EOG processing module includes two cascaded fully connected layers for processing electrooculography (EOG) features x in multimodal sample pairs. eog Extracting external behavioral modal features F from electrooculogram signals eog ;

[0018] Step 6: Construct a pseudo-twin network, and train, validate, and test it;

[0019] The pseudo-twin network includes two branch subnets and a weighted fusion module, wherein the two branch subnets are an EEG processing module and an EOG processing module, respectively.

[0020] The weighted fusion module processes the two modal features F output by the EEG processing module and the EOG processing module. eeg and F eog Perform automatic learning and weighted fusion to obtain better combinations of F eeg and F eog The fusion characteristics;

[0021] Step 7: Implement multimodal sentiment analysis using a well-trained, validated, and tested pseudo-twin network.

[0022] Preferably, in step two, the EEG features are derived from the differential entropy features of the EEG signal.

[0023] Preferably, in step two, the electrooculography (EOG) features are the result of splicing the mean, standard deviation, mean and standard deviation of the first and second order differences of the EOG signal, normalized length density (NLD) features, and Hjorth features.

[0024] Preferably, in step four, the EEG processing module includes three paths and an output layer, namely a bottom-up path, a top-down path, and a horizontal connection.

[0025] The bottom-up path includes a first to a third convolutional block connected in series; the first convolutional block accepts EEG features x. eeg After processing, the result is output to the second convolutional block, and simultaneously input to the top-down path via lateral connections; the second convolutional block receives the output result of the first convolutional block, processes it, and outputs the result to the third convolutional block, and simultaneously input to the top-down path via lateral connections; the third convolutional block receives the output result of the second convolutional block, and input to the top-down path via lateral connections.

[0026] The top-down path uses bilinear interpolation to upsample and adjust feature maps of different sizes with respect to the output of the third convolutional block in the bottom-up path, including the first to third multi-scale fusion layers connected in sequence.

[0027] The lateral connection includes a 1×1 convolution used to adjust the channel size of the feature maps of different scales output by the bottom-up path;

[0028] The output layer flattens the information-rich feature DF1 output by the third multi-scale fusion layer into a one-dimensional vector through a flatten layer; then, it passes through two fully connected layers with ReLU activation function to extract the vector representation F. eeg Specifically:

[0029]

[0030] Where N(s) represents the number of neurons in layer s, where s = {1, 2, 3}, corresponding to the flattened layer and fully connected layers with 32 and 16 neurons respectively; p represents the p-th neuron in the input layer, and DF1 is used as input, which can be divided into the feature value DF input by the p-th neuron in the input layer. 1_p F eeg Indicates output; w qp w represents the connection weight between the p-th neuron in the input layer and the q-th neuron in the first fully connected layer; kq σ represents the connection weight between the q-th neuron in the first fully connected layer and the k-th neuron in the second fully connected layer; σ is the ReLU activation function;

[0031] Preferably, in the bottom-up path described in step four, the first to third convolutional blocks have the same structure, each including sequentially connected depthwise separable convolutions, batch normalization, and dropout layers, wherein the depthwise separable convolutions include sequentially connected depthwise convolutions and pointwise convolutions; specifically:

[0032] First, the EEG feature sample x eeg The feature map F is fed into the depthwise convolution of the i-th convolutional block. depthwisei ;

[0033] F depthwisei =DepthwiseConv2D(F i-1 Kernel di )

[0034] Where DepthwiseConv2D represents the depthwise convolution operation, Kernel di F represents the kernel of the depthwise convolution of the i-th convolutional block. i-1 F0 represents the output feature map of the (i-1)th convolutional block. eeg_ab x eeg_ab This represents the data information of the a-th channel and the b-th feature of the EEG feature; i = 1, 2, or 3;

[0035] Next, feature map F depthwisei It is fed into the pointwise convolution in the current convolutional block to obtain the feature map F. i :

[0036]

[0037] Where Conv2D represents pointwise convolution operation, Kernel pi The kernel represents the pointwise convolution of the i-th convolutional block, N(H) represents the number of EEG signal channels, N(W) represents the number of frequency band features, and bias... pi F represents the bias term used in the pointwise convolution of the i-th convolutional block, σ represents the ReLU activation function;depthwisei_a b represents the feature map F depthwisei Characteristic values ​​of the b band in channel a;

[0038] The feature map F generated by pointwise convolution is processed by batch normalization. i Optimize the model to speed up its convergence, and then use dropout layers to reduce overfitting.

[0039] In the top-down path, the first multi-scale fusion layer includes upsampling;

[0040] The second and third multi-scale fusion layers have the same structure, including upsampling and fusion modules; specifically, the received output of the previous multi-scale fusion layer is upsampled and then added element-wise with the horizontally connected output of the same layer.

[0041] Preferably, in step six, the weighted fusion module is specifically represented as follows:

[0042] F fusion =w1F eeg +w2F eog

[0043] Where w1 is the weight of the learned EEG signal and w2 is the weight of the learned electrooculogram (EOG) feature.

[0044] Preferably, the loss function L of the pseudo-twin network is:

[0045] L=μ1L ce_eeg +μ2L ce_eog +αL cl +L imp_ce

[0046] L cl =YD 2 +(1-Y)max(margin-D, 0) 2

[0047]

[0048]

[0049]

[0050] Where μ1, μ2, and α are respectively L ce_eeg L ce_eog L cl The coefficient of L; cl Indicates the contrast loss between EEG and EEG features; L ce-eeg L represents the cross-entropy loss of the EEG processing module; ce-eog L represents the cross-entropy loss of the EOG processing module;imp_ce D represents the improved cross-entropy loss; D represents the modal feature F. eeg and F eog Euclidean distance between them;

[0051] Margin represents the feature distance parameter that balances between samples of the same category and samples of different categories; n represents the number of samples; c represents the category of sentiment. These represent the predicted probabilities that sample i belongs to class r, sample j belongs to class t, and sample h belongs to class v, respectively; i,r ,y j,t ,y h,v Let represent the true probabilities that sample i belongs to category r, sample j belongs to category t, and sample h belongs to category v, respectively; Y represents the pair label of the sample pair.

[0052] Secondly, the present invention provides a multimodal sentiment analysis system, comprising:

[0053] The data acquisition module acquires multimodal physiological signals;

[0054] The data preprocessing module preprocesses the collected multimodal physiological signals and extracts features to obtain EEG and electrooculography features.

[0055] The data analysis module inputs EEG and EEG features into a trained, validated, and tested pseudo-twin network to achieve multimodal sentiment analysis.

[0056] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described thereon.

[0057] Fourthly, the present invention provides a computing device, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method described above.

[0058] The beneficial effects of this invention are as follows:

[0059] 1. This invention integrates a depthwise separable convolution strategy into a pyramid network to process complex EEG signals. This not only reduces parameters and computational complexity and endows it with powerful temporal and spatial feature extraction capabilities, but also integrates multi-scale feature information, providing a deeper abstraction of emotion-related features for EEG signals and significantly improving the accuracy of emotion classification.

[0060] 2. This invention uses a flexible pseudo-twin network with a dual-branch subnet structure, which can fully explore the consistency and complementarity of cross-modal features and emotional interaction information, learn more class-discriminative information, and improve the generalization performance of the method under various data modalities.

[0061] The multimodal sentiment analysis method proposed in this invention can extract and integrate intrinsic emotional features at different levels and scales from EEG and extrinsic emotional features from EEG. It can also integrate a pseudo-twin network with two branch subnetworks, allowing them to synergistically utilize EEG and EEG emotional features, achieving consistency and complementarity in multimodal sentiment analysis. Ultimately, this improves the accuracy of multimodal sentiment analysis. Compared to other multimodal methods, the pseudo-twin network-based multimodal sentiment analysis method provided in this invention contributes to improved sentiment analysis accuracy, achieving 96.02% and 96.45% accuracy in valence and arousal dimensions respectively on the DEAP dataset, and 77.81% accuracy on the SEED-IV dataset. Attached Figure Description

[0062] Figure 1 The overall processing flowchart of a multimodal sentiment analysis method based on pseudo-twin networks provided by this invention is shown.

[0063] Figure 2 The location of the EEG channel provided by this invention.

[0064] Figure 3-1 This is a schematic diagram of the EEG processing module provided by the present invention.

[0065] Figure 3-2 This is a schematic diagram of the convolutional blocks in the bottom-up path of the EEG processing module provided by the present invention. Detailed Implementation

[0066] The technical solution of the present invention will be further clearly described below with reference to the accompanying drawings in the embodiments of the present invention.

[0067] Example 1

[0068] The goal of this method is to fully leverage the advantages of EEG and EEG to explore and utilize intramodal and intermodal feature information to achieve higher accuracy in sentiment analysis. EEG and EEG data from 32 participants were used to assess the valence and arousal dimensions of the emotion. Figure 1 The overall flowchart of this method is shown below. Following the above overall flowchart, the specific steps of a multimodal sentiment analysis method based on pseudo-Twin networks are as follows:

[0069] Step 1: Data Collection

[0070] Physiological signals induced by watching 40 one-minute music video clips with emotional labels were collected from 32 participants. These signals included 32 channels of electroencephalography (EEG) and 8 channels of peripheral physiological signals, including 2 channels of electrooculography (EOG), forming the DEAP dataset. Participants were also asked to rate their emotional state within 15 seconds after watching the videos, including two dimensions: valence and arousal, with scores ranging from 1 to 9 for each dimension. Figure 2 This shows the location of the 32-lead EEG channels acquired in this invention. EEG signals were acquired using a 32-lead electrode cap conforming to the 10-20 international lead standard. Electrodes were placed around the eyes to record eye movements using an electrophysiological technique with a sampling frequency of 512 Hz.

[0071] Step 2: Data Preprocessing and Feature Extraction

[0072] Step (2-1) For EEG signals, a Fast Fourier Transform (FFT) is first used to convert the EEG signal from the time domain to the frequency domain, extracting differential entropy features and then filtering the data within a frequency range of 4.0 Hz to 45.0 Hz. Differential entropy (DE) features are extracted from 32 electrodes from the θ (4-7 Hz), α (8-13 Hz), β (14-30 Hz), and γ (31-45 Hz) frequency bands. DE features are a generalized form of Shannon entropy over continuous variables, used to describe the uncertainty or information content of a probability distribution. Typically, EEG signals over a period of time are considered to approximate a Gaussian distribution. Therefore, for EEG signals with a probability density function, DE can be approximated as the logarithm of the power spectrum within a certain frequency band. The duration of the EEG signal is 63 seconds, including 60 seconds of test signal plus 3 seconds of baseline. The formula for calculating DE from EEG is as shown in equation (1):

[0073]

[0074] Where p(x) is the probability density function that follows a Gaussian distribution. a and b are the value ranges of the continuous EEG signal. A 3-second non-overlapping time window was used to segment the collected EEG signal. Finally, preprocessed features with a dimension of 32×4 were obtained.

[0075] Step (2-2) For the electrooculogram (EOG) signal, combine and calculate the mean, standard deviation, mean and standard deviation of the first and second differences, normalized length density (NLD) features and Hjorth features, as shown in Equation (2-9), to improve more comprehensive and richer EOG information features.

[0076] Mean:

[0077] Standard deviation:

[0078] Mean of the first-order difference absolute value sequence:

[0079] Standard deviation of the first-order difference absolute value sequence:

[0080] Mean of a second-order difference absolute value sequence: Standard deviation of the second-order difference absolute value sequence:

[0081] NLD characteristics:

[0082] Hjorth characteristic: activity = σ 2

[0083] mobility=σ(s′(n)) / σ(s(n))

[0084] complexity=mobility(s′(n)) / mobility(s(n)) (9)

[0085] Where N represents N time points, s(n) represents the signal value of time series s at time n, and y(n) represents the signal value of time series s at time n after standardization. s'(n) is the first derivative of s(n). After preprocessing, a feature dimension of 2×10 is obtained.

[0086] Step 3: Generation of multimodal sample pairs.

[0087] EEG and EEG signals from the same subject at the same time segment were recorded as positive samples, while EEG and EEG signals from the same subject with different emotions were recorded as negative sample pairs. The differential entropy feature of a single time segment was used as an input sample, with dimensions of time × channel × frequency band. Each EEG signal sample x was iterated over. eeg Electroocular signal samples x were selected from the same subject at the same time. eog(m) , for multimodal positive sample pairs {x eeg x eog(m)}, and label Y as 1; randomly select electrooculography (EOG) signal samples x of different emotions from the same subject. eog(n) As a multimodal negative sample pair {x eeg x eog(n)}, and the label Y is denoted as 0. For ease of description, multimodal samples are used for {x}. eeg x eog Let} represent the value of x. eeg and x eog It may belong to the same type of emotion or a different type of emotion. Let label Y represent x. eeg and xeog Whether two modal samples belong to the same emotion is determined by a value of 1 if they belong to the same emotion and 0 if they belong to different emotions.

[0088] Step 4: Define the EEG processing module. Integrate depthwise separable convolutions into the pyramid network as the EEG processing module, serving as subnet 1 of the pseudo-twin network. Figure 3-1 The diagram illustrates the process of the EEG processing module extracting EEG features. This process begins with step three, x. eeg As input, temporal and spatial features of EEG are extracted through depthwise separable convolution. Pyramid network extracts and fuses spatiotemporal features at different levels of EEG, thereby obtaining the multi-scale comprehensive spatiotemporal intrinsic modal features F of EEG. eeg .

[0089] The EEG processing module mainly includes three paths: a bottom-up path, a top-down path, and lateral connections. Depthwise separable convolution consists of depthwise convolution and pointwise convolution.

[0090] The bottom-up path consists of three convolutional blocks, such as Figure 3-2 Each convolutional block consists of depthwise convolutions, pointwise convolutions, batch normalization, and dropout layers; specifically:

[0091] First, the EEG feature sample x eeg The feature map F is fed into the depthwise convolution of the i-th convolutional block. depthwisei ;

[0092] F depthwisei =DepthwiseConv2D(F i-1 Kernel di (10)

[0093] Where DepthwiseConv2D represents the depthwise convolution operation, Kernel di F represents the kernel of the depthwise convolution of the i-th convolutional block. i-1 F0 represents the output feature map of the (i-1)th convolutional block. eeg_ab x eeg_ab This represents the data information of the a-th channel and the b-th feature of the EEG feature; i = 1, 2, or 3;

[0094] Next, feature map F depthwisei It is fed into the pointwise convolution in the current convolutional block to obtain the feature map F. i :

[0095]

[0096] Where Conv2D represents pointwise convolution operation, Kernel piThis represents the convolution kernel for pointwise convolution, N(H) represents the number of EEG signal channels, N(W) represents the number of frequency band features, and bias... pi F represents the bias term used in pointwise convolution, σ represents the ReLU activation function; depthwisei_ab This represents the characteristic value of channel a in frequency band b;

[0097] The feature map F generated by pointwise convolution is processed by batch normalization. i Optimize the model to speed up its convergence, and then use dropout layers to reduce overfitting, for example, by using a dropout rate of 0.2 during the learning phase.

[0098] In the top-down path, upsampling operations using bilinear interpolation are employed to adjust feature maps of different sizes output from the third convolutional block in the bottom-up path, aligning them with the lower-level feature maps. Typically, the spatial resolution of the upsampling is set to twice the original resolution. That is, based on the original feature map pixels, new elements are inserted between pixels using an appropriate interpolation algorithm, thereby expanding the size of the original feature map.

[0099] The lateral connections play a crucial role, fusing the upsampling results with the bottom-up generated features to sequentially acquire more global and multi-scale features {DF3, DF2, DF1}. The lateral connections adjust the channel sizes of the original high-level features {F1, F2, F3} using 1×1 convolutional kernels. Subsequently, the adjusted feature maps are combined with the upsampled feature maps using element-wise addition to generate features {DF3, DF2, DF1} with multi-scale information. The feature DF1, containing the richest information, is selected as the final feature map.

[0100] The feature map DF1 is first flattened into a one-dimensional vector, which serves as the input data for the input layer. Then, it passes through two fully connected layers with ReLU activation function to extract the 16-dimensional electrooculography modal features F. eeg As shown in equation (12):

[0101]

[0102] Where N(s) represents the number of neurons in layer s, where s = {1, 2, 3}, corresponding to the flattened layer and fully connected layers with 32 and 16 neurons respectively; p represents the p-th neuron in the input layer, and DF1 is used as input, which can be divided into the feature value DF input by the p-th neuron in the input layer. 1_p F eeg Indicates output; w qp w represents the connection weight between the p-th neuron in the input layer and the q-th neuron in the first fully connected layer; kqσ represents the connection weight between the q-th neuron in the first fully connected layer and the k-th neuron in the second fully connected layer; σ is the ReLU activation function;

[0103] Step 5: Define the EOG processing module; introduce two fully connected layers as the EOG processing module, serving as branch subnet 2 of the pseudo-twin network. This module uses x eog As input, the relatively simple EOG signal is processed through a fully connected layer to obtain the external behavioral modal features F of the EOG. eog .

[0104] x eog As input, the vector is first flattened into a 1D vector; then it passes through an EOG processing module consisting of two fully connected layers with dimensions of 128 and 16 respectively, outputting an EOG modal feature F with a dimension of 16. eog As shown in equation (13):

[0105]

[0106] Among them, the parameters are similar to those in formula (12), w mp w represents the weights between the p-th neuron in the input layer and the m-th neuron in the first fully connected layer. km x represents the weights between the m-th neuron in the second fully connected layer and the k-th neuron in the third fully connected layer. eog_p This represents the input data value corresponding to the p-th neuron in the input layer.

[0107] Step 6: Define the pseudo-twin network. The pseudo-twin network includes two branch subnets and a weighted fusion module. The two branch subnets are the EEG processing module and the EOG processing module, respectively.

[0108] Step (6-1): Calculate F eeg and F eog The Euclidean distance D between two modal features is used to learn contrastive loss to make modal features with the same emotion closer together and modal features with different emotions further apart, thereby learning the correlation between modalities and achieving consistency and complementarity of modal emotional features.

[0109] It is known that steps 4 and 5 have already obtained two modal features F. eeg and F eog Use equation (14) to calculate the Euclidean distance:

[0110] D = ||F eeg -F eog ||2 (14)

[0111] Step (6-2): For feature F eeg and F eogPerform automatic learning and weighted fusion to obtain a better fused feature F that combines information from various features. fusion The fused feature F is output after passing through a fully connected layer and a softmax layer. fusion The probability of belonging to a certain emotion This improves the accuracy of sentiment analysis. Calculate using formula (15):

[0112] F fusion =w1F eeg +w2F eog (15)

[0113] Where w1 is the weight of the learned EEG signal and w2 is the weight of the learned EEG signal.

[0114] Step 7: Define the model input, output, and loss function. The model input consists of two parts: one part is the multimodal sample pairs x. eeg and x eog The input to the pseudo-Twin network consists of two parts: one part is the input labels Y of the currently input multimodal sample pairs; the other part is the input labels Y of the multimodal sample pairs. The output of the model consists of three parts, the first part being two intermediate features F. eeg and F eog Predicted probability and The second part is the Euclidean distance D between the two intermediate features; the third part is the fused feature F. fusion Probability of belonging to each emotion category

[0115] The model's loss function consists of three parts. The first part is the cross-entropy loss L generated by the EEG and EOG processing modules respectively. ce-eeg and L ce-eog The second part is the contrastive loss L for pseudo-twin networks. cl The third part is the improved cross-entropy loss L. imp_ce .

[0116] Contrastive loss is used to optimize the learning process of two modalities, so that the distance between two modalities of the same emotion is close, and the distance between two modalities of different emotions is higher than a given threshold margin, as shown in Equation (16):

[0117] L cl =YD 2 +(1-Y)max(margin-D,0) 2 (16)

[0118] The cross-entropy loss of the electrooculogram (EOG) and electroencephalogram (EEG) modules is used to measure the modality sample feature x. eeg and x eog

[0119] The error between the predicted sentiment label and the true label is shown in equation (17-18):

[0120]

[0121]

[0122] The improved cross-entropy loss works by calculating the error between the model's prediction and the true label when the input mode pair is positive. However, when the input mode pair is negative, it means that the information fusion of the two modes is meaningless, so the cross-entropy loss is not calculated, as shown in equation (19).

[0123]

[0124] In equation (16-19), L cl Indicates the contrast loss between EEG and EEG features; L ce-eeg L represents the cross-entropy loss of the EEG processing module; ce-eog L represents the cross-entropy loss of the EOG processing module; imp_ce D represents the improved cross-entropy loss; D represents the modal feature F. eeg and F eog The Euclidean distance between samples; margin represents the feature distance parameter that balances the distance between samples of the same and different classes; n represents the number of samples; c represents the sentiment category; These represent the predicted probabilities that sample i belongs to class r, sample j belongs to class t, and sample h belongs to class v, respectively; i,r ,y j ,t,y h,v Let represent the true probabilities that sample i belongs to category r, sample j belongs to category t, and sample h belongs to category v, respectively; Y represents the pair label of the sample pair.

[0125] The final loss function of the model is shown in equation (20):

[0126] L=μ1L ce_eeg +μ2L ce_eog +αL cl +L imp_ce (20)

[0127] Where μ1 and μ2 are the cross-entropy loss coefficients of the adjustable EEG and EOG network modules, and α is the contrast loss coefficient of the adjustable EEG and EOG features. The larger α is, the greater the contrast loss.

[0128] Step 8: Training, Testing, and Evaluation.

[0129] First, the dataset was divided into two categories based on its dimensions, with the dimension labels set to 5, namely low / high valence and low / high arousal. Second, 5-fold cross-validation was performed on the data of 32 participants in the dataset, meaning 32 experiments were conducted. Finally, the mean accuracy was used as the evaluation criterion for all participants in the dataset. Accuracy measures the proportion of samples where the predicted value matches the true value. It is a commonly used metric for evaluating classification performance. The higher the accuracy, the more precise the classification result. Its accuracy is represented by the following equation (21):

[0130]

[0131] Where TP represents the case where the true label is positive and the prediction is positive, FN represents the case where the true label is positive but the prediction is negative, FP represents the case where the true label is negative but the prediction is positive, and TN represents the case where the true label is negative and the prediction is negative.

[0132] Example 2

[0133] The 62 EEG channels, eye movement signals, and emotional state scores of 15 subjects in the DEAP dataset of Example 1 were evaluated according to discrete emotions to form the SEED-IV dataset; Step (2) extracts EEG features and extracts eye blink, pupil diameter, annotation and saccade information as electrooculogram features through power spectrum features; The dataset in Step (8) is trained and tested with an in-subject 9-after-6 experimental setting; Other steps are the same as in Example 1.

[0134] To demonstrate the effectiveness of this invention, it was compared with other methods. Tables 1 and 2 compare the average accuracy of the methods in Examples 1-2 of this invention with other existing methods on the DEAP and SEED-IV datasets, respectively. This invention was compared not only with single-modal EEG but also with other multimodal emotion analysis (EEG + all peripheral physiological signals: EEG + PERI; EEG + EOG). The experimental results confirm the advantages of this invention in multimodal emotion analysis.

[0135] Table 1. Comparison of the average accuracy of the method of Embodiment 1 of the present invention with other existing methods.

[0136]

[0137] Table 2 Comparison of the average accuracy of the method in Example 2 of the present invention with other existing methods

[0138] SVM EEG 56.61 DBN EEG 66.77 DGCNN EEG 69.88 GCN EEG 72.23 BiHDM EEG 74.35 BiLSTM EEG 76.27 STGATE EEG 76.43 DLF-SUM EEG+EOG 69.97 EmitionMeter EEG+EOG 70.58 MMDA-VAE EEG+EOG 73.82 This invention EEG+EOG 77.81

Claims

1. A multimodal sentiment analysis method based on pseudo-twin networks, characterized in that... The method includes the following steps: Step 1: Data Collection; Multimodal physiological signals induced in subjects while watching video clips were collected, and the physiological signals were labeled with emotional tags; wherein the multimodal physiological signals include electroencephalogram (EEG) signals and electrooculogram (EOG) signals; Step 2: Preprocess the collected multimodal physiological signals and extract features to obtain EEG and EEG features; Step 3: Generation of multimodal sample pairs; Use multimodal samples to pair {x eeg x eog Let} represent the value of x. eog =x eog(m) or x eog(n) For the label Y = 1 or 0, 1 represents x eeg and x eog The two modal samples belong to the same sentiment, and 0 represents x. eeg and x eog The two modal samples do not belong to the same sentiment; where x eog(m) x represents a sample of electrooculography (EOG) characteristics of the same subject at the same time. eog(n) x represents electrooculography (EOG) feature samples representing different emotions of the same subject. eeg This represents a sample of electroencephalogram (EEG) features. Step 4: Construct the EEG processing module: The EEG processing module employs a pyramid network integrating depthwise separable convolutions to process EEG features x in multimodal sample pairs. eeg Extracting multi-scale integrated spatiotemporal intrinsic modal features F eeg ; Step 5: Construct the EOG processing module; The EOG processing module includes two cascaded fully connected layers for processing electrooculography (EOG) features x in multimodal sample pairs. eog Extracting external behavioral modal features F from electrooculogram signals eog ; Step 6: Construct a pseudo-twin network, and train, validate, and test it; The pseudo-twin network includes two branch subnets and a weighted fusion module, wherein the two branch subnets are an EEG processing module and an EOG processing module, respectively. The weighted fusion module processes the two modal features F output by the EEG processing module and the EOG processing module. eeg and F eog Perform automatic learning and weighted fusion to obtain better combinations of F eeg and F eog The fusion characteristics; Step 7: Implement multimodal sentiment analysis using a well-trained, validated, and tested pseudo-twin network.

2. The method according to claim 1, characterized in that... In step two, the EEG features are derived from the differential entropy features of the EEG signals.

3. The method according to claim 1 or 2, characterized in that... In step two, the electrooculography (EOG) features are obtained by splicing the mean, standard deviation, mean and standard deviation of the first and second order differences of the EOG signal, normalized length density features, and Hjorth features.

4. The method according to claim 1, characterized in that... In step four, the EEG processing module includes three paths and an output layer, namely a bottom-up path, a top-down path, and a horizontal connection. The bottom-up path includes a first to a third convolutional block connected in series; the first convolutional block accepts EEG features x. eeg After processing, the result is output to the second convolutional block, and simultaneously input to the top-down path via lateral connections; the second convolutional block receives the output result of the first convolutional block, processes it, and outputs the result to the third convolutional block, and simultaneously input to the top-down path via lateral connections; the third convolutional block receives the output result of the second convolutional block, and input to the top-down path via lateral connections. The top-down path uses upsampling operations to adjust the feature maps of different sizes with respect to the output of the third convolutional block in the bottom-up path; The lateral connection includes a 1×1 convolution used to adjust the channel size of the feature maps of different scales output by the bottom-up path; The output layer flattens the information-rich feature DF1 output from the third multi-scale fusion layer into a one-dimensional vector through a flatten layer; then, it passes through two fully connected layers with ReLU activation function to extract the EEG modality feature F. eeg .

5. The method according to claim 4, characterized in that... In step four, the bottom-up path describes the same structure for the first to third convolutional blocks, each consisting of sequentially connected depthwise separable convolutions, batch normalization, and dropout layers. The depthwise separable convolutions include sequential depthwise convolutions and pointwise convolutions. Specifically: First, the EEG feature sample x eeg The feature map F is fed into the depthwise convolution of the i-th convolutional block. depthwisei ; F depthwisei =DepthwiseConv2D(F i-1 ,Kernel di ) (1) Where DepthwiseConv2D represents the depthwise convolution operation, Kernel di F represents the kernel of the depthwise convolution of the i-th convolutional block. i-1 F0 represents the output feature map of the (i-1)th convolutional block. eeg_ab x eeg_ab This represents the data information of the a-th channel and the b-th feature of the EEG feature; i = 1, 2, or 3; Next, feature map F depthwisei It is fed into the pointwise convolution in the current convolutional block to obtain the feature map F. i : Where Conv2D represents pointwise convolution operation, Kernel pi Let N(H) represent the convolution kernel of the pointwise convolution of the i-th convolutional block, N(W) represent the number of EEG signal channels, N(W) represent the number of frequency band features, and bias represent the bias. pi F represents the bias term used in the pointwise convolution of the i-th convolutional block, σ represents the ReLU activation function; depthwisei_ab Representing feature map F depthwisei Characteristic values ​​of channel a in band b; The feature map F generated by pointwise convolution is processed by batch normalization. i Optimize the model to speed up its convergence, and then use dropout layers to reduce overfitting. The top-down path includes a first to a third multi-scale fusion layer connected in series, wherein the first multi-scale fusion layer includes upsampling; The second and third multi-scale fusion layers have the same structure, including upsampling and fusion modules; Specifically, the received output of the previous multi-scale fusion layer is upsampled and then element-wise added to the horizontally connected output of the same layer.

6. The method according to claim 4, characterized in that... In step six, the weighted fusion module is specifically represented as follows: F fusion =w1F eeg +w2F eog (3) Where w1 is the weight of the learned EEG signal and w2 is the weight of the learned electrooculogram (EOG) feature.

7. The method according to claim 1, characterized in that... The loss function L of the pseudo-twin network is specifically: L=μ1L ce_eeg +μ2L ce_eog +αL cl +L imp_ce (4) L cl =YD 2 +(1-Y)max(margin-D,0) 2 (5) Where μ1, μ2, and α are respectively L ce_eeg L ce_eog L cl The coefficient of L; cl Represents modal features F eeg and F eog Contrast loss; L ce-eeg L represents the cross-entropy loss of the EEG processing module; ce-eog L represents the cross-entropy loss of the EOG processing module; imp_ce D represents the improved cross-entropy loss; D represents the modal feature F. eeg and F eog The Euclidean distance between samples; margin represents the feature distance parameter that balances the distance between samples of the same and different classes; n represents the number of samples; c represents the sentiment category; These represent the predicted probabilities that sample i belongs to class r, sample j belongs to class t, and sample h belongs to class v, respectively; i，r y j，t y h，v Let represent the true probabilities that sample i belongs to category r, sample j belongs to category t, and sample h belongs to category v, respectively; Y represents the pair label of the sample pair.

8. A multimodal sentiment analysis system implementing the method of any one of claims 1-7, characterized in that... include: The data acquisition module acquires multimodal physiological signals; The data preprocessing module preprocesses the collected multimodal physiological signals and extracts features to obtain EEG and electrooculography features. The data analysis module inputs EEG and EEG features into a trained, validated, and tested pseudo-twin network to achieve multimodal sentiment analysis.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-7.

10. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-7.

Images