Sentiment analysis method and apparatus, machine readable storage medium, and electronic device

Abstract

The application discloses an emotion analysis method and device, a machine readable storage medium and an electronic device. The emotion analysis method comprises the following steps: determining electroencephalogram fusion features based on electroencephalogram signals of multiple channels, wherein the determined electroencephalogram fusion features fuse time domain information, frequency domain information and spatial information of the electroencephalogram signals; determining text semantic features based on a read text; and determining emotion classification based on the determined electroencephalogram fusion features and the determined text semantic features to perform emotion analysis. In this way, the accuracy of emotion analysis is improved.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction, and more specifically, to an emotion analysis method and apparatus, a machine-readable storage medium, and an electronic device. Background Technology

[0002] Sentiment analysis has become an important research direction in the field of human-computer interaction, playing a crucial role in various application scenarios such as mental health monitoring and affective computing. Traditional methods typically rely on single-modal data sources, such as facial expressions, speech signals, or physiological signals, to infer an individual's emotional state. However, sentiment analysis based on single-modal data suffers from insufficient accuracy.

[0003] Therefore, improving the accuracy of sentiment analysis has become a technical problem that needs to be solved in this field. Summary of the Invention

[0004] In view of this, this application proposes a sentiment analysis method and apparatus, a machine-readable storage medium and an electronic device to improve the accuracy of sentiment analysis.

[0005] In a first aspect, this application provides a sentiment analysis method, which includes: determining EEG fusion features based on EEG signals from multiple channels, wherein the determined EEG fusion features fuse temporal, frequency, and spatial information of the EEG signals; determining text semantic features based on the text being read; and determining sentiment classification based on the determined EEG fusion features and the determined text semantic features, so as to perform sentiment analysis.

[0006] Optionally, determining brainwave fusion features based on EEG signals from multiple channels includes: constructing a brain network map of the multiple channels based on the EEG signals from multiple channels; determining the weight of each node in the constructed brain network map; determining the frequency band energy for each of the multiple channels based on the corresponding EEG signal; and determining the weighted frequency band energy for each determined frequency band energy based on the weight of the corresponding node in the constructed brain network map, so as to determine the brainwave fusion features.

[0007] Optionally, constructing a brain network map of the multiple channels based on EEG signals from multiple channels includes: constructing a brain network map of the multiple channels based on EEG signals from multiple channels by calculating the DCCA cross-correlation coefficient.

[0008] Optionally, determining the weight of each node in the constructed brain network graph includes: determining the weight of each node in the constructed brain network graph based on the PageRank algorithm.

[0009] Optionally, based on the determined EEG fusion features and the determined text semantic features, an emotion classification is determined, including: fusing the determined EEG fusion features and the determined text semantic features to obtain EEG-text fusion features; and determining the emotion classification based on the obtained EEG-text fusion features.

[0010] Optionally, the determined EEG fusion features and the determined text semantic features are fused to obtain EEG-text fusion features, including: fusing the determined EEG fusion features and the determined text semantic features based on a bidirectional multi-head cross-attention mechanism to obtain EEG-text fusion features.

[0011] Optionally, based on the obtained EEG-text fusion features, the emotion classification is determined, including: based on the obtained EEG-text fusion features, combined with a BiLSTM model with an attention mechanism, the emotion classification is determined.

[0012] Secondly, this application also provides an emotion analysis device, which includes: an EEG fusion feature determination module, used to determine EEG fusion features based on EEG signals from multiple channels, wherein the determined EEG fusion features fuse temporal, frequency, and spatial information of the EEG signals; a text semantic feature determination module, used to determine text semantic features based on the text being read; and an emotion classification determination module, used to determine an emotion classification based on the determined EEG fusion features and the determined text semantic features, for emotion analysis.

[0013] Optionally, determining brainwave fusion features based on EEG signals from multiple channels includes: constructing a brain network map of the multiple channels based on the EEG signals from multiple channels; determining the weight of each node in the constructed brain network map; determining the frequency band energy for each of the multiple channels based on the corresponding EEG signal; and determining the weighted frequency band energy for each determined frequency band energy based on the weight of the corresponding node in the constructed brain network map, so as to determine the brainwave fusion features.

[0014] Optionally, constructing a brain network map of the multiple channels based on EEG signals from multiple channels includes: constructing a brain network map of the multiple channels based on EEG signals from multiple channels by calculating the DCCA cross-correlation coefficient.

[0015] Optionally, determining the weight of each node in the constructed brain network graph includes: determining the weight of each node in the constructed brain network graph based on the PageRank algorithm.

[0016] Optionally, based on the determined EEG fusion features and the determined text semantic features, an emotion classification is determined, including: fusing the determined EEG fusion features and the determined text semantic features to obtain EEG-text fusion features; and determining the emotion classification based on the obtained EEG-text fusion features.

[0017] Optionally, the determined EEG fusion features and the determined text semantic features are fused to obtain EEG-text fusion features, including: fusing the determined EEG fusion features and the determined text semantic features based on a bidirectional multi-head cross-attention mechanism to obtain EEG-text fusion features.

[0018] Optionally, based on the obtained EEG-text fusion features, the emotion classification is determined, including: based on the obtained EEG-text fusion features, combined with a BiLSTM model with an attention mechanism, the emotion classification is determined.

[0019] Thirdly, this application also provides a machine-readable storage medium storing instructions that cause a machine to perform the aforementioned sentiment analysis method.

[0020] Fourthly, this application also provides an electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the executable instructions to implement the aforementioned sentiment analysis method.

[0021] According to the technical solution of this application, EEG fusion features are determined based on EEG signals from multiple channels, textual semantic features are determined based on the text being read, and emotion classification is determined based on the determined EEG fusion features and textual semantic features for emotion analysis. Thus, emotion analysis is achieved. Furthermore, in performing emotion analysis, combining EEG signals and text integrates the objective physiological responses of EEG signals with the subjective semantic expression of the text, achieving more comprehensive and robust emotion state recognition and improving the accuracy of emotion analysis. Moreover, emotion analysis through the combination of EEG signals and text can also distinguish complex emotional states, enhancing the ability to differentiate between complex emotional states.

[0022] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0023] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application, and the illustrative embodiments and descriptions thereof are used to explain this application. In the drawings: Figure 1 A flowchart of a preferred embodiment of the sentiment analysis method according to this application; Figure 2a This is a brain network diagram of a positive emotional state without threshold screening according to a preferred embodiment of this application; Figure 2b This is a binary brain network diagram of a positive emotional state after threshold filtering according to a preferred embodiment of this application; Figure 2c This is a binary brain network diagram of a positive emotional state according to a preferred embodiment of this application, after threshold filtering and calculation of node weights; Figure 3a This is a brain network diagram of a neutral emotional state without threshold screening according to a preferred embodiment of this application; Figure 3b This is a binary brain network diagram of a neutral emotional state after threshold filtering according to a preferred embodiment of this application; Figure 3c This is a binary brain network diagram of a neutral emotional state after threshold filtering and node weight calculation according to a preferred embodiment of this application. Figure 4a This is a brain network diagram of a negative emotional state without threshold screening according to a preferred embodiment of this application; Figure 4b This is a binary brain network diagram of a negative emotional state after threshold filtering according to a preferred embodiment of this application; Figure 4c This is a binary brain network diagram of a negative emotional state after threshold filtering and node weight calculation according to a preferred embodiment of this application. Figure 5a The training set loss function iteration curve according to the preferred embodiment of this application; Figure 5b The training set accuracy iteration curve according to the preferred embodiment of this application; Figure 6a The verification set loss function iteration curve according to the preferred embodiment of this application; Figure 6b The verification set accuracy iteration curve according to the preferred embodiment of this application; Figure 7 The confusion matrix results of the multimodal sentiment analysis model according to the preferred embodiment of this application on the test set; Figure 8a The confusion matrix result of the BiLSTM model according to the preferred embodiment of this application on the test set; Figure 8b The confusion matrix result of the LSTM model according to the preferred embodiment of this application on the test set; Figure 8c The confusion matrix result of the GRU model according to the preferred embodiment of this application on the test set; Figure 8d The confusion matrix result of the CNN model according to the preferred embodiment of this application on the test set; Figure 8e The confusion matrix result of the BiGRU model according to the preferred embodiment of this application on the test set; Figure 9 This is a structural block diagram of a sentiment analysis device according to a preferred embodiment of this application. Detailed Implementation

[0024] The technical solution of this application will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] Electroencephalograms (EEGs) are a promising means of emotion recognition due to their ability to directly reflect neural activity and capture an individual's internal emotional state. This application's implementation method performs emotion analysis based on EEG signals and text.

[0026] Firstly, this application provides a sentiment analysis method.

[0027] Figure 1 This is a flowchart of a preferred embodiment of a sentiment analysis method according to this application. Figure 1 As shown, this sentiment analysis method includes the following.

[0028] In step S10, EEG fusion features are determined based on EEG signals from multiple channels. These determined EEG fusion features fuse temporal, frequency, and spatial information from the EEG signals. This application utilizes multiple channels to acquire EEG signals. For example, 16 electrode channels can be arranged.

[0029] In step S11, semantic features of the text are determined based on the text being read. These determined semantic features express the semantic information within the text being read.

[0030] In step S12, based on the determined EEG fusion features and the determined text semantic features, an emotion classification is determined for emotion analysis.

[0031] Optionally, in embodiments of this application, determining brainwave fusion features based on EEG signals from multiple channels may include the following:

[0032] A multi-channel brain network map was constructed based on EEG signals from multiple channels. The constructed brain network map reflects the spatial information of the EEG signals from multiple channels. In the constructed brain network map, each channel corresponds to a node.

[0033] Determine the weight of each node in the constructed brain network graph. For each node, its corresponding weight reflects the degree to which its corresponding channel contributes to the emotional state.

[0034] For each of the multiple channels, the frequency band energy is determined based on the corresponding EEG signal. In this application, the frequency bands for which the frequency band energy needs to be determined can be pre-set. For example, the frequency bands for which the frequency band energy needs to be determined may include the α band, β band, θ band, and γ band, a total of four frequency bands. Alternatively, the α band, β band, θ band, and γ band can be segmented separately, and the specific number of segments depends on the specific situation. For example, each frequency band can be divided into two segments, determining the frequency band energy of a total of eight frequency bands.

[0035] Frequency band energy is a hallmark feature of electroencephalogram (EEG) signals. The energy of different frequency bands in EEG signals varies under different emotional states. The calculation formula is as follows: Where f1 is the starting frequency of the frequency band, f2 is the ending frequency of the frequency band, and X(f) is the frequency domain signal obtained after performing a Fourier transform on the time-domain EEG signal.

[0036] For each determined frequency band energy, a weighted frequency band energy is determined based on the weight of the corresponding node in the constructed brain network graph, thereby determining the EEG fusion feature. The EEG fusion feature includes all weighted frequency band energies for each channel. In this embodiment, the weighted frequency band energy can be the product of the weight and the frequency band energy. For any frequency band of any channel, the frequency band energy corresponding to that frequency band is multiplied by the weight of the node corresponding to that channel to obtain the weighted frequency band energy corresponding to that frequency band.

[0037] Information transmission between different brain regions cannot be obtained solely from frequency domain features. Therefore, this application combines analysis with a constructed brain network map, which can preserve spatial information and long-range dependencies between electrode nodes in EEG signals. To reduce the impact of redundant channel connections in the network on sentiment analysis, an appropriate DCCA cross-correlation coefficient threshold is set based on the actual situation to perform edge filtering on the brain network map. Then, node weights are calculated based on the edge-filtered brain network map, which can reflect the importance of channels. Finally, the frequency band energy of each channel is weighted using the weights of the corresponding nodes to obtain EEG fusion features that integrate spatial, temporal, and frequency domain information.

[0038] Temporal features reflect the instantaneous changes in EEG signals, frequency domain features provide the energy distribution of the signal across different frequency ranges, and spatial domain features characterize the coordinated activity of different brain regions. Fusion of these features provides a more comprehensive description of the characteristics of EEG signals. EEG fusion features reflect both the temporal and spatial information of EEG signals and effectively eliminate the volumetric effect of brain networks. These fusion features are then used as input features to train and predict a classification model. The effectiveness of the proposed weighting method is initially evaluated through classification accuracy. The proposed classification models include machine learning models SVM and Random Forest, as well as deep learning models including 1DCNN, 1DCNN-LSTM, and 1DCNN-GRU. The fusion features also serve as the input feature vectors for the EEG modalities in subsequent multimodal sentiment analysis models.

[0039] Optionally, in embodiments of this application, constructing a multi-channel brain network map based on multiple channels of EEG signals may include the following: Constructing a multi-channel brain network map based on multiple channels of EEG signals by calculating detrended cross-correlation analysis (DCCA) cross-correlation coefficients. Specifically, multiple channels are paired, and for each pair, DCCA cross-correlation coefficients are calculated to construct the brain network map.

[0040] Specifically, a multi-channel brain network map can be constructed based on the following information and combined with the DCCA cross-relation coefficient.

[0041] Multiple channels are paired, with each pair corresponding to an edge in the brain network graph. For each pair, the DCCA cross-correlation coefficient is calculated. This is equivalent to each edge in the desired brain network graph corresponding to a DCCA cross-correlation coefficient. Based on the calculated DCCA cross-correlation coefficients, a matrix is ​​constructed, with values ​​ranging from -1 to 1.

[0042] The matrix is ​​normalized by mapping all values ​​to the range of 0 to 1, resulting in a normalized matrix. Specifically, the absolute value of the calculated DCCA cross-correlation coefficient is taken and normalized.

[0043] A threshold is pre-set for the DCCA cross-correlation coefficient. Edges in the matrix with DCCA cross-correlation coefficients greater than or equal to the set threshold are marked as 1, and edges with DCCA cross-correlation coefficients less than the set threshold are marked as 0, constructing a binary matrix. By setting an appropriate threshold, redundant connections are eliminated, reducing network complexity and the impact of volumetric effects.

[0044] By retaining the edges with a value of 1 in the binary matrix, a brain network graph based on the DCCA cross-correlation coefficient is constructed.

[0045] The DCCA cross-correlation coefficient is calculated using signals after trend removal. It can remove the interference of trends on fluctuation characteristics, reveal the true dynamic relationship of EEG signals, and reflect the long-term correlation between signals.

[0046] Optionally, the DCCA cross-correlation coefficient can be calculated based on the following: Detrended Cross-Correlation Analysis (DCCA) extends the research object to two non-stationary time series, building upon Detrended Fluctuation Analysis (DFA). This method is suitable for analyzing the correlation and dependence of EEG signals in different channels across various frequency bands under different emotional states.

[0047] First, calculate the DFA value of the EEG signal for a single channel. Specifically, the DFA value of the EEG signal for a single channel can be calculated based on the following:

[0048] Let a certain EEG signal sequence be x(t), where x(t) is the EEG signal at time t, and the length of the EEG signal sequence is N, that is, the range of t is t1-t2. N A total of N EEG signal values ​​were collected. First, the differences between the EEG signal sequence and the mean of the entire sequence were accumulated and summed to generate its integral form, resulting in an integral sequence denoted as X(t). The formula for calculating the integral form at time t is as follows: , where x(i) is the EEG signal at time i; , This is the average value across the entire EEG signal sequence. This integration process transforms the original sequence into a random walk process, which is more beneficial for subsequent trend modeling and fluctuation analysis.

[0049] The integral sequence X(t) is divided into several non-overlapping intervals of length n. Within each sub-interval, a polynomial trend curve (trend value X) is fitted to the data using the least squares method. n (t)). Then, by subtracting the theoretical value X given by the regression. n (t), to achieve local detrending of sequence X(t). For a given interval length n, the magnitude of the volatility characteristic of the detrended sequence is calculated as follows: ,in, F represents the number of divisible intervals; DFA (n) represents the magnitude of the fluctuation characteristic of the detrended sequence with an interval length of n. For EEG signals, F DFA (n) can be used to measure the volatility and activity of a signal at a specific scale.

[0050] The above process is repeated at specified intervals. In this application, the interval length n of each sub-interval ranges from 5 to 1000, encompassing the entire frequency band of EEG and being sufficiently long to reflect the long-range characteristics of EEG signals. Based on the sampling rate of the EEG signal f0 = 500Hz, the formula is used... The correspondence between the frequency bands alpha (α), beta (β), theta (θ), and gamma (γ) and the interval length can be calculated as shown in Table 1, where f represents the frequency of the EEG signal.

[0051] Table 1. Correspondence between frequency bands of EEG signals and interval length n In this application, the interval length n can be determined as needed, based on the frequency band of the frequency band energy. Specifically, each of the alpha, beta, theta, and gamma bands can be divided into two segments, resulting in eight frequency bands. For each frequency band, the value of the interval length n is determined according to the content shown in Table 1, resulting in a total of eight possible values.

[0052] After obtaining the DFA features of a single EEG signal sequence, detrended cross-correlation analysis (DCCA) is further introduced to calculate the DCCA cross-correlation coefficient between two EEG signal sequences (to assess the multi-scale correlation between sequences). Specifically, for each pairwise combination of multiple channels, the DCCA cross-correlation coefficient can be calculated based on the following.

[0053] Two EEG signal sequences, each of length N, are labeled x(t) and y(t), where t = t1, t2, ..., tn. N The differences between the two EEG signal sequences and their respective overall mean values ​​are accumulated and summed to generate their integral forms, resulting in integral sequences labeled X(t) and Y(t). , .in, Let x(t) be the mean of the EEG signal sequence. y(t) is the mean of the EEG signal sequence.

[0054] Two integral sequences X(t) and Y(t) are divided into k non-overlapping sub-intervals of length n. Within each sub-interval, a local trend is fitted using the least squares method to obtain the trend value X. n (t), Y n (t). After removing the trend, the average of the products of the detrended residuals is calculated as the cross-correlation volatility for an interval length n. For a given interval length n, the detrended cross-correlation is calculated as follows: , This represents the detrended cross-correlation fluctuation over an interval of length n.

[0055] The above calculation steps are repeated over a specified interval length to calculate the detrended cross-correlation fluctuation for each interval length. In this application, the interval length n ranges from 5 to 1000, which includes all frequency bands of EEG and is long enough to reflect the long-range characteristics of EEG signals.

[0056] because It only reflects the trend of correlation with scale. To further quantify the cross-correlation strength between two EEG signal sequences, the detrended cross-correlation coefficient (DCCA) between the two EEG signal sequences is calculated, i.e., the DCCA cross-correlation coefficient. , .

[0057] Among them, F DFA x(n) represents the detrended cross-correlation fluctuation of the EEG signal sequence x(t) over an interval of length n, F DFA y(n) represents the detrended cross-correlation fluctuation of the EEG signal sequence y(t) with an interval length of n.

[0058] It is a set of coefficients, taking values ​​in the range [-1, 1], representing the change in their relationship with the interval length n. If If , it indicates that the sequences are completely negatively correlated; if If , it indicates perfect cross-correlation; if If , it indicates that there is no significant correlation.

[0059] Optionally, in this embodiment, determining the weight of each node in the constructed brain network graph may include the following: The weight of each node in the constructed brain network graph is determined based on the PageRank algorithm.

[0060] Optionally, the weight of each node in the brain network can be calculated based on the following: The PageRank algorithm was first applied to Google's webpage ranking problem, and it has solved the ranking problem in the webpage search process quite well. In this application, the PageRank algorithm is transferred to the field of brain science, treating each node in the brain network as a webpage, and the connections between nodes as links between webpages. The weight of the nodes is calculated by calculating the importance ranking vector. The specific calculation steps of the channel importance ranking vector include the following aspects, which are illustrated by taking a channel count of 16 as an example.

[0061] For the constructed brain network graph containing 16 nodes, firstly, an adjacency matrix is ​​constructed based on the connection strength between all node pairs. , of which elements This represents the connection strength from node p to node q.

[0062] To model the probability propagation of PageRank, the adjacency matrix needs to be normalized column-wise. Transform into a transition probability matrix , in, .

[0063] Let the importance ranking vector of the 16 nodes be... , where h1 (s) This represents the PageRank value of the l-th channel in the s-th iteration, indicating its importance score. Initially, all nodes are assumed to have equal weights. .

[0064] The iterative formula is Where d is the damping factor. d is the damping factor, typically taken as 0.85, used to control the proportion of random jumps; e represents the uniform distribution vector. ; The weight vector is propagated according to the network structure; R represents...; H (s) H represents the importance ranking vector in the s-th iteration; (s-1) This represents the importance ranking vector for the (s-1)th iteration.

[0065] Iterate until the convergence condition is met: . It depends on the specific circumstances. For example, setting... .

[0066] Finally, we obtain the vector. The values ​​in this vector represent the PageRank weights of the corresponding nodes in the current emotional state, reflecting their relative importance in the brain's functional connectivity network.

[0067] In this application's embodiments, whether a threshold screening is performed and whether node weights are calculated when constructing the brain network map have a significant impact on the emotion capture capabilities of the technical solution provided in this application's embodiments. This can be referred to... Figures 2a to 4c To understand. Figure 2a This is a brain network diagram of a positive emotional state without threshold screening according to a preferred embodiment of this application. The thickness of the edges in the diagram represents the magnitude of the DCCA cross-correlation coefficient between nodes. The thicker the edge, the larger the DCCA cross-correlation coefficient. Figure 2bThe image shows a binary brain network diagram under a positive emotional state after threshold filtering according to a preferred embodiment of this application. It can be seen that after reasonable threshold filtering, weakly correlated node connections are deleted, which greatly reduces the complexity of the brain network diagram while retaining strong node dependencies. Figure 2c This is a binary brain network diagram of a positive emotional state according to a preferred embodiment of this application, after threshold filtering and calculation of node weights. The color of a node represents its weight; the darker the color, the greater the weight and importance of the node. Figures 3a to 3c , Figures 4a to 4c Examples of neutral and negative affective states are provided below. For analysis and explanation of each figure, please refer to [link / reference needed]. Figures 2a to 2c As shown. Comparison Figures 2a to 2c , Figures 3a to 3c , Figures 4a to 4c As can be seen, the brain network diagrams based on DCCA cross-correlation coefficients and the weights of each node differ under different emotional states. This verifies the sensitivity of the DCCA cross-correlation coefficient combined with the PageRank method in capturing the dynamic features of emotion-related brain networks.

[0068] Table 2 compares the accuracy of emotion classification using frequency domain energy before and after weighting. As shown in Table 2, the accuracy of emotion classification using weighted frequency domain energy is higher than that using unweighted frequency domain energy. This demonstrates the effectiveness of the proposed brain network graph-based and weighted EEG feature extraction method in improving the accuracy of emotion classification. It should be noted that the data in Table 2 are based on a brain network graph constructed using DCCA cross-correlation coefficients and node weights calculated using the PageRank method.

[0069] Table 2 Comparison of sentiment classification accuracy before and after weighting Optionally, in embodiments of this application, determining the emotion classification based on the determined EEG fusion features and the determined text semantic features may include the following: fusing the determined EEG fusion features and the determined text semantic features to obtain EEG-text fusion features; and determining the emotion classification based on the obtained EEG-text fusion features.

[0070] Optionally, in this embodiment of the application, fusing the determined EEG fusion features and the determined text semantic features to obtain EEG-text fusion features may include the following: Based on a bidirectional multi-head cross-attention mechanism, fusing the determined EEG fusion features and the determined text semantic features to obtain EEG-text fusion features.

[0071] To achieve deep complementary fusion between EEG fusion features and text semantic features, this application designs a cross-modal bidirectional multi-head cross-attention mechanism module. This module includes cross-attention in two directions: the first direction uses text semantic features as the query and EEG features as the key and value to capture the neural response features of EEG signals guided by language; the second direction reverses this by using EEG fusion features as the query and text semantic features as the key and value to achieve language feature focusing driven by EEG signals.

[0072] The advantage of the bidirectional multi-head cross-attention mechanism is that it allows each modality to dynamically adjust its attention area according to the semantic or neural patterns of another modality during the fusion process, achieving true semantic alignment and information supplementation, and is especially suitable for multimodal semantic misalignment.

[0073] Optionally, in this embodiment of the application, before fusing the determined EEG fusion features and the determined text semantic features, the determined EEG fusion features may be encoded to make the determined EEG fusion features more suitable for inputting into the deep learning model and to improve the expressive power of the two modal signal features.

[0074] Specifically, the identified EEG fusion features can be encoded based on the following:

[0075] To address the characteristic of the input weighted frequency band energy features, which are structured, low-dimensional vector data after being weighted by spatial and temporal information, the EEG fusion features are encoded (i.e., the EEG encoder) using a one-dimensional convolutional neural network (1D CNN) combined with a gated recurrent unit (GRU) network for modeling. It should be noted that, in this embodiment, each frequency band of each channel corresponds to a weighted frequency band energy. The weighted frequency band energies corresponding to all frequency bands of all channels are used to construct a weighted frequency band energy vector as the EEG fusion feature for subsequent processing.

[0076] Specifically, a 1D CNN is first used to extract local features from the weighted frequency band energy vector. 1D CNN can simulate local dependency patterns between frequency bands or channels, which helps in extracting local spatially perceptual features.

[0077] Next, the feature sequences extracted by the 1D CNN are input into the GRU network for time series modeling. Although the original input is static frequency domain features, GRU can effectively simulate the sequential relationship between different features and identify the potential contextual dependencies between frequency combinations and channels.

[0078] Finally, the dimension is adjusted to be the same as the output dimension of the text encoder through linear mapping, so as to be suitable for input feature fusion module for further modality fusion.

[0079] Through a 1D CNN-GRU cascaded architecture, the EEG encoder enhances the network's perception of local feature structures by representing weighted frequency domain energy as structured two-dimensional features and introducing local convolutions. GRU further incorporates global dependency information, enabling EEG representations to possess both local accuracy and global contextual awareness, thus improving the ability to distinguish between different emotional states.

[0080] Optionally, in this embodiment, determining the semantic features of the text based on the text being read may include the following: Specifically, the semantic features of the text are obtained by encoding the text being read. The module that encodes the text being read is defined as a text encoder.

[0081] A text encoder is used to extract sentiment-related deep semantic features from raw sentences. Recognizing that sentiment in natural language is often distributed across multiple levels of semantic information, this application employs a pre-trained RoBERTa model to extract context-dependent word vector representations. A Deep Pyramid Convolutional Neural Network (DPCNN) is then cascaded onto this model to further model the local and long-range dependency structures in short texts. This structure aims to preserve the expressive power of context-dependent language while leveraging deep convolutional structures to enhance local semantic patterns and sentiment discrimination capabilities, overcoming the problem of local information dilution in short text sentiment recognition caused by the RoBERTa model.

[0082] Specifically, the input text sequence is first encoded into a sequence of word embedding vectors and then fed into the RoBERTa model. The RoBERTa model maps these vectors into a context-aware sequence of word vectors. To fully preserve sentence-level semantic features, this application uses the entire word vector sequence as input to the subsequent DPCNN module.

[0083] To further enhance the semantic expressive power of word vectors and model long-distance dependencies, DPCNN then expands the receptive field layer by layer by stacking convolutional modules of equal length and pooling layers with a stride of 2, thereby extracting hierarchical syntactic and deep semantic structural information.

[0084] To enhance information flow and alleviate the gradient vanishing problem, DPCNN introduces residual connections in each convolutional module.

[0085] Downsampling is performed after the residual connection.

[0086] Each time a sample is downsampled, the receptive field doubles, and the sequence length is reduced to half of its original length, forming a pyramid structure.

[0087] Finally, the dimension is adjusted to be the same as the output dimension of the EEG encoder through linear mapping, so as to be suitable for input feature fusion module for further modality fusion.

[0088] Optionally, in embodiments of this application, fusing the determined EEG fusion features and the determined text semantic features based on a bidirectional multi-head cross-attention mechanism may include the following:

[0089] To achieve deep complementary fusion between EEG fusion features and text semantic features, this application designs a cross-modal bidirectional multi-head cross-attention mechanism module. This module includes cross-attention in two directions: the first direction uses text semantic features as the query and EEG features as the key and value to capture the neural response features of EEG signals guided by language; the second direction reverses this by using EEG fusion features as the query and text features as the key and value to achieve language feature focusing driven by EEG signals.

[0090] For direction one, this process uses textual semantic features as the query and EEG fusion features as the key and value. In other words, it uses text to "question" EEG features, searching for EEG information related to the text. The attention mechanism calculates the correlation between each element in the textual semantic features and the EEG fusion features, and then performs a weighted sum of the EEG features based on the calculated correlation, i.e., the attention score. Let the output vector obtained after multi-head attention calculation, concatenation, and linear transformation be... .therefore, It contains information extracted from EEG features that is most relevant to the semantic features of the text, i.e., text-guided EEG feature focusing.

[0091] For direction two, this process uses EEG fusion features as the query and text semantic features as the key and value. This means we use EEG to "question" text features, searching for text information related to EEG. The attention mechanism calculates the correlation between each element in the EEG fusion features and the text semantic features, and then performs a weighted sum of these scores on the text semantic features. Let the output vector obtained after multi-head attention calculation, concatenation, and linear transformation be... .therefore, It contains information extracted from text semantic features that is most relevant to EEG features, highlighting those parts of the text features that are related to EEG signals.

[0092] Finally, these two vectors are concatenated to obtain a fused feature vector. This fused feature vector serves as the input vector for the subsequent classification module, used for sentiment state recognition.

[0093] In addition, to achieve bidirectional attention interaction, the multi-head attention mechanism can perform scaling dot product attention operations in parallel by multiple heads to capture the interaction relationships between different subspaces.

[0094] Optionally, in embodiments of this application, determining the emotion classification based on the obtained EEG-text fusion features may include the following: determining the emotion classification based on the obtained EEG-text fusion features, combined with a BiLSTM model incorporating an attention mechanism.

[0095] Specifically, emotion categories can be determined based on the following:

[0096] The fused bidirectional attention features are concatenated into a unified feature vector and then input into an Attention Bidirectional Long Short-Term Memory (Att-BiLSTM) network with an attention mechanism for sequence modeling and sentiment classification. This module is designed to utilize BiLSTM to model the contextual relationships within the fused feature vector, while simultaneously introducing an attention mechanism to highlight key features that are discriminative for sentiment classification.

[0097] BiLSTM consists of two LSTMs in opposite directions, which can simultaneously capture the forward and backward semantic information of the input sequence.

[0098] To further highlight the time step features that contribute more to the judgment of emotional state, this application introduces an attention mechanism to weight the time series output by BiLSTM.

[0099] Finally, the attention-weighted feature vector is input into a fully connected layer and a softmax classifier to predict the final sentiment category. Furthermore, the predicted sentiment category can be compared with the ground truth, and performance can be evaluated using accuracy, precision, recall, and F1 score. The performance of the multimodal sentiment analysis model proposed in this application is compared with that of other models. The training and validation results on the dataset are as follows: Figure 5a , Figure 5b , Figure 6a , Figure 6b As shown in the iterative curves, the model proposed in this application achieves optimal performance on both the training and validation sets. Specifically, for Figure 5a and Figure 6a The horizontal axis represents Epochs, and the vertical axis represents the loss; for Figure 5b and Figure 6b The horizontal axis represents Epochs, and the vertical axis represents accuracy.

[0100] The confusion matrix is ​​used to evaluate the model's performance on the test set. The values ​​on the diagonal of the confusion matrix represent the proportion of correctly predicted categories; higher values ​​indicate better classification performance for that category. The values ​​off-diagonal reveal category pairs that the model easily confuses, reflecting systematic biases. The confusion matrix results of the proposed model on the test set are as follows: Figure 7 As shown, the confusion matrix results for other models on the test set are as follows: Figures 8a to 8e As shown. Among them, for Figure 7 , Figures 8a to 8e The horizontal axis represents the predicted result, and the vertical axis represents the actual result.

[0101] The confusion matrix results show that other models exhibit varying degrees of recognition for the three emotions on the test set, while the model proposed in this application achieves good recognition for all three emotions, with the best overall classification accuracy. These experimental results effectively demonstrate that the multimodal sentiment analysis model based on a dual-encoder, cross-modal, bidirectional, multi-head cross-attention mechanism proposed in this application possesses superior performance and can effectively complete sentiment analysis tasks.

[0102] In summary, the embodiments of this application provide a multimodal emotion analysis method based on EEG signals and text, and the main inventive objectives include the following aspects.

[0103] (1) A method for calculating EEG signal features based on detrended cross-correlation analysis is proposed. This method utilizes the advantage of DCCA in characterizing the long-term cross-correlation and dependence between various channels of EEG signals to construct a brain network diagram that can simultaneously reflect temporal and spatial information. Then, the PageRank algorithm is used to calculate the weight of each node in the brain network diagram to obtain the different contribution levels of each channel under a certain emotional state. Next, the frequency band energy of the EEG signal of each channel in the alpha, beta, theta, and gamma bands is calculated, and the PageRank weights are used to weight them to obtain the frequency band energy weighted by spatial and temporal information.

[0104] EEG fusion features combine information from the time, frequency, and spatial domains, effectively improving the multi-dimensional representation ability of features and thus enhancing the classification performance of deep learning models for multimodal sentiment analysis. Experimental results show that, compared to unweighted energy features, using frequency domain energy features weighted with spatial and temporal information significantly improves the classification performance of sentiment analysis.

[0105] (2) An innovative deep learning model for multimodal sentiment analysis based on EEG signals and text is proposed. The EEG feature vector is weighted by spatial and temporal information and encoded temporally using a neural network combining CNN and GRU. The input text data is first processed by a pre-trained BERT model to extract initial semantic representations, forming text vectors. Then, multi-granularity sentiment semantic features are extracted using a deep pyramid convolutional neural network. Next, the model achieves information interaction and fusion between EEG and text features through a cross-modal bidirectional attention mechanism, and the fused features are input into a bidirectional long short-term memory network with an attention mechanism for final sentiment classification prediction.

[0106] This multimodal sentiment analysis deep learning model combines the objective physiological responses of EEG with the subjective semantic expression of text, achieving a more comprehensive and robust method for identifying emotional states. It effectively addresses the shortcomings of existing sentiment recognition technologies, such as insufficient accuracy under single-modal conditions and limited ability to distinguish complex emotional states. Furthermore, the multimodal sentiment analysis model provided in this application can fully explore the deep complementary relationship between the frequency-domain temporal features of EEG signals and the semantic information of text.

[0107] Secondly, this application also provides an emotion analysis device.

[0108] Figure 9 This is a structural block diagram of a sentiment analysis device according to a preferred embodiment of this application. Figure 9 As shown, the emotion analysis device includes an EEG fusion feature determination module 10, a text semantic feature determination module 20, and an emotion classification determination module 30. The EEG fusion feature determination module 10 determines EEG fusion features based on EEG signals from multiple channels, wherein the determined EEG fusion features fuse temporal, frequency, and spatial information of the EEG signals. The text semantic feature determination module 20 determines text semantic features based on the text being read. The emotion classification determination module 30 determines the emotion classification based on the determined EEG fusion features and the determined text semantic features, for emotion analysis.

[0109] Optionally, determining brainwave fusion features based on EEG signals from multiple channels includes: constructing a brain network map of multiple channels based on EEG signals from multiple channels; determining the weight of each node in the constructed brain network map; determining the frequency band energy for each channel based on the corresponding EEG signal; and determining the weighted frequency band energy for each determined frequency band energy based on the weight of the corresponding node in the constructed brain network map, so as to determine the brainwave fusion features.

[0110] Optionally, a multi-channel brain network map is constructed based on the EEG signals from multiple channels, including: constructing a multi-channel brain network map by calculating the DCCA cross-correlation coefficient based on the EEG signals from multiple channels.

[0111] Optionally, determining the weight of each node in the constructed brain network graph includes: determining the weight of each node in the constructed brain network graph based on the PageRank algorithm.

[0112] Optionally, based on the determined EEG fusion features and the determined text semantic features, an emotion classification is determined, including: fusing the determined EEG fusion features and the determined text semantic features to obtain EEG-text fusion features; and determining the emotion classification based on the obtained EEG-text fusion features.

[0113] Optionally, the determined EEG fusion features and the determined text semantic features are fused to obtain EEG-text fusion features, including: fusing the determined EEG fusion features and the determined text semantic features based on a bidirectional multi-head cross-attention mechanism to obtain EEG-text fusion features.

[0114] Optionally, based on the obtained EEG-text fusion features, the emotion classification is determined, including: based on the obtained EEG-text fusion features, combined with a BiLSTM model with an attention mechanism, the emotion classification is determined.

[0115] The specific working principle and benefits of the sentiment analysis device provided in this application are similar to those of the sentiment analysis method provided in this application, and will not be repeated here.

[0116] Thirdly, this application also provides a machine-readable storage medium storing instructions that cause a machine to perform the aforementioned sentiment analysis method.

[0117] Fourthly, this application also provides an electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the executable instructions to implement the aforementioned sentiment analysis method.

[0118] The preferred embodiments of this application have been described in detail above. However, this application is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solution of this application, and these simple modifications all fall within the protection scope of this application.

[0119] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, and they should also be regarded as the content disclosed in this application.

Claims

1. A sentiment analysis method, characterized in that, This sentiment analysis method includes: Based on EEG signals from multiple channels, EEG fusion features are determined, wherein the determined EEG fusion features integrate the time domain information, frequency domain information and spatial information of the EEG signals; Based on the text being read, determine the semantic features of the text; and Based on the identified EEG fusion features and the identified text semantic features, emotion classification is determined for emotion analysis.

2. The sentiment analysis method according to claim 1, characterized in that, Based on EEG signals from multiple channels, EEG fusion features were determined, including: Based on EEG signals from multiple channels, a brain network diagram of the multiple channels is constructed; Determine the weight of each node in the constructed brain network graph; For each of the multiple channels, the frequency band energy is determined based on the corresponding EEG signal; and For each determined frequency band energy, a weighted frequency band energy is determined based on the weight of the corresponding node in the constructed brain network graph, in order to determine the brain-electric fusion characteristics.

3. The sentiment analysis method according to claim 2, characterized in that, Based on EEG signals from multiple channels, a brain network map of the multiple channels is constructed, including: Based on EEG signals from multiple channels, a brain network map of the multiple channels is constructed by calculating the DCCA cross-correlation coefficient.

4. The sentiment analysis method according to claim 2, characterized in that, Determine the weight of each node in the constructed brain network graph, including: The weight of each node in the constructed brain network graph is determined based on the PageRank algorithm.

5. The sentiment analysis method according to claim 1, characterized in that, Based on the identified EEG fusion features and the identified text semantic features, emotion classification is determined, including: The determined EEG fusion features and the determined text semantic features are fused to obtain EEG-text fusion features; and Based on the obtained EEG-text fusion features, the emotion classification was determined.

6. The sentiment analysis method according to claim 5, characterized in that, The determined EEG fusion features and the determined text semantic features are fused to obtain EEG-text fusion features, including: Based on the bidirectional multi-head cross-attention mechanism, the determined EEG fusion features and the determined text semantic features are fused to obtain EEG-text fusion features.

7. The sentiment analysis method according to claim 5, characterized in that, Based on the obtained EEG-text fusion features, emotion classification is determined, including: Based on the obtained EEG-text fusion features, combined with a BiLSTM model with an attention mechanism, the emotion classification is determined.

8. An emotion analysis device, characterized in that, The sentiment analysis device includes: The EEG fusion feature determination module is used to determine EEG fusion features based on EEG signals from multiple channels. The determined EEG fusion features integrate the time domain information, frequency domain information, and spatial information of the EEG signals. A text semantic feature determination module is used to determine the text semantic features based on the text being read; and The emotion classification determination module is used to determine the emotion classification based on the determined EEG fusion features and the determined text semantic features for emotion analysis.

9. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to perform the sentiment analysis method according to any one of claims 1-7.

10. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the sentiment analysis method according to any one of claims 1-7.

Images