Emotion recognition method and system of multi-scale convolutional network based on bidirectional attention

Abstract

The invention belongs to the field of electroencephalogram emotion recognition, and provides an emotion recognition method and system for a multi-scale convolutional network based on bidirectional attention, and the method comprises the steps: segmenting an electroencephalogram signal according to a time window, extracting differential entropy (DE) and power spectral density (PSD) features, introducing an electroencephalogram map to embed spatial information, and capturing a spatial dependency relationship in the signal. Then, the preprocessed features are input into an MSA-TCN model, and the model combines the attention in the bidirectional features, the attention between the bidirectional features, a multi-scale convolution network and a time sequence convolution network to effectively extract dynamic time features and cross-channel spatial features; the invention aims to overcome the limitation of the prior art in emotion recognition through an innovative bidirectional attention mechanism and a multi-scale convolutional network structure (MSA-TCN).

Description

Technical Field

[0001] The present invention belongs to the technical field of electroencephalogram emotion recognition, and specifically relates to an emotion recognition method and system based on a multi-scale convolutional network with bidirectional attention. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, with the rapid development of artificial intelligence and deep learning technology, emotion recognition has gradually become a research field that has attracted much attention. Emotion, as a complex psychological process, can truly reflect an individual's cognitive, decision-making and behavioral state, and plays a vital role in daily activities. With the widespread application of intelligent human-computer interaction technology, giving computers emotional perception and cognitive abilities close to humans has become a research hotspot, aiming to enable computers to accurately understand human emotional intentions and thus perform various tasks more efficiently. This technology has important application significance in many fields such as disease assessment, user emotion monitoring, and medical health.

[0004] At present, emotion recognition technology is mainly divided into two categories: based on non-physiological signals and based on physiological signals. Non-physiological signals include analysis of facial expressions, voice intonation, body posture and other features. Although such methods perform well in certain specific scenarios, they are easily interfered by external factors, and the signals are highly controllable, resulting in unstable robustness and reliability. In contrast, emotion recognition methods based on physiological signals use physiological signals such as electroencephalogram (EEG), electrocardiogram (ECG), and electromyogram (EMG) to judge emotional states, which can more directly reflect the individual's internal physiological state and are less susceptible to interference from external factors. Therefore, they show higher reliability and stability in emotion recognition tasks.

[0005] Among them, EEG signals, as a physiological signal with high temporal resolution, can reflect the neural activity of the cerebral cortex in real time and are closely related to emotional states. Therefore, EEG signals have gradually become an important research object in the field of emotion recognition. In the study of neuroscience and psychology, EEG signals are considered to be an objective physiological reflection of individual cognitive and emotional changes, which can help researchers to deeply explore the neural mechanisms of emotions. With the continuous advancement of EEG data acquisition and analysis technology, a variety of advanced methods have emerged in the study of emotion recognition based on EEG.

[0006] However, traditional emotion recognition models often have some shortcomings when processing EEG signals. First, these models usually ignore the spatial topological relationship between brain electrodes and fail to fully utilize the deep spatial information between adjacent electrodes, resulting in limited feature extraction capabilities. Secondly, most existing methods focus on a certain type of features in the time domain or frequency domain and cannot simultaneously process the complex spatiotemporal relationship of EEG signals, which in turn limits the accuracy and robustness of emotion recognition. Finally, although some deep learning models have made progress in emotion recognition tasks, the generalization ability of the model still needs to be improved due to the failure to fully consider the interaction and fusion between features. Summary of the invention

[0007] In order to solve the above problems, the present invention proposes an emotion recognition method and system based on a multi-scale convolutional network with bidirectional attention. The present invention aims to overcome the limitations of the existing technology in emotion recognition through an innovative bidirectional attention mechanism and a multi-scale convolutional network structure, and provide an efficient and robust EEG signal processing solution, which shows broad application prospects in multiple application scenarios.

[0008] According to some embodiments, a first solution of the present invention provides an emotion recognition method based on a multi-scale convolutional network with bidirectional attention, which adopts the following technical solution:

[0009] The emotion recognition method based on multi-scale convolutional network with bidirectional attention includes:

[0010] Obtain raw EEG signals and perform preprocessing;

[0011] Based on the preprocessed EEG signal, the power spectrum density feature and differential entropy feature of the EEG signal are extracted;

[0012] According to the power spectral density characteristics and differential entropy characteristics of EEG signals, EEG emotion recognition is performed using the trained multi-scale convolutional network emotion recognition model based on the bidirectional attention mechanism, specifically:

[0013] The differential entropy feature and the power spectral density feature are aligned in time series through the bidirectional inter-feature attention mechanism to obtain a joint representation across features. The key features within the differential entropy feature and the power spectral density feature are identified based on the intra-feature attention mechanism.

[0014] After the key features and joint representations are fused, they are processed through convolutions of different scales to capture the multi-level information of the signal in time and frequency. After the multi-level information is fused, it is processed through the self-attention mechanism to obtain the attention fusion feature.

[0015] The attention fusion features are processed by dilated causal convolution using a temporal convolutional network, and then the processed features are flattened and sentiment classification and recognition are performed through a fully connected layer.

[0016] Furthermore, based on the preprocessed EEG signal, the power spectrum density feature and differential entropy feature of the EEG signal are extracted, specifically:

[0017] By calculating the logarithm of the energy spectrum of the preprocessed EEG signal, the differential entropy characteristics of different frequency bands are determined;

[0018] By calculating the square of the frequency domain signal, the power of each frequency component is obtained and the power spectrum density characteristics of different frequency bands are determined;

[0019] The differential entropy features of different frequency bands are mapped to the two-dimensional EEG map, and a three-dimensional feature matrix is ​​generated to obtain the differential entropy features of the EEG signal;

[0020] The power spectral density features of different frequency bands are mapped to the two-dimensional brain electrode map, and a three-dimensional feature matrix is ​​generated to obtain the power spectral density features of the EEG signal.

[0021] Furthermore, the bidirectional feature-to-feature attention mechanism is used to align the differential entropy feature and the power spectral density feature in time series, thereby obtaining a joint representation across features, specifically:

[0022] A time slice feature of the differential entropy feature is used as a query, and all time slice features of the power spectrum density feature are used as keys and values. The power spectrum density feature is fused into the differential entropy feature using the attention mechanism. The differential entropy feature after fusion is summed with the differential entropy feature before fusion to obtain the differential entropy joint representation.

[0023] A certain time slice feature of the power spectral density feature is used as the query, and all time slice features of the differential entropy feature are used as the key and value. The attention mechanism is used to fuse the differential entropy feature into the power spectral density feature. After the fusion, the fused power spectral density feature and the power spectral density feature before the fusion are summed to obtain the power spectral density joint representation.

[0024] Furthermore, the identification of key features in differential entropy features and power spectral density features based on the intra-feature attention mechanism is specifically as follows:

[0025] Based on the intra-feature attention mechanism, the differential entropy feature is self-attention calculated to obtain the differential entropy key features;

[0026] Based on the intra-feature attention mechanism, self-attention calculation is performed on the power spectral density features to obtain the key features of the power spectral density.

[0027] Furthermore, the multi-scale convolutional network emotion recognition model based on the bidirectional attention mechanism includes:

[0028] The bidirectional inter-feature attention module is connected to the intra-feature attention module, and the output features of the two are fused and processed through two convolutional layers with the same convolution kernel size, and an average pooling layer is connected after each convolutional layer;

[0029] After the second average pooling layer, two convolutional layers of different scales are connected. After the two convolutional layers of different scales, an average pooling layer is connected respectively. The features processed by the two convolutional layers of different scales and the average pooling layer are spliced ​​in the channel dimension.

[0030] The concatenated fusion features are input into the self-attention module, the self-attention module is connected to the temporal convolutional network layer, and the temporal convolutional network layer is connected to the fully connected layer.

[0031] Furthermore, the temporal convolutional network layer includes an expanded causal convolutional layer and a residual connection composed of convolutions;

[0032] The input features input into the temporal convolutional network layer are processed by the dilated causal convolutional layer and then concatenated with the input features processed by the residual connection to obtain the output of the temporal convolutional network layer.

[0033] Furthermore, the key features and the joint representation are integrated as follows:

[0034]

[0035] Among them, Z i is the fusion result of key features and joint representation, and the attention weight γ 1 , γ 2 , γ 3 , γ 4 ; is the differential entropy joint representation, is the power spectral density joint representation, is the key feature of differential entropy, It is the key feature of power spectral density.

[0036] According to some embodiments, the second solution of the present invention provides an emotion recognition system based on a multi-scale convolutional network with bidirectional attention, which adopts the following technical solution:

[0037] Emotion recognition system based on multi-scale convolutional network with bidirectional attention, including:

[0038] A signal processing module is configured to obtain raw EEG signals and perform preprocessing;

[0039] A signal feature extraction module is configured to extract a power spectrum density feature and a differential entropy feature of the EEG signal based on the preprocessed EEG signal;

[0040] The emotion recognition module is configured to perform EEG emotion recognition based on the power spectral density characteristics and differential entropy characteristics of the EEG signal using the trained multi-scale convolutional network emotion recognition model based on the bidirectional attention mechanism. Specifically:

[0041] The differential entropy feature and the power spectral density feature are aligned in time series through the bidirectional inter-feature attention mechanism to obtain a joint representation across features. The key features within the differential entropy feature and the power spectral density feature are identified based on the intra-feature attention mechanism.

[0042] After the key features and joint representations are fused, they are processed through convolutions of different scales to capture the multi-level information of the signal in time and frequency. After the multi-level information is fused, it is processed through the self-attention mechanism to obtain the attention fusion feature.

[0043] The attention fusion features are processed by dilated causal convolution using a temporal convolutional network, and then the processed features are flattened and sentiment classification and recognition are performed through a fully connected layer.

[0044] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium.

[0045] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the emotion recognition method based on a multi-scale convolutional network with bidirectional attention as described in the first aspect above.

[0046] According to some embodiments, a fourth aspect of the present invention provides a computer device.

[0047] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the emotion recognition method based on a multi-scale convolutional network of bidirectional attention as described in the first aspect above are implemented.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] The present invention proposes an EEG emotion recognition model that combines a bidirectional attention mechanism with a multi-scale convolutional temporal network (MSA-TCN). The model uses a bidirectional attention mechanism to align differential entropy and power spectral density features in time, promote information fusion between features, and effectively enhance the complementarity of different features. At the same time, through the attention mechanism within the feature, the model can weaken noise and redundant information and highlight the key features of emotion recognition. The multi-scale convolution structure extracts the features of EEG signals from different spatial and temporal scales, enhancing the model's perception and processing capabilities for complex spatiotemporal patterns. This design can not only significantly improve the accuracy and robustness of emotion recognition, but also has good real-time processing capabilities, and is suitable for multiple fields such as intelligent human-computer interaction, medical health, and psychological emotion management. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0051] Figure 1 1 is a diagram of the architecture of an EEG emotion recognition model based on a multi-scale convolutional network with a bidirectional attention mechanism in an embodiment of the present invention;

[0052] Figure 2 This is a block diagram of the overall architecture of the TCN network model in an embodiment of the present invention;

[0053] Figure 3 is an electrode mapping diagram in an embodiment of the present invention;

[0054] Figure 4 is a training flow chart of the method described in an embodiment of the present invention;

[0055] Figure 5 It is a schematic diagram of the confusion matrix of the emotion recognition model of the method described in the embodiment of the present invention in high / low emotion classification. DETAILED DESCRIPTION

[0056] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0057] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0058] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0059] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0060] Embodiment 1

[0061] The present embodiment provides an emotion recognition method based on a multi-scale convolutional network with bidirectional attention. The present embodiment takes the application of the method to a server as an example. It is understandable that the method can also be applied to a terminal, and can also be applied to a system including a terminal, a server, and a server, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application. In the present embodiment, the method includes the following steps:

[0062] Obtain raw EEG signals and perform preprocessing;

[0063] Based on the preprocessed EEG signal, the power spectrum density feature and differential entropy feature of the EEG signal are extracted;

[0064] According to the power spectral density characteristics and differential entropy characteristics of EEG signals, EEG emotion recognition is performed using the trained multi-scale convolutional network emotion recognition model based on the bidirectional attention mechanism, specifically:

[0065] The differential entropy feature and the power spectral density feature are aligned in time series through the bidirectional inter-feature attention mechanism to obtain a joint representation across features. The key features within the differential entropy feature and the power spectral density feature are identified based on the intra-feature attention mechanism.

[0066] After the key features and joint representations are fused, they are processed through convolutions of different scales to capture the multi-level information of the signal in time and frequency. After the multi-level information is fused, it is processed through the self-attention mechanism to obtain the attention fusion feature.

[0067] The attention fusion features are processed by dilated causal convolution using a temporal convolutional network, and then the processed features are flattened and sentiment classification and recognition are performed through a fully connected layer.

[0068] like Figure 4 As shown, the overall process of the method described in this embodiment is specifically as follows:

[0069] Step 1: Data preprocessing.

[0070] First, the original EEG signal x(t) is preprocessed. In order to extract the α, β, γ and θ frequency band information in the signal, we need to design four bandpass filters. The typical frequency range of each band is as follows: θ band: 4Hz–8Hz, α band: 8Hz–13Hz, β band: 13Hz–30Hz, γ band: 30Hz–45Hz. When using the Butterworth filter, its order N and cutoff frequency need to be determined. Assume that the sampling frequency is f s (Unit: Hz), the filter parameters can be calculated using the following steps.

[0071] 1. Frequency Normalization

[0072] In order to design a digital filter, the frequency must first be normalized. The normalized cutoff frequency formula is:

[0073]

[0074] Where: f c are the upper and lower cutoff frequencies of the bandpass filter, in Hz. s is the sampling frequency in Hz.

[0075] 2. The Butterworth filter is designed based on the following transfer function:

[0076]

[0077] Where: s is a complex frequency variable. ω c is the normalized cutoff frequency, and N is the order of the filter. After designing the filter, divide the signal of each frequency band into segments according to the time window. Assuming that each time window is 0.5 seconds, the signal can be divided into multiple data segments: 3 seconds of baseline can be divided into 6 segments, and 60 seconds of data can be divided into 120 segments.

[0078] Step 2: Feature extraction and transformation.

[0079] The power spectrum density feature and differential entropy feature of the EEG signal are extracted based on the preprocessed EEG signal, specifically:

[0080] By calculating the logarithm of the energy spectrum of the preprocessed EEG signal, the differential entropy characteristics of different frequency bands are determined;

[0081] By calculating the square of the frequency domain signal, the power of each frequency component is obtained and the power spectrum density characteristics of different frequency bands are determined;

[0082] The differential entropy features of different frequency bands are mapped to the two-dimensional EEG map, and a three-dimensional feature matrix is ​​generated to obtain the differential entropy features of the EEG signal;

[0083] The power spectral density features of different frequency bands are mapped to the two-dimensional brain electrode map, and a three-dimensional feature matrix is ​​generated to obtain the power spectral density features of the EEG signal.

[0084] Differential entropy is a tool for measuring the uncertainty of continuous random variables and is widely used in signal processing and information theory. In electroencephalogram (EEG) analysis, differential entropy is used to quantify the complexity and information content of the signal. It is a generalized form of Shannon information entropy on continuous variables. The expression of Shannon entropy is:

[0085]

[0086] For continuous variables, the differential entropy is of the form:

[0087]

[0088] Among them, p(x) represents the probability density function of the continuous signal, and [a,b] is the interval of the signal value. For the EEG signal obtained after preprocessing, it can be regarded as approximately following the Gaussian distribution N(μ,σ 2 ), the differential entropy DE(X) of the EEG signal can be calculated by the logarithm of its energy spectrum.

[0089]

[0090] Among them, π and e are constants, σ 2 Represents the variance of the EEG signal sequence in the frequency band.

[0091] In the emotion recognition task, the differential entropy of EEG signals is an important dynamic feature that can reflect the complexity and information content of the signal. Changes in emotional state may lead to differences in differential entropy, so it is one of the effective features in emotion classification.

[0092] Power spectral density is an important tool in EEG signal analysis, which can effectively extract and quantify the frequency characteristics of EEG signals. By analyzing PSD, researchers and clinicians can gain a deep understanding of the activity and state of the brain, providing strong support for emotion recognition, cognitive monitoring, and the diagnosis of neurological diseases. For continuous-time signals, Fourier transform can usually be used to convert the time domain signal into a frequency domain signal, and then calculate the power spectral density of the signal. The common method for calculating PSD is based on discrete Fourier transform (DFT) or fast Fourier transform (FFT). Specifically: Given a continuous time domain signal x(t), its Fourier transform is:

[0093]

[0094] X(f) is the spectral component of the EEG signal x(t) at frequency f. Power spectral density is the power distribution of the signal at each frequency. The power of each frequency component can be obtained by calculating the square of the frequency domain signal. The formula for PSD is:

[0095]

[0096] Where A is the total number of signal points, f s is the sampling frequency, X(f) is the frequency domain signal after Fourier transformation, |X(f)| 2 is the square of the amplitude of the frequency domain signal. In order to obtain a smoother power spectrum, the commonly used Welch method is to divide the signal into overlapping segments, calculate the PSD for each segment and take the average. The PSD calculation formula of the Welch method is:

[0097]

[0098] Where M is the number of segments, L is the length of each segment, and X m (f) is the Fourier transform result of the mth segment. In Python, PSD and DE can be calculated by calling the corresponding library.

[0099] The international 10-20 system is a standardized scalp electrode placement system used to describe the correspondence between electrode locations and underlying areas of the cerebral cortex. The electrode locations used in the DEAP dataset are as follows: Figure 2 As shown, the blue-labeled electrodes represent the test points. In the EEG electrode plan, each electrode is physically adjacent to multiple electrodes that record EEG signals from a specific brain region.

[0100] By mapping the power spectral density (PSD) and differential entropy (DE) features of different frequency bands onto two-dimensional EEG maps (such as Figure 3As shown in Figure 2, a three-dimensional feature matrix is ​​generated, which can more comprehensively capture the EEG signal information related to emotions. This process helps improve the model's ability to distinguish emotional states, thereby improving the accuracy of classification.

[0101] Step 3: Constructing the overall architecture of the emotion recognition network aims to achieve efficient and accurate emotion recognition by combining advanced technologies such as bidirectional attention mechanism and multi-scale convolutional network. First, the input EEG signal is preprocessed to extract power spectral density (PSD) and differential entropy (DE) features. Next, the processed features are fed into the multi-scale convolutional network (MSA-TCN), which captures the multi-level information of the signal in time and frequency through bidirectional inter-feature and intra-feature attention mechanisms, as well as convolution kernels of different scales, and further enhances the ability to distinguish emotional states. Finally, the extracted features are mapped to emotion categories through a fully connected layer, and an activation function is applied to achieve nonlinear transformation, and the prediction results are finally output. The design of the overall network architecture fully considers the complexity of signal processing and the diversity of emotion recognition, and strives to achieve a good balance between accuracy and efficiency.

[0102] This embodiment proposes an EEG emotion recognition model based on a multi-scale convolutional network (MSA-TCN) with a bidirectional attention mechanism. First, the preprocessed EEG signal features are analyzed to extract key features, including power spectral density (PSD) and differential entropy (DE). These features are input into the MSA-TCN network for deep processing. The model introduces bidirectional inter-feature and intra-feature fusion modules to enhance the interaction and effectiveness of information between features. Through the bidirectional attention mechanism, the differential entropy and power spectral density features are aligned in time series to achieve joint representation across features and enhance the complementarity of features. The intra-feature attention mechanism focuses on reducing noise and redundant information and highlighting the core information that is critical to emotion recognition.

[0103] This fusion module can not only accurately capture the key features closely related to the emotional state, but also flexibly adjust the internal weights of the features by building deep associations between different features, significantly improving the accuracy of feature extraction. Under the guidance of bidirectional attention, the model can focus on channels and feature areas with greater emotional representation value, avoid information loss, and improve the sensitivity of emotional signal processing, thereby significantly improving the accuracy of classification tasks.

[0104] In the multi-scale convolution structure, the fused features are first extracted through two 3×3 convolutional layers to extract local spatiotemporal features. This operation focuses on capturing the short-term dynamic changes in EEG signals and their local spatial associations, retaining more detailed information, and is particularly suitable for extracting short-term spatial change patterns, thereby improving the perception of local feature differences.

[0105] The model then uses 2×1 and 4×1 convolutional layers to further broaden the scope of capturing spatial dimensional features. These convolutional layers have different receptive fields, accurately capturing the spatiotemporal changes in EEG signals at multiple scales, significantly enhancing the ability to understand and process complex features. By fusing local and global features, the model forms a more comprehensive feature representation and enhances the ability to recognize complex patterns. This diverse information enables the model to capture richer spatiotemporal changes and effectively analyze and understand EEG signals in different contexts.

[0106] The intra-feature attention mechanism dynamically calculates the importance weights of each feature, allowing the model to focus on the key features of sentiment classification. This mechanism effectively suppresses noise and redundant information and improves the model's sensitivity to important features. At the same time, the attention mechanism gives the model context-awareness, enabling it to flexibly adjust its focus on features according to different contexts, thereby improving overall performance.

[0107] By combining the dilated convolutional network, the model successfully captures the long-term dependencies in EEG signals. This feature enables the model to perform well in the task of emotional state classification, and can perform comprehensive and in-depth feature analysis from local to global, from short-term to long-term.

[0108] The model of the present invention has end-to-end learning capabilities, can automatically learn and adjust parameters from data, and avoids the limitations of manually designed feature extraction rules. In addition, the MSA-TCN model has strong real-time processing capabilities, is suitable for online emotion recognition tasks, and is widely used in intelligent human-computer interaction, emotional computing, and medical health. The model can help medical practitioners monitor the emotional state of patients, or be used in the emotional response system of smart devices to improve user experience.

[0109] Regardless of the training phase or the application phase, the overall framework of the network model in this embodiment is as follows: Figure 1 The network model input defined in the present invention is X = [x 0 ,x 1 ,…,x n ]∈R n×c×l , Y=[y 0 ,y 1 ,…,y n ]∈R n×c×l , n is the number of time slices, c is the number of channels, and l is the feature length of each channel.

[0110] Assume that X is the three-dimensional DE feature sample of the EEG signal, that is, the differential entropy feature of the EEG signal, and Y is the three-dimensional PSD feature sample of the EEG signal, that is, the power spectral density feature of the EEG signal, X = [x 0 ,x 1 ,…,x n ], xn ∈R c×l , Y=[y 0 ,y 1 ,…,y n ],y n ∈R c×l Where n is the number of feature samples, c is the number of channels of the feature map, l is the number of feature points in each channel, and the number of feature points is the product of the height and width of the feature map h×w.

[0111] The differential entropy feature and the power spectral density feature are aligned in time series through the bidirectional feature-to-feature attention mechanism to obtain a joint representation across features, specifically:

[0112] A time slice feature of the differential entropy feature is used as a query, and all time slice features of the power spectrum density feature are used as keys and values. The power spectrum density feature is fused into the differential entropy feature using the attention mechanism. The differential entropy feature after fusion is summed with the differential entropy feature before fusion to obtain the differential entropy joint representation.

[0113] A certain time slice feature of the power spectral density feature is used as the query, and all time slice features of the differential entropy feature are used as the key and value. The attention mechanism is used to fuse the differential entropy feature into the power spectral density feature. After the fusion, the fused power spectral density feature and the power spectral density feature before the fusion are summed to obtain the power spectral density joint representation.

[0114] The bidirectional attention mechanism not only needs to realize the fusion between DE and PSD features, but also requires information interaction between DE itself and PSD itself, and finally fuses this information together.

[0115] In the i-th time slice x of the DE feature i ∈R c×l As a query, all time slices y of the PSD feature j ∈R c×l As the key and value, the attention mechanism is used to fuse the PSD features into the DE features. First, the query weight W is used Q and PSD feature key weight W K x i and j Map to query and key space respectively, and calculate similarity:

[0116]

[0117] in: and is the learned weight matrix, d k It is a hyperparameter used to define the size of the projected key vector space, that is, after the PSD feature key weight matrix W KThe dimension of the projected feature vector can be understood as the PSD feature y j ∈R c×l The PSD feature key weight matrix W is K The dimension of the new feature representation obtained after projection. The dimension d after projection k is a scalar value, usually the PSD feature key weight matrix W K The output dimension of . The similarity scores are normalized using the softmax function to obtain the differential entropy attention weight:

[0118]

[0119] According to the differential entropy attention weight α ij Perform weighted summation on the PSD features to obtain the fusion result:

[0120]

[0121] Among them, W V is the weight matrix of the PSD eigenvalue vector, which converts each time slice y in the PSD feature j Projected into the value space, a value vector is generated. With this weight matrix W V and the weight matrix of query and PSD feature key and Similarly, they are learned during the training process and are used to map input features to different spaces for better information integration in the attention mechanism.

[0122] The fusion result z i Add to original DE feature x i In the differential entropy joint representation

[0123]

[0124] in, It is the DE feature after the PSD feature information is enhanced. Similarly, reverse fusion is performed. That is, the PSD feature is used as the query, and the DE feature is used as the key and value. The power spectrum density joint representation is obtained.

[0125]

[0126] Among them, β ij is the power spectral density attention weight, W' V The same goes for W V Similarly, it is the weight matrix of the DE eigenvalue vector, which is used to convert the input feature x j Convert to a value vector.

[0127] The key features in the differential entropy feature and the power spectral density feature are identified based on the intra-feature attention mechanism, specifically:

[0128] Based on the intra-feature attention mechanism, the differential entropy feature is self-attention calculated to obtain the differential entropy key features;

[0129] Based on the intra-feature attention mechanism, self-attention calculation is performed on the power spectral density features to obtain the key features of the power spectral density.

[0130] Similarly, in order to capture the temporal and spatial dependencies within DE and PSD features, self-attention calculations need to be performed on DE and PSD features. That is:

[0131]

[0132] in, is the key feature of differential entropy, is the key feature of power spectral density, the jth time slice x of DE feature j ∈R c×l As a query, all time slices y of the PSD feature j ∈R c×l as key and value.

[0133] By introducing trainable attention weights γ 1 , γ 2 , γ 3 , γ 4 , perform weighted summation on the fusion results:

[0134]

[0135] Among them, Z i is the fusion result of key features and joint representation, is the differential entropy joint representation, is the power spectral density joint representation, is the key feature of differential entropy, It is the key feature of power spectral density.

[0136] For the fused feature Z i Perform two 3x3 convolutions and add an average pooling layer after each convolution:

[0137] Z 1 =AvgPool(Conv 3×3 (Z)

[0138] Z 2 =AvgPool(Conv 3×3 (Z 1 ))

[0139] Next, Z 2 Perform 2x1 convolution and 4x1 convolution respectively, and add an average pooling layer after each convolution:

[0140] Z 2×1 =AvgPool(Conv 2×1 (Z)

[0141] Z 4×1 =AvgPool(Conv 4×1 (Z 2 ))

[0142] Z 2×1 and Z 4×1 Concatenate in the channel dimension to get the fused features:

[0143] Z Concat =Concat(Z 2×1 ,Z 4×1 )

[0144] The fused features are then passed through the self-attention mechanism to obtain the attention fusion features:

[0145] z′ j =Self-Attention(z j ,{z 1 ,z 2 ,...,z N}),j∈N

[0146] Among them, N is the sequence length, which can be understood as all moments j in the sequence represent a time slice at a certain moment.

[0147] In the overall architecture of the temporal convolutional network (TCN), Figure 2 As shown, a block diagram of the overall architecture of a residual convolution block in a temporal convolutional network model (TCN) in an embodiment of the present invention is shown. In the TCN model, the dilation rate of each layer of convolution is set to d=1, d=2, and d=3, respectively, that is, the expansion distance of the convolution kernel increases layer by layer. This design enables the receptive field of the model to expand layer by layer, thereby gradually enhancing the ability to capture long-distance dependencies in the input sequence without increasing the depth of the network. In the TCN model of the present invention, the increasing dilation rate layer by layer enables the model to effectively model the long-term dependency features in time series data, which is very suitable for processing and analyzing time series tasks with long-distance dependency requirements.

[0148] like Figure 2As shown in the figure, the left side is the dilated causal convolution layer, and the right side is the residual connection composed of 1x1 convolution. The core advantage of the dilated causal convolution is that it can expand the receptive field without significantly increasing the number of parameters, thereby effectively capturing the long-range dependencies in time series data. Its basic operation formula is:

[0149]

[0150] Among them, d is the expansion rate. By controlling the size of the expansion rate, the receptive field can be flexibly adjusted to capture dependencies at different time scales. In the temporal convolutional network, r represents the length of the convolution kernel of the convolution operation, that is, the number of time steps considered. In order to further improve the training effect of the model, TCN introduces a residual connection structure. The residual block composed of a 1x1 convolutional layer directly adds the input to the convolution output, alleviating the problem of gradient vanishing and gradient exploding, and promoting the effective training of deep networks. The formula for residual connection is:

[0151] y(t)=ReLU(h(t)+x(t))

[0152] Through residual connections, the model can be trained more stably while retaining input information, thereby improving the generalization ability of the model.

[0153] In this network, the input attention fusion feature z' j First, after the dilated causal convolutional layer processing, we get Z TCN These features are then flattened and finally passed through a fully connected layer for classification or prediction:

[0154]

[0155] The overall architecture combines dilated convolutions and residual connections to effectively capture long-range dependencies in time series data and improve model performance by optimizing network training stability.

[0156] Step 4: Model optimization.

[0157] Loss function and optimizer: This paper adopts the cross entropy loss function (CrossEntropyLoss) as the loss function of the classification task to measure the difference between the model prediction and the actual category. At the same time, the Adam optimizer is used for gradient descent. Adam combines the advantages of momentum and adaptive learning rate, and can achieve fast and stable convergence in complex loss surfaces.

[0158]

[0159] Among them, C is the category parameter, y i is the true label encoding, is the predicted probability of the model, θ is the model parameter, and η is the learning rate.

[0160] Learning rate scheduling: Introduce a learning rate scheduler (such as StepLR) to dynamically adjust the learning rate according to a predetermined strategy during model training to avoid model oscillation caused by too high a learning rate or slow convergence caused by too low a learning rate.

[0161] Regularization technology: Dropout and layer normalization (LayerNorm) technology are used in model training to prevent model overfitting. Dropout reduces the co-adaptability between neurons and enhances the generalization ability of the model by randomly discarding some neurons during the training process. Layer normalization can stabilize the model training process and accelerate model convergence.

[0162] Step 5: Model training and evaluation

[0163] Model training: The present invention uses the pytorch deep learning framework in the environment of NVDIA GeForce RTX4090, and uses multiple rounds of iterative training on the training data set. The number of categories is set to 2, the random seed is set to 1234, the learning rate is set to 0.0001, the number of training rounds is 500, the batch size is 512, and the dropout is set to 0.2 to prevent overfitting, so that the model continuously adjusts parameters to minimize the loss function. During the training process, after each epoch, the performance of the model on the validation set will be recorded in order to select the best model parameters.

[0164] Step 6: Model Evaluation

[0165] After the model is trained, it is applied to the test data set and common classification performance indicators are calculated, including accuracy, F1-score, etc. In addition, a confusion matrix is ​​constructed to deeply analyze the classification performance of the model on different categories to identify possible misclassification problems.

[0166] In order to verify the feasibility and effectiveness of the present invention, the DEAP dataset was used to evaluate the model performance, mainly using the following indicators: accuracy, which indicates the proportion of samples correctly predicted by the model to the total number of samples. By calculating the number of true positive examples and true negative examples, the overall performance of the model can be understood. The F1 score is the harmonic mean of precision and recall, and is suitable for cases of class imbalance. The F1 score can better reflect the model's ability to recognize minority classes. The confusion matrix visualizes the model's predictions on different categories and helps analyze the classification effect of the model. It shows the correspondence between the true label and the model's predicted label, and provides specific information on misclassification.

[0167] Table 1 Comparison results of this method with other methods

[0168] Accuracy F1-score MS-Timesnet 0.8666 0.885 MLP 0.6104 0.7315 Tsception 0.7636 0.7994 EEGnet 0.6393 0.7246 LSTM 0.5903 0.7292 LogisticRegression 0.6216 0.7098 MSA-TCN (this method) 0.9014 0.9245

[0169] As can be seen from Table 1, the models all showed the best accuracy in arousal classification, significantly better than several other classification models. In contrast, MS-Timesnet and EEGnet achieved scores of 0.8666 and 0.763 respectively, while the traditional LSTM, MLP and Logistic Regression models performed relatively poorly, with accuracies of 0.8045, 0.799 and 0.7706 respectively. This shows that the MSA-TCN model in the present invention has significant advantages in emotion recognition tasks, especially when processing complex EEG signals, it can better capture key features and improve classification accuracy.

[0170] like Figure 5 The confusion matrix shown shows that the model performs well in the emotion classification task, correctly classifying 6231 low emotion samples and 7794 high emotion samples, with relatively few misclassifications. This performance fully demonstrates the superiority of the model in the emotion recognition task.

[0171] The core advantage of the model lies in the combination of bidirectional inter-feature and intra-feature attention mechanisms with the Multi-Scale Convolutional Temporal Convolutional Network (MSA-TCN). The bidirectional attention mechanism can not only capture the associations between different features, but also flexibly allocate weights within a single feature, thereby more accurately extracting key information related to emotions. This mechanism helps the model focus on the most valuable parts of the signal for emotion recognition, significantly improving the accuracy of classification.

[0172] In addition, multi-scale convolution TCN uses convolution kernels of different sizes to capture time domain and frequency domain information at multiple scales. It can not only extract local features of emotional changes, but also effectively process long-term dependencies in complex time series data, further enhancing the ability to recognize emotional states. The combination of these two mechanisms enables the model to show higher robustness and accuracy when processing complex EEG signals, providing strong support for emotion recognition.

[0173] Embodiment 2

[0174] This embodiment provides an emotion recognition system based on a multi-scale convolutional network with bidirectional attention, including:

[0175] A signal processing module is configured to obtain raw EEG signals and perform preprocessing;

[0176] A signal feature extraction module is configured to extract a power spectrum density feature and a differential entropy feature of the EEG signal based on the preprocessed EEG signal;

[0177] The emotion recognition module is configured to perform EEG emotion recognition based on the power spectral density characteristics and differential entropy characteristics of the EEG signal using the trained multi-scale convolutional network emotion recognition model based on the bidirectional attention mechanism. Specifically:

[0178] The differential entropy feature and the power spectral density feature are aligned in time series through the bidirectional inter-feature attention mechanism to obtain a joint representation across features. The key features within the differential entropy feature and the power spectral density feature are identified based on the intra-feature attention mechanism.

[0179] After the key features and joint representations are fused, they are processed through convolutions of different scales to capture the multi-level information of the signal in time and frequency. After the multi-level information is fused, it is processed through the self-attention mechanism to obtain the attention fusion feature.

[0180] The attention fusion features are processed by dilated causal convolution using a temporal convolutional network, and then the processed features are flattened and sentiment classification and recognition are performed through a fully connected layer.

[0181] The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment 1. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0182] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0183] The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0184] Embodiment 3

[0185] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps in the emotion recognition method based on a multi-scale convolutional network with bidirectional attention as described in the first embodiment above are implemented.

[0186] Embodiment 4

[0187] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the emotion recognition method based on a multi-scale convolutional network of bidirectional attention as described in the first embodiment are implemented.

[0188] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0189] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0190] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0191] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0192] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0193] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. Emotion recognition method based on multi-scale convolutional network with bidirectional attention, characterized in that: include: Obtain raw EEG signals and perform preprocessing; Based on the preprocessed EEG signal, the power spectrum density feature and differential entropy feature of the EEG signal are extracted; According to the power spectral density characteristics and differential entropy characteristics of EEG signals, EEG emotion recognition is performed using the trained multi-scale convolutional network emotion recognition model based on the bidirectional attention mechanism, specifically: The differential entropy feature and the power spectral density feature are aligned in time series through the bidirectional inter-feature attention mechanism to obtain a joint representation across features. The key features within the differential entropy feature and the power spectral density feature are identified based on the intra-feature attention mechanism. After the key features and joint representations are fused, they are processed through convolutions of different scales to capture the multi-level information of the signal in time and frequency. After the multi-level information is fused, it is processed through the self-attention mechanism to obtain the attention fusion feature. The attention fusion features are processed by dilated causal convolution using a temporal convolutional network, and then the processed features are flattened and sentiment classification and recognition are performed through a fully connected layer.

2. The emotion recognition method based on a multi-scale convolutional network based on bidirectional attention as claimed in claim 1, characterized in that: The power spectrum density feature and differential entropy feature of the EEG signal are extracted based on the preprocessed EEG signal, specifically: By calculating the logarithm of the energy spectrum of the preprocessed EEG signal, the differential entropy characteristics of different frequency bands are determined; By calculating the square of the frequency domain signal, the power of each frequency component is obtained and the power spectrum density characteristics of different frequency bands are determined; The differential entropy features of different frequency bands are mapped to the two-dimensional EEG map, and a three-dimensional feature matrix is ​​generated to obtain the differential entropy features of the EEG signal; The power spectral density features of different frequency bands are mapped to the two-dimensional brain electrode map, and a three-dimensional feature matrix is ​​generated to obtain the power spectral density features of the EEG signal.

3. The emotion recognition method based on a multi-scale convolutional network based on bidirectional attention as claimed in claim 1, characterized in that: The bidirectional feature-to-feature attention mechanism is used to align the differential entropy feature and the power spectral density feature in time series, and obtain a joint representation across features, specifically: A time slice feature of the differential entropy feature is used as a query, and all time slice features of the power spectrum density feature are used as keys and values. The power spectrum density feature is fused into the differential entropy feature using the attention mechanism. The differential entropy feature after fusion is summed with the differential entropy feature before fusion to obtain the differential entropy joint representation. A certain time slice feature of the power spectral density feature is used as the query, and all time slice features of the differential entropy feature are used as the key and value. The attention mechanism is used to fuse the differential entropy feature into the power spectral density feature. After the fusion, the fused power spectral density feature and the power spectral density feature before the fusion are summed to obtain the power spectral density joint representation.

4. The emotion recognition method based on a multi-scale convolutional network based on bidirectional attention as claimed in claim 1, characterized in that: The key features in the differential entropy feature and the power spectral density feature are identified based on the intra-feature attention mechanism, specifically: Based on the intra-feature attention mechanism, the differential entropy feature is self-attention calculated to obtain the differential entropy key features; Based on the intra-feature attention mechanism, self-attention calculation is performed on the power spectral density features to obtain the key features of the power spectral density.

5. The emotion recognition method based on a multi-scale convolutional network based on bidirectional attention as claimed in claim 1, characterized in that: The multi-scale convolutional network emotion recognition model based on the bidirectional attention mechanism includes: The bidirectional inter-feature attention module is connected to the intra-feature attention module. The output features of the two are fused and processed through two convolutional layers with the same convolution kernel size. An average pooling layer is connected after each convolutional layer. After the second average pooling layer, two convolutional layers of different scales are connected. After the two convolutional layers of different scales, an average pooling layer is connected respectively. The features processed by the two convolutional layers of different scales and the average pooling layer are concatenated in the channel dimension. The concatenated fusion features are input into the self-attention module, the self-attention module is connected to the temporal convolutional network layer, and the temporal convolutional network layer is connected to the fully connected layer.

6. The emotion recognition method based on a multi-scale convolutional network based on bidirectional attention as claimed in claim 5, characterized in that: The temporal convolutional network layer includes an expanded causal convolutional layer and a residual connection composed of convolutions; The input features input into the temporal convolutional network layer are processed by the dilated causal convolutional layer and then concatenated with the input features processed by the residual connection to obtain the output of the temporal convolutional network layer.

7. The emotion recognition method based on a multi-scale convolutional network based on bidirectional attention as claimed in claim 1, characterized in that: The key features and joint representations are fused as follows: Among them, Z i is the fusion result of key features and joint representation, with attention weights γ1, γ2, γ3, γ4; is the differential entropy joint representation, is the power spectral density joint representation, is the key feature of differential entropy, It is the key feature of power spectral density.

8. Emotion recognition system based on multi-scale convolutional network with bidirectional attention, characterized by: include: A signal processing module is configured to obtain raw EEG signals and perform preprocessing; A signal feature extraction module is configured to extract a power spectrum density feature and a differential entropy feature of the EEG signal based on the preprocessed EEG signal; The emotion recognition module is configured to perform EEG emotion recognition based on the power spectral density characteristics and differential entropy characteristics of the EEG signal using the trained multi-scale convolutional network emotion recognition model based on the bidirectional attention mechanism. Specifically: The differential entropy feature and the power spectral density feature are aligned in time series through the bidirectional inter-feature attention mechanism to obtain a joint representation across features. The key features within the differential entropy feature and the power spectral density feature are identified based on the intra-feature attention mechanism. After the key features and joint representations are fused, they are processed through convolutions of different scales to capture the multi-level information of the signal in time and frequency. After the multi-level information is fused, it is processed through the self-attention mechanism to obtain the attention fusion feature. The attention fusion features are processed by dilated causal convolution using a temporal convolutional network, and then the processed features are flattened and sentiment classification and recognition are performed through a fully connected layer.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the emotion recognition method based on a multi-scale convolutional network based on bidirectional attention are implemented as described in any one of claims 1 to 7.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the emotion recognition method based on a multi-scale convolutional network based on bidirectional attention are implemented as described in any one of claims 1-7.

Images