A Method and System for EEG Emotion Recognition and Detection Based on Lightweight Residual Convolutional Capsule Networks

Abstract

This invention discloses a method for EEG emotion recognition and detection based on a lightweight residual convolutional capsule network, characterized by the following steps: S1: Extracting the original waveform of the EEG signal and performing data preprocessing; S2: Using a feature extractor based on a residual convolutional network to model the interdependencies between feature channels and automatically extract deep emotion features; S3: Using convolutional capsule layers to convert the feature data into original capsules, the length and orientation of which reflect the existence and characteristics of the related deep features; S4: Using an emotion capsule layer as a classifier, using a weight matrix to capture the positional relationships between local features, and using the length of the capsule output vector to represent the probability of the feature appearing in the current capsule layer, obtaining the corresponding prediction result. A system for EEG emotion recognition and detection based on a lightweight residual convolutional capsule network is also disclosed. This invention can significantly improve the accuracy of emotion recognition level detection.

Description

Technical Field

[0001] This invention relates to the field of EEG emotion recognition and detection technology, and in particular to an EEG emotion recognition and detection method and system based on lightweight residual convolutional capsule networks. Background Technology

[0002] Emotional phenomena have a profound impact on human cognition, decision-making, and behavior, reflecting subjective feelings and objective physiological responses. Emotion analysis is an important part of daily life, especially in the context of human-computer interaction. In recent years, significant progress has been made in electroencephalography (EEG) emotion recognition. However, the shallow convolutional neural networks (CNNs) used cannot effectively represent the spatial relationships between different features, which affects the performance of these models. Furthermore, due to the limited amount of EEG sample data, networks with many parameters are prone to overfitting; traditional CNNs applied to EEG emotion recognition research have shallow layers and cannot fully learn the effective information from the EEG data; pooling layers lose a large amount of valuable information; and simple CNNs cannot distinguish the spatial relationships between features in emotion recognition tasks.

[0003] Representative deep emotion features can improve prediction performance, making it crucial to design an effective method for directly extracting deep features from raw EEG signals. Therefore, a novel emotion recognition and detection method is urgently needed to address these issues. This paper proposes a lightweight residual convolution-based capsule network (Light-ResNet-Capsule) for EEG emotion recognition. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a method and system for EEG emotion recognition and detection based on a lightweight residual convolutional capsule network. For the first time, this invention proposes a lightweight residual convolutional network for emotion recognition and detection, which can significantly improve the accuracy of emotion recognition technology, addressing the problem of small EEG sample data volume.

[0005] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is: to provide a brainwave emotion recognition and detection method based on a lightweight residual convolutional capsule network, comprising the following steps:

[0006] S1: Extract the raw waveform of the EEG signal and perform data preprocessing, which serves as the input feature for predicting the emotion recognition level;

[0007] S2: A feature extractor based on residual convolutional networks is used to model the interdependencies between feature channels and automatically extract deep emotional features.

[0008] S3: A convolutional capsule layer is used to convert the deep emotional feature data obtained in S2 into original capsules, promoting the inclusion of more important emotional information in the capsules. The length and orientation of the original capsules reflect the existence and characteristics of the related deep features.

[0009] S4: The emotion capsule layer is used as a classifier. The weight matrix is ​​used to capture the positional relationship between local features and strengthen the connection between capsules. The length of the capsule output vector represents the probability of the feature appearing in the current capsule layer, and the corresponding prediction result is obtained.

[0010] In a preferred embodiment of the present invention, the specific steps of extracting the raw waveform of the EEG signal and performing data preprocessing in step S1 include:

[0011] Remove the baseline and split the data using a sliding window:

[0012]

[0013]

[0014] Among them, X p This represents data segmented using a sliding window. x′ represents the baseline signal per second. q This indicates the time slice from which the baseline signal per second is removed.

[0015] In a preferred embodiment of the present invention, step S2 specifically includes the following steps:

[0016] The input data is convolved and downsampled to extract its spatiotemporal features; using four residual convolutional blocks, the number of channels in the feature map is increased to 512, which can capture more deep emotional features.

[0017] y = F(x, {W i})+x (3)

[0018] F=σ(W1x) (4)

[0019] Where x represents the input, and the function F(x, {W}) is... i}) represents the residual mapping to be learned, W1 represents the weight matrix, σ represents the ReLU activation function, and F+x is executed by concatenating shortcuts and adding elements.

[0020] In a preferred embodiment of the present invention, step S3 specifically includes:

[0021] S301: The feature map is convolved with 512 3×3 convolution kernels with a stride of 2, and the shape of the output feature map is (8, 2, 512).

[0022] S302: Divide the above feature map into 64 sub-feature maps with a channel dimension of 8, i.e., the shape is (8, 2, 8). Taking one of the sub-feature maps as an example, its tensor data structure can be regarded as multiple 8-dimensional vectors arranged in an 8×2 grid. That is, each carrier (vector) represents a primary capsule containing 8 elements. Through the calculation of the convolutional capsule layer, a total of 1024 original capsules are obtained. These original capsules will be used as the input of the emotion capsule layer.

[0023] In a preferred embodiment of the present invention, step S4 specifically includes:

[0024] The entire original capsule output from the convolutional capsule layer is routed to multiple capsules containing 16 elements, implemented by a dynamic routing algorithm:

[0025] S401: Using the weight matrix W ij Learn the input features and compute the primary capsule m. i Prediction vector for emotion capsule j

[0026]

[0027] S402: By applying all those with different weights Summation is performed to generate capsule S j :

[0028]

[0029]

[0030] c ij =softmax(b ij (8)

[0031] In this process, updating the capsule network actually involves calculating the coupling coefficient, c. ij This represents the coupling coefficient between main capsule i and emotion capsule j, and its value can be updated by b in each iteration. ij The value is used to calculate, and then a softmax operation is performed to obtain the result;

[0032] S403: Compressing capsule S using a nonlinear compression function j The length of the capsule is such that the length of the short capsule approaches 0 and the length of the long capsule approaches 1. S is calculated using a nonlinear compression function. j Then, the probability value n of emotion capsule j is obtained. j .

[0033]

[0034] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is: to provide a brainwave emotion recognition and detection system based on a lightweight residual convolutional capsule network, comprising:

[0035] The EEG signal input module is used to extract the raw waveform of the EEG signal and perform data preprocessing, which serves as the input feature for predicting the emotion recognition level.

[0036] The feature processing module is used to model the interdependencies between feature channels using a feature extractor based on a residual convolutional network, and automatically extract deep emotional features.

[0037] The capsule classifier module is used to convert feature data into original capsules using convolutional capsule layers, promoting the inclusion of more important emotional information in the capsules; it uses an emotion capsule layer as a classifier, and obtains the corresponding prediction result by using the length of the capsule output vector to represent the probability of the feature appearing in the current capsule layer.

[0038] In a preferred embodiment of the present invention, the specific steps of the EEG signal input module in extracting the original waveform of the EEG signal and performing data preprocessing include:

[0039] Remove the baseline and split the data using a sliding window:

[0040]

[0041]

[0042] Among them, X p This represents data segmented using a sliding window. X′ represents the baseline signal per second. q This indicates the time slice from which the baseline signal per second is removed.

[0043] In a preferred embodiment of the present invention, the feature processing module employs a feature extractor based on a residual convolutional network to model the interdependencies between feature channels, and the specific steps for automatically extracting deep emotional features include:

[0044] The input data is convolved and downsampled to extract its spatiotemporal features; using four residual convolutional blocks, the number of channels in the feature map is increased to 512, which can capture more deep emotional features.

[0045] y = F(x, {W i})+x (3)

[0046] F=σ(W1x) (4)

[0047] Where x represents the input, and the function F(x, {W}) is... i}) represents the residual mapping to be learned, W1 represents the weight matrix, σ represents the ReLU activation function, and F+x is executed by concatenating shortcuts and adding elements.

[0048] In a preferred embodiment of the present invention, the capsule classifier module uses convolutional capsule layers to convert feature data into raw capsules, and the specific steps to promote the inclusion of more important emotional information in the capsules include:

[0049] First, the feature map is convolved with 512 3×3 convolution kernels with a stride of 2, and the shape of the output feature map is (8, 2, 512).

[0050] Secondly, the above feature map is divided into 64 sub-feature maps with a channel dimension of 8, i.e., the shape is (8, 2, 8). Taking one of the sub-feature maps as an example, its tensor data structure can be regarded as multiple 8-dimensional vectors arranged in an 8×2 grid. That is, each carrier (vector) represents a primary capsule containing 8 elements. Through the calculation of the convolutional capsule layer, a total of 1024 original capsules are obtained. These original capsules will be used as the input of the emotion capsule layer.

[0051] In a preferred embodiment of the present invention, the capsule classifier module uses an emotion capsule layer as the classifier, and obtains the corresponding prediction result by using the length of the capsule output vector to indicate the probability of the feature appearing in the current capsule layer. The specific steps include:

[0052] The entire original capsule output from the convolutional capsule layer is routed to multiple capsules containing 16 elements, implemented by a dynamic routing algorithm:

[0053] First, using the weight matrix W ij Learn the input features and compute the primary capsule m. i Prediction vector for emotion capsule j

[0054]

[0055] Secondly, by applying all those with different weights Summation is performed to generate capsule S j :

[0056]

[0057]

[0058] c ij =softrmax(b ij (8)

[0059] In this process, updating the capsule network actually involves calculating the coupling coefficient, c.ij This represents the coupling coefficient between main capsule i and emotion capsule j, and its value can be updated by b in each iteration. ii The value is used to calculate, and then a softmax operation is performed to obtain the result;

[0060] Finally, the capsule S is compressed using a nonlinear compression function. i The length of the capsule is such that the length of the short capsule approaches 0 and the length of the long capsule approaches 1. S is calculated using a nonlinear compression function. j Then, the probability value n of emotion capsule j is obtained. i .

[0061]

[0062] The beneficial effects of this invention are:

[0063] (1) This invention proposes a lightweight residual convolutional capsule network (Light-ResNet-Capsule) for emotion recognition. This network uses raw EEG signals as input features because raw signals contain more emotional information. Using raw signals can extract more discriminative features, thereby improving prediction accuracy.

[0064] (2) In addition, in order to obtain deep distinguishable features of emotions, a feature extractor based on residual convolutional network (Light-ResNet) is used to model the interdependence between feature channels to obtain deep emotion features.

[0065] (3) Finally, a convolutional capsule layer is used to convert the feature data into original capsules, which promotes the inclusion of more emotional information in the capsules; an emotion capsule layer is used as a classifier, and the probability of the feature appearing in the current capsule layer is represented by the length of the capsule output vector to obtain the corresponding prediction result;

[0066] (4) This invention is very helpful for the detection of EEG emotion recognition and can significantly improve the accuracy of EEG emotion recognition detection technology. Attached Figure Description

[0067] Figure 1 This is a flowchart of the EEG emotion recognition and detection method based on lightweight residual convolutional capsule networks of the present invention;

[0068] Figure 2 This is a schematic diagram of the Light-ResNet-Capsule model of the EEG emotion recognition and detection method and system based on lightweight residual convolutional capsule network;

[0069] Figure 3 This is a block diagram of the EEG emotion recognition and detection system based on a lightweight residual convolutional capsule network. Detailed Implementation

[0070] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0071] Please see Figure 1 and Figure 2 The embodiments of the present invention include:

[0072] A method for EEG emotion recognition and detection based on lightweight residual convolutional capsule networks includes the following steps:

[0073] S1: Extract the raw waveform of the EEG signal and perform data preprocessing, which serves as the input feature for predicting the emotion recognition level.

[0074] The data preprocessing steps for extracting the raw waveforms of EEG signals include the following: removing the baseline and segmenting the data using a sliding window:

[0075]

[0076]

[0077] Among them, X p This represents data segmented using a sliding window. x′ represents the baseline signal per second. q This indicates the time slice from which the baseline signal per second is removed.

[0078] Preferably, the EEG signal includes a baseline signal and a stimulation signal. During the preprocessing, the baseline signal is removed for 3 seconds, and a 1-second sliding window is selected to divide the EEG signal into several segments. The sampling frequency of the signal is 128Hz.

[0079] S2: A feature extractor based on a residual convolutional network is used to model the interdependencies between feature channels, automatically extracting deep emotional features; such as... Figure 2 As shown in the Light-ResNet-based feature extractor, this module includes 4 residual convolutional blocks.

[0080] The input data is convolved and downsampled to extract its spatiotemporal features; using four residual convolutional blocks, the number of channels in the feature map is increased to 512, which can capture more deep emotional features.

[0081] y = F(x, {W i})+x (3)

[0082] F=σ(W1x) (4)

[0083] Where x represents the input, and the function F(x, {W}) is... i}) represents the residual mapping to be learned, W1 represents the weight matrix, σ represents the ReLU activation function, and F+x is executed by concatenating shortcuts and adding elements.

[0084] It should be noted that, in order to minimize model parameters and avoid overfitting, the ResNet18 network is used as the basic framework, and the number of filters is reduced to implement Light-ResNet, such as... Figure 2 As shown, Light-ResNet consists of four blocks (N=4), each of which is a residual learning framework. Each residual learning framework contains a 3×3 filter, a non-linear activation layer, and two normalization layers. Gradients are passed to lower layers through skip connections in residual learning, allowing for deeper features to be obtained in a relatively simple model. Max pooling layers are removed from this residual learning framework because CNNs lose a significant amount of information in pooling layers, causing small changes in the input to remain almost constant.

[0085] S3: Convolutional capsule layers are used to convert the deep emotional feature data obtained in S2 into original capsules, which helps the capsules contain more important emotional information. The length and orientation of the original capsules reflect the existence and characteristics of the related deep features.

[0086] S301: The feature maps are first convolved with 512 3×3 convolutional kernels. These kernels have a stride of 2, so their output feature maps have a shape of (8, 2, 512).

[0087] S302: Divide these feature maps into 64 sub-feature maps with a channel dimension of 8, i.e., a shape of (8, 2, 8). Taking one of the sub-feature maps as an example, its tensor data structure can be viewed as multiple 8-dimensional vectors arranged in an 8×2 grid, i.e., each vector represents a primary capsule containing 8 elements. Through the calculation of the convolutional capsule layer, a total of 1024 primary capsules are obtained, which will be used as the input of the emotion capsule layer.

[0088] S4: The entire original capsule output by the convolutional capsule layer is routed to multiple capsules containing 16 elements. The sentiment capsule layer is used as a classifier. The weight matrix is ​​used to capture the positional relationship between local features and strengthen the connection between capsules. The length of the capsule output vector represents the probability of the feature appearing in the current capsule layer, and the corresponding prediction result is obtained.

[0089] Specifically, the entire original capsule output by the convolutional capsule layer is routed to multiple capsules containing 16 elements, implemented by a dynamic routing algorithm:

[0090] S401: Using the weight matrix Wij Learn the input features and compute the primary capsule m. i Prediction vector for emotion capsule j

[0091]

[0092] By calculating the weights to obtain the importance of each input feature, and selectively assigning weights to features based on their importance, useful features are improved while features that are not useful for the current task are suppressed, thereby improving the quality of deep emotion feature representation.

[0093] S402: By applying all those with different weights Summation is performed to generate capsule S j :

[0094]

[0095]

[0096] c ij =softmax(b ij (8)

[0097] In this process, updating the capsule network actually involves calculating the coupling coefficient, c. ij This represents the coupling coefficient between main capsule i and emotion capsule j, and its value can be updated by b in each iteration. ij The value is used to calculate, and then a softmax operation is performed to obtain the result.

[0098] S403: Compressing capsule S using a nonlinear squashing function. j The length of the capsule is such that the length of the short capsule approaches 0, and the length of the long capsule approaches 1. S is calculated using a nonlinear compression function. j Then, the probability value n of emotion capsule j is obtained. j :

[0099]

[0100] It should be noted that in step S4, the weight matrix W ii The size is 8*16. Furthermore, the extractor uses Adam as the optimizer, DEAP has an input size of 128*32 and a learning rate of 0.00001, and DREAMER has an input size of 128*14 and a learning rate of 0.0001. The raw EEG signals are used as input to the deep neural network classifier, which is then trained for a certain number of epochs. Finally, the best-performing model from the training is selected for testing, and the test results are used as the corresponding prediction results.

[0101] In the embodiments of the present invention, see Figure 3 Furthermore, a brainwave emotion recognition and detection system based on a lightweight residual convolutional capsule network is provided, comprising:

[0102] The EEG signal input module is used to extract the raw waveform of the EEG signal as input features for predicting the emotion recognition level.

[0103] The feature processing module uses a feature extractor based on residual convolutional networks to model the interdependencies between feature channels and automatically extract deep emotional features.

[0104] The capsule classifier module uses convolutional capsule layers to convert feature data into original capsules, promoting the inclusion of more important emotional information in the capsules; it uses an emotion capsule layer as a classifier, and obtains the corresponding prediction result by using the length of the capsule output vector to represent the probability of the feature appearing in the current capsule layer.

[0105] Using the method described in this invention, experiments were conducted on two publicly available datasets, DEAP and DREAMER. To quantitatively evaluate the emotion recognition results, mean accuracy and standard deviation (std) were used as evaluation metrics.

[0106] Table 1

[0107]

[0108] Table 2

[0109]

[0110] In this invention, the Light-Resnet-Capsule mainly consists of raw EEG signals, a feature extractor, and a capsule classifier.

[0111] Tables 1 and 2 show the mean accuracy and standard deviation of the subjects' arousal, valence, and dominance classification tasks. The results show that this proposed method achieves better performance on all three classification tasks compared to six other methods. First, as shown in Table 1, compared to DT, SVM, and MLP, the method of this invention improves the mean accuracy on the three classification tasks by 20%, 22%, and 18%, respectively. Second, compared to 3DCNN and DGCNN, the method of this invention achieves mean accuracy improvements of approximately 6%, 7%, and 6% on the three classification tasks. The results indicate that in emotion recognition tasks, reducing model parameters and adding residual connections can improve the accuracy and effectiveness of the task. Furthermore, compared to the original CapsNet, the proposed Light-Resnet-Capsule shows improved accuracy and stability on both arousal and valence tasks. As shown in Table 2, on the DREAMER dataset, compared to DT, SVM, and MLP, the method of this invention improves the mean accuracy on the three classification tasks by approximately 16%, 16%, and 16%, respectively. Second, compared with 3DCNN and DGCNN, the method described in this invention improves the average accuracy by 10%, 9%, and 10% on the three classification tasks, respectively. Furthermore, compared with the original CapsNet, the proposed Light-Resnet-Capsule demonstrates higher accuracy and stability on the three classification tasks. These results prove the effectiveness of the method proposed in this invention.

[0112] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for EEG emotion recognition and detection based on lightweight residual convolutional capsule networks, characterized in that, Includes the following steps: S1: Extract the raw waveform of the EEG signal and perform data preprocessing, which serves as the input feature for predicting the emotion recognition level; S2: A feature extractor based on residual convolutional networks is used to model the interdependencies between feature channels and automatically extract deep emotional features. S3: Convolutional capsule layers are used to convert the deep emotional feature data obtained in S2 into original capsules, promoting the inclusion of more important emotional information in the capsules. The length and orientation of the original capsules reflect the existence and characteristics of the associated deep features. Specific steps include: S301: The feature map is convolved with 512 3×3 convolution kernels with a stride of 2, and the shape of the output feature map is (8, 2, 512). S302: Divide the above feature map into 64 sub-feature maps with a channel dimension of 8, i.e., the shape is (8, 2, 8). Taking one of the sub-feature maps as an example, its tensor data structure can be regarded as multiple 8-dimensional vectors arranged in an 8×2 grid. That is, each carrier (vector) represents a primary capsule containing 8 elements. Through the calculation of the convolutional capsule layer, a total of 1024 primary capsules are obtained. These primary capsules will be used as the input of the emotion capsule layer. S4: An emotion capsule layer is used as a classifier. A weight matrix is ​​used to capture the positional relationships between local features, strengthening the connections between capsules. The length of the capsule output vector represents the probability of that feature appearing in the current capsule layer, thus obtaining the corresponding prediction result. Specific steps include: routing the entire original capsule output from the convolutional capsule layer to multiple capsules containing 16 elements, implemented by a dynamic routing algorithm. S401: Using the weight matrix Learn input features and compute primary capsules. Prediction vector for emotion capsule j : (5) S402: By applying all those with different weights Summation to generate capsules : (6) (7) (8) The update of the capsule network is actually the calculation of the coupling coefficient. This represents the coupling coefficient between main capsule i and emotion capsule j, and its value can be updated in each iteration. The value is used to calculate, and then a softmax operation is performed to obtain the result; S403: Compressing capsules using a nonlinear compression function The length of the capsule is such that the length of the short capsule approaches 0 and the length of the long capsule approaches 1, calculated using a nonlinear compression function. Then, the probability value of emotion capsule j is obtained. ; (9)。 2. The EEG emotion recognition and detection method based on lightweight residual convolutional capsule networks according to claim 1, characterized in that, In step S1, the specific steps for extracting the raw waveform of the EEG signal and performing data preprocessing include: Remove the baseline and split the data using a sliding window: (1) (2) in, This represents data segmented using a sliding window. This represents the baseline signal per second. This indicates the time slice from which the baseline signal per second is removed.

3. The EEG emotion recognition and detection method based on lightweight residual convolutional capsule networks according to claim 1, characterized in that, The specific steps of step S2 include: The input data is convolved and downsampled to extract its spatiotemporal features; using four residual convolutional blocks, the number of channels in the feature map is increased to 512, which can capture more deep emotional features. (3) (4) Where x represents the input, and the function This represents the residual mapping to be learned. Represents the weight matrix. This represents the ReLU activation function, F + x, which is executed by concatenating shortcuts and adding elements.

4. A brainwave emotion recognition and detection system based on a lightweight residual convolutional capsule network, characterized in that, include: The EEG signal input module is used to extract the raw waveform of the EEG signal and perform data preprocessing, which serves as the input feature for predicting the emotion recognition level. The feature processing module is used to model the interdependencies between feature channels using a feature extractor based on a residual convolutional network, and automatically extract deep emotional features. The capsule classifier module uses convolutional capsule layers to transform feature data into raw capsules, promoting the inclusion of more important sentiment information in the capsules. Specific steps include: First, the feature map is convolved with 512 3×3 convolution kernels with a stride of 2, and the shape of the output feature map is (8, 2, 512). Secondly, the above feature map is divided into 64 sub-feature maps with a channel dimension of 8, i.e., the shape is (8, 2, 8). Taking one of the sub-feature maps as an example, its tensor data structure can be regarded as multiple 8-dimensional vectors arranged in an 8×2 grid. That is, each carrier (vector) represents a primary capsule containing 8 elements. Through the calculation of the convolutional capsule layer, a total of 1024 original capsules are obtained. These original capsules will be used as the input of the emotion capsule layer. The emotion capsule layer is used as a classifier. The length of the capsule output vector represents the probability of the feature appearing in the current capsule layer, and the corresponding prediction result is obtained. The specific steps include: The entire original capsule output from the convolutional capsule layer is routed to multiple capsules containing 16 elements, implemented by a dynamic routing algorithm: First, using the weight matrix Learn input features and compute primary capsules. Prediction vector for emotion capsule j : (5) Secondly, by applying all those with different weights Summation to generate capsules : (6) (7) (8) The update of the capsule network is actually the calculation of the coupling coefficient. This represents the coupling coefficient between main capsule i and emotion capsule j, and its value can be updated in each iteration. The value is used to calculate, and then a softmax operation is performed to obtain the result; Finally, the capsule is compressed using a nonlinear compression function. The length of the capsule is such that the length of the short capsule approaches 0 and the length of the long capsule approaches 1, calculated using a nonlinear compression function. Then, the probability value of emotion capsule j is obtained. : (9)。 5. The EEG emotion recognition and detection system based on lightweight residual convolutional capsule networks according to claim 4, characterized in that, The specific steps for the EEG signal input module to extract the raw waveform of the EEG signal and perform data preprocessing include: Remove the baseline and split the data using a sliding window: (1) (2) in, This represents data segmented using a sliding window. This represents the baseline signal per second. This indicates the time slice from which the baseline signal per second is removed.

6. The EEG emotion recognition and detection system based on lightweight residual convolutional capsule networks according to claim 4, characterized in that, The feature processing module uses a feature extractor based on a residual convolutional network to model the interdependencies between feature channels. The specific steps for automatically extracting deep emotional features include: The input data is convolved and downsampled to extract its spatiotemporal features; using four residual convolutional blocks, the number of channels in the feature map is increased to 512, which can capture more deep emotional features. (3) (4) Where x represents the input, and the function This represents the residual mapping to be learned. Represents the weight matrix. This represents the ReLU activation function, F + x, which is executed by concatenating shortcuts and adding elements.

Images