Multimodal emotion recognition method based on recurrent capsule network based on channel expansion and fusion

Abstract

The present invention discloses a multimodal emotion recognition method based on a recurrent capsule network with channel expansion and fusion, specifically comprising the following steps: extracting audio modal feature sequences, visual modal feature sequences, and text modal feature sequences from each video sample; inputting the feature sequences one by one into a time-domain convolutional network to obtain modal feature sequences, visual modal feature sequences, and text modal feature sequences containing contextual information; utilizing a multi-channel deformable convolutional network to learn from the above modal feature sequences one by one to obtain audio modal feature sequences, visual modal feature sequences, and text modal feature sequences containing high-level semantic information; utilizing a constructed multimodal semantic capsule fusion network to obtain global information capsules, which are input into a bidirectional long short-term memory network for compression, and finally, are fed into a dense layer for multimodal emotion classification. The recurrent capsule network based on channel expansion and fusion of the present invention is reasonable and effective, and can greatly improve the accuracy of multimodal emotion recognition.

Description

Technical Field

[0001] The present invention belongs to the field of emotion recognition technology, and specifically relates to a recurrent capsule network multimodal emotion recognition method based on channel expansion and fusion. Background Art

[0002] In recent years, with the advancement of computer hardware resources, the field of artificial intelligence has experienced rapid development. Today, AI has become integrated into every aspect of life. For example, multimodal emotion recognition in AI can effectively identify the emotional state of a subject by combining information from multiple modalities. It plays a vital role in scenarios such as medical diagnosis, intelligent driving, human-computer interaction, and public opinion monitoring. However, while multimodal emotion recognition has performed well in various industrial applications, two challenges remain. The first is how to effectively extract high-level semantic information from feature sequences of each modality, providing a more powerful high-level representation that incorporates more in-depth semantic information for the subsequent fusion stage. The second is how to learn the underlying relationships between feature sequences of different modalities and effectively fuse them to obtain a more representative multimodal fusion representation. Summary of the Invention

[0003] The purpose of the present invention is to provide a multimodal emotion recognition method based on a recurrent capsule network with channel expansion and fusion, which processes each modal feature sequence by channel expansion and fusion to effectively extract high-level semantic information of each modal feature sequence.

[0004] The technical solution adopted by the present invention is a multimodal emotion recognition method based on a recurrent capsule network with channel expansion and fusion, which is specifically implemented in the following steps:

[0005] Step 1: Extract multiple video samples from the multimodal emotional video database and extract audio modality feature sequences from each video sample. Visual modality feature sequence and text modality feature sequences

[0006] T is the sequence length of each modal feature sequence, d a d l and d v are the feature vector dimensions of audio modality, text modality, and visual modality respectively;

[0007] Step 2: Establish three time domain convolutional networks respectively, and transform the audio modal feature sequence Z of each video sample into a , visual modality feature sequence Z v and text modality feature sequence Z lThey are respectively sent into the corresponding time domain convolutional network to learn the time domain relationship of each modal feature sequence. The feature vector dimension of each modal feature sequence is unified to d. Finally, the audio modal feature sequence X containing context information is obtained. a , visual modality feature sequence X v and text modality feature sequence X l ;

[0008] Step 3: Construct L multi-channel deformable convolutional networks and stack them together in sequence. Each multi-channel deformable convolutional network consists of three “channel expansion” modules and one “channel fusion” module. Then, the audio modality feature sequence X containing contextual information is a , visual modality feature sequence X v and text modality feature sequence X l At the same time, it is input into the stacked multi-channel deformable convolutional network for processing to obtain an audio modality feature sequence containing high-level semantic information. Visual modality feature sequence and text modality feature sequences

[0009] Step 4: Construct a multimodal semantic capsule fusion network to obtain the global information capsule V;

[0010] Step 5: Input the global semantic capsule V into the bidirectional long short-term memory network for compression to obtain compressed multimodal fusion

[0011] Step 6: Fuse the compressed multimodal Feed it into the dense layer for multimodal sentiment classification.

[0012] The present invention is also characterized in that:

[0013] In step 3, each "channel expansion" module is composed of a convolutional neural network with convolution kernel sizes of 3×3 and 5×5 and a maximum pooling layer with a pooling size of 3×3; the "channel fusion" module is composed of five deformable convolutional networks with a convolution kernel size of 3×3;

[0014] The specific processing process of each multi-channel deformable convolutional network is:

[0015] Step A: The audio modality feature sequence containing contextual information Visual modality feature sequence and text modality feature sequences They are input into the "channel expansion" module respectively; in the "channel expansion" module, convolution kernels of sizes 3×3 and 5×5 and maximum pooling operations of size 3×3 are used to process each modal feature sequence respectively, and semantic information of different granularities of each modal feature sequence is extracted. The specific calculation process is shown in formulas (1)-(3):

[0016]

[0017]

[0018]

[0019] Among them, i∈{a, v, l}, 2Conv represents a two-dimensional convolutional neural network, and These are the output results of 3×3 convolution, 5×5 convolution, and 3×3 maximum pooling, respectively. is the number of feature channels after the 3×3 convolution operation, is the number of feature channels after the 5×5 convolution operation, is the number of feature channels after using the 3×3 maximum pooling operation;

[0020] Step B: and Respectively with X i Stitched together to get the final output of the "Channel Expansion" module That is, and The specific calculation process is shown in formula (4):

[0021]

[0022] Among them, C is the dimension of the feature channel after splicing, and the calculation formula is

[0023] Step C: Output of the three "Channel Expansion" modules and The input is sent to the "channel fusion" module, in which the channel fusion is firstly categorized according to the granularity of the semantic information. and Divide and splice, and send them to the corresponding deformable convolutional network for processing. The specific calculation process is shown in formulas (5)-(9):

[0024]

[0025]

[0026]

[0027]

[0028]

[0029] Among them, X 3×3 、X 5×5 、X pool and X avl They are the results of fusion of feature sequences of the same granularity in each modality. DeforConv is represented by a deformable convolutional network. and These are the feature sequences of the audio modality, text modality, and visual modality output by the “channel fusion” module respectively;

[0030] Step D: Introduce the residual network to ensure that the multi-channel deformable convolutional network does not experience performance degradation as the number of network layers increases. The specific calculation process is shown in formula (10):

[0031]

[0032] in, represents the matrix addition operation, and are the feature sequences of each modality output by the multi-channel deformable convolutional network;

[0033] Step E: and As input, repeat steps A-D L times to get the final output of the stacked multi-channel deformable convolutional network and

[0034] In step 4, specifically:

[0035] Step A: Sequence the audio modal features Visual modality feature sequence Text modality feature sequence Audio modal feature sequence X a , visual modality feature sequence X v and text modality feature sequence X l Splice together according to dimension T to get the output feature sequence Then divide the feature sequence X from dimension T to obtain T feature sequences where i∈{1,2,...,};

[0036] Step B: Use one-dimensional convolutional neural network to process feature sequences separately Get T local semantic information capsules Where r is the feature dimension after one-dimensional convolution;

[0037] Step C: Give each local information capsule an initial weight value, and sum these local information capsules and the corresponding weight values to obtain a new feature sequence. It is called the global information capsule; then calculate the global information capsule P and each local information capsule U i The Pearson correlation coefficient between them is used to update the weight value corresponding to each local information capsule; secondly, all updated weight values are re-weighted and summed with the corresponding local information capsule to obtain a new global information capsule. Finally, calculate the new global information capsule M and each local information capsule U i The Pearson correlation coefficient is calculated and the corresponding weight value is updated continuously. Repeat N times to obtain the final output global information capsule. Where j∈{1,2,...,}, the specific calculation process is shown in equations (11)-(15):

[0038]

[0039]

[0040]

[0041]

[0042]

[0043] in, represents the weight value, For the middle capsule, c ij Indicates the capsule coefficient, used for the intermediate capsule Weighted summation generates global information capsule j j , b ij is 1, “←” means updating the value. yes and V j The Pearson correlation coefficient, yes The mean of It is V j The mean of yes The mean of It is V j 2 The mean of yes The mean of .

[0044] In step 6, the dense layer consists of a fully connected layer with ReLU activation, two linear fully connected layers and a Dropout layer.

[0045] The beneficial effects of the present invention are:

[0046] 1) This invention innovatively proposes a network for learning high-level semantic information of feature sequences of each modality, namely a multi-channel deformable convolutional network. This network can fully extract semantic information of different granularities of each modality feature sequence through "channel expansion" and "channel fusion", and learn the implicit semantic information in each modality feature sequence by combining the semantic information of other modalities through a deformable convolutional network. In addition, stacking multiple multi-channel deformable convolutional networks to repeatedly process each modality feature sequence can fully extract the high-level semantic information of each modality feature sequence, making the learned high-level representation of each modality more representative and containing more abstract semantic information;

[0047] 2) Compared with previous emotion recognition work using capsule networks, this paper improves on standard capsule networks by combining all modal feature sequences for each word to generate local information capsules. Furthermore, the capsules are converted from vector form to matrix form, enhancing the representational capabilities of the capsule network. Furthermore, the Pearson correlation coefficient is used to calculate the relationship between local information capsules and global information capsules, making the generation process of global information capsules more rigorous and reasonable.

[0048] 3) Through a large number of experimental analyses and verifications, the recurrent capsule network based on channel expansion and fusion proposed in this invention is reasonable and effective, and can greatly improve the accuracy of multimodal emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of the multimodal emotion recognition method of the recurrent capsule network based on channel expansion and fusion of the present invention;

[0050] Figure 2 It is a structural diagram of the "channel expansion" module of the present invention;

[0051] Figure 3 It is a structural diagram of the "channel fusion" module of the present invention;

[0052] Figure 4 is a structural diagram of the multimodal semantic capsule fusion network of the present invention;

[0053] Figure 5 It is a flow chart of the division of the characteristic sequence X of the present invention. DETAILED DESCRIPTION

[0054] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] Example 1

[0056] The present invention is based on a multimodal emotion recognition method of a recurrent capsule network based on channel expansion and fusion, such as Figure 1 As shown, please follow the steps below:

[0057] Step 1: Extract multiple video samples from the multimodal emotional video database and extract audio modality feature sequences from each video sample using the multimodal software development kit. Visual modality feature sequence and text modality feature sequences

[0058] Among them, T is the sequence length of each modal feature sequence, d a d l and d v are the feature vector dimensions of audio modality, text modality, and visual modality respectively;

[0059] Multimodal emotional video databases include CMU-MOSI, CMU-MOSEI, and IEMOCAP;

[0060] Step 2: Establish three time domain convolutional networks respectively, and transform the audio modal feature sequence Z of each video sample into a , visual modality feature sequence Z v and text modality feature sequence Z l They are respectively sent to the corresponding established time domain convolutional network to learn the time domain relationship of each modal feature sequence and filter out the redundant information in each modal feature sequence. In addition, through the time domain convolutional network processing, the feature vector dimension of each modal feature sequence is unified to d. Finally, the audio modal feature sequence X containing context information is obtained a , visual modality feature sequence X v and text modality feature sequence X l ;

[0061] Step 3: Construct L multi-channel deformable convolutional networks and stack them together in sequence. Each multi-channel deformable convolutional network consists of three “channel expansion” modules and one “channel fusion” module. Then, the audio modality feature sequence X containing contextual information is a , visual modality feature sequence X v and text modality feature sequence X l At the same time, it is input into the stacked multi-channel deformable convolutional network for processing. In each multi-channel deformable convolutional network, each modal feature sequence X a 、X v and X lFirst, they are sent to the corresponding "channel expansion" module, and multi-granularity semantic information is extracted through convolution kernels of different sizes. Then, the output feature sequences of all "channel expansion" modules are sent to the "channel fusion" module for further learning and processing, and finally the audio modal feature sequence containing high-level semantic information is obtained. Visual modality feature sequence and text modality feature sequences

[0062] Among them, each "channel expansion" module is composed of a convolutional neural network with a convolution kernel size of 3×3 and 5×5 and a maximum pooling layer with a pooling size of 3×3. The detailed structure is as follows Figure 2 As shown in the figure, the “channel fusion” module is composed of five deformable convolution networks with a convolution kernel size of 3×3. The detailed structure is shown in the figure. Figure 3 As shown; specifically, the specific process of each multi-channel deformable convolutional network is:

[0063] Step A: The audio modality feature sequence containing contextual information Visual modality feature sequence and text modality feature sequences They are input into the "channel expansion" module respectively; in the "channel expansion" module, convolution kernels of sizes 3×3 and 5×5 and maximum pooling operations of size 3×3 are used to process each modal feature sequence respectively, and semantic information of different granularities of each modal feature sequence is extracted. The specific calculation process is shown in formulas (1)-(3):

[0064]

[0065]

[0066]

[0067] Among them, i∈{a, v, l}, 2Conv represents a two-dimensional convolutional neural network, and These are the output results of 3×3 convolution, 5×5 convolution, and 3×3 maximum pooling, respectively. is the number of feature channels after the 3×3 convolution operation, is the number of feature channels after the 5×5 convolution operation, is the number of feature channels after the 3×3 maximum pooling operation.

[0068] Step B: and Respectively with X i Stitched together to get the final output of the "Channel Expansion" module That is, and The specific calculation process is shown in formula (4):

[0069]

[0070] Among them, C is the dimension of the feature channel after splicing, and the calculation formula is

[0071] Step C: Output of the three "Channel Expansion" modules and The input is sent to the "channel fusion" module, in which the channel fusion is firstly categorized according to the granularity of the semantic information. and Divide and splice, and send them to the corresponding deformable convolutional network for processing. The specific calculation process is shown in formulas (5)-(9):

[0072]

[0073]

[0074]

[0075]

[0076]

[0077] Among them, X 3×3 、X 5×5 、X pool and X avl They are the results of fusion of feature sequences of the same granularity in each modality. DeforConv is represented by a deformable convolutional network, and its structure is shown in Table 1. and These are the feature sequences of the audio modality, text modality, and visual modality output by the “channel fusion” module respectively.

[0078] Step D: In order to prevent network degradation, a residual network is introduced to ensure that the multi-channel deformable convolutional network will not experience performance degradation as the number of network layers increases. The specific calculation process is shown in formula (10):

[0079]

[0080] in, represents the matrix addition operation, and They are the feature sequences of each modality output by the multi-channel deformable convolutional network.

[0081] Step E: and As input, repeat steps A-D L times to get the final output of the stacked multi-channel deformable convolutional network and

[0082] Table 1 Structure of deformable convolutional network, 1 and C2 represent the number of channels of feature map

[0083] Layer Kernel Activation Filters Padding Output Input - - - - <![CDATA[C1×T×d <!-- 6 -->]]> Conv 1 3×3 - <![CDATA[C1×2]]> 1 <![CDATA[C1×2×T×d]]> Offset 1 - - - - <![CDATA[C1×T×d]]> Conv 2 3×3 ReLU <![CDATA[C2]]> 1 <![CDATA[C2×T×d]]> Batch Norm 1 - - - - <![CDATA[C2×T×d]]> Conv 3 3×3 - <![CDATA[C2×2]]> 1 <![CDATA[C2×2×T×d]]> Offset 2 - - - - <![CDATA[C2×T×d]]> Conv 4 3×3 ReLU 1 1 1×T×d Batch Norm 2 - - - - 1×T×d

[0084] Step 4: Construct a multimodal semantic capsule fusion network, such as Figure 4 As shown in the figure, the network divides the input feature sequence into T parts, processes each part using a one-dimensional convolutional neural network, and outputs T feature sequences, which are called local information capsules; then, each local information capsule is given a weight value, and these local information capsules are weighted and summed to obtain a new feature sequence, which is called a global information capsule; secondly, the Pearson correlation coefficient of the global information capsule and each local information capsule is calculated, and the Pearson correlation coefficient is used to update the corresponding weight value, and then the updated weight value is used to re-weight and sum with the corresponding local information capsule to obtain a new global information capsule; finally, the Pearson correlation coefficient of the new global information capsule and each local information capsule is calculated, and the corresponding weight value is updated, and this is repeated N times, and the global information capsule obtained for the last time is output;

[0085] Step 5: Input the global semantic capsule V into the bidirectional long short-term memory network for compression to obtain compressed multimodal fusion The specific calculation process is shown in formula (16):

[0086]

[0087] Among them, Bi-LSTM represents a bidirectional long short-term memory network; through this method, the number of parameters of the model is reduced from (b*6*r) 2 Reduced to (6*) 2 , reducing b 2 In addition, since the number of global information capsules b is very small, there will be no long-range dependency problem in the learning process of the bidirectional long short-term memory network.

[0088] Step 6: Fuse the compressed multimodal The data is sent to the dense layer for multimodal sentiment classification. The specific calculation process is shown in formula (17):

[0089]

[0090] Where Dense represents a dense layer, which consists of a fully connected layer with ReLU activation, two linear fully connected layers, and a Dropout layer. The dimension of o varies for different datasets and target tasks.

[0091] Example 2

[0092] The difference from Example 1 is that the specific construction process of the multimodal semantic capsule fusion network is:

[0093] Step A: Audio modal feature sequence Visual modality feature sequence Text modality feature sequence Audio modal feature sequence X a , visual modality feature sequence X v and text modality feature sequence X l Splice together according to dimension T to get the output feature sequence Then the feature sequence X is divided from dimension T. The specific division steps are as follows: Figure 5 As shown, T feature sequences are obtained where i∈{1,2,...,}.

[0094] Step B: Use one-dimensional convolutional neural network to process feature sequences separately Get T local semantic information capsules Where r is the feature dimension after one-dimensional convolution.

[0095] Step C: Give each local information capsule an initial weight value, and sum these local information capsules and the corresponding weight values to obtain a new feature sequence. It is called the global information capsule; then calculate the global information capsule P and each local information capsule U i The Pearson correlation coefficient between them is used to update the weight value corresponding to each local information capsule; secondly, all updated weight values are re-weighted and summed with the corresponding local information capsule to obtain a new global information capsule. Finally, calculate the new global information capsule M and each local information capsule U i The Pearson correlation coefficient is calculated and the corresponding weight value is updated continuously. Repeat N times to obtain the final output global information capsule. Where j∈{1,2,...,}, the specific calculation process is shown in equations (11)-(15):

[0096]

[0097]

[0098]

[0099]

[0100]

[0101] in, represents the weight value, For the middle capsule, c ij Indicates the capsule coefficient, used for the intermediate capsule Weighted summation generates global information capsule V j , b ij It is a scalar with an initial value of 1, used to measure How much information should be sent to V j ,when and V j The more similar they are, the better. j Include The more semantic information there is, the The greater the contribution of b ij The larger it should be, the “←” indicates updating the value. is and V j The Pearson correlation coefficient, yes The mean of It is V j The mean of yes The mean of yes The mean of yes The mean of .

[0102] Example 3

[0103] The present invention conducts a large number of experiments and analyzes and discusses on three multimodal emotion datasets: CMU-MOSI, CMU-MOSEI, and IEMOCAP, to evaluate the performance of the present invention in various indicators.

[0104] The experimental results are compared as follows:

[0105] For the CMU-MOSI and CMU-MOSEI datasets, the present invention treats it as a regression task, so the evaluation indicators include the common correlation coefficient (Corr) and mean absolute error (MAE). In addition, since the value range of the continuous value labels of the CMU-MOSI and CMU-MOSEI datasets is between [-3,3], most multimodal emotion recognition works divide this value range to obtain new evaluation indicators: seven-category accuracy (Acc7, [-3,3] is divided into seven categories), two-category accuracy (Acc2, [-3,3] is divided into two categories) and F1 score (F1). In order to compare the performance with most models more fairly, the present invention also chooses to use these three indicators to evaluate the performance of the recurrent capsule network based on channel expansion and fusion. For the IEMOCAP dataset, the present invention follows the evaluation method of most existing works and uses accuracy (Acc) and F1 to evaluate the recognition performance of the recurrent capsule network based on channel expansion and fusion on this dataset.

[0106] The performance comparison results of different network models on the CMU-MOSI dataset are shown in Table 2; the performance comparison results of different network models on the CMU-MOSEI dataset are shown in Table 3; the performance comparison results of different network models on the IEMOCAP dataset are shown in Table 4.

[0107] Table 2 Performance comparison of different network models on the CMU-MOSI dataset

[0108] Methods <![CDATA[Acc7]]> <![CDATA[Acc2]]> <![CDATA[F1]]> Corr MAE EF-LSTM 33.7 75.3 75.2 0.608 1.023 LF-LSTM 35.3 76.8 76.7 0.625 1.015 TFN 34.9 75.6 75.5 0.605 1.009 LMF 30.5 75.3 75.2 0.605 1.018 QMF 33.5 79.7 79.6 0.696 0.915 CIA 38.9 79.8 79.5 0.689 0.914 MCTN 35.6 79.3 79.1 0.676 0.909 MFM 31.9 76.2 75.8 0.622 0.988 RAVEN 33.8 78.8 76.9 0.667 0.968 MARN 34.7 77.1 77.0 0.625 0.968 RCN-CEF 34.4 81.1 81.4 0.712 0.908

[0109] Table 3 Performance comparison results of different network models on the CMU-MOSEI dataset

[0110] Methods <![CDATA[Acc7]]> <![CDATA[Acc2]]> <![CDATA[F1]]> Corr MAE EF-LSTM 47.5 79.2 79.0 0.627 0.656 LF-LSTM 47.2 79.1 78.6 0.610 0.667 Graph-MFN 45.0 76.9 77.0 0.54 0.71 TFN 47.3 79.3 78.2 0.618 0.657 LMF 47.6 78.2 77.6 0.623 0.660 QMF 47.9 80.7 79.8 0.658 0.640 MMMU-BA 48.4 80.7 80.2 0.672 0.627 MCTN 49.6 79.8 80.6 0.670 0.609 RCN-CEF 50.8 79.7 80.7 0.681 0.596

[0111] Table 4 Performance comparison results of different network models on the IEMOCAP dataset

[0112]

[0113]

[0114] It can be seen from the experimental data in Tables 2 to 4 that the multimodal emotion recognition method based on the recurrent capsule network of channel expansion and fusion of the present invention can accurately identify the emotional state of the user. In addition, the "channel expansion" and "channel fusion" modules of the multi-channel deformable convolutional network can respectively extract the semantic information of different granularities of each modal feature sequence and the implicit semantic information in each modal feature sequence, and the multi-channel deformable convolutional network module can ensure the learning of high-level semantic information of each modal feature sequence by stacking. At the same time, the multimodal semantic capsule fusion network first fuses the feature sequences of each modality in the local area, and then obtains a complete multimodal fusion representation by integrating these local fusion representations. In this way, the model can fully learn all the local semantic information, while ensuring that the learned global semantic information is more complete and sufficient.

[0115] In summary, the recurrent capsule network multimodal emotion recognition method based on channel expansion and fusion in the present invention is superior to the existing methods as a whole, which verifies the effectiveness of the method proposed in the present invention. It can effectively extract the high-level semantic information of each modal feature sequence, and at the same time learn the potential relationship between different modal feature sequences, effectively fuse different modal feature sequences, and obtain a more representative multimodal fusion representation.

Claims

1. A multimodal emotion recognition method based on a recurrent capsule network with channel expansion and fusion, characterized by: Specifically, multiple video samples are extracted, and audio modal feature sequences, visual modal feature sequences, and text modal feature sequences are extracted from each video sample respectively; the above modal feature sequences are input into the time domain convolutional network one by one to obtain modal feature sequences containing contextual information, visual modal feature sequences, and text modal feature sequences respectively; a multi-channel deformable convolutional network is used to learn from the above modal feature sequences one by one to obtain audio modal feature sequences, visual modal feature sequences, and text modal feature sequences containing high-level semantic information respectively, and a global information capsule is obtained by using the constructed multimodal semantic capsule fusion network, and is input into the bidirectional long short-term memory network for compression, and finally, is sent to the dense layer for multimodal sentiment classification; the specific implementation is as follows: Step 1: Extract multiple video samples from the multimodal emotional video database and extract audio modality feature sequences from each video sample. Visual modality feature sequence and text modality feature sequences T is the sequence length of each modal feature sequence, d a d l and d v are the feature vector dimensions of audio modality, text modality, and visual modality respectively; Step 2: Establish three time domain convolutional networks respectively, and transform the audio modal feature sequence Z of each video sample into a , visual modality feature sequence Z v and text modality feature sequence Z l They are respectively sent into the corresponding time domain convolutional network to learn the time domain relationship of each modal feature sequence. The feature vector dimension of each modal feature sequence is unified to d. Finally, the audio modal feature sequence X containing context information is obtained. a , visual modality feature sequence X v and text modality feature sequence X l ; Step 3: Construct L multi-channel deformable convolutional networks and stack them together in sequence. Each multi-channel deformable convolutional network consists of three "channel expansion" modules and one "channel fusion" module. Then, the audio modality feature sequence X containing context information is a , visual modality feature sequence X v and text modality feature sequence X l At the same time, it is input into the stacked multi-channel deformable convolutional network for processing to obtain an audio modality feature sequence containing high-level semantic information. Visual modality feature sequence and text modality feature sequences Step 4: Construct a multimodal semantic capsule fusion network to obtain the global information capsule V; Step 5: Input the global semantic capsule V into the bidirectional long short-term memory network for compression to obtain compressed multimodal fusion Step 6: Fuse the compressed multimodal Feed it into the dense layer for multimodal sentiment classification.

2. The multimodal emotion recognition method based on recurrent capsule network with channel expansion and fusion according to claim 1 is characterized in that: In step 3, each "channel expansion" module is composed of a convolutional neural network with convolution kernel sizes of 3×3 and 5×5 and a maximum pooling layer with a pooling size of 3×3; the "channel fusion" module is composed of five deformable convolutional networks with a convolution kernel size of 3×3; The specific processing process of each multi-channel deformable convolutional network is: Step A: The audio modality feature sequence containing contextual information Visual modality feature sequence and text modality feature sequences They are input into the "channel expansion" module respectively; in the "channel expansion" module, convolution kernels of sizes 3×3 and 5×5 and maximum pooling operations of size 3×3 are used to process each modal feature sequence respectively, and semantic information of different granularities of each modal feature sequence is extracted. The specific calculation process is shown in formulas (1)-(3): Among them, i∈{a, v, l}, 2DConv represents a two-dimensional convolutional neural network, and These are the output results of 3×3 convolution, 5×5 convolution, and 3×3 maximum pooling, respectively. is the number of feature channels after the 3×3 convolution operation, is the number of feature channels after the 5×5 convolution operation, is the number of feature channels after using the 3×3 maximum pooling operation; Step B: and Respectively with X i Stitched together to get the final output of the "Channel Expansion" module That is, and The specific calculation process is shown in formula (4): Among them, C is the dimension of the feature channel after splicing, and the calculation formula is Step C: Output of the three "Channel Expansion" modules and The input is sent to the "channel fusion" module, in which the channel fusion module is firstly categorized according to the granularity of the semantic information. and Divide and splice, and send them to the corresponding deformable convolutional network for processing. The specific calculation process is shown in formulas (5)-(9): Among them, X 3×3 、X 5×5 、X pool and X avl They are the results of fusion of feature sequences of the same granularity in each modality. DeforConv is represented by a deformable convolutional network. and are the feature sequences of the audio, text, and visual modalities output by the "channel fusion" module respectively; Step D: Introduce the residual network to ensure that the multi-channel deformable convolutional network does not experience performance degradation as the number of network layers increases. The specific calculation process is shown in formula (10): in, represents the matrix addition operation, and are the feature sequences of each modality output by the multi-channel deformable convolutional network; Step E: and As input, repeat steps A-D L times to get the final output of the stacked multi-channel deformable convolutional network and 3. The multimodal emotion recognition method based on recurrent capsule network with channel expansion and fusion according to claim 1 is characterized in that: In the step 4, specifically: Step A: Sequence the audio modal features Visual modality feature sequence Text modality feature sequence Audio modal feature sequence X a , visual modality feature sequence X v and text modality feature sequence X l Splice together according to dimension T to get the output feature sequence Then divide the feature sequence X from dimension T to obtain T feature sequences where i∈{1,2,...,T}; Step B: Use one-dimensional convolutional neural network to process feature sequences separately Get T local semantic information capsules Where r is the feature dimension after one-dimensional convolution; Step C: Give each local information capsule an initial weight value, and sum these local information capsules and the corresponding weight values to obtain a new feature sequence. It is called the global information capsule; then calculate the global information capsule P and each local information capsule U i The Pearson correlation coefficient between them is used to update the weight value corresponding to each local information capsule; secondly, all updated weight values are re-weighted and summed with the corresponding local information capsule to obtain a new global information capsule. Finally, calculate the new global information capsule M and each local information capsule U i The Pearson correlation coefficient is calculated and the corresponding weight value is updated continuously. Repeat N times to obtain the final output global information capsule. Where j∈{1,2,...,b}, the specific calculation process is shown in equations (11)-(15): in, represents the weight value, For the middle capsule, c ij Indicates the capsule coefficient, used for the intermediate capsule Weighted summation generates global information capsule V j , b ij is 1, "←" means updating the value. yes and V j The Pearson correlation coefficient, yes The mean, EV j It is V j The mean of yes The mean of It is V j 2 The mean of yes The mean of .

4. The multimodal emotion recognition method based on recurrent capsule network with channel expansion and fusion according to claim 1 is characterized in that: In step 6, the dense layer consists of a ReLU activated fully connected layer, two linear fully connected layers and a Dropout layer.

Images