Emotion recognition method based on multi-mode split attention fusion

Abstract

The invention discloses an emotion recognition method based on multi-modal split attention fusion. The method comprises the following steps: acquiring multi-modal data to be recognized; the emotion recognition model is used for carrying out emotion recognition on multi-modal data to be recognized, the emotion recognition model comprises an initial feature extraction module, an advanced feature extraction module, an MSAF module and an emotion classifier, the initial feature extraction module comprises an initial text feature extraction unit, an initial audio feature extraction unit and an initial video feature extraction unit, and the initial text feature extraction unit comprises an initial text feature extraction unit; the initial text feature extraction unit adopts a Transform model as an encoder; the initial audio feature extraction unit and the initial video feature extraction unit adopt an AKFC network as an encoder; the advanced feature extraction module comprises a context information acquisition unit and a pairwise cross-modal feature interaction unit, and the context information acquisition unit comprises an RGCN model and a Graph Transform model; the MSAF module is used for determining fusion features; the emotion classifier module is used for emotion recognition. The emotion recognition accuracy can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of emotion recognition, and in particular relates to an emotion recognition method based on multimodal split attention fusion. Background Art

[0002] In the multimodal emotion recognition task, how to extract audio features and visual features from audio and video datasets respectively, and then effectively learn the complex nonlinear relationship between different modalities has always been a major problem in real-world scenarios. The traditional solution is to extract features from audio and video through a fully connected layer. Although the fully connected layer has the advantages of being able to process feature vectors of any input size and being able to learn complex nonlinear relationships between features, it has problems such as a large number of parameters and a large amount of computation.

[0003] Multimodal feature fusion generally requires learning data from at least two modalities. How to exploit the complementarity and redundancy of multiple modalities to represent and summarize multimodal data is an important topic in the multimodal field. Learning representations for multimodal data is particularly challenging due to the inherent heterogeneity between modalities, such as the difference between speech and audio signals. In the context of multimodal emotion recognition tasks, effectively fusing the obtained multimodal representations poses a significant challenge. Studies have shown that different feature fusion methods can significantly affect the robustness, stickiness, and recognition accuracy of the model. In addition, the feature fusion design within different network architectures also plays a crucial role. Currently, there are four main popular feature fusion methods: (1) early fusion, which integrates the modal features immediately after they are extracted, usually by simply concatenating these features; (2) late fusion, which integrates the results after each modality makes a decision (e.g., regression or classification). Common methods include average decision, voting model, weighting based on channel noise and signal variance, or combining additional networks to learn decisions; (3) Hybrid fusion, which combines the principles of early fusion and late fusion, using the mixed features of early fusion and multiple modes of late fusion. The output of this fusion is then used for network prediction; (4) Model fusion, which is an integrated learning method that combines the prediction results of multiple independent models to produce a more accurate and robust overall prediction or decision.

[0004] However, existing multimodal emotion recognition models suffer from problems such as insufficient feature fusion and high model complexity, which limit the performance and application scope of the models. Summary of the invention

[0005] In view of this, in order to address the problem of insufficient feature fusion in traditional multimodal emotion recognition models, the present invention adopts a fusion method of modal fusion features and provides an emotion recognition method based on multimodal split attention fusion.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] An emotion recognition method based on multimodal split attention fusion, comprising:

[0008] Acquire multimodal data to be identified, wherein the multimodal data to be identified is associated with three modalities, wherein the three modalities include audio, video, and text, and the multimodal data to be identified is a conversation, and the conversation includes two or more utterances;

[0009] The emotion recognition model is used to perform emotion recognition on the multimodal data to be recognized, wherein the emotion recognition model includes: an initial feature extraction module, an advanced feature extraction module, an MSAF module and an emotion classifier.

[0010] The initial feature extraction module includes an initial text feature extraction unit, an initial audio feature extraction unit and an initial video feature extraction unit. The initial text feature extraction unit uses the Transformer model as an encoder to extract the initial text features of each utterance. The initial audio feature extraction unit and the initial video feature extraction unit both use the AKFC network as an encoder to extract the initial audio features and initial video features of each utterance respectively. In the AKFC network, the AKConv module is used to replace the convolutional layer in the benchmark model, and the initial audio features and initial video features are extracted in combination with the fully connected layer.

[0011] The advanced feature extraction module includes a context information acquisition unit and a paired cross-modal feature interaction unit, wherein the context information acquisition unit includes a relational graph convolutional network RGCN and a Graph Transformer model, the RGCN is used to obtain the local context information of each utterance in the conversation and obtain the first feature representation of each modality of each utterance; the Graph Transformer model is used to obtain the second feature representation of each modality of each utterance according to the first feature representation of each modality of each utterance;

[0012] The paired cross-modal feature interaction unit is used to perform cross-modal attention on any two modalities to obtain a third feature representation;

[0013] The MSAF module is used to fuse the second feature representation and the third feature representation obtained by the context information acquisition unit and the paired cross-modal feature interaction unit to obtain a fused feature;

[0014] The sentiment classifier module is used to perform sentiment recognition on each utterance based on the fused features.

[0015] Furthermore, the emotion recognition is four-category emotion recognition.

[0016] Furthermore, the emotion recognition is six-category emotion recognition.

[0017] Furthermore, the baseline model is Transformer, and the AKConv module is used to replace the convolution layer in the baseline model, and combined with the fully connected layer to extract the initial audio features or the initial video features. Specifically, it includes using the AKConv module to replace the convolution layer of the preprocessing part of the input data in the baseline model, and directly connecting the fully connected layer after each AKConv module to realize the extraction and preliminary integration of local features to form an initial feature representation for subsequent processing.

[0018] Furthermore, the method further includes training the emotion recognition model, wherein the training process includes:

[0019] Obtain training samples, each of which is multimodal data, each of which is associated with three modalities, wherein the three modalities include audio, video, and text, each of which is a conversation, each of which includes two or more utterances, and each utterance in each training sample corresponds to an emotion label;

[0020] Build an emotion recognition model;

[0021] For each training sample,

[0022] Extracting initial text features, initial audio features, and initial video features from each utterance of the training sample using an initial feature extraction module;

[0023] Performing a cascade operation on the initial text feature, the initial audio feature, and the initial video feature for fusion, and generating a first fusion feature corresponding to each utterance in the training sample;

[0024] Inputting the first fused feature into a context information acquisition unit and a paired cross-modal feature interaction unit to generate a second feature representation and a third feature representation corresponding to each utterance in the training sample;

[0025] Inputting the second feature representation and the third feature representation into the MSAF module to generate a second fusion feature corresponding to the utterance;

[0026] Inputting the second fusion feature into an emotion classifier to obtain an emotion prediction result corresponding to the speech;

[0027] According to the emotion prediction results and corresponding emotion labels corresponding to all training samples, the parameters in the emotion recognition model are adjusted until the preset conditions are met.

[0028] Furthermore, the initial feature extraction module is used to extract the initial text features, initial audio features and initial video features in the training sample, which is achieved by the following formula:

[0029]

[0030] Among them, r i t 、r i a and r i v Respectively represent the set of utterances in the training sample {r1,r2,...,r i ,...,r N Any multimodal data r in i , that is, discourse r i , the text data, audio data and video data in, i = 1, 2, ..., N, N represents the number of utterances in a conversation; and Respectively represent the discourse r i corresponding initial text features, initial audio features, and initial video features; represents the Transformer model used to extract initial text features, i.e., the learnable parameters of Transformer(·); represents the learnable parameters of the fully connected layer in the AKFC network used to extract the initial audio features, i.e., AKFC1(·); represents the learnable parameters of the fully connected layer in the AKFC network used to extract the initial video features, i.e., AKFC2(·).

[0031] Furthermore, inputting the first fusion feature into the context information acquisition unit to generate a second feature representation corresponding to the utterance includes:

[0032] For each conversation, a multimodal graph is established, represented as G(V, R, E), where V represents the set of discourse nodes with three modal types in the multimodal graph, E represents the set of edges in the multimodal graph, and R represents the set of relationship types in the multimodal graph. i Generate three nodes, namely r i t 、r i a and r i v , a, v and t correspond to audio mode, video mode and text mode respectively, r ij ∈R, R contains R multi and R temp , R multi represents the set of internal connections between the three modalities in the same utterance, reflecting multimodal interaction, R temp represents the set of interconnected utterances of the same modality within a specified time window.

[0033] According to the multimodal graph, determine the first feature representation of any modality τ in the i-th utterance Right now:

[0034]

[0035] Among them, r represents any relationship type, N r (i) represents the set of neighbor nodes of the ith node under relation r, |·| represents the number of elements in the set, and W r and W0 represent the learnable parameters in RGCN;

[0036] The first feature is represented as Input the Graph Transformer model to obtain the second feature representation of any modality τ in the i-th utterance Right now:

[0037]

[0038] Among them, W1 and W2 represent the learnable parameters in the Graph Transformer model. represents the cascade of D-head attention, represents the attention coefficient of node j, which is calculated by the softmax activation function, that is:

[0039]

[0040] Where d represents the scaling factor, W3 and W4 represent learnable parameters;

[0041] Then, with the training samples {r1,r2,...,r i ,...,r N The second feature corresponding to} is expressed as

[0042] Furthermore, the first fused features are input into a paired cross-modal feature interaction unit to generate a third feature representation, including:

[0043] Perform cross-modal attention between any two modalities τ1 and τ2 to generate a third feature representation and in,

[0044]

[0045] τ1 and τ2 take any two modes in {a, v, t}, and τ1 and τ2 take different modes;

[0046] queue key value

[0047] queue key value

[0048] and represents the learnable parameters of the pairwise cross-modal feature interaction unit, σ represents the softmax function, and d k1 express The dimension, d k2 express Dimension.

[0049] Furthermore, inputting the second feature representation and the third feature representation into the MSAF module to generate a second fusion feature corresponding to the speech in the training sample includes:

[0050] The MSAF module obtains the feature map set {F1, F2, ..., F Mod}, where Mod represents the total number of modalities, i.e., Mod=3, each feature map in the feature map set corresponds to a modality, and the feature map contains a set of second feature representations and third feature representations corresponding to the modality;

[0051] Each feature map is divided into equal channel feature blocks according to the channel. The number of channels in each equal channel feature block is C. The set of feature blocks belonging to the modality mod is represented as B mod , D represents the total number of channels corresponding to the modal features, Indicates the number of divided equal channel feature blocks, Indicates rounding up, C mod represents the number of channels corresponding to a single modal feature, mod∈{1,2,...,Mod}, Indicates B mod The i-th equal channel feature block in ;

[0052] Connect all equal-channel blocks corresponding to the modality mod, and add the equal-channel feature blocks belonging to the modality mod to the shared representation D mod By calculating B mod Element-wise sum S mod , and perform global average pooling in the spatial dimension, which can be expressed as:

[0053]

[0054] Among them, N i represents the size of each spatial dimension, K represents the number of spatial dimensions of the modal mod, n1,n2,...,n Krepresents the position index of each spatial dimension of the feature map, c represents the channel index of the feature map, and D mod (c) represents the aggregate eigenvalue of the modality on a specific channel c;

[0055] For each Generated by applying a linear transformation to Z And use softmax activation to get the block-wise attention weights The formula is:

[0056]

[0057] in, and Indicates that Convert to The weights and biases of the fully connected layers;

[0058] Using block-wise attention weights and the regularization factor λ of the minimum attention of the split, determining the optimized feature block The formula is:

[0059]

[0060] Among them, ⊙ represents the element-by-element product;

[0061] The optimized feature blocks of mod belonging to the modality Generate optimized feature maps by combining channel-by-channel

[0062] Optimizing the collection of feature maps That is, the second fusion feature of the training sample generated by the MSAF module.

[0063] Further, the second fusion feature is input into the emotion classifier to obtain the emotion prediction result corresponding to the speech, including using the following formula for prediction:

[0064] v i =ReLU(Φ0h i +b0),

[0065] p i =softmax(Φ1v i +b1),

[0066]

[0067] Among them, h i represents the second fusion feature corresponding to the i-th utterance, v irepresents the result after using the ReLU activation function, Φ0 and b0 represent the learnable parameters of ReLU, and p i represents the probability distribution obtained by the softmax activation function, Φ1 and b1 represent the learnable parameters of softmax, Table 1 is the predicted emotional label for the i-th utterance, argmax represents the value that makes p i The function that finds the predicted label when the value is the maximum.

[0068] The beneficial effects of the present invention are:

[0069] The AKFC network proposed in this invention not only takes into account the advantages of the fully connected layer for audio and video feature extraction, but also effectively reduces the calculation time of the model. Ultimately, the audio and video feature extraction capability of the fully connected layer model is improved;

[0070] The modal fusion feature fusion method is adopted to solve the problems of partial information loss of multimodal features, redundancy of different modal features, and limitation of feature representation. By using multimodal split attention fusion to perform multimodal feature fusion on audio, video, and text, MSAF allows the model to dynamically assign weights of different modal features during the feature fusion process, thereby better capturing the complex relationship between multimodal information.

[0071] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0073] Figure 1 This is the architecture diagram of the emotion recognition method based on multimodal split attention fusion;

[0074] Figure 2 It is a flexible parameter and adaptive sampling shape map;

[0075] Figure 3 This is the AKConv architecture diagram;

[0076] Figure 4 This is a schematic diagram of AKFC audio and video feature extraction;

[0077] Figure 5 It is a schematic diagram of the relationship between the three-modal discourse nodes of text, video and audio;

[0078] Figure 6 It is a schematic diagram of pairwise cross-modal feature interaction;

[0079] Figure 7 It is a pairwise cross-modal feature interaction flow chart;

[0080] Figure 8 It is a schematic diagram of the MSAF model;

[0081] Fig. 9 This is the IEMOCAP (4-classification) emotion recognition accuracy graph under different modal features;

[0082] Fig.10 This is the IEMOCAP (6 categories) emotion recognition accuracy diagram under different modal features. DETAILED DESCRIPTION

[0083] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments are only for illustrating the present invention, rather than for limiting the protection scope of the present invention.

[0084] This application proposes an emotion recognition method based on multimodal split attention fusion. This method uses the AKFC network to extract audio and video features, thereby solving the problem of high model complexity in the multimodal emotion recognition model; then, RGCN+Graph Transformer and paired cross-modal feature interaction modules are used to further extract audio, video, and text features. RGCN+Graph Transformer is used to capture contextual information, and paired cross-modal feature interaction modules are used to realize cross-modal feature interaction. The output of RGCN+Graph Transformer and paired cross-modal feature interaction modules is used as the input of multimodal split attention fusion (i.e., Multi-modal Split Attention Fusion, MSAF) for multimodal feature fusion. Finally, it is used as the input of the linear classifier to output the recognition result of the emotion.

[0085] Figure 1 This is the architecture diagram of the emotion recognition method based on multimodal split attention fusion. Figure 1 , an emotion recognition method based on multimodal split attention fusion, comprising:

[0086] Acquire multimodal data to be identified, wherein the multimodal data to be identified is associated with three modalities, wherein the three modalities include audio, video, and text, and the multimodal data to be identified is a conversation, and the conversation includes two or more utterances;

[0087] The emotion recognition model is used to perform emotion recognition on the multimodal data to be recognized.

[0088] The emotion recognition model includes: initial feature extraction module, advanced feature extraction module, MSAF module and emotion classifier.

[0089] The initial feature extraction module includes an initial text feature extraction unit, an initial audio feature extraction unit and an initial video feature extraction unit. The initial text feature extraction unit uses the Transformer model as an encoder to extract the initial text features of each utterance; the initial audio feature extraction unit and the initial video feature extraction unit both use the AKFC network as an encoder to extract the initial audio features and initial video features of each utterance respectively.

[0090] The advanced feature extraction module includes a context information acquisition unit and a paired cross-modal feature interaction unit, wherein the context information acquisition unit includes a relational graph convolutional network RGCN and a Graph Transformer model, the RGCN is used to obtain the local context information of each utterance in the conversation and obtain the first feature representation of each modality of each utterance; the Graph Transformer model is used to obtain the second feature representation of each modality of each utterance according to the first feature representation of each modality of each utterance;

[0091] The paired cross-modal feature interaction unit is used to perform cross-modal attention on any two modalities to obtain a third feature representation;

[0092] The MSAF module is used to fuse the second feature representation and the third feature representation obtained by the context information acquisition unit and the paired cross-modal feature interaction unit to obtain a fused feature;

[0093] The emotion classifier module is used to perform emotion recognition on each utterance based on the fusion features. Emotion recognition can be either four-class emotion recognition or six-class emotion recognition.

[0094] Since the sampling position of traditional standard convolution is fixed, the convolution can only extract the local information of the current window, but cannot fully capture the information of other positions. And the traditional standard convolution does not allow the convolution kernel to have any number of parameters. As the size of the convolution kernel increases, the number of convolution parameters of the traditional standard convolution tends to increase by a square, which is more demanding on the hardware environment. That is, the traditional convolution has two defects. The first is that the convolution operation runs on a fixed-size window, cannot capture the information of other windows, and the shape of the window is fixed; the second is that the size of the convolution kernel is fixed to k×k, and the window size is fixed to k. As k increases, the parameters will increase rapidly.

[0095] Traditional convolutions usually have fixed sizes and shapes, such as 3x3 or 5x5 square grids. The core principle of AKConv is to allow convolution kernels to have a deformable number of parameters. This means that we are no longer limited to standard square networks, but can adopt more diverse and flexible shapes according to the characteristics of the graph and the requirements of the task, such as Figure 2 The number of flexible parameters shown in . When processing different images and objects, the AKConv convolution kernel can automatically adjust its sampling shape. This is achieved by introducing a new coordinate generation algorithm that can generate initial sampling coordinates for convolution kernels of different sizes and shapes, such as Figure 1 The adaptive sampling shape is shown in Figure 2 Adaptive sampling shape shown.

[0096] AKConv first determines the initial sampling positions of the convolution kernel through a coordinate generation algorithm. These positions are no longer fixed, but can change dynamically according to the features and targets in the image. In order to better adapt to the size and shape changes of the target in the image, AKConv will adjust the sampling positions of the convolution kernel according to the characteristics of the target. This adjustment is achieved by adding an offset, making the convolution operation more flexible and practical.

[0097] AKConv can adapt to any shape and has unparalleled applicability when processing visual and audio data compared to traditional convolution. Standard convolution is a k×k rectangle, while deformable convolution (Deformable Conv) can adjust the shape. Similar to deformable convolution, AKConv also learns offsets to change the shape of the convolution kernel. The number of parameters of AKConv is arbitrary, so it can get rid of the limitation of traditional convolution k×k, thereby effectively improving the operation speed. N is the number of AKConv convolution parameters. The feature map is convolved to obtain the position offset of the convolution, and then the convolution operation is performed, just like deformable convolution. Deformable convolution can change the position of the convolution calculation by learning the offset, so that the shape of the convolution kernel is not fixed, but deformable convolution has a defect: the convolution kernel parameters are fixed, such as 1, 9, 27. Therefore, the present invention adopts AKConv convolution.

[0098] Figure 3 This is the AKConv architecture diagram. Figure 3 As shown, first, the offset of the corresponding kernel is obtained through a convolution operation, which has dimensions (B, 2N, H, W), where N is the convolution kernel size, B represents the number of offset groups, H represents the height of the data, and W represents the width of the data. Figure 3 In the example, N = 5. By adding the offset and the original coordinates P0+P n To obtain the modified coordinates. Finally, the features of the corresponding positions are obtained through interpolation and resampling.

[0099] The features corresponding to the sampling positions of irregular convolution kernels are difficult to extract. The features are stacked on rows or columns, and column convolution or row convolution is used to extract features corresponding to irregular sampling shapes. Extract features to use convolution kernels of appropriate size and stride. In addition, the features are converted to four dimensions (C, N, H, W), where C represents the number of channels, and then Conv3d with stride and convolution size (N, 1, 1) is used to extract features.

[0100] Convolutional neural networks are based on convolution operations, which locate features at corresponding locations through regular sampling grids. Assuming that Grid represents a sampling grid, Grid is represented as:

[0101] Grid={(-1,-1),(-1,0),...,(0,1),(1,1)}

[0102] The sampling grid of a common convolutional neural network is regular, while AKConv targets irregularly shaped convolution kernels. In order to allow irregular convolution kernels to have sampling grids, an algorithm for convolution of arbitrary size is designed, which generates a convolution kernel P n The sampling grid is first generated as a regular sampling grid, then irregular grids are created for the remaining sampling points, and finally they are spliced ​​together to generate the overall sampling grid.

[0103] like Figure 3 As shown in Figure 1, it shows that the initial sampling coordinates are generated for convolutions of any size. The sampling grid of regular convolution is centered at point (0,0). Irregular convolution has no center at many sizes. In order to adapt to the size of the convolution used, the upper left corner (0,0) point is set as the sampling origin in the algorithm. n Afterwards, the corresponding convolution operation at position P0 is defined as shown in Formula 1:

[0104] Conv(P0)=∑w×(P0+P n ), (Formula 1)

[0105] Among them, w represents the convolution parameter.

[0106] The algorithm for initial sampling coordinates is shown in Algorithm 1 in Table 1.

[0107] Table 1 Algorithm for initial sampling coordinates

[0108]

[0109]

[0110] Since AKConv has certain advantages over conventional convolution operations, AKConv is also used in the audio and video feature extraction network of the present invention. The present invention proposes an AKFC network, in which an AKConv module and a fully connected layer are included. Among them, the AKConv module uses a convolution kernel of variable shape and size for convolution as a convolution layer in the input preprocessing stage, and extracts initial audio features or initial video features in combination with a fully connected layer. The present invention replaces the traditional convolution layer with an AKConv module, so that feature vectors of any size can be processed, and more complex nonlinear relationships between features can be learned, thereby improving the expressive power of feature extraction. At the same time, the introduction of the AKConv module also effectively reduces the number of parameters and the amount of calculation of the model, further improving the efficiency of the model.

[0111] Figure 4 This is a schematic diagram of AKFC audio and video feature extraction. Figure 4 As shown in the figure, the AKFC network contains the AKConv module and the fully connected layer (FC layer). First, a video stream can be split into three modal data: text, audio, and video. The audio and video data are feature extracted by the AKFC network. Figure 4 As shown in the figure, the audio and video are first convolved through the AKConv module. AKConv allows convolution to have any shape and size, so it can process feature vectors of any size and dimension. Therefore, the audio and video data can be convolved to obtain the initial feature vector, which is then used as the input of the FC layer to finally obtain the visual features and audio features. The visual and audio features are then simply concatenated as the input of the subsequent RGCN and paired cross-modal feature interaction module.

[0112] Given N multimodal data samples {r1,r2,...,r i ,...,r N}, each multimodal data r i Associated with three modalities, where the three modalities include audio (denoted as a), video (denoted as v) and text (denoted as t), the data corresponding to the three modalities can be obtained, namely r i a 、r i v and r i t It should be noted that the symbols or formulas expressed above and below may represent multimodal data samples to be identified or multimodal data in training samples. The same operation is performed on the multimodal data samples to be identified and the multimodal data in training samples.

[0113] For text information, Transformer is used as an encoder to extract features from the text information and obtain the initial text features. As shown in formula (2):

[0114]

[0115] in, represents the learnable parameters of the Transformer model (i.e., Transformer(·)) used to extract the initial text features.

[0116] For auditory (i.e., audio) and visual (i.e., video) modal information, the AKFC network is used as a unimodal encoder to extract features from audio and video streams. The context features of each modality type are extracted using the following formulas (3) and (4). The context features of audio and video (i.e., initial audio features) and initial video features )

[0117]

[0118] in, represents the learnable parameters of the fully connected layer in the AKFC network (i.e., AKFC1(·)) used to extract the initial audio features; represents the learnable parameters of the fully connected layer in the AKFC network (i.e., AKFC2(·)) used to extract the initial video features; and d a ,d v ,d t Represents the dimensions of the features extracted from the three modalities, Represents the set of real numbers.

[0119] The data sets of the three modalities of text, audio, and video are first subjected to initial feature extraction through their respective unimodal networks to obtain the initial features corresponding to their respective modalities (initial video features, initial audio features, initial text features). Then, the initial features corresponding to the three modalities are simply fused through a simple cascade as the input of the subsequent advanced feature extraction module.

[0120] After primary feature extraction, advanced feature extraction can be further performed.

[0121] The relational graph convolutional network (RGCN) captures the local context information of each utterance in a conversation by utilizing the multimodal graph between utterances and their modalities, and builds a multimodal graph through RGCN. Figure 5As shown, it represents the relationship between the three modal discourse nodes of text, video and audio.

[0122] exist Figure 5 In the figure, the black solid line, orange solid line and orange dashed line arrows represent cross-modal connections, past time connections and future time connections, respectively. For each session, a multimodal graph is established, represented as G(V, R, E), where V represents the set of discourse nodes with three modal types in the multimodal graph (where the number of nodes |V| = 3×N, N represents the number of discourses in a session), E represents the set of edges in the multimodal graph, R represents the set of relationship types in the multimodal graph, and each discourse r i Generate three nodes, namely r i t 、r i a and r i v , a, v and t correspond to audio mode, video mode and text mode respectively, r ij ∈R, R contains R multi and R temp , R multi represents the set of internal connections between the three modalities in the same utterance, reflecting multimodal interaction, R temp represents the set of interconnections between utterances of the same modality within a specified time window. This temporal relationship includes past / previous utterances, which are labeled P, and next / future utterances, which are labeled F. Therefore, 15 edge types are created using the definitions of these two groups. When there are relations between three different modalities for the same utterance (i.e., the same i), this situation contains 6 edge relations, i.e., reference Figure 5 The black solid line in the middle shows the six edge relationships between the three pairs of modalities v and a, v and t, and a and t. For the interconnected relationship between utterances of the same modality in the same time window, this situation contains six variable relationships, namely, Figure 5 The orange solid line and the orange dotted line in the middle contain two edge relations (past time connection and future time connection) for one mode, so the three modes have a total of 6 variable relations, as shown in formula (5):

[0123]

[0124] Among them, τ∈{a,v,t}, i,j range from 1 to N, and future and past represent past and future relations respectively.

[0125] In order to capture the nuances and variations of heterogeneous interactions between discourses and modalities in a multimodal graph, a relational graph convolutional network (RGCN) is employed. For each relation type r∈R, a mapping function f(H,W r) infers the node representation, where W is the weight matrix. Aggregating all 15 edge types, we can use ∑ r∈R f(H,W r ) Calculate the final node representation. Determine the first feature representation of any modality τ in the i-th utterance As shown in formula (6):

[0126]

[0127] Among them, r represents any relationship type, N r (i) represents the set of neighbors of the i-th node, |·| represents the number of elements in the set, and W r and W0 represent the learnable parameters in RGCN. d h1 is the dimension of the hidden layer used by RGCN,

[0128] To extract rich representations from node features, a Graph Transformer model is utilized, where each layer includes a self-attention mechanism followed by a feed-forward neural network. The self-attention mechanism allows nodes to exploit information from the neighborhood (i.e., the set of nodes that each node is directly connected to in the graph) and capture local and global patterns in the graph.

[0129] Therefore, the first feature can be expressed as Input the Graph Transformer model to obtain the second feature representation of any modality τ in the i-th utterance That is, formula (7):

[0130]

[0131] Among them, W1 and W2 represent the learnable parameters in the Graph Transformer model. d h2 represents the dimension of the hidden layer used by Graph Transformer, represents the cascade of D-head attention, represents the attention coefficient of node j, which can be calculated by the softmax activation function, as shown in formula (8):

[0132]

[0133] Where d represents the scaling factor, W3 and W4 represent learnable parameters, d ρ Indicates the dimensions of each head.

[0134] After aggregation in the entire graph, a new representation vector is obtained, which is consistent with the multimodal data {r1,r2,...,ri ,...,r N The second feature corresponding to} is expressed as

[0135] Cross-modal heterogeneity often increases the difficulty of human language analysis. Exploiting cross-modal interactions may help reveal the "inconsistent" nature and long-term dependencies of cross-modality, so pairwise cross-modal feature interaction (i.e., P-CM) method design can be added to the pairwise cross-modal feature interaction framework. Figure 6 It is a schematic diagram of pairwise cross-modal feature interaction; Figure 7 is a flowchart of pairwise cross-modal feature interaction. Figure 6 and Figure 7 , inputting the first fused feature into a paired cross-modal feature interaction unit, and generating a third feature representation may include:

[0136] Perform cross-modal attention between any two modalities τ1 and τ2 to generate a third feature representation and As shown in formula (9) and formula (10):

[0137]

[0138] τ1 and τ2 take any two modes in {a, v, t}, and τ1 and τ2 take different modes;

[0139] queue key value

[0140] queue key value

[0141] and represents the learnable parameters of the pairwise cross-modal feature interaction unit, σ represents the softmax function, and d k1 express The dimension, d k2 express The dimension represents the scaling factor.

[0142] For example, given two modalities, such as audio a and text t, then, The superscript T indicates transpose. as the corresponding modality-sensitive representation of the entire session using a unimodal encoder. Based on the Transformer architecture, the key is defined as Once cross-modal attention is performed on modality a (i.e. modality τ2) through modality t (i.e. modality τ1), M aThe rich representation of (i.e.) can be expressed as a formula as shown in formula (11) with reference to formula (9), i.e.:

[0143]

[0144] Back to Figure 1 After determining the second feature representation and the third feature representation, the second feature representation and the third feature representation can be input into the MSAF module to generate a second fusion feature corresponding to the discourse. Figure 8 It is a schematic diagram of the MSAF model (i.e., MSAF module).

[0145] Reference Figure 8 Inputting the second feature representation and the third feature representation into the MSAF module to generate a second fusion feature corresponding to the utterance in the training sample may include:

[0146] The MSAF module obtains the feature map set {F1, F2, ..., F Mod}, where Mod represents the total number of modalities, i.e., Mod=3, each feature map in the feature map set corresponds to a modality, and the feature map contains a set of second feature representations and third feature representations corresponding to the modality;

[0147] First, perform a split operation (i.e. Figure 8 Step 1 in , that is, each feature map is divided into equal channel feature blocks according to the channel, the number of channels in each equal channel feature block is C, and the set of feature blocks belonging to the mode mod is represented as B mod , Indicates the number of divided equal channel feature blocks, Indicates rounding up, C mod represents the total number of channels corresponding to the modal mod feature, mod∈{1,2,...,Mod}, Indicates B mod The i-th equal channel feature block in C mod If C is not a multiple of C, the last block is filled with zeros in the missing channels.

[0148] Next, perform the connection operation (i.e. Figure 8 In step 2), all equal channel blocks corresponding to the modality mod are connected, and the equal channel feature blocks belonging to the modality mod are added to the shared representation D mod By calculating B mod Element-wise sum S mod , and perform global average pooling in the spatial dimension, expressed by formula (12):

[0149]

[0150] Among them, N i represents the size of the i-th spatial dimension, K represents the number of spatial dimensions of the modal mod, n1,n2,...,n K represents the index of each spatial dimension, c represents the channel index, D mod (c) represents the global feature description of mode mod under channel c;

[0151] Then, perform the highlight operation (i.e. Figure 8 Step 3), that is, for each Generated by applying a linear transformation to Z And use softmax activation to get the block-wise attention weights It can be expressed by formula (13) and formula (14):

[0152]

[0153] in, and Indicates that Convert to The weights and biases of the fully connected layers, C' represents the output dimension of the fully connected layer;

[0154] Since the soft attention value depends on the total number of feature blocks, the features may be over-suppressed. This effect is more obvious in complex tasks, which leads to insufficient information for accurate prediction. For the above problem, a hyperparameter λ∈[0,1] is proposed to control the suppression ability of MSAF. Therefore, the block-wise attention weights can be used and the regularization factor λ of the minimum attention of the split, determining the optimized feature block It can be expressed as:

[0155]

[0156] in, ⊙ represents element-wise product.

[0157] Next, the optimized feature blocks belonging to the modal mod Generate optimized feature maps by combining channel-by-channel Optimizing the collection of feature maps That is, the second fusion feature of the training sample generated by the MSAF module.

[0158] After the feature fusion of MASF, the emotion classifier receives the multimodal fused features of the MSAF module (i.e., the second fused features) as the input of the classifier to obtain the emotion prediction result corresponding to the speech, including prediction using the following formulas (16) to (18):

[0159] v i =ReLU(Φ0h i +b0), (Formula 16)

[0160] p i =softmax(Φ1v i +b1), (Formula 17)

[0161]

[0162] Among them, h i represents the second fusion feature corresponding to the i-th utterance, v i represents the result after using the ReLU activation function, Φ0 and b0 represent the learnable parameters of ReLU, and p i represents the probability distribution obtained by the softmax activation function, Φ1 and b1 represent the learnable parameters of softmax, Table 1 is the predicted emotional label for the i-th utterance, argmax represents the value that makes p i The function that finds the predicted label when the value is the maximum.

[0163] An emotion recognition method based on multimodal split attention fusion also includes training an emotion recognition model, wherein the training process includes:

[0164] Acquire training samples, each training sample is multimodal data, each multimodal data is associated with three modalities, wherein the three modalities include audio, video and text, each training sample is a conversation, each conversation includes two or more utterances, and each utterance in each training sample corresponds to an emotion label (one or more emotion labels);

[0165] Build an emotion recognition model;

[0166] For each training sample,

[0167] Extracting initial text features, initial audio features, and initial video features from each utterance of the training sample using an initial feature extraction module;

[0168] Performing a cascade operation on the initial text feature, the initial audio feature, and the initial video feature for fusion, and generating a first fusion feature corresponding to each utterance in the training sample;

[0169] Inputting the first fused feature into a context information acquisition unit and a paired cross-modal feature interaction unit to generate a second feature representation and a third feature representation corresponding to each utterance in the training sample;

[0170] Inputting the second feature representation and the third feature representation into the MSAF module to generate a second fusion feature corresponding to the utterance;

[0171] Inputting the second fusion feature into an emotion classifier to obtain an emotion prediction result corresponding to the speech;

[0172] According to the emotion prediction results and corresponding emotion labels corresponding to all training samples, the parameters in the emotion recognition model are adjusted until the preset conditions are met.

[0173] The processing of the utterances in the training samples is similar to that in the above appendix. Figure 3-8 The representation and related operations described in are the same and will not be repeated here. The advantages of the present invention are illustrated by specific experiments.

[0174] The present invention uses the IEMOCAP (Interactive Emotional Dyadic Motion Capture) dataset. The dataset consists of conversations between 10 pairs of English-speaking college students from the western United States. Each pair of conversation participants is asked to express emotions in several emotion-stimulating scenarios. These emotion scenarios simulate various emotional situations that may be encountered in real life, including positive and negative emotions, such as happiness, sadness, neutrality, anger, excitement, frustration, etc.

[0175] The IEMOCAP dataset file mainly contains two folders: Documentiation and Session. Documentiation contains five files: corpus.dic, FIVE_face_markers2.png, HumaineInfo.txt, phonemes.txt, and timeinfo.txt. Session mainly contains two folders: dialog and sentences.

[0176] corpus.dic is a dictionary file used to record the corpus in the IEMOCAP dataset. It contains the definitions of some terms, symbols or tags used in the dataset, as well as the usage of these terms in the dataset. The FIVE_face_markers2.png file is an image file that contains information related to facial expressions. The HumaineInfo.txt file contains general information or metadata related to the dataset, such as the source, creator, and collection method of the dataset. The phonemes.txt file contains information related to the phonemes used in the speech data. Phonemes are the basic units in speech and are commonly used in speech recognition and analysis. The timeinfo.txt file contains content related to time information. In emotion recognition, time information may refer to the duration of each segment in speech or facial expression data, or information related to the temporal relationship of emotional expression.

[0177] The session is divided into 5 parts (Session1, Session2, Session3, Session4, Session5), and each session is recorded by 1 actor and 1 actress. The total recording time is about 12 hours. Each session includes 2 folders: dialog and sentences. The dialog folder contains the audio and video files for each session (dialog), as well as the annotation information related to these sessions. The specific information is shown in Table 2.

[0178] Table 2 Main distribution of dialog dataset

[0179]

[0180] Among them, avi is an audio and video interleaved format, emo evaluation stores emotional evaluation labels, transcriptions is text transcription, stored in txt format, and wav is an audio file format.

[0181] Each session corresponds to a conversation (or conversation) or communication scene in the IEMOCAP dataset. These sessions are usually conducted by multiple participants and involve natural conversations and emotional expressions. In the study of emotion recognition, researchers can use these audio and video files for training and evaluation of emotion recognition models.

[0182] The sentences folder contains sentence-level annotation information in the conversation, usually the transcript of the conversation. This sentence-level annotation information may include the speaker, content, and sentiment label of each sentence. This sentence-level annotation information can be used to build a dataset for emotion recognition tasks and evaluate the emotion recognition performance of the model at the sentence level. Some of the transcriptions in the sentences file are shown in Table 3.

[0183] In Table 3, Seg.[sec] refers to turn segmentation (in seconds). Turn refers to the exchange of speech between speakers, where F refers to female and M refers to male. Transcription refers to transcribed text. Lable refers to categorical sentiment evaluation (three themes). [v,a,d] refers to attribute sentiment evaluation (valence, activation, dominance, three themes).

[0184] Table 3 Partial transcription content in the sentences file

[0185]

[0186]

[0187] In Table 3, Dialogues refers to the dialogue scenes included in the dataset. Each dialogue scene is conducted by two actors and simulates a specific emotional exchange or scenario. These dialogues are designed to simulate human communication and interaction in different emotional states in order to capture rich and natural emotional expressions. Utterances represent individual speeches or statements (referred to as utterances) in the dialogue. Each utterance is a speech segment in the dialogue, uttered by an actor in a specific emotional state. Each statement is designed to express or elicit a specific emotional response and can be used to train and test emotion recognition models.

[0188] The IEMOCAP dataset uses Dialogues and Utterances to analyze and understand the complexity of human emotional expression and the role of emotions in interpersonal communication. These data help develop more accurate emotion recognition algorithms, especially when processing voice and facial expressions.

[0189] Table 4 Dialogues dataset division

[0190]

[0191] As shown in Table 4, Dialogues divides the training set, validation set, and test set into a ratio of 7:1:2 in the four-category and six-category tasks based on the IEMOCAP dataset.

[0192] Table 5 Utterances dataset division

[0193]

[0194] As shown in Table 5, Utterances divides the training set, validation set, and test set into 7:1:2 ratios in the four-category and six-category tasks based on the IEMOCAP dataset. Among them, Utterances removes some datasets that are only applicable to the six-category task in the four-category task.

[0195] The evaluation indicators are Acc and w-F1 score. Accuracy Acc represents the proportion of all correctly classified samples to the total samples, which is a simple and intuitive indicator for evaluating classifier performance. For the multimodal emotion recognition task, its calculation formula (19) is shown as follows:

[0196]

[0197] Among them, TP (True Positives) refers to the number of samples correctly identified as a certain emotion category. TN (True Negatives) refers to the number of samples correctly identified as not belonging to a certain emotion category. FP (False Positives) refers to the number of samples incorrectly identified as a certain emotion category. FN (False Negatives) refers to the number of samples incorrectly identified as not belonging to a certain emotion category.

[0198] The w-F1 score (Weighted F1 Score) is the weighted average of the F1 scores, where the w-F1 scores of different categories are weighted and summed according to the number of samples in the data set. The w-F1 score is the harmonic mean of precision and recall, and is used to measure the model performance in binary or multi-classification problems. As shown in formula (20):

[0199]

[0200] Among them, freq k is the relative frequency of category k.

[0201] Tables 6, 7, 8 and 9 respectively show the comparison of the accuracy and training effect of four-category and six-category emotion recognition in the four-category and six-category emotion recognition tasks between the model after adding the AKFC audio and video feature extraction network and the SOTA (State Of The Art) model with the IEMOCAP dataset as the specific task. The bold data represents the best accuracy in this table. According to the situation, this embodiment selects the following models: bc-LSTM, CMN, ICON, DialogueRNN, DialogueGCN

[47] , MMGCN, DialougueCRN, CHFusion, COGMEN, and compares the accuracy of four-category and six-category emotion recognition with the model of the present invention (Transformer+AKFC).

[0202] Table 6 Comparison of model accuracy for four-category emotion recognition task

[0203]

[0204] In Table 6, the Acc and w-F1 accuracies in the four-category emotion recognition scenario are 0.69% and 2.81% higher than the best model COGMEN studied by previous researchers, which is 0.4% and 2.39% higher than the baseline model (Transformer).

[0205] Table 7 Comparison of accuracy of six-category emotion recognition task models

[0206]

[0207] In Table 7, the accuracy of Acc and w-F1 in the six-category emotion recognition scenario is 0.42% and 0.04% higher than the best model COGMEN studied by previous researchers, and is 0.73% and 0.86% higher than the baseline model. According to Tables 6 and 7, after the baseline model is added with the AKFC audio and video feature extraction network, the accuracy of Acc and w-F1 for emotion recognition in four / six categories on the IEMOCAP dataset have been significantly improved. In the four-category emotion recognition task, Acc and w-F1 reached 82.98% and 84.96% respectively. In the six-category emotion recognition task, Acc and w-F1 reached 67.46% and 67.31% respectively.

[0208] Table 8 Comparison of training results between four-category emotion recognition task and baseline model

[0209]

[0210] Table 9 Comparison of training results between six-category emotion recognition task and baseline model

[0211]

[0212]

[0213] According to the comparison of the results in Table 8 and Table 9, based on the training round epoch=50, learning rate lr=0.0003, 3060 GPU training parameters and training process, in the four-category and six-category emotion recognition scenarios, after adding the AKFC audio and video feature extraction network, compared with the baseline model, the average training losses of the four-category and six-category classification were reduced by 0.0406 and 0.0424 respectively, and the average training time was reduced by 0.031s and 0.1231s respectively, and the model parameters (that is, the number of learnable parameters included in the training and inference process, which measures the complexity and computing requirements of the model) did not increase.

[0214] In the four-category scenario, after adding the AKFC audio and video feature extraction network, compared with the bc-LSTM, CHFusion, and COGMEN models, it can be clearly seen that the average training loss, average training time, and model parameters have all been significantly reduced. In the six-category scenario, after adding the AKFC audio and video feature extraction network, compared with the bc-LSTM, CMN, ICON, DialogueRNN, DialogueGCN, MMGCN, DialogueCRN, and COGMEN models, it can be clearly seen that the average training loss, average training time, and model parameters have all been significantly reduced, and all achieved good results.

[0215] Therefore, it can be judged that after adding the AKFC audio and video feature extraction network to the baseline model, the model recognition accuracy training effect of the baseline model in the four-category and six-category emotion recognition scenarios has been improved.

[0216] The following public datasets are used to conduct four-category / six-category emotion recognition ablation experiments on IEMOCAP. This experiment mainly verifies the ablation of different modules of the multimodal emotion recognition model proposed in this invention. The training process uses the Adam optimizer, with the learning rate set to 0.0003, the drop_rate to 0.5, the batch_size to 8, and the epoch to 50. The designed network is trained on a single RTX3060 GPU with 12GB video memory.

[0217] First, the effects of different modules of the model are verified, and the results are shown in Tables 10 and 11. The bold data represent the best data in this table.

[0218] Table 10 Comparison of ablation test accuracy of different modules of the model

[0219]

[0220] Table 11 Comparison of ablation experiment training effects of different modules of the model

[0221]

[0222]

[0223] It is obvious from Tables 10 and 11 that after adding AKConv to the baseline model (i.e. Transformer), in the four-category and six-category emotion recognition scenarios, for the four-category and six-category emotion recognition scenarios, Acc and w-F1 have been improved by 1.86% and 1.86% (four-category results), and 1.78% and 1.43% (six-category results) respectively compared with the baseline model, and the average training loss and average training time have been improved. Compared with the baseline model, in the four-category and six-category emotion recognition scenarios, the average training loss and average training time are shortened by 0.09 seconds and 0.03 seconds (four-category results) and 0.04 seconds and 0.06 seconds (six-category results).

[0224] After adding MSAF to the baseline model, in the four-category and six-category emotion recognition scenarios, Acc and w-F1 were improved by 2.34% and 2.34% (four-category results) and 0.62% and 0.5% (six-category results) compared with the baseline model. Compared with the baseline model, in the four-category and six-category emotion recognition scenarios, the average training loss was reduced by 0.15 (four-category) and 0.89 (six-category), respectively, but the average training time was increased by 1.81 seconds (four-category) and 1.56 seconds (six-category), respectively. After adding AKConv and MSAF to the baseline model, in the four-category and six-category emotion recognition scenarios, Acc and w-F1 are improved by 3.08% and 3.09% (four-category) and 3.33% and 3.31% (six-category) compared with the baseline model, and the average training loss and average training time are improved (i.e., the time is shortened) compared with the case when only the MSAF module is added. Compared with the baseline model, in the four-category and six-category emotion recognition scenarios, the average training loss is reduced by 0.27 and 1.03, respectively, but the average training time is increased (i.e., shortened) by 0.33 seconds and 0.11 seconds, respectively, compared with the case when only the MSAF module is added. Although the average training time is increased by 1.48 seconds and 1.44 seconds compared with the baseline model, after adding the MSAF module, the emotion recognition model based on multimodal split attention fusion improves the model's attention control ability for different modal features compared with the baseline model, and can more effectively integrate multimodal information. It can flexibly adjust the weights of different modal features and make adaptive adjustments according to task requirements. And through the attention mechanism, it can dynamically capture the importance of each modal feature, thereby improving the performance of the model. Specifically, it improves the accuracy in four-category and six-category emotion recognition tasks, and effectively reduces the average training loss during training. Therefore, introducing AKConv and MSAF on the basis of the baseline model can effectively improve the model performance while minimizing the computational cost.

[0225] The training process uses the Adam optimizer, with the learning rate set to 0.0003, the drop_rate set to 0.5, the batch_size set to 8, and the epoch set to 50. The designed network is on a single RTX 3060G with 12GB of video memory. The experiment uses the IEMOCAP dataset. First, the IEMOCAP dataset is preprocessed, and the data needs to be loaded and cleaned to ensure the integrity and consistency of the data.

[0226] In a multimodal emotion recognition model, different unimodal features may have different effects on accuracy, depending on factors such as the characteristics of the data set, task requirements, and model design. In general, each unimodal feature will have a certain impact on the final emotion recognition accuracy, but its importance and degree of influence may vary. Features between different modalities may be correlated and complementary, so combining them can improve the accuracy and robustness of emotion recognition. When designing a multimodal emotion recognition model, it is necessary to consider the importance and degree of influence of each unimodal feature, and take appropriate methods to fuse them together to achieve better recognition results. Based on this, this embodiment explores the impact of different unimodal features on emotion recognition effects and performance.

[0227] According to Table 12, Fig. 9 , Fig.10 It can be seen that among the single-modal features, the text features perform better in the four-category / six-category emotion recognition tasks, and among the bimodal features, the audio+text bimodal features perform better in the four-category / six-category emotion recognition tasks. And it can be clearly seen that the accuracy of the audio+video+text multimodal features in the four-category and six-category emotion recognition tasks of IEMOCAP is better than that of the audio+text bimodal features. In the four-category scenario, the Acc and w-F1 indicators are 1.57% and 1.56% higher, respectively, and in the six-category scenario, the Acc and w-F1 indicators are 3.02% and 3.24% higher, respectively.

[0228] Table 12 Four-class / six-class emotion recognition accuracy table under different modal features

[0229]

[0230] like Fig. 9 As shown in the figure, in the IEMOCAP (4-category) emotion recognition scenario, the Acc and w-F1 accuracies of single-modal emotion recognition through text reached 80.76% and 80.85% respectively (green in the figure represents ACC, and yellow represents w-F1), which is much higher than the Acc and w-F1 accuracies of single-modal emotion recognition through audio or video, which are 63.68%, 43.65% and 61.40%, 43.08%. The Acc and w-F1 accuracies of multi-modal emotion recognition through three modalities of text + audio + video reached 84.14% and 84.12% respectively, which is better than the accuracies of other emotion recognition based on single or multi-modality. It can be concluded that it is necessary to perform multi-modal emotion recognition in the four-category emotion recognition task.

[0231] like Fig.10As shown in the figure, in the IEMOCAP (6-category) emotion recognition scenario, the Acc and w-F1 accuracies of single-modal emotion recognition through text reached 65.38% and 65.08% respectively, which are much higher than the Acc and w-F1 accuracies of single-modal emotion recognition through audio or video, which are 49.07%, 32.83% and 48.03%, 32.78%. The Acc and w-F1 accuracies of multi-modal emotion recognition through text + audio + video three modalities reached 69.01% and 69.19% respectively, which are better than the accuracies of emotion recognition based on single or multi-modality. It can be concluded that multi-modal emotion recognition is necessary in the six-category emotion recognition task.

[0232] Next, the model of the present invention is compared with the SOTA benchmark model using the IEMOCAP dataset for four / six-category emotion recognition accuracy. The experimental results show that the model proposed by the present invention achieves better results than the above models. The specific comparison of the four / six-category emotion recognition accuracy is shown in Tables 13 and 14.

[0233] Table 13 Comparison of the accuracy of the proposed model (Ours) and the existing model in four emotion recognition on the IEMOCAP dataset

[0234]

[0235] Table 14 Comparison of the accuracy of our model (Ours) and existing models in recognizing 6 emotions on the IEMOCAP dataset

[0236]

[0237] The bold data in Table 13 and Table 14 represent the best accuracy in this table. According to Table 13 and Table 14, the multimodal emotion recognition model based on MSAF and RGCN proposed in the present invention has achieved good accuracy for emotion recognition of four / six categories on the IEMOCAP dataset.

[0238] In Table 14, the emotion recognition accuracy of the model proposed in this paper is slightly lower than that of COGMEN in Neutral, but the other emotion recognition accuracies are better than those of COGMEN model, especially in Acc and w-F1 accuracy, which are increased by 1.97% and 1.73% (relative to COGMEN). Therefore, it can be judged that the model proposed in this paper has achieved good performance in both the four-category emotion recognition classification and the six-category emotion recognition classification tasks.

[0239] It should be noted that the six-category emotions in Tables 13 and 14 include: happiness, sadness, neutral, anger, excitement and frustration; the four-category emotions include happiness, sadness, neutral and anger.

[0240] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. An emotion recognition method based on multimodal split attention fusion, characterized in that: include: Acquire multimodal data to be identified, wherein the multimodal data to be identified is associated with three modalities, wherein the three modalities include audio, video, and text, and the multimodal data to be identified is a conversation, and the conversation includes two or more utterances; The emotion recognition model is used to perform emotion recognition on the multimodal data to be recognized, wherein the emotion recognition model includes: an initial feature extraction module, an advanced feature extraction module, an MSAF module and an emotion classifier. The initial feature extraction module includes an initial text feature extraction unit, an initial audio feature extraction unit and an initial video feature extraction unit, wherein the initial text feature extraction unit uses the Transformer model as an encoder to extract the initial text features of each utterance; the initial audio feature extraction unit and the initial video feature extraction unit both use the AKFC network as an encoder to extract the initial audio features and initial video features of each utterance respectively, and in the AKFC network, the AKConv module and the fully connected layer are used to extract the initial audio features and initial video features; The advanced feature extraction module includes a context information acquisition unit and a paired cross-modal feature interaction unit, wherein the context information acquisition unit includes a relational graph convolutional network RGCN and a Graph Transformer model, the RGCN is used to obtain the local context information of each utterance in the conversation and obtain the first feature representation of each modality of each utterance; the Graph Transformer model is used to obtain the second feature representation of each modality of each utterance according to the first feature representation of each modality of each utterance; The paired cross-modal feature interaction unit is used to perform cross-modal attention on any two modalities to obtain a third feature representation; The MSAF module is used to fuse the second feature representation and the third feature representation obtained by the context information acquisition unit and the paired cross-modal feature interaction unit to obtain a fused feature; The sentiment classifier module is used to perform sentiment recognition on each utterance based on the fused features.

2. The emotion recognition method based on multimodal split attention fusion according to claim 1, characterized in that: The emotion recognition is a four-category emotion recognition.

3. The emotion recognition method based on multimodal split attention fusion according to claim 1, characterized in that: The emotion recognition is six-category emotion recognition.

4. The emotion recognition method based on multimodal split attention fusion according to claim 1, characterized in that: The AKConv module uses convolution kernels of variable shape and size for convolution, and a fully connected layer is directly connected after each AKConv module to extract and initially integrate local features of audio and video to form an initial feature representation for subsequent processing.

5. The emotion recognition method based on multimodal split attention fusion according to claim 1, characterized in that: The method further includes training the emotion recognition model, wherein the training process includes: Obtain training samples, each of which is multimodal data, each of which is associated with three modalities, wherein the three modalities include audio, video, and text, each of which is a conversation, each of which includes two or more utterances, and each utterance in each training sample corresponds to an emotion label; Build an emotion recognition model; For each training sample, Extracting initial text features, initial audio features, and initial video features from each utterance of the training sample using an initial feature extraction module; Performing a cascade operation on the initial text feature, the initial audio feature, and the initial video feature for fusion, and generating a first fusion feature corresponding to each utterance in the training sample; Inputting the first fused feature into a context information acquisition unit and a paired cross-modal feature interaction unit to generate a second feature representation and a third feature representation corresponding to each utterance in the training sample; Inputting the second feature representation and the third feature representation into the MSAF module to generate a second fusion feature corresponding to the utterance; Inputting the second fusion feature into an emotion classifier to obtain an emotion prediction result corresponding to the speech; According to the emotion prediction results and corresponding emotion labels corresponding to all training samples, the parameters in the emotion recognition model are adjusted until the preset conditions are met.

6. The emotion recognition method based on multimodal split attention fusion according to claim 5, characterized in that: The initial feature extraction module is used to extract the initial text features, initial audio features, and initial video features in the training samples, which is achieved through the following formula: in, r i a and r i v Respectively represent the set of utterances in the training sample {r1,r2,...,r i ,...,r N Any multimodal data r in i , that is, discourse r i , the text data, audio data and video data in, i = 1, 2, ..., N, N represents the number of utterances in a conversation; and Respectively represent the discourse r i corresponding initial text features, initial audio features, and initial video features; represents the Transformer model used to extract initial text features, i.e., the learnable parameters of Transformer(·); represents the learnable parameters of the fully connected layer in the AKFC network used to extract the initial audio features, i.e., AKFC1(·); represents the learnable parameters of the fully connected layer in the AKFC network used to extract the initial video features, i.e., AKFC2(·).

7. The emotion recognition method based on multimodal split attention fusion according to claim 6, characterized in that: Inputting the first fused feature into the context information acquisition unit to generate a second feature representation corresponding to the utterance includes: For each conversation, a multimodal graph is established, represented as G(V, R, E), where V represents the set of discourse nodes with three modal types in the multimodal graph, E represents the set of edges in the multimodal graph, and R represents the set of relationship types in the multimodal graph. i Generate three nodes, namely r i t 、r i a and r i v , τ∈{a,v,t}, a, v and t correspond to the audio modality, video modality and text modality respectively, r ij ∈R, R contains R multi and R temp , R multi represents the set of internal connections between the three modalities in the same utterance, reflecting multimodal interaction, R temp represents the set of interconnected utterances of the same modality within a specified time window. According to the multimodal graph, determine the first feature representation of any modality τ in the i-th utterance Right now: Among them, r represents any relationship type, N r (i) represents the set of neighbor nodes of the ith node under relation r, · represents the number of elements in the set, W r and W0 represent the learnable parameters in RGCN; The first feature is represented as Input the GraphTransformer model to obtain the second feature representation of any modality τ in the i-th utterance Right now: Among them, W1 and W2 represent the learnable parameters in the GraphTransformer model. represents the cascade of D-head attention, represents the attention coefficient of node j, which is calculated by the softmax activation function, that is: Where d represents the scaling factor, W3 and W4 represent learnable parameters; Then, with the training samples {r1,r2,...,r i ,...,r N The second feature corresponding to} is expressed as 8. The emotion recognition method based on multimodal split attention fusion according to claim 7, characterized in that: The first fused features are input into a paired cross-modal feature interaction unit to generate a third feature representation, including: Perform cross-modal attention between any two modalities τ1 and τ2 to generate a third feature representation and in, τ1 and τ2 take any two modes in {a, v, t}, and τ1 and τ2 take different modes; queue key value queue key value and represents the learnable parameters of the pairwise cross-modal feature interaction unit, σ represents the softmax function, and d k1 express The dimension, d k2 express Dimension.

9. The emotion recognition method based on multimodal split attention fusion according to claim 8, characterized in that: Inputting the second feature representation and the third feature representation into the MSAF module to generate a second fusion feature corresponding to the speech in the training sample includes: The MSAF module obtains the feature map set {F1, F2, ..., F Mod }, where Mod represents the total number of modalities, i.e., Mod=3, each feature map in the feature map set corresponds to a modality, and the feature map contains a set of second feature representations and third feature representations corresponding to the modality; Each feature map is divided into equal channel feature blocks according to the channel. The number of channels in each equal channel feature block is C. The set of feature blocks belonging to the modality mod is represented as B mod , D represents the total number of channels corresponding to the modal features, Indicates the number of divided equal channel feature blocks, Indicates rounding up, C mod represents the number of channels corresponding to a single modal feature, mod∈{1,2,...,Mod}, i∈{1,2,...,|B mod |}, Indicates B mod The i-th equal channel feature block in ; Connect all equal-channel blocks corresponding to the modality mod, and add the equal-channel feature blocks belonging to the modality mod to the shared representation D mod By calculating B mod Element-wise sum S mod , and perform global average pooling in the spatial dimension, which can be expressed as: Among them, N i represents the size of each spatial dimension, K represents the number of spatial dimensions of the modal mod, n1,n2,...,n K represents the position index of each spatial dimension of the feature map, c represents the channel index of the feature map, and D mod (c) represents the aggregate eigenvalue of the modality on a specific channel c; For each Generated by applying a linear transformation to Z And use softmax activation to get the block-by-block attention weights The formula is: in, and Indicates that Convert to The weights and biases of the fully connected layers; Using block-wise attention weights and the regularization factor λ of the minimum attention of the split, determining the optimized feature block The formula is: Among them, ⊙ represents the element-by-element product; The optimized feature blocks of mod belonging to the modality Generate optimized feature maps by combining channel-by-channel Optimizing the collection of feature maps That is, the second fusion feature of the training sample generated by the MSAF module.

10. The emotion recognition method based on multimodal split attention fusion according to claim 8, characterized in that: The second fusion feature is input into the emotion classifier to obtain the emotion prediction result corresponding to the speech, including using the following formula for prediction: in i =ReLU(Φ0h i +b0), p i =softmax(Φ1v i +b1), Among them, h i represents the second fusion feature corresponding to the i-th utterance, v i represents the result after using the ReLU activation function, Φ0 and b0 represent the learnable parameters of ReLU, and p i represents the probability distribution obtained by the softmax activation function, Φ1 and b1 represent the learnable parameters of softmax, Table 1 is the predicted emotional label for the i-th utterance, argmax represents the value that makes p i The function that finds the predicted label when the value is the maximum.

Images