An Emotion Recognition Method Based on Multimodal Split Attention Fusion

Abstract

This invention discloses an emotion recognition method based on multimodal split attention fusion, comprising: acquiring multimodal data to be recognized; performing emotion recognition on the multimodal data using an emotion recognition model, the emotion recognition model comprising: an initial feature extraction module, an advanced feature extraction module, an MSAF module, and an emotion classifier; the initial feature extraction module comprising an initial text feature extraction unit, an initial audio feature extraction unit, and an initial video feature extraction unit, the initial text feature extraction unit employing a Transformer model as the encoder; the initial audio feature extraction unit and the initial video feature extraction unit employing an AKFC network as the encoder; the advanced feature extraction module comprising a context information acquisition unit and a pairwise cross-modal feature interaction unit, the context information acquisition unit comprising an RGCN and a Graph Transformer model; the MSAF module for determining fusion features; and the emotion classifier module for performing emotion recognition. This invention can improve the accuracy of emotion recognition.

Description

Technical Field

[0001] This invention belongs to the field of emotion recognition, specifically relating to an emotion recognition method based on multimodal split attention fusion. Background Technology

[0002] In multimodal emotion recognition tasks, effectively learning the complex nonlinear relationships between different modalities after extracting audio and visual features from audio and video datasets respectively has always been a major problem in real-world scenarios. The traditional solution is to use fully connected layers for feature extraction from audio and video. While fully connected layers have advantages such as handling feature vectors of arbitrary size and learning complex nonlinear relationships between features, they also suffer from problems such as a large number of parameters and high computational cost.

[0003] Multimodal feature fusion generally requires learning data from at least two modalities. How to utilize the complementarity and redundancy of multiple modalities to represent and summarize multimodal data for multimodal learning is an important issue in the multimodal field. Due to the inherent heterogeneity between modalities, such as the difference between speech and audio signals, learning representations of multimodal data is particularly challenging. In the context of multimodal emotion recognition tasks, effectively fusing the obtained multimodal representations poses a significant challenge. Studies have shown that different feature fusion methods can significantly affect the robustness, stickiness, and recognition accuracy of the model. In addition, the feature fusion design within different network architectures also plays a crucial role. Currently, the popular feature fusion methods are mainly of the following four types: (1) early fusion, which integrates modal features immediately after extraction, usually by simply concatenating these features; (2) late fusion, which integrates the results after each modality makes a decision (e.g., regression or classification). Common methods include average decision, voting model, weighted average based on channel noise and signal variance, or combining additional networks to learn decisions; (3) Hybrid fusion, which combines the principles of early fusion and late fusion, utilizing the hybrid features of early fusion and the multiple modes of late fusion. The output of this fusion is then used for network prediction; (4) Model fusion, which is an ensemble learning method that combines the prediction results of multiple independent models to produce a more accurate and robust overall prediction or decision.

[0004] However, existing multimodal emotion recognition models suffer from problems such as insufficient feature fusion and high model complexity, which limit the performance and application scope of the models. Summary of the Invention

[0005] In view of this, in order to address the problem of insufficient feature fusion in traditional multimodal emotion recognition models, this invention adopts a modal fusion feature fusion method and provides an emotion recognition method based on multimodal split attention fusion.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] An emotion recognition method based on multimodal split attention fusion includes:

[0008] Acquire multimodal data to be identified, which is associated with three modalities, including audio, video and text, and the multimodal data to be identified is a session, which includes two or more utterances;

[0009] An emotion recognition model is used to perform emotion recognition on multimodal data to be identified. This model includes: an initial feature extraction module, a high-level feature extraction module, an MSAF module, and an emotion classifier.

[0010] The initial feature extraction module includes an initial text feature extraction unit, an initial audio feature extraction unit, and an initial video feature extraction unit. The initial text feature extraction unit uses a Transformer model as an encoder to extract the initial text features of each utterance. The initial audio feature extraction unit and the initial video feature extraction unit both use an AKFC network as an encoder to extract the initial audio features and initial video features of each utterance, respectively. In the AKFC network, the AKConv module replaces the convolutional layers in the baseline model and combines them with fully connected layers to extract the initial audio features and initial video features.

[0011] The advanced feature extraction module includes a context information acquisition unit and a pairwise cross-modal feature interaction unit. The context information acquisition unit includes a relational graph convolutional network RGCN and a graph transformer model. RGCN is used to acquire the local context information of each utterance in the session and obtain the first feature representation of each modality of each utterance. The graph transformer model is used to obtain the second feature representation of each modality of each utterance based on the first feature representation of each modality of each utterance.

[0012] Paired cross-modal feature interaction units are used to perform cross-modal attention on any two modalities to obtain a third feature representation;

[0013] The MSAF module is used to fuse the second and third feature representations obtained by the context information acquisition unit and the paired cross-modal feature interaction unit to obtain fused features.

[0014] The emotion classifier module is used to identify the emotion of each utterance based on fused features.

[0015] Furthermore, the emotion recognition is a four-class emotion recognition.

[0016] Furthermore, the emotion recognition is a six-category emotion recognition.

[0017] Furthermore, the baseline model is Transformer. The AKConv module is used to replace the convolutional layers in the baseline model, and a fully connected layer is used to extract the initial audio or video features. Specifically, the AKConv module is used to replace the convolutional layers in the preprocessing part of the input data in the baseline model. A fully connected layer is directly connected after each AKConv module to achieve the extraction and preliminary integration of local features, forming an initial feature representation for subsequent processing.

[0018] Furthermore, the method also includes training an emotion recognition model, wherein the training process includes:

[0019] Acquire training samples. Each training sample is multimodal data, and each multimodal data is associated with three modalities, including audio, video and text. Each training sample is a conversation, and each conversation includes two or more utterances. Each utterance in each training sample corresponds to an emotion label.

[0020] Build an emotion recognition model;

[0021] For each training sample,

[0022] The initial text features, initial audio features, and initial video features of each utterance in the training samples are extracted using the initial feature extraction module.

[0023] The initial text features, initial audio features, and initial video features are concatenated and fused to generate the first fused feature corresponding to each utterance in the training samples.

[0024] The first fusion feature is input into the context information acquisition unit and the pairwise cross-modal feature interaction unit to generate the second feature representation and the third feature representation corresponding to each utterance in the training samples;

[0025] The second and third feature representations are input into the MSAF module to generate a second fusion feature corresponding to the utterance.

[0026] The second fusion feature is input into the emotion classifier to obtain the emotion prediction result corresponding to the utterance;

[0027] Based on the emotion prediction results and corresponding emotion labels of all training samples, the parameters in the emotion recognition model are adjusted until the preset conditions are met.

[0028] Furthermore, the initial text features, initial audio features, and initial video features in the training samples are extracted using the initial feature extraction module, achieved through the following formula:

[0029]

[0030] Where, r i t r i a and r i v Let {r1, r2, ..., r} represent the sets of utterances in the training samples. i ,...,r N Any multimodal data r in} i That is, discourse r i The data in the , i = 1, 2, ..., N, is text data, audio data and video data, i = 1, 2, ..., N, where N represents the number of utterances in a session; and Representing the discourse r i The corresponding initial text features, initial audio features, and initial video features; This represents the Transformer model used to extract initial text features, i.e., the learnable parameters of Transformer(·); This represents the learnable parameters of the fully connected layers in the AKFC network used to extract initial audio features, i.e., AKFC1(·). This represents the learnable parameters of the fully connected layers in the AKFC network used to extract initial video features, specifically in AKFC2(·).

[0031] Furthermore, the first fusion feature is input into the context information acquisition unit to generate a second feature representation corresponding to the utterance, including:

[0032] For each session, a multimodal graph is constructed, denoted as G(V, R, E), where V represents the set of utterance nodes with three modalities in the multimodal graph, E represents the set of edges in the multimodal graph, R represents the set of relation types in the multimodal graph, and each utterance r i Generate three nodes, namely r i t r i a and r i v , a, v, and t correspond to the audio modality, video modality, and text modality, respectively, and r ij ∈R, R contains R multi and R temp R multi Representing the set of internal connections between the three modalities within the same discourse, reflecting multimodal interaction, R temp This represents the set of interconnections among utterances of the same modality within a specified time window.

[0033] Based on the multimodal graph, determine the first feature representation of any modality τ in the i-th utterance. Right now:

[0034]

[0035] Where r represents any relation type, N r (i) represents the set of neighboring nodes of the i-th node under relation r, |·| represents the number of elements in the set, W r W0 represents the learnable parameters in RGCN;

[0036] Represent the first feature Inputting the Graph Transformer model yields the second feature representation of any modality τ in the i-th utterance. Right now:

[0037]

[0038] Where W1 and W2 represent the learnable parameters in the Graph Transformer model, This represents the cascade of D-head attention. The attention coefficient of node j is calculated by the softmax activation function, i.e.:

[0039]

[0040] Where d represents the scaling factor, and W3 and W4 represent learnable parameters;

[0041] Then, with the training samples {r1,r2,...,r...} i ,...,r N The second feature corresponding to} is represented as

[0042] Furthermore, the first fused feature is input into a pairwise cross-modal feature interaction unit to generate a third feature representation, including:

[0043] Perform cross-modal attention between any two modes τ1 and τ2 to generate a third feature representation. and in,

[0044]

[0045] τ1 and τ2 can take any two modes from {a,v,t}, and τ1 and τ2 can take different modes;

[0046] queue key value

[0047] queue key value

[0048] and The learnable parameters of the pairwise cross-modal feature interaction units are represented by σ, which represents the softmax function, and d k1 express The dimension, d k2 express Dimensions.

[0049] Further, the second and third feature representations are input into the MSAF module to generate a second fusion feature corresponding to the utterances in the training samples, including:

[0050] The MSAF module obtains the feature map set {F1, F2, ..., F...} Mod}, where Mod represents the total number of modes, i.e. Mod = 3, each feature map in the feature map set corresponds to a mode, and the feature map contains a set of second feature representations and third feature representations corresponding to the mode;

[0051] Each feature map is divided into equal-channel feature blocks, with C channels in each block. The set of feature blocks belonging to the modality is represented as B. mod D represents the total number of channels corresponding to the modal feature. This indicates the number of equal-channel feature blocks in the division. Indicates rounding up, C mod This represents the number of channels corresponding to a single modal feature, where mod ∈ {1, 2, ..., Mod}. B mod The i-th isochannel feature block in;

[0052] Perform a concatenation operation on all isochannel blocks corresponding to the modality (mod), and add the isochannel feature blocks belonging to the modality (mod) to the shared representation D. mod In the middle, by calculating B mod element-wise sum of S mod And perform global average pooling in the spatial dimension, expressed by the formula:

[0053]

[0054] Where, N i Let K represent the size of each spatial dimension, and let K represent the number of spatial dimensions of the modality, n1, n2, ..., n. KLet c represent the position index in each spatial dimension of the feature map, and let D represent the channel index of the feature map. mod (c) represents the aggregated feature value of the mode on a specific channel c;

[0055] For each Generate by applying a linear transformation to Z And use softmax activation to obtain block-by-block attention weights Expressed as a formula:

[0056]

[0057] in, and Indicates will Convert to The weights and biases of the fully connected layer;

[0058] Use block-by-block attention weights The minimum attention regularization factor λ for splitting is used to determine the optimal feature blocks. Expressed as a formula:

[0059]

[0060] Where ⊙ represents element-wise multiplication;

[0061] Optimize the feature blocks belonging to the modality. Optimized feature maps are generated by channel-by-channel joining.

[0062] Optimize the set of feature maps This refers to the second fusion feature of the training samples generated by the MSAF module.

[0063] Furthermore, the second fused feature is input into the emotion classifier to obtain the emotion prediction result corresponding to the utterance, including prediction using the following formula:

[0064] v i =ReLU(Φ0h) i +b0),

[0065] p i =softmax(Φ1v) i +b1),

[0066]

[0067] Among them, h i v represents the second fusion feature corresponding to the i-th utterance. iThis represents the result after applying the ReLU activation function, where Φ0 and b0 represent the learnable parameters of ReLU, and p i Let represent the probability distribution obtained by the softmax activation function, and Φ1 and b1 represent the learnable parameters of softmax. The table represents the predicted sentiment label for the i-th utterance, and argmax represents the expression that makes p... i The function that calculates the predicted label when the maximum value is reached.

[0068] The beneficial effects of this invention are:

[0069] This invention proposes the AKFC network, which combines the advantages of fully connected layers for audio and video feature extraction while effectively reducing the model's computation time. Ultimately, this improves the audio and video feature extraction capabilities of fully connected layer models.

[0070] The modality fusion feature fusion method solves the problems of loss of multimodal partial information, redundancy of different modal features, and limitation of feature representation. By using multimodal split attention fusion to perform multimodal feature fusion of audio, video and text, MSAF allows the model to dynamically allocate the weights of different modal features during the feature fusion process, thereby better capturing the complex relationships between multimodal information.

[0071] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0072] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0073] Figure 1 This is an architecture diagram of an emotion recognition method based on multimodal split attention fusion;

[0074] Figure 2 It features flexible parameters and an adaptive sampling shape graph;

[0075] Figure 3 This is the AKConv architecture diagram;

[0076] Figure 4 This is a schematic diagram of AKFC audio and video feature extraction;

[0077] Figure 5 This is a diagram illustrating the relationship between text, video, and audio discourse nodes.

[0078] Figure 6 This is a schematic diagram of paired cross-modal feature interactions;

[0079] Figure 7 It is a flowchart of paired cross-modal feature interactions;

[0080] Figure 8 This is a schematic diagram of the MSAF model;

[0081] Figure 9 These are the IEMOCAP (4-class) emotion recognition accuracy maps under different modal features;

[0082] Figure 10 This is an IEMOCAP (6-class) emotion recognition accuracy map under different modal features. Detailed Implementation

[0083] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be understood that the preferred embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0084] This application proposes an emotion recognition method based on multimodal split attention fusion. The method uses an AKFC network for audio and video feature extraction, thus addressing the high model complexity issue in multimodal emotion recognition models. Next, it employs an RGCN+Graph Transformer and a pairwise cross-modal feature interaction module to further extract audio, video, and text features. The RGCN+Graph Transformer captures contextual information, while the pairwise cross-modal feature interaction module enables cross-modal feature interaction. The output of the RGCN+Graph Transformer and the pairwise cross-modal feature interaction module serves as the input to Multimodal Split Attention Fusion (MSAF) for multimodal feature fusion. Finally, this input is used as the input to a linear classifier to output the emotion recognition result.

[0085] Figure 1 This is an architecture diagram of an emotion recognition method based on multimodal split attention fusion. Figure 1 An emotion recognition method based on multimodal split attention fusion includes:

[0086] Acquire multimodal data to be identified, which is associated with three modalities, including audio, video and text, and the multimodal data to be identified is a session, which includes two or more utterances;

[0087] An emotion recognition model is used to perform emotion recognition on multimodal data to be identified.

[0088] The emotion recognition model includes: an initial feature extraction module, an advanced feature extraction module, an MSAF module, and an emotion classifier.

[0089] The initial feature extraction module includes an initial text feature extraction unit, an initial audio feature extraction unit, and an initial video feature extraction unit. The initial text feature extraction unit uses a Transformer model as an encoder to extract the initial text features of each utterance. The initial audio feature extraction unit and the initial video feature extraction unit both use an AKFC network as an encoder to extract the initial audio features and initial video features of each utterance, respectively.

[0090] The advanced feature extraction module includes a context information acquisition unit and a pairwise cross-modal feature interaction unit. The context information acquisition unit includes a relational graph convolutional network RGCN and a graph transformer model. RGCN is used to acquire the local context information of each utterance in the session and obtain the first feature representation of each modality of each utterance. The graph transformer model is used to obtain the second feature representation of each modality of each utterance based on the first feature representation of each modality of each utterance.

[0091] Paired cross-modal feature interaction units are used to perform cross-modal attention on any two modalities to obtain a third feature representation;

[0092] The MSAF module is used to fuse the second and third feature representations obtained by the context information acquisition unit and the paired cross-modal feature interaction unit to obtain fused features.

[0093] The emotion classifier module is used to identify the emotion of each utterance based on fused features. Emotion recognition can be either four-class or six-class emotion recognition.

[0094] Because traditional standard convolution sampling positions are fixed, convolution can only extract local information from the current window and cannot fully capture information from other locations. Furthermore, traditional standard convolution does not allow the kernel to have an arbitrary number of parameters. As the kernel size increases, the number of parameters in traditional standard convolution often increases quadratically, which places stringent demands on the hardware environment. In other words, traditional convolution has two drawbacks: first, the convolution operation runs within a fixed-size window, unable to capture information from other windows, and the window shape is fixed; second, the kernel size is fixed at k×k, the window size is fixed at k, and as k increases, the number of parameters increases rapidly.

[0095] Traditional convolutions typically have fixed sizes and shapes, such as 3x3 or 5x5 square grids. The core principle of AKConv, however, is to allow convolutional kernels to have a deformable number of parameters. This means it's no longer limited to standard square networks, but can adopt more diverse and flexible shapes based on graphical features and task requirements, such as... Figure 2 The number of flexible parameters is shown. The AKConv convolutional kernel can automatically adjust its sampling shape when processing different images and targets. This is achieved by introducing a novel coordinate generation algorithm that can generate initial sampling coordinates for convolutional kernels of different sizes and shapes, such as... Figure 1 The adaptive sampling shape is shown in the image. Figure 2 The adaptive sampling shape shown.

[0096] AKConv first determines the initial sampling positions of the convolution kernel using a coordinate generation algorithm. These positions are no longer fixed but can dynamically change based on the features and targets in the image. To better adapt to changes in the size and shape of targets in the image, AKConv adjusts the sampling positions of the convolution kernel according to the characteristics of the target. This adjustment is achieved by adding an offset, making the convolution operation more flexible and practical.

[0097] AKConv can adapt to any shape, offering unparalleled applicability in processing visual and audio data compared to traditional convolution. Standard convolution is a k×k rectangle, while deformable convolution (Deformable Conv) can adjust its shape. Similar to deformable convolution, AKConv learns offsets to change the shape of the convolution kernel. The number of parameters in AKConv is arbitrary, thus overcoming the k×k limitation of traditional convolution and effectively improving computational speed. N is the number of AKConv convolution parameters. The feature map undergoes convolution to obtain the convolution position offset, and then the convolution operation is performed, just like with deformable convolution. Deformable convolution can change the position of the convolution calculation by learning offsets, making the shape of the convolution kernel flexible. However, deformable convolution has a drawback: the kernel parameters are fixed, such as 1, 9, 27. Therefore, this invention uses AKConv convolution.

[0098] Figure 3 This is the AKConv architecture diagram. (Example) Figure 3 As shown, firstly, the offset of the corresponding kernel is obtained through convolution operation, which has dimensions (B, 2N, H, W), where N is the size of the convolution kernel, B represents the number of offset groups, H represents the height of the data, and W represents the width of the data. Figure 3 In this case, N = 5. The offset and the original coordinates are added together: P0 + P n This is used to obtain the modified coordinates. Finally, interpolation and resampling are used to obtain the features at the corresponding locations.

[0099] Features corresponding to sampling locations of irregular convolutional kernels are difficult to extract. Features are stacked on rows or columns, and column convolution or row convolution is used to extract features corresponding to irregular sampling shapes. Features are extracted using convolutional kernels of appropriate size and stride. Furthermore, the features are transformed into four dimensions (C, N, H, W), where C represents the number of channels, and then Conv3d with stride and convolution size (N, 1, 1) is used to extract features.

[0100] Convolutional neural networks (CNNs) are based on convolution operations, which locate features at corresponding positions using a regular sampling grid. Assuming Grid represents the sampling grid, then Grid is represented as:

[0101] Grid={(-1,-1),(-1,0),...,(0,1),(1,1)}

[0102] Typical convolutional neural networks use regular sampling grids, while AKConv targets irregularly shaped convolutional kernels. To allow irregularly shaped kernels to have sampling grids, an algorithm for convolutions of arbitrary sizes was designed, which generates convolutional kernel P. n The initial sampling coordinates are determined by: first, generating a regular sampling grid; then, creating an irregular grid for the remaining sampling points; and finally, stitching them together to generate the overall sampling grid.

[0103] like Figure 3 As shown, this indicates that the initial sampling coordinates are generated for convolutions of arbitrary sizes. The sampling grid of a regular convolution is centered at (0, 0). However, irregular convolutions do not have a center at many sizes; to accommodate the size of the convolution used, the top-left corner (0, 0) is set as the sampling origin in the algorithm. The initial coordinates P of the irregular convolution are defined... n Then, the corresponding convolution operation at position P0 is defined as shown in Equation 1:

[0104] Conv(P0)=∑w×(P0+P n ), (Formula 1)

[0105] Where w represents the convolution parameters.

[0106] The algorithm for initial sampling coordinates is shown in Table 1 as Algorithm 1.

[0107] Table 1 Algorithm for Initial Sampling Coordinates

[0108]

[0109]

[0110] Because AKConv has certain advantages over conventional convolutional operations, it is also used in the audio and video feature extraction network of this invention. This invention proposes an AKFC network, which includes an AKConv module and fully connected layers. The AKConv module uses convolutional kernels of variable shape and size as the convolutional layer in the input preprocessing stage, and is combined with fully connected layers to extract initial audio or video features. This invention replaces traditional convolutional layers with AKConv modules, which allows for the processing of feature vectors of arbitrary size and the learning of more complex nonlinear relationships between features, thereby improving the expressive power of feature extraction. At the same time, the introduction of the AKConv module effectively reduces the number of model parameters and computational cost, further improving the model's efficiency.

[0111] Figure 4 This is a schematic diagram of AKFC audio and video feature extraction. For example... Figure 4 As shown, the AKFC network includes the AKConv module and a fully connected layer (FC layer). First, a video stream can be split into three modalities: text, audio, and video. The audio and video data are then feature-extracted by the AKFC network. For example... Figure 4 As shown, audio and video data are first convolved using the AKConv module. AKConv allows convolutions to have arbitrary shapes and sizes, thus enabling the processing of feature vectors of arbitrary size and dimension. Therefore, audio and video data can be convolved to obtain initial feature vectors. These feature vectors are then used as input to the FC layer, ultimately yielding visual and audio features. These visual and audio features are then first simplely concatenated and used as input to the subsequent RGCN and pairwise cross-modal feature interaction modules.

[0112] Given N multimodal data samples {r1, r2, ..., r... i ,...,r N}, each multimodal data r i Associated with three modalities—audio (denoted as a), video (denoted as v), and text (denoted as t)—data corresponding to the three modalities, i.e., r, can be obtained. i a r i v and r i t It should be noted that the symbols or formulas used above and below can represent either the multimodal data sample to be identified or the multimodal data in the training samples. The same operation is performed for both the multimodal data sample to be identified and the multimodal data in the training samples.

[0113] For text information, a Transformer is used as the encoder to extract features from the text information, obtaining initial text features. That is, as shown in formula (2):

[0114]

[0115] in, This represents the learnable parameters of the Transformer model (i.e., Transformer(·)) used to extract initial text features.

[0116] For auditory (i.e., audio) and visual (i.e., video) modal information, the AKFC network is used as a single-peak encoder to extract features from the audio and video streams. Contextual features for each modality type are extracted using the following formulas (3) and (4). The contextual features of the audio and video (i.e., the initial audio features) are... and initial video features )

[0117]

[0118] in, represents the learnable parameters of the fully connected layers in the AKFC network (i.e., AKFC1(·)) used to extract initial audio features; represents the learnable parameters of the fully connected layers in the AKFC network (i.e., AKFC2(·)) used to extract initial video features; and d a ,d v ,d t The dimensions representing the features extracted from the three modalities. It represents the set of real numbers.

[0119] The datasets of text, audio, and video are first processed by their respective single-peak networks to extract initial features, resulting in initial features for each modality (initial video features, initial audio features, and initial text features). Then, the initial features of the three modalities are simply fused together through a concatenation process and used as input for the subsequent advanced feature extraction module.

[0120] After performing primary feature extraction, advanced feature extraction can be further performed.

[0121] Relational graph convolutional networks (RGCNs) capture local contextual information for each utterance in a session by utilizing a multimodal graph between utterances and their modalities, and construct the multimodal graph through RGCNs. For example... Figure 5As shown, this illustrates the relationship between text, video, and audio discourse nodes.

[0122] exist Figure 5 In the diagram, black solid lines, orange solid lines, and orange dashed arrows represent cross-modal connections, past-time connections, and future-time connections, respectively. For each session, a multimodal graph is constructed, denoted as G(V, R, E), where V represents the set of utterance nodes with all three modalities in the multimodal graph (where the number of nodes |V| = 3 × N, and N represents the number of utterances in a session), E represents the set of edges in the multimodal graph, R represents the set of relation types in the multimodal graph, and each utterance r i Generate three nodes, namely r i t r i a and r i v , a, v, and t correspond to the audio modality, video modality, and text modality, respectively, and r ij ∈R, R contains R multi and R temp R multi R represents the set of internal connections between the three modalities within the same discourse, reflecting multimodal interaction. temp This represents the set of interconnections between utterances of the same modality within a specified time window. This temporal relationship includes past / previous utterances, denoted as P, and the next / future utterance, denoted as F. Therefore, 15 edge types are created using the definitions of these two groups. When considering the relationship between three different modalities for the same utterance (i.e., the same i), this case includes 6 edge relationships, namely referential... Figure 5 The solid black line represents six edge relationships: three pairs of modal relationships (v and a, v and t, and a and t). For the same time window, the interconnections between utterances of the same modality involve six variable relationships, namely, reference relationships. Figure 5 The solid orange line and the dashed orange line represent two edge relationships for each modality (past time connection and future time connection). Therefore, there are a total of six variable relationships for the three modalities, as shown in formula (5):

[0123]

[0124] Where τ∈{a,v,t}, i,j take values ​​from 1 to N, and future and past represent past and future relations, respectively.

[0125] To capture the subtle differences and variations in heterogeneous interactions between utterances and modalities in a multimodal graph, a relational graph convolutional network (RGCN) is employed. For each relation type r∈R, a mapping function f(H,W) is used. rThe node representation is inferred, where W is the weighting matrix. Aggregating all 15 edge types can be done using ∑ r∈R f(H,W r Calculate the final node representation. Determine the first feature representation of any modality τ in the i-th utterance. As in formula (6):

[0126]

[0127] Where r represents any relation type, N r (i) represents the set of neighbors of the i-th node, |·| represents the number of elements in the set, and W r W0 represents the learnable parameters in RGCN. d h1 This refers to the dimension of the hidden layers used by RGCN.

[0128] To extract rich representations from node features, a Graph Transformer model is used, where each layer includes a self-attention mechanism followed by a feedforward neural network. The self-attention mechanism allows nodes to utilize information from their neighborhood (i.e., the set of nodes directly connected to each node in the graph) and to capture local and global patterns in the graph.

[0129] Therefore, the first feature can be represented Inputting the Graph Transformer model yields the second feature representation of any modality τ in the i-th utterance. That is, formula (7):

[0130]

[0131] Where W1 and W2 represent the learnable parameters in the Graph Transformer model, d h2 This indicates the dimension of the hidden layer used by the Graph Transformer. This represents the cascade of D-head attention. The attention coefficient of node j can be calculated by the softmax activation function, as shown in formula (8):

[0132]

[0133] Where d represents the scaling factor, and W3 and W4 represent the learnable parameters. d ρ This indicates the dimension of each head.

[0134] After aggregation across the entire graph, a new representation vector is obtained, which is the vector corresponding to the multimodal data {r1, r2, ..., r...}.i ,...,r N The second feature corresponding to} is represented as

[0135] Cross-modal heterogeneity often increases the difficulty of human language analysis. Utilizing cross-modal interactions may help reveal the “inconsistent” nature and long-term dependencies of cross-modal interactions. Therefore, pairwise cross-modal feature interaction (P-CM) methods can be incorporated into the P-CM framework. Figure 6 This is a schematic diagram of paired cross-modal feature interactions; Figure 7 This is a flowchart of paired cross-modal feature interactions. Combined with... Figure 6 and Figure 7 The process of inputting the first fused feature into a pair of cross-modal feature interaction units to generate a third feature representation may include:

[0136] Perform cross-modal attention between any two modes τ1 and τ2 to generate a third feature representation. and For example, formulas (9) and (10):

[0137]

[0138] τ1 and τ2 can take any two modes from {a,v,t}, and τ1 and τ2 can take different modes;

[0139] queue key value

[0140] queue key value

[0141] and The learnable parameters of the pairwise cross-modal feature interaction units are represented by σ, which represents the softmax function, and d k1 express The dimension, d k2 express The dimension represents the scaling factor.

[0142] For example, given two modalities, such as audio 'a' and text 't', then, The superscript T indicates transpose. As the corresponding modality-sensitive representation for the entire session using a single-peak encoder. Based on the Transformer architecture, the key is defined as... Once cross-modal attention is performed on mode a (i.e., mode τ2) via mode t (i.e., mode τ1), then M aThe rich representation (i.e.) can be expressed with reference to formula (9) as shown in formula (11), that is:

[0143]

[0144] Back Figure 1 After determining the second and third feature representations, the second and third feature representations can be input into the MSAF module to generate the second fusion feature corresponding to the utterance. Figure 8 This is a schematic diagram of the MSAF model (i.e., MSAF module).

[0145] Reference Figure 8 Inputting the second and third feature representations into the MSAF module to generate a second fusion feature corresponding to the utterances in the training samples may include:

[0146] The MSAF module obtains the feature map set {F1, F2, ..., F...} Mod}, where Mod represents the total number of modes, i.e. Mod = 3, each feature map in the feature map set corresponds to a mode, and the feature map contains a set of second feature representations and third feature representations corresponding to the mode;

[0147] First, perform the splitting operation (i.e. Figure 8 Step 1) involves dividing each feature map into equal-channel feature blocks, with C channels in each block. The set of feature blocks belonging to the modality is represented as B. mod , This indicates the number of equal-channel feature blocks in the division. Indicates rounding up, C mod This represents the total number of channels corresponding to the modal feature, where mod ∈ {1, 2, ..., Mod}. B mod The i-th isochannel feature block in C, when C mod If it is not a multiple of C, the last block is filled with zeros in the missing channel.

[0148] Next, the connection operation is performed (i.e. Figure 8 Step 2) involves connecting all isochannel blocks corresponding to the modality (mod), and adding isochannel feature blocks belonging to the modality (mod) to the shared representation D. mod In the middle, by calculating B mod element-wise sum of S mod And perform global average pooling in the spatial dimension, expressed by formula (12):

[0149]

[0150] Where, N i Let n1, n2, ..., n represent the size of the i-th spatial dimension, K represent the number of spatial dimensions of the modal mod, and n represent the number of spatial dimensions of the modal mod. K This represents the index of each spatial dimension, where c represents the channel index, and D represents the index of each spatial dimension. mod (c) represents the global feature description of the modality mod under channel c;

[0151] Then, perform the highlighted part operation (i.e. Figure 8 Step 3 in the text, that is, for each Generate by applying a linear transformation to Z And use softmax activation to obtain block-by-block attention weights This can be expressed using formulas (13) and (14):

[0152]

[0153] in, and Indicates will Convert to The weights and biases of the fully connected layer C' represents the output dimension of the fully connected layer;

[0154] Since the soft attention value depends on the total number of feature blocks, features may be oversuppressed. This effect is more pronounced in complex tasks, leading to insufficient information for accurate prediction. To address this issue, a hyperparameter λ∈[0,1] is proposed to control the suppression capability of MSAF. Therefore, block-wise attention weights can be used. The minimum attention regularization factor λ for splitting is used to determine the optimal feature blocks. This can be expressed as formula (15):

[0155]

[0156] in, ⊙ represents element-wise multiplication.

[0157] Next, the optimized feature blocks belonging to the mod are... Optimized feature maps are generated by channel-by-channel joining. Optimize the set of feature maps This refers to the second fusion feature of the training samples generated by the MSAF module.

[0158] After feature fusion using MASF, the emotion classifier takes the features fused from the multimodal fusion of the MSAF module (i.e., the second fused features) as input to the classifier to obtain the emotion prediction result corresponding to the utterance, including prediction using the following formulas (16)-(18):

[0159] v i =ReLU(Φ0h) i +b0), (Formula 16)

[0160] p i =softmax(Φ1v) i +b1), (Formula 17)

[0161]

[0162] Among them, h i v represents the second fusion feature corresponding to the i-th utterance. i This represents the result after applying the ReLU activation function, where Φ0 and b0 represent the learnable parameters of ReLU, and p i Let represent the probability distribution obtained by the softmax activation function, and Φ1 and b1 represent the learnable parameters of softmax. The table represents the predicted sentiment label for the i-th utterance, and argmax represents the expression that makes p... i The function that calculates the predicted label when the maximum value is reached.

[0163] An emotion recognition method based on multimodal split attention fusion further includes training an emotion recognition model, wherein the training process includes:

[0164] Acquire training samples. Each training sample is multimodal data. Each multimodal data is associated with three modalities, including audio, video and text. Each training sample is a session. Each session includes two or more utterances. Each utterance in each training sample corresponds to an emotion label (one or more emotion labels).

[0165] Build an emotion recognition model;

[0166] For each training sample,

[0167] The initial text features, initial audio features, and initial video features of each utterance in the training samples are extracted using the initial feature extraction module.

[0168] The initial text features, initial audio features, and initial video features are concatenated and fused to generate the first fused feature corresponding to each utterance in the training samples.

[0169] The first fusion feature is input into the context information acquisition unit and the pairwise cross-modal feature interaction unit to generate the second feature representation and the third feature representation corresponding to each utterance in the training samples;

[0170] The second and third feature representations are input into the MSAF module to generate a second fusion feature corresponding to the utterance.

[0171] The second fusion feature is input into the emotion classifier to obtain the emotion prediction result corresponding to the utterance;

[0172] Based on the emotion prediction results and corresponding emotion labels of all training samples, the parameters in the emotion recognition model are adjusted until the preset conditions are met.

[0173] The processing of utterances in the training samples is the same as above. Figure 3-8 The representations and related operations described herein are the same and will not be repeated here. The advantages of this invention will be illustrated below through specific experiments.

[0174] This invention utilizes the IEMOCAP (Interactive Emotional Dyadic Motion Capture) dataset. This dataset consists of dialogues from 10 pairs of native English speakers among college students in the western United States. Each pair of participants was asked to express emotions in several emotionally charged scenarios. These scenarios simulate various emotional situations that might be encountered in real life, including both positive and negative emotions such as happiness, sadness, neutrality, anger, excitement, and frustration.

[0175] The IEMOCAP dataset primarily contains two folders: Documentation and Session. Documentation contains five files: corpus.dic, FIVE_face_markers2.png, HumaineInfo.txt, phonemes.txt, and timeinfo.txt. Session contains two folders: dialog and sentences.

[0176] `corpus.dic` is a dictionary file used to record the corpus of the IEMOCAP dataset. It contains definitions of terms, symbols, or tags used in the dataset, as well as their usage within the dataset. `FIVE_face_markers2.png` is an image file containing information related to facial expressions. `HumaineInfo.txt` contains general information or metadata related to the dataset, such as its source, creator, and collection method. `phonemes.txt` contains information related to phonemes used in the speech data. Phonemes are the basic units of speech and are commonly used in speech recognition and analysis. `timeinfo.txt` contains information related to time. In emotion recognition, time information may refer to the duration of each segment in the speech or facial expression data, or information related to the temporal relationship of emotion expression.

[0177] The session is divided into 5 parts (Session 1, Session 2, Session 3, Session 4, Session 5), each recorded by one male actor and one female actor. The total recording time is approximately 12 hours. Each session includes two folders: dialog and sentences. The dialog folder contains the audio and video files for each session (dialog), as well as annotation information related to these sessions. Specific information is shown in Table 2.

[0178] Table 2 shows the main distribution of the dialog dataset.

[0179]

[0180] Among them, avi is an audio-video interleaved format, emo evaluation stores sentiment evaluation tags, transcriptions are text transcriptions stored in txt format, and wav is an audio file format.

[0181] Each session corresponds to a dialogue (or conversation) or communication scenario in the IEMOCAP dataset. These sessions typically involve multiple participants and involve natural conversation and emotional expression. In emotion recognition research, researchers can use these audio and video files to train and evaluate emotion recognition models.

[0182] The `sentences` folder contains sentence-level annotations from dialogues, typically transcribed text. These annotations may include the speaker, content, and sentiment label for each sentence. This annotation information can be used to construct datasets for sentiment recognition tasks and to evaluate the model's sentiment recognition performance at the sentence level. Some transcribed content from the `sentences` folder is shown in Table 3.

[0183] In Table 3, Seg. [sec] refers to turn segmentation (in seconds). Turn refers to the exchange of words between speakers; in Turn, F indicates female and M indicates male. Transcription refers to the transcribed text. Label refers to categorical emotion assessment (three themes). [v,a,d] refers to attribute emotion assessment (valence, activation, dominance, three themes).

[0184] Table 3 contains partial transcripts from the sentences file.

[0185]

[0186]

[0187] In Table 3, Dialogues refer to the dialogue scenarios included in the dataset. Each dialogue scenario is performed by two actors, simulating a specific emotional exchange or situation. These dialogues are designed to simulate human communication and interaction in different emotional states in order to capture rich and natural emotional expressions. Utterances represent individual utterances or statements (denoted as utterances) in the dialogue. Each utterance is a segment of speech in the dialogue, delivered by an actor in a specific emotional state. Each statement is designed to express or evoke a specific emotional response and can be used to train and test the emotion recognition model.

[0188] The IEMOCAP dataset analyzes and understands the complexity of human emotional expression and the role of emotion in interpersonal communication through dialogues and utmost urging. This data contributes to the development of more accurate emotion recognition algorithms, particularly demonstrating good performance in processing speech and facial expressions.

[0189] Table 4. Dialogues Dataset Partitioning

[0190]

[0191] As shown in Table 4, Dialogues divides the training, validation, and test sets in a 7:1:2 ratio for the four-class and six-class classification tasks based on the IEMOCAP dataset.

[0192] Table 5. Utterances dataset partitioning

[0193]

[0194] As shown in Table 5, Utterances divides the training, validation, and test sets in a 7:1:2 ratio for the four-class and six-class classification tasks based on the IEMOCAP dataset. Utterances also removed some datasets from the four-class classification task that were only applicable to the six-class classification task.

[0195] The evaluation metrics are Acc and w-F1 score. Accuracy Acc represents the proportion of all correctly classified samples out of the total samples, and is a simple and intuitive metric for evaluating the performance of a classifier. For the multimodal emotion recognition task, its calculation formula (19) is shown:

[0196]

[0197] In this context, TP (True Positives) refers to the number of samples correctly identified as a specific emotion category. TN (True Negatives) refers to the number of samples correctly identified as not belonging to a specific emotion category. FP (False Positives) refers to the number of samples incorrectly identified as a specific emotion category. FN (False Negatives) refers to the number of samples incorrectly identified as not belonging to a specific emotion category.

[0198] The w-F1 score is a weighted average of F1 scores, where the w-F1 scores for each class are summed weighted by the number of samples in the dataset. The w-F1 score is the harmonic mean of precision and recall, used to measure model performance in binary or multi-class classification problems. See formula (20):

[0199]

[0200] Where, freq k It is the relative frequency of category k.

[0201] Tables 6, 7, 8, and 9 respectively show the comparison of the accuracy and training effect of the model after adding the AKFC audio and video feature extraction network in the four-class and six-class emotion recognition tasks with the SOTA (State of the Art) model with the IEMOCAP dataset as the specific model. The bolded data represents the part with the best accuracy in this table. Depending on the situation, the following models were selected in this embodiment: bc-LSTM, CMN, ICON, DialogueRNN, DialogueGCN

[47] , MMGCN, DialogueCRN, CHFusion, and COGMEN, and compared with the model of the present invention (Transformer+AKFC) in the four-class and six-class emotion recognition accuracy.

[0202] Table 6. Accuracy Comparison of Four-Class Emotion Recognition Task Models

[0203]

[0204] In Table 6, the Acc and w-F1 accuracy in the four-class emotion recognition scenario are 0.69% and 2.81% higher than the best model COGMEN studied by previous researchers, respectively, and are 0.4% and 2.39% higher than the baseline model (Transformer).

[0205] Table 7 Comparison of Accuracy of Six-Class Emotion Recognition Task Models

[0206]

[0207] In Table 7, the Acc and w-F1 accuracies in the six-class emotion recognition scenario are 0.42% and 0.04% higher than the previous best model COGMEN, respectively, and represent improvements of 0.73% and 0.86% compared to the baseline model. As shown in Tables 6 and 7, the addition of the AKFC audio / video feature extraction network to the baseline model significantly improves both Acc and w-F1 accuracy for four / six-class emotion recognition on the IEMOCAP dataset. In the four-class emotion recognition task, Acc and w-F1 reach 82.98% and 84.96%, respectively. In the six-class emotion recognition task, Acc and w-F1 reach 67.46% and 67.31%, respectively.

[0208] Table 8 Comparison of Training Results between Four-Class Emotion Recognition Task and Baseline Model

[0209]

[0210] Table 9 Comparison of Training Performance between the Six-Class Emotion Recognition Task and the Baseline Model

[0211]

[0212]

[0213] Comparing the results in Tables 8 and 9, it can be seen that, based on the training epoch=50, learning rate lr=0.0003, training parameters of 3060 GPUs, and the training process, in the four-class and six-class emotion recognition scenarios, after adding the AKFC audio and video feature extraction network, compared with the baseline model, the average training loss for four-class and six-class classification decreased by 0.0406 and 0.0424 respectively, and the average training time decreased by 0.031s and 0.1231s respectively, and the model parameters (i.e., the number of learnable parameters included in the training and inference process, which measures the complexity and computational requirements of the model) did not increase.

[0214] In the four-class classification scenario, after adding the AKFC audio / video feature extraction network, compared with the bc-LSTM, CHFusion, and COGMEN models, a significant reduction is observed in all three dimensions: average training loss, average training time, and model parameters. In the six-class classification scenario, after adding the AKFC audio / video feature extraction network, compared with the bc-LSTM, CMN, ICON, DialogueRNN, DialogueGCN, MMGCN, DialogueCRN, and COGMEN models, a significant reduction is observed in all three dimensions: average training loss, average training time, and model parameters, achieving good results.

[0215] Therefore, it can be concluded that after adding the AKFC audio and video feature extraction network to the baseline model, the training effect of the baseline model in both four-class and six-class emotion recognition scenarios was improved.

[0216] The following publicly available datasets were used for ablation experiments on IEMOCAP to perform four-class / six-class emotion recognition. The ablation tests primarily validated different modules of the multimodal emotion recognition model proposed in this invention. The training process used the Adam optimizer with a learning rate of 0.0003, a drop rate of 0.5, a batch size of 8, and 50 epochs. The designed network was trained on a single RTX 3060 GPU with 12GB of VRAM.

[0217] First, the function of different modules of the model was verified, and the results are shown in Tables 10 and 11. The bolded data represents the best performance in this table.

[0218] Table 10 Comparison of ablation experimental accuracy of different modules of the model

[0219]

[0220] Table 11 Comparison of ablation experiment training effects of different modules of the model

[0221]

[0222]

[0223] Tables 10 and 11 clearly show that after adding AKConv to the baseline model (i.e., Transformer), in both four-class and six-class emotion recognition scenarios, Acc and w-F1 scores improved by 1.86% and 1.86% (four-class results) and 1.78% and 1.43% (six-class results), respectively, compared to the baseline model. Furthermore, improvements were seen in both average training loss and average training time. Compared to the baseline model, the average training loss and average training time were reduced by 0.09 seconds and 0.03 seconds (four-class results) and 0.04 seconds and 0.06 seconds (six-class results), respectively, in four-class and six-class emotion recognition scenarios.

[0224] After incorporating MSAF into the baseline model, both Acc and w-F1 scores improved by 2.34% and 2.34% (for four-class classification) and 0.62% and 0.5% (for six-class classification) respectively in four-class and six-class emotion recognition scenarios compared to the baseline model. Compared to the baseline model, the average training loss decreased by 0.15 (for four-class classification) and 0.89 (for six-class classification) in four-class and six-class emotion recognition scenarios, respectively, while the average training time improved by 1.81 seconds (for four-class classification) and 1.56 seconds (for six-class classification), respectively. After incorporating AKConv and MSAF into the baseline model, both Acc and w-F1 scores improved by 3.08% and 3.09% (four-class) and 3.33% and 3.31% (six-class) respectively in four-class and six-class emotion recognition scenarios. Furthermore, both average training loss and average training time were improved compared to the model with only MSAF (i.e., training time was reduced). Compared to the baseline model, the average training loss decreased by 0.27 and 1.03 in four-class and six-class emotion recognition scenarios, respectively, while the average training time improved (i.e., shortened) by 0.33 seconds and 0.11 seconds respectively compared to the model with only MSAF. Although the average training time improved by 1.48 seconds and 1.44 seconds compared to the baseline model, the addition of the MSAF module, based on multimodal split attention fusion, enhanced the model's attention control over different modal features, enabling more effective integration of multimodal information. It can flexibly adjust the weights of different modal features and adaptively adjust them according to task requirements. Furthermore, through an attention mechanism, it can dynamically capture the importance of each modal feature, thereby improving model performance. Specifically, this manifests as improved accuracy in four-class and six-class emotion recognition tasks, and an effective reduction in average training loss during training. Therefore, introducing AKConv and MSAF onto the baseline model can effectively improve model performance while minimizing computational costs.

[0225] The training process used the Adam optimizer with a learning rate of 0.0003, drop rate of 0.5, batch size of 8, and epochs of 50. The designed network was run on a single RTX 3060G with 12GB of VRAM. The experiments used the IEMOCAP dataset. Preprocessing of the IEMOCAP dataset was performed first, requiring data loading and cleaning to ensure data integrity and consistency.

[0226] In multimodal emotion recognition models, the impact of different unimodal features on accuracy can vary, depending on factors such as dataset characteristics, task requirements, and model design. Generally, each unimodal feature has some influence on the final emotion recognition accuracy, but their importance and degree of influence may differ. Features from different modalities may exhibit correlation and complementarity; therefore, combining them can improve the accuracy and robustness of emotion recognition. When designing a multimodal emotion recognition model, it is necessary to consider the importance and degree of influence of each unimodal feature and adopt appropriate methods to fuse them together to achieve better recognition results. Based on this, this embodiment explores the impact of different unimodal features on emotion recognition performance and effectiveness.

[0227] According to Table 12, Figure 9 , Figure 10 It can be seen that among unimodal features, text features perform better in four-class / six-class emotion recognition tasks, while among bimodal features, audio + text bimodal features perform better in four-class / six-class emotion recognition tasks. Furthermore, it is evident that audio + video + text multimodal features outperform audio + text bimodal features in IEMOCAP's four-class and six-class emotion recognition tasks, achieving 1.57% and 1.56% higher Acc and w-F1 scores, respectively, in the four-class scenario, and 3.02% and 3.24% higher Acc and w-F1 scores, respectively, in the six-class scenario.

[0228] Table 12 Accuracy of Four / Six Classification Emotion Recognition under Different Modal Features

[0229]

[0230] like Figure 9 As shown, in the IEMOCAP (4-class) emotion recognition scenario, the Acc and w-F1 accuracies for unimodal emotion recognition using text reached 80.76% and 80.85% respectively (green represents ACC, yellow represents w-F1), significantly higher than the 63.68% and 43.65% and 61.40% and 43.08% Acc and w-F1 accuracies for unimodal emotion recognition using audio or video (61.40% and 43.08% respectively). The Acc and w-F1 accuracies for multimodal emotion recognition using text + audio + video reached 84.14% and 84.12% respectively, outperforming other unimodal or multimodal emotion recognition methods. This demonstrates the necessity of multimodal emotion recognition in 4-class emotion recognition tasks.

[0231] like Figure 10As shown, in the IEMOCAP (6-class) emotion recognition scenario, the Acc and w-F1 accuracies for unimodal emotion recognition using text reached 65.38% and 65.08%, respectively, significantly higher than the 49.07% and 32.83% and 48.03% and 32.78% for audio or video-based unimodal emotion recognition. The Acc and w-F1 accuracies for multimodal emotion recognition using text + audio + video reached 69.01% and 69.19%, respectively, outperforming other unimodal or multimodal emotion recognition methods. This demonstrates the necessity of multimodal emotion recognition in six-class emotion recognition tasks.

[0232] Next, the model of this invention is compared with the state-of-the-art (SOTA) benchmark model using the IEMOCAP dataset as a specific example to demonstrate the accuracy of four / six-class emotion recognition. Experimental results show that the model proposed in this invention achieves better performance compared to the aforementioned models. Specific comparisons of the accuracy of four / six-class emotion recognition are shown in Tables 13 and 14.

[0233] Table 13 Comparison of the accuracy of the proposed model (Ours) and existing models in four emotion recognition scenarios on the IEMOCAP dataset.

[0234]

[0235] Table 14 compares the accuracy of our model (Ours) with existing models in six emotion recognition scenarios on the IEMOCAP dataset.

[0236]

[0237] The bolded data in Tables 13 and 14 represent the best-performing data in these tables. As shown in Tables 13 and 14, the multimodal emotion recognition model based on MSAF and RGCN proposed in this invention achieves good accuracy in both four- and six-class emotion recognition on the IEMOCAP dataset.

[0238] In Table 14, the model proposed in this invention achieves slightly lower emotion recognition accuracy than COGMEN in Neutral (Neu) cases, but superior accuracy in all other emotion recognition tasks, particularly in Acc and w-F1 accuracy, where it increases by 1.97% and 1.73% respectively (relative to COGMEN). Therefore, it can be concluded that the model proposed in this paper achieves good performance in both four-class and six-class emotion recognition tasks.

[0239] It should be noted that the six-category emotions in Tables 13 and 14 include: Happiness, Sadness, Neutral, Anger, Excited, and Frustrated; the four-category emotions include Happiness, Sadness, Neutral, and Anger.

[0240] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An emotion recognition method based on multimodal split attention fusion, characterized in that, include: Acquire multimodal data to be identified, which is associated with three modalities, including audio, video and text, and the multimodal data to be identified is a session, which includes two or more utterances; An emotion recognition model is used to perform emotion recognition on multimodal data to be identified. This model includes: an initial feature extraction module, a high-level feature extraction module, an MSAF module, and an emotion classifier. The initial feature extraction module includes an initial text feature extraction unit, an initial audio feature extraction unit, and an initial video feature extraction unit. The initial text feature extraction unit uses a Transformer model as an encoder to extract the initial text features of each utterance. The initial audio feature extraction unit and the initial video feature extraction unit both use an AKFC network as an encoder to extract the initial audio features and initial video features of each utterance, respectively. In the AKFC network, the AKConv module and fully connected layers are used to extract the initial audio features and initial video features. The advanced feature extraction module includes a context information acquisition unit and a pairwise cross-modal feature interaction unit. The context information acquisition unit includes a relational graph convolutional network RGCN and a graph transformer model. RGCN is used to acquire the local context information of each utterance in the session and obtain the first feature representation of each modality of each utterance. The graph transformer model is used to obtain the second feature representation of each modality of each utterance based on the first feature representation of each modality of each utterance. Paired cross-modal feature interaction units are used to perform cross-modal attention on any two modalities to obtain a third feature representation; The MSAF module is used to fuse the second and third feature representations obtained by the context information acquisition unit and the paired cross-modal feature interaction unit to obtain fused features. The emotion classifier module is used to identify the emotion of each utterance based on fused features.

2. The emotion recognition method based on multimodal split attention fusion according to claim 1, characterized in that, The emotion recognition described is a four-category emotion recognition.

3. The emotion recognition method based on multimodal split attention fusion according to claim 1, characterized in that, The emotion recognition described is a six-category emotion recognition.

4. The emotion recognition method based on multimodal split attention fusion according to claim 1, characterized in that, The AKConv module uses convolutional kernels with variable shapes and sizes for convolution. Each AKConv module is directly connected to a fully connected layer to extract and initially integrate local features of audio and video, forming an initial feature representation for subsequent processing.

5. The emotion recognition method based on multimodal split attention fusion according to claim 1, characterized in that, The method further includes training an emotion recognition model, wherein the training process includes: Acquire training samples. Each training sample is multimodal data, and each multimodal data is associated with three modalities, including audio, video and text. Each training sample is a conversation, and each conversation includes two or more utterances. Each utterance in each training sample corresponds to an emotion label. Build an emotion recognition model; For each training sample, The initial text features, initial audio features, and initial video features of each utterance in the training samples are extracted using the initial feature extraction module. The initial text features, initial audio features, and initial video features are concatenated and fused to generate the first fused feature corresponding to each utterance in the training samples. The first fusion feature is input into the context information acquisition unit and the pairwise cross-modal feature interaction unit to generate the second feature representation and the third feature representation corresponding to each utterance in the training samples; The second and third feature representations are input into the MSAF module to generate a second fusion feature corresponding to the utterance. The second fusion feature is input into the emotion classifier to obtain the emotion prediction result corresponding to the utterance; Based on the emotion prediction results and corresponding emotion labels of all training samples, the parameters in the emotion recognition model are adjusted until the preset conditions are met.

6. The emotion recognition method based on multimodal split attention fusion according to claim 5, characterized in that, The initial text features, initial audio features, and initial video features in the training samples are extracted using the initial feature extraction module, which is achieved through the following formula: in, r i a and r i v Let {r1, r2, ..., r} represent the sets of utterances in the training samples. i ,...,r N Any multimodal data r in} i That is, discourse r i The data in the , i = 1, 2, ..., N, is text data, audio data and video data, i = 1, 2, ..., N, where N represents the number of utterances in a session; and Representing the discourse r i The corresponding initial text features, initial audio features, and initial video features; This represents the Transformer model used to extract initial text features, i.e., the learnable parameters of Transformer(·); This represents the learnable parameters of the fully connected layers in the AKFC network used to extract initial audio features, i.e., AKFC1(·). This represents the learnable parameters of the fully connected layers in the AKFC network used to extract initial video features, specifically in AKFC2(·).

7. The emotion recognition method based on multimodal split attention fusion according to claim 6, characterized in that, The first fusion feature is input into the context information acquisition unit to generate a second feature representation corresponding to the utterance, including: For each session, a multimodal graph is constructed, denoted as G(V, R, E), where V represents the set of utterance nodes with three modalities in the multimodal graph, E represents the set of edges in the multimodal graph, R represents the set of relation types in the multimodal graph, and each utterance r i Generate three nodes, namely r i t r i a and r i v , τ∈{a,v,t}, where a, v, and t correspond to the audio modality, video modality, and text modality, respectively, and r ij ∈R, R contains R multi and R temp R multi R represents the set of internal connections between the three modalities within the same discourse, reflecting multimodal interaction. temp This represents the set of interconnections among utterances of the same modality within a specified time window. Based on the multimodal graph, determine the first feature representation of any modality τ in the i-th utterance. Right now: Where r represents any relation type, N r (i) represents the set of neighboring nodes of the i-th node under relation r, · represents the number of elements in the set, W r W0 represents the learnable parameters in RGCN; Represent the first feature Inputting the GraphTransformer model yields the second feature representation of any modality τ in the i-th utterance. Right now: Where W1 and W2 represent the learnable parameters in the GraphTransformer model, This represents the cascade of D-head attention. The attention coefficient of node j is calculated by the softmax activation function, i.e.: Where d represents the scaling factor, and W3 and W4 represent learnable parameters; Then, with the training samples {r1,r2,...,r...} i ,...,r N The second feature corresponding to} is represented as 8. The emotion recognition method based on multimodal split attention fusion according to claim 7, characterized in that, The first fused feature is input into a pairwise cross-modal feature interaction unit to generate a third feature representation, including: Perform cross-modal attention between any two modes τ1 and τ2 to generate a third feature representation. and in, τ1 and τ2 can take any two modes from {a,v,t}, and τ1 and τ2 can take different modes; queue key value queue key value and The learnable parameters of the pairwise cross-modal feature interaction units are represented by σ, which represents the softmax function, and d k1 express The dimension, d k2 express Dimensions.

9. The emotion recognition method based on multimodal split attention fusion according to claim 8, characterized in that, The second and third feature representations are input into the MSAF module to generate a second fusion feature corresponding to the utterances in the training samples, including: The MSAF module obtains the feature map set {F1, F2, ..., F...} Mod }, where Mod represents the total number of modes, i.e. Mod = 3, each feature map in the feature map set corresponds to a mode, and the feature map contains a set of second feature representations and third feature representations corresponding to the mode; Each feature map is divided into equal-channel feature blocks, with C channels in each block. The set of feature blocks belonging to the modality is represented as B. mod D represents the total number of channels corresponding to the modal feature. This indicates the number of equal-channel feature blocks in the division. Indicates rounding up, C mod This represents the number of channels corresponding to a single modal feature, where mod ∈ {1, 2, ..., Mod}. i∈{1,2,...,|B mod |}, B mod The i-th isochannel feature block in; Perform a concatenation operation on all isochannel blocks corresponding to the modality (mod), and add the isochannel feature blocks belonging to the modality (mod) to the shared representation D. mod In the middle, by calculating B mod element-wise sum of S mod And perform global average pooling in the spatial dimension, expressed by the formula: Where, N i Let K represent the size of each spatial dimension, and let K represent the number of spatial dimensions of the modality, n1, n2, ..., n. K Let c represent the position index in each spatial dimension of the feature map, and let D represent the channel index of the feature map. mod (c) represents the aggregated feature value of the mode on a specific channel c; For each Generate by applying a linear transformation to Z And use softmax activation to obtain block-by-block attention weights. Expressed as a formula: in, and Indicates will Convert to The weights and biases of the fully connected layer; Using block-by-block attention weights The minimum attention regularization factor λ for splitting is used to determine the optimal feature block. Expressed as a formula: Where ⊙ represents element-wise multiplication; Optimize the feature blocks belonging to the modality. Optimized feature maps are generated by channel-by-channel fusion. Optimize the set of feature maps This refers to the second fusion feature of the training samples generated by the MSAF module.

10. The emotion recognition method based on multimodal split attention fusion according to claim 8, characterized in that, The second fused feature is input into the emotion classifier to obtain the emotion prediction result corresponding to the utterance, including prediction using the following formula: in i =ReLU(Φ0h i +b0), p i =softmax(Φ1v i +b1), Among them, h i v represents the second fusion feature corresponding to the i-th utterance. i This represents the result after applying the ReLU activation function, where Φ0 and b0 represent the learnable parameters of ReLU, and p i Let represent the probability distribution obtained by the softmax activation function, and Φ1 and b1 represent the learnable parameters of softmax. The table represents the predicted sentiment label for the i-th utterance, and argmax represents the expression that makes p... i The function that calculates the predicted label when the maximum value is reached.

Images