Multi-modal emotion understanding method and system with consistent hierarchical relationship

Abstract

The invention discloses a multi-modal emotion understanding method and a multi-modal emotion understanding system with a consistent hierarchical relationship, designs a multi-level consistency method, and solves the problem by setting consistency at three different levels, i.e., between features, between modals and between samples. Firstly, semantic consistency is set between features to enhance aggregation of same emotions in a mode, so that interaction in the mode is enhanced. Secondly, the characterization consistency is set between the modes, and the mode synchronization is enhanced by enhancing the characterization consistency; in order to further enhance the synchronization of different modes, a geometric consistency algorithm is designed between samples, and the synchronization between the modes is enhanced by ensuring the geometric consistency between different modes of different samples. The strategy alleviates the influence of lack of intra-modal interaction and inter-modal synchronization. The effectiveness and superiority of the multi-modal emotion understanding method and system with the consistent hierarchical relationship in the aspect of multimedia multi-modal emotion understanding are verified through wide experiments.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and relates to a multimodal emotion understanding method and system, and specifically to a multimodal emotion understanding method and system with consistent hierarchical relationships. Background Art

[0002] In the digital age, multimedia has become essential in information dissemination and human interaction, especially multimedia data with character themes. Multimodal emotion understanding lies at the core of interpreting multimedia. By integrating language, audio, and video modalities, multimodal emotion understanding aims to determine the polarity (positive or negative) and intensity (the degree of positivity or negativity) of emotions in multimodal data clips to better understand these data. The integration of multimodal emotion understanding in multimedia applications makes content and services closer to users' emotional states and needs, thereby enhancing the naturalness and effectiveness of human-computer interaction.

[0003] For multimodal emotion understanding, the main challenge lies in the inherent heterogeneity between different modalities. Each modality uniquely conveys semantic information and emotional cues, which makes it difficult to form a complete and coherent emotion understanding. To address this problem, mainstream methods can be roughly divided into two categories: cross-modal attention-based methods and fusion strategy-based methods. Cross-modal attention-based methods focus on discovering and exploiting connections between various modalities. The attention mechanisms in these methods explore and utilize features in a single modality and gain guidance from the context provided by other modalities. These methods strengthen the development of more integrated multimodal representations. On the other hand, fusion strategy-based methods aim to create a comprehensive representation strategy to generate unique multimodal representations.

[0004] Although these methods are good at integrating information from multiple modalities to create richer emotion representations, they share a common shortcoming: the lack of depth in emotion understanding within each single modality. This is mainly due to the neglect of intra-modal interactions. The way emotions are conveyed in each modality is different. For example, in the language modality, positive emotions may be conveyed through direct expression or subtle details embedded in the content. Without sufficient intra-modal interactions, the neglect of these intra-modal interactions will lead to shallow understanding within each modality, thereby weakening the model's ability to thoroughly interpret the information presented by each modality. In addition, another key problem of existing methods is the difference in predictions between different modalities. This defect stems from their insufficient alignment of representations between modalities. We call this phenomenon modality synchronization loss, which can lead to significant differences in predictions between different modalities, resulting in significant errors in predictions and poor model robustness.

[0005] In summary, it is crucial to design a new method and system for multimodal emotion understanding to solve the above two problems. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a multimodal emotion understanding method and system with consistent hierarchical relationships.

[0007] The technical solution adopted by the method of the present invention is: a multimodal emotion understanding method with consistent hierarchical relationship, comprising the following steps:

[0008] Step 1: Extract the original multimodal data features X that preserve the time dimension m ; where m∈{L, A, V}, X L , X A , X V Respectively represent text data features, voice data features and visual data features;

[0009] Step 2: Use a common encoder ε shared To extract three modal data X m The modal common part X m irr =ε shared (X m );Use three modality-specific encoders to extract three modality data X respectively m The modality-specific part X m exc =ε m private (X m );

[0010] The shared encoder ε shared and three modality-specific encoders, all of which are trained encoders; during the training process, the common part and the unique part corresponding to each modality are connected, and then the reconstructed feature R is obtained through the decoder m =D m ([X m exc , X m irr ]), where [·,·] means connected, D m Represents the decoder corresponding to each modality;

[0011] Cosine similarity is used as an indicator to measure the semantic consistency between feature vectors of different modalities, and the cosine similarity between feature vectors of different modalities is maximized; square difference is used as an indicator to measure the consistency of feature representation under different modalities in the same sample, and the square difference of feature representation under different modalities of the same sample is minimized; the square difference of inner product is used as an indicator to measure the geometric consistency between different modal sub-samples of different samples, and the square difference of inner product of different modal features of different samples is minimized;

[0012] The reconstructed feature R mAccording to the above three indicators, consistency processing is performed, and then compared with the original multimodal data feature X m The consistency loss calculation of the three indicators is loaded together for training, and the training ends after reaching the preset convergence standard;

[0013] Step 3: For the common parts of each modality, use the self-attention mechanism network Θ to obtain the intermediate result a C =[Θ(X L irr ), Θ(X A irr ), Θ(X V irr )];

[0014] For the unique parts of each modality, a cross-modal attention mechanism network Φ is used to calculate the cross-modal result a corresponding to each modality m =[Φ(X m exc , X m1 exc ),Φ(X m exc , X m2 exc )], where m1 and m2 represent two modes other than m;

[0015] Final comprehensive sentiment estimate MLP is a multi-layer perceptron, a′ m =Sigmoid(a m ·W m )m∈{L,A,V,C},Sigmoid() is the activation function, W m are the weights set for each group of modes.

[0016] Preferably, in step 1, one-dimensional time convolution is used to extract original features of multimedia data retaining the time dimension, and the data is further aligned.

[0017] As a preference, in step 2, a common encoder ε is used shared To extract three modal data X m The modal common part X m irr =ε shared (X m ), where m∈{L, A, V}, L, A, V represent text data, speech data, and visual data respectively;

[0018] The shared encoder ε shared, consisting of a multi-head self-attention layer and a feedforward neural network layer connected in sequence, wherein the feedforward neural network layer consists of a first convolutional layer, a first activation function layer, a second convolutional layer, a second activation function layer and a third convolutional layer connected in sequence.

[0019] Preferably, in step 2, three modality-specific encoders are used to extract the three modality data X respectively. m The modality-specific part X m exc =ε m private (X m );

[0020] The text modality-specific encoder is composed of a plurality of sequentially connected multi-layer self-attention mechanism layers and a convolutional layer; the multi-layer self-attention mechanism layer is composed of sequentially connected Transformer layers and convolutional layers, and is output after being connected through residuals;

[0021] The speech modality-specific encoder is composed of a plurality of sequentially connected convolutional layers and a bidirectional long short-term memory network;

[0022] The visual modality-specific encoder is composed of a plurality of sequentially connected convolutional layers, and each convolutional layer has a residual connection.

[0023] Preferably, in step 2, the decoder is composed of a plurality of sequentially connected hybrid layer structures and a deconvolution layer; the hybrid layer structure is composed of sequentially connected convolution layers, deconvolution layers and activation function layers, wherein residual connections are formed between the convolution layers and the deconvolution layers.

[0024] Preferably, in step 3, the self-attention mechanism network Θ sets a dedicated self-attention module for each modal data to perform data transposition and reconstruction processing; the processed data are respectively passed through a linear projection layer to further abstract and enhance the features from different modalities; the processed features of the three modalities are spliced ​​at the fusion point using the torch.cat function to generate a comprehensive feature representation a C ; The fusion process includes linear transformation and nonlinear activation operations.

[0025] Preferably, in step 3, the cross-modal attention mechanism network Φ is composed of a cross-modal conversion network, a memory enhancement network, and a feature fusion and processing network connected in sequence;

[0026] The cross-modal conversion network includes six cross-modal conversion modules, each of which is responsible for converting information of one modality into information formats of the other two modalities;

[0027] The memory enhancement network, each mode is provided with a memory enhancement module having a three-layer network structure;

[0028] The feature fusion and processing network consists of a feature splicing and fusion layer and a feature projection layer; the feature splicing and fusion layer is used to splice the modal features output by the memory enhancement network in dimension, and then output them after linear projection with the feature projection layer.

[0029] Preferably, in step 2, during the training process, cosine similarity is used as an indicator to measure the semantic consistency between different features, and a similarity matrix S is constructed, where S[i, j] is the cosine similarity between feature i and feature j, i, j∈[0, N), N is the number of features of each modality of each sample, and S[i, i]=0;

[0030] Get the loss function L sem (X m )for:

[0031]

[0032] During the training process, the square error is used to measure the consistency of representations between different modalities in the same sample, and the loss function L is obtained. rep (X m ):

[0033]

[0034] During the training process, the squared difference of the inner product is used as an indicator to measure the geometric consistency between different modalities of different samples, and the loss function L is obtained. geo (X m ):

[0035]

[0036] Where X m and represent the modal features of two different samples respectively, and <·, ·> refers to the matrix inner product.

[0037] The technical solution adopted by the system of the present invention is: a multimodal emotion understanding system with consistent hierarchical relationships, comprising:

[0038] one or more processors;

[0039] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal emotion understanding method with consistent hierarchical relationships.

[0040] Compared with the prior art, the beneficial effects of the present invention include:

[0041] (1) The present invention applies different levels of consistency algorithms to the original features and reconstructed features during the training process. This hierarchical approach makes emotion understanding across multiple modalities more accurate.

[0042] (2) We propose a semantic consistency strategy to aggregate the internal information of a single modality. By enforcing semantic consistency, we ensure a more accurate and nuanced interpretation of the emotional state in each modality.

[0043] (3) The present invention designs a representation consistency algorithm at the modal level and a geometric consistency algorithm at the sample level to address the challenge of lack of modal synchronization and significantly improve the model performance.

[0044] (4) We conduct comprehensive experiments on public multimodal emotion understanding datasets and obtain superior or comparable results, which demonstrates the feasibility of our approach and the effectiveness of the hierarchical consistency we maintain in multimodal emotion understanding. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The technical solution of the present invention is further described below using embodiments and specific implementation methods. In addition, in the process of describing the technical solution, some drawings are also used. For those skilled in the art, other drawings and the intention of the present invention can also be obtained based on these drawings without paying creative work.

[0046] Figure 1 A schematic diagram of a method according to an embodiment of the present invention;

[0047] Figure 2 The overall network framework diagram used in the embodiment of the present invention;

[0048] Figure 3 Schematic diagram of the method for semantic consistency within a modality in an embodiment of the present invention;

[0049] Figure 4 Schematic diagram of the inter-modal representation consistency and inter-sample geometric consistency method according to an embodiment of the present invention;

[0050] Figure 5 A schematic diagram of the structure of a shared encoder according to an embodiment of the present invention;

[0051] Figure 6 A schematic diagram of the structure of a text modality-specific encoder according to an embodiment of the present invention;

[0052] Figure 7 A schematic diagram of the structure of an audio modality-specific encoder according to an embodiment of the present invention;

[0053] Figure 8 A schematic diagram of the structure of a video modality-specific encoder according to an embodiment of the present invention;

[0054] Fig. 9 A schematic diagram of the decoder structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0056] Please see Figure 1 and Figure 2 , this embodiment provides a multimodal emotion understanding method with consistent hierarchical relationships, comprising the following steps:

[0057] Step 1: Extract the original multimodal data features X that preserve the time dimension m ; where m∈{L, A, V}, X L , X A , X V Respectively represent text data features, voice data features and visual data features;

[0058] In one embodiment, since multimedia data is streaming data with a time dimension, retaining the time dimension when extracting raw features from the data is critical to capturing the inherent dynamics and patterns in the data stream. To this end, this embodiment uses a 1d convolution layer for each modality of data. By using a one-dimensional time convolution to derive these shallow encoding features, the time dimension of the input is effectively retained, so that aligned and unaligned data can be processed.

[0059] Step 2: Use a common encoder ε shared To extract three modal data X m The modal common part X m irr =ε shared (X m );Use three modality-specific encoders to extract three modality data X respectively m The modality-specific part X m exc =ε m private (X m );

[0060] The shared encoder ε shared and three modality-specific encoders, all of which are trained encoders; during the training process, the common part and the unique part corresponding to each modality are connected, and then the reconstructed feature R is obtained through the decoder m =D m ([X m exc , X mirr ]), where [·,·] means connected, D m Represents the decoder corresponding to each modality;

[0061] Please see Figure 3 and Figure 4 In one implementation, cosine similarity is used as a measure of semantic consistency between feature vectors of different modalities, and the cosine similarity between feature vectors of different modalities is maximized; square difference is used as a measure of consistency of feature representations of different modalities within the same sample, and the square difference of feature representations of the same sample under different modalities is minimized; square difference of inner product is used as a measure of geometric consistency between different modalities of different samples, and the square difference of inner products of different modal features of different samples is minimized;

[0062] Taking into account that the decoupling process may affect the consistency of the original feature processing settings, this embodiment processes the common parts and unique parts corresponding to each modality obtained by decoupling to obtain reconstructed features, and also performs consistency on them, and loads them together with the consistency loss calculation of the original features for training the entire model.

[0063] In this embodiment, the feature R m According to the above three indicators, consistency processing is performed, and then compared with the original multimodal data feature X m The consistency loss calculations of the three indicators are loaded together for training, and the training ends after reaching the preset convergence standard.

[0064] This embodiment uses cosine similarity to measure the semantic consistency between features of different modalities. By maximizing the cosine similarity between feature vectors of different modalities, the model can capture the semantic similarity between different features within the modality.

[0065] This embodiment uses the squared error (mean square error) to measure the consistency of feature representation of the same sample in different modalities. By minimizing the squared error of feature representation of the same sample in different modalities, the consistency of the model's understanding of different modal data of the same object is enhanced.

[0066] This embodiment uses the square difference of inner products to measure the geometric consistency between different modalities of different samples. By minimizing the square difference of inner products of different modal features of different samples, the geometric relationship between mismatched modalities between samples is kept stable, and the consistency of understanding between models is enhanced.

[0067] In this embodiment, the model shows good performance indicators on an independent validation set, such as high precision, F1 score, etc. And on the validation set, the model does not show overfitting.

[0068] Please see Figure 5 In one embodiment, a common encoder ε is used.shared To extract three modal data X m The modal common part X m irr =ε shared (X m ), where m∈{L, A, V}, L, A, V represent text data, speech data, and visual data respectively;

[0069] The shared encoder ε shared , consisting of a multi-head self-attention layer and a feedforward neural network layer connected in sequence, wherein the feedforward neural network layer consists of a first convolutional layer, a first activation function layer, a second convolutional layer, a second activation function layer and a third convolutional layer connected in sequence.

[0070] The multi-head self-attention layer allows the model to consider information at different positions simultaneously when encoding, so as to more effectively process time series data of various modalities. The feedforward neural network layer further processes the output of the self-attention layer to enhance the feature expression capability.

[0071] The multi-head self-attention layer uses a similar mechanism to Transformers, but adjusts the focus to accommodate multimodal input. The weights of each head are initialized differently to focus on different features of text, speech, and visual data respectively.

[0072] The feedforward neural network layer uses the ReLU activation function to enhance nonlinear expression capabilities. In order to be common to the three modalities, the number of input and output channels of the feedforward neural network layer is set to 768, which is consistent with the text modality after BERT preprocessing with the richest feature number. The step size of the convolutional layer is kept at 1, and the convolution kernel size is adjusted to 3 to better capture the correlation and contextual information between the modalities. In the public encoder of the present invention, the feedforward neural network layer contains three convolutional layers.

[0073] In one embodiment, three modality-specific encoders are used to extract three modality data X respectively. m The modality-specific part X m exc =ε m private (X m );

[0074] Please see Figure 6 In one embodiment, the text modality-specific encoder is composed of a plurality of sequentially connected multi-layer self-attention mechanism layers and a convolutional layer; the multi-layer self-attention mechanism layer is composed of sequentially connected Transformer layers and convolutional layers, and is output after being connected through residuals;

[0075] Multi-layer self-attention mechanism layer sets a multi-layer Transformer layer, each layer includes a self-attention mechanism and a convolutional neural network, and a residual connection is set at the end, which can extract the semantics and context information of the text data at a deeper level. In this embodiment, 3 layers of self-attention mechanism layers are set, and the input and output channels of each layer are consistent with the data preprocessed by BERT, which is set to 768.

[0076] The convolution layer sets the input and output channels to 768, which is consistent with the BERT preprocessed data; the convolution kernel size is set to 3 to better capture local context features.

[0077] Please see Figure 7 In one embodiment, the speech modality specific encoder is composed of a plurality of sequentially connected convolutional layers and a bidirectional long short-term memory network;

[0078] This embodiment uses multiple convolution layers to extract the time-frequency features of speech signals. Each layer is followed by batch normalization and ReLU activation function to enhance the stability and nonlinear expression ability of feature extraction. There are three convolution layers here. The number of input and output channels of the convolution layer is set to 128, and the convolution kernel size is set to 5, which is more suitable for capturing the dynamic changes of speech.

[0079] Bidirectional Long Short-Term Memory (Bi-LSTM): A bidirectional LSTM is added after the convolutional layer to capture the front-end and back-end dependencies of time series data and adapt to complex speech changes.

[0080] Please see Figure 8 In one embodiment, the visual modality-specific encoder is composed of a number of sequentially connected convolutional layers, and each convolutional layer has a residual connection.

[0081] This embodiment uses a multi-level convolutional layer to gradually extract visual features, and each layer of convolution is followed by maximum pooling and batch normalization to effectively extract key information in the image. Here, the number of input and output channels of the convolutional layer is set to 128 to process richer image features, and the convolution kernel size is set to 3 to better capture local visual features. In this embodiment, a total of 3 convolutional layers are set to form a deep convolutional neural network. At the same time, residual connections are introduced in the convolutional layer to avoid the problem of gradient disappearance in training, improve training efficiency and model depth.

[0082] Please see Fig. 9 In one embodiment, the decoder is composed of several sequentially connected hybrid layer structures and a deconvolution layer; the hybrid layer structure is composed of sequentially connected convolution layers, deconvolution layers and activation function layers, wherein the convolution layers and deconvolution layers are residually connected.

[0083] This embodiment first uses the convolution layer to continue to compress features and enhance feature fusion between modalities. Since the input is composed of the modal public part decoupled from the corresponding modality and the modal private part linked together, the number of input channels of this convolution layer is twice that of the corresponding modality after preprocessing, the convolution kernel size is 3, and the step size is 1.

[0084] This embodiment introduces a deconvolution layer after the convolution layer to upsample the data and restore it to its original size. The deconvolution layer can increase the generation capability of the model, expand the feature map, and help restore the temporal characteristics of the data.

[0085] This embodiment adds a residual connection between each convolution and deconvolution layer, which helps the decoder to maintain information flow in the deep network structure, reduce information loss, and improve decoding accuracy.

[0086] In this embodiment, the number of output channels of the last deconvolution layer is adjusted to match the original modal data to ensure that the dimension of the output data is consistent with the corresponding original modal data, which is convenient for subsequent sentiment classification or sentiment regression processing.

[0087] This embodiment adds batch normalization after each convolution and deconvolution layer, which helps the stability and convergence speed of the model training process. At the same time, the ReLU activation function is used to provide nonlinear processing capabilities and enhance the expressiveness of the model.

[0088] Step 3: For the common parts of each modality, use the self-attention mechanism network Θ to obtain the intermediate result a C =[Θ(X L irr ), Θ(X A irr ), Θ(X V irr )];

[0089] For the unique parts of each modality, a cross-modal attention mechanism network Φ is used to calculate the cross-modal result a corresponding to each modality m =[Φ(X m exc , X m1 exc ),Φ(X m exc , X m2 exc )], where m1 and m2 represent two modes other than m;

[0090] Final comprehensive sentiment estimate MLP is a multi-layer perceptron, a′ m =Sigmoid(a m ·Wm )m∈{L,A,V,C},Sigmoid() is the activation function, W m are the weights set for each group of modes.

[0091] In one embodiment, the self-attention mechanism network Θ sets a dedicated self-attention module for each modal data to perform data transposition and reconstruction processing. These modules focus on capturing the key features of each modality and its internal semantic relationship to improve the accuracy of sentiment analysis; the processed data are respectively passed through a linear projection layer to further abstract and enhance the features from different modalities; the processed features of the three modalities are spliced ​​at the fusion point using the torch.cat function to generate a comprehensive feature representation a C ; The fusion process includes linear transformation and nonlinear activation operations, as well as the dropout mechanism used during training to enhance the generalization ability of the model and prevent overfitting.

[0092] In one embodiment, the cross-modal attention mechanism network Φ is composed of a sequentially connected cross-modal conversion network, a memory enhancement network, and a feature fusion and processing network;

[0093] The cross-modal conversion network includes six cross-modal conversion modules, each of which is responsible for converting the information of one modality into the information format of the other two modalities; the design of these modules ensures that information can be effectively converted and integrated between different modalities, enhancing the model's ability to process multiple data types.

[0094] The memory enhancement network is provided with a memory enhancement module having a three-layer network structure for each modality; these modules are used to deeply process and store information after cross-modal conversion, thereby improving the network's ability to remember and reproduce complex emotional features.

[0095] The feature fusion and processing network consists of a feature splicing and fusion layer and a feature projection layer; the feature splicing and fusion layer is used to splice the modal features output by the memory enhancement network in dimensions, and then output them after linear projection with the feature projection layer, so as to enhance the model's ability to express emotions.

[0096] In one embodiment, during the training process, cosine similarity is used as an indicator to measure the semantic consistency between different features, and a similarity matrix S is constructed, where S[i,j] is the cosine similarity between feature i and feature j, i,j∈[0,N), N is the number of features of each modality of each sample, and S[i,i]=0;

[0097] Get the loss function L sem (X m )for:

[0098]

[0099] The element S[i,j] in the similarity matrix S represents the cosine similarity between feature i and feature j. Cosine similarity is obtained by calculating the dot product of two feature vectors and dividing them by their respective moduli. This metric evaluates the similarity of two feature vectors in direction, thereby indirectly reflecting their semantic similarity. In this method, S[i,i] is set to 0 to exclude the comparison of the features themselves and ensure that only the similarity between different features is compared.

[0100] The similarity matrix S is used to enhance the aggregation of the same emotion within the modality. By constructing the similarity matrix S, the model can identify and aggregate features with high similarity (i.e., similar emotions). For example, when processing the text modality, if the feature vectors generated by different words or sentence fragments are semantically close, the cosine similarity between them will be high. This recognition of similarity enables the model to cluster semantic features with similar emotions together, thereby enhancing the representation of emotional consistency within the modality.

[0101] Within the modality, through feature aggregation, the model can more easily identify and learn the features that are most representative of specific emotional expressions. For example, in the visual modality, similar expressions or colors may be aggregated, which enhances the efficiency and accuracy of the model in parsing and expressing the same emotion.

[0102] The loss function uses the similarity matrix S to optimize the model to ensure that emotion-similar features that should be clustered are not mistakenly scattered during model training. This is achieved by minimizing the high cosine similarity between emotion-inconsistent features (i.e., different emotion features that are mistakenly identified as semantically similar).

[0103] In one embodiment, during the training process, the square error is used to measure the consistency of representations between different modalities in the same sample, and the loss function L is obtained. rep (X m ):

[0104]

[0105] Squared difference is a common mathematical method used to calculate the difference between two variables. In multimodal sentiment understanding, the representations generated by different modalities (such as text, vision, and audio) may differ due to the inherent characteristics of the modalities, so each modality is first normalized and then the squared difference is calculated. By calculating the squared difference between these representations, the inconsistency between their representations can be quantified.

[0106] The loss function optimizes the model by minimizing the squared difference between representations of different modalities. This means that during the learning process, the model is driven to minimize the difference between the representations of each modality.

[0107] When representation differences between modalities are minimized, these modalities will be more consistent in expressing the sentiment of the same sample. For example, a positive sentiment description in text should closely match the corresponding visual expression (e.g., a smile) and audio representation (e.g., a pleasant tone of voice).

[0108] In one embodiment, during the training process, the squared difference of the inner product is used as an indicator to measure the geometric consistency between different modalities of different samples, and the loss function L is obtained. geo (X m ):

[0109]

[0110] Where X m and They represent the modal features of two different samples respectively, and <·,·> refers to the matrix inner product.

[0111] The inner product is a mathematical operation that is usually used to calculate the dot product of two vectors and can characterize the angular relationship between the two vectors. In multimodal data processing, the inner product can be used to measure the similarity or consistency between feature vectors of different modalities. Here, this embodiment is used to quantify the geometric consistency between modalities.

[0112] The loss function is designed to minimize the squared difference of the inner product of features between different modalities of different samples, so that the model can make the features between corresponding modalities geometrically consistent when processing different samples. For example, if the feature vectors of textual modality and visual modality can maintain similar inner product values ​​between different samples, it indicates that these modalities have consistent geometric structures when representing the sentiment of the sample.

[0113] Geometric consistency means that the feature spaces of different modalities have consistent directionality or alignment for the representation of similar or related emotions. This alignment ensures that the emotional state observed in one modality is expressed in a similar way in other modalities, thereby enhancing the synchronization between different modalities. This synchronization between modalities makes the entire model's judgment of emotions more accurate and consistent, because each modality is reinforcing the other's emotional signal. During the model training process, by adjusting the model parameters to minimize the loss function, that is, reducing the squared difference of the inner product between different modalities of different samples, the model can be systematically guided to learn how to synchronize the feature representations of different modalities from a geometric perspective.

[0114] This embodiment also provides a multimodal emotion understanding system with consistent hierarchical relationships, including:

[0115] one or more processors;

[0116] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal emotion understanding method with consistent hierarchical relationships.

[0117] The present invention is further described below through specific experiments.

[0118] The deep learning framework used in this experiment is Pytorch. The hardware environment of the experiment is NVIDIA GeForce RTX3090*8 graphics card and the processor is Intel(R)Xeon(R)Gold 6240.

[0119] For the input multimodal multimedia data, this experiment took a series of preprocessing steps. For language data, the present invention uses the GloVe tool to convert text into 300-dimensional word vectors. In addition, in order to obtain more complex word features, the present invention uses the BERT-base-uncased model, which generates a 768-dimensional hidden state representation for each word. For the video modality, this experiment uses the Facet tool to encode video frames. This tool is good at capturing and understanding 35 unique facial action units, providing a comprehensive analysis of facial expressions. For the audio modality, this experiment uses the COVAREP tool to process audio data and extract 74-dimensional feature vectors to make the audio expression more detailed. The code implementation of this experiment is built using the PyTorch framework version 1.10.1+cu111 and Transformer version 4.29.2. In all experiments, the batch size is set to 16 and trained for 30 epochs or until convergence to ensure the stability of the model performance. The results of the model trained for multiple epochs are selected by testing the multimodal emotion understanding performance on the validation set.

[0120] The data set for this experiment is:

[0121] CMU-MOSI: This dataset contains 2,199 short video clips from online stand-up videos of multiple subjects, with audio and video modalities sampled at 12.5Hz and 15Hz, respectively. In the experiment, the CMU-MOSI dataset is divided into training, validation, and test sets, containing 1,284, 229, and 686 instances, respectively.

[0122] CMU-MOSEI: This dataset includes 22,856 video segments extracted from YouTube movie reviews, where the sampling rates of audio and video data are 20Hz and 15Hz respectively. Considering its large size, the dataset is divided into 16,326 training instances, 1,871 validation instances, and 4,659 test instances, providing a comprehensive scope for rigorous analysis.

[0123] CH-SIMS: Different from the other two datasets, the CH-SIMS dataset contains 60 original videos and 2,281 refined video segments, each of which is no less than one second and no more than ten seconds long. In the experiment, we divide the CH-SIMS dataset into training, validation, and test sets in a ratio of 6:2:2, containing 1,368, 456, and 457 sentences, respectively.

[0124] Experimental evaluation indicators:

[0125] In the experiment, the performance evaluation indicators used include accuracy (ACC i represents i-unit accuracy), F1 score, mean absolute error (MAE), and Pearson correlation coefficient (Corr). Consistent with previous work on multimodal emotion understanding, both CMU-MOSI and CMU-MOSEI datasets are annotated with emotion intensity scores ranging from -3 to +3, covering the emotion spectrum from extremely negative to extremely positive. In addition, following the work on the CH-SIMS dataset, this experiment annotates the CH-SIMS dataset with emotion intensity scores ranging from -1 to +1.

[0126] Comparison with existing methods

[0127] In order to verify the effectiveness of the present invention, the search results of the present invention are compared with those of the existing methods (the best result is marked in bold and the second best is marked in Underline For details, please see Tables 1, 2 and 3 below.

[0128] Table 1: Comparison results with state-of-the-art methods on the CMU-MOSI dataset

[0129]

[0130]

[0131] Table 2: Comparison results with state-of-the-art methods on the CMU-MOSEI dataset

[0132]

[0133]

[0134] Table 3: Comparison results with state-of-the-art methods on the CH-SIMS dataset

[0135]

[0136]

[0137] Tables 1, 2, and 3 show the comparative results obtained on the CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets, respectively. The present invention achieves superior or comparable performance under different settings, compared with existing multimodal emotion understanding (MEM) methods. The reasons why the present invention can significantly outperform previous multimodal emotion understanding methods can be roughly summarized as follows: First, the present invention uses feature decoupling technology to process the original features, which enhances the discriminative power of the model by reducing feature redundancy and focusing on significant emotion indicators. In addition, the present invention considers the consistency of the original features and the reconstructed features at different levels during the training process. The hierarchical structure consistency algorithm of the present invention solves the problem of lack of intra-modal interaction in mainstream multimodal emotion understanding methods, further strengthens the modal synchronization, and significantly improves the performance of our model. Unfortunately, SUGRM slightly outperforms the present invention in the non-aligned setting experiment on the CMU-MOSI dataset, which has limited features and significant temporal differences between modalities. This is mainly because the modality recalibration module (MRM) and unimodal label generation module (ULGM) of SUGRM are good at processing datasets with significant differences between modalities. However, our method excels on larger and more complex datasets such as CH-SIMS and CMU-MOSEI, and also shows the best performance on the aligned CMU-MOSI dataset, demonstrating its versatility and adaptability in multimodal emotion understanding.

[0138] In order to better verify the importance of the proposed module, ablation experiments are also conducted here to show the effectiveness of each component, including the hierarchical structure consistency algorithm: semantic consistency, representation consistency, and geometric consistency. In order to more clearly show the impact of our hierarchical relationship consistency algorithm on feature reconstruction, we also analyze it separately as a component in the ablation experiment. This experiment selects binary accuracy, F1 score, and mean absolute error as performance evaluation indicators, and the results are shown in Table 4 (Sem-, Rep-, Geo-represent semantic consistency, representation consistency, and geometric consistency applied to original features, respectively, and Rec-represents our HSCL applied to reconstructed features).

[0139] Table 4: Comparison of different configuration components

[0140]

[0141] Through experiments, the following conclusions were drawn:

[0142] First, applying the proposed semantic consistency algorithm to the original features significantly improves the performance compared to using only the decoupled neural network for emotion understanding. This improvement shows that the proposed semantic consistency algorithm effectively strengthens the interaction within the modality, enabling the model to more deeply understand the emotional cues conveyed by each modality.

[0143] Regarding the algorithm for solving the lack of modal synchronization, the results show that the use of the cross-modal representation consistency of the present invention or the geometric consistency between samples alone can effectively improve the performance of the model on various data sets. When both are applied to the original features, the performance of the model is further optimized, which is particularly significant in improving the prediction accuracy of MOSI and MOSEI. This result verifies the effectiveness of the algorithm of the present invention in solving the synchronization challenges between different modalities and helps to improve the performance of the model. In addition, the experiment shows that when the hierarchical structure consistency algorithm is applied to the reconstructed features in training, the model performance is significantly improved. For example, on the MOSI dataset, the use of the reconstruction consistency algorithm alone improves the performance from the original ACC282.2, F1 score 82.1, and MAE 0.775 to ACC283.7, F1 83.7, and MAE 0.726. When used in combination with the consistency algorithm of the original features, the F1 score is improved from 84.2 to 85.2, ACC2 is improved from 84.2 to 85.2, and MAE is reduced from 0.743 to 0.716. This finding further validates the rationale of the proposed hierarchical structured consistency learning (HSCL) approach and decoupled neural network design, highlighting its applicability in enhancing the training process and extending the hierarchical consistency approach to reconstructed features.

[0144] In order to comprehensively evaluate the effectiveness of the present invention, this experiment conducted a quantitative comparison experiment using the feature decoupling neural network FD. The results are shown in Table 5. Although FD demonstrates efficient information extraction capabilities through feature decoupling and attention mechanisms, it fails to address the challenges of insufficient intra-modal interaction and ensuring modal synchronization. In contrast, the present invention HSCL cleverly solves these challenges. The results confirm the effectiveness of the method proposed in the present invention, indicating that hierarchical structure consistency learning is beneficial for multimodal emotion understanding.

[0145] Table 5: Comparison of experimental results between the present invention and the simple decoupled neural network

[0146]

[0147]

[0148] In order to effectively evaluate the performance of the present invention under single-modal conditions and understand the performance of the hierarchical structure consistency algorithm of the present invention on a single modality and its contribution to the final result, a single-modal input comparison experiment with and without the use of the hierarchical structure consistency algorithm of the present invention is also carried out, and the results are shown in Table 6.

[0149] Table 6: Single-mode experimental results

[0150]

[0151] Experiments show that the language modality performs well when using only decoupled neural networks. The performance is further improved by applying the hierarchical consistency algorithms of the present invention, as evidenced by the improved metrics. In contrast, the audio and video modalities initially had limited performance when using only decoupled neural networks, but improved significantly after implementing the present invention. This enhancement is attributed to better modal aggregation and interaction, especially the consistency algorithms of the present invention that address the lack of modal synchronization. These algorithms not only synchronize different modalities, but also enhance the weaker modality, thereby improving the overall prediction performance and the robustness of the model.

[0152] It should be understood that the embodiments described above are part of the embodiments of the present invention, rather than all of the embodiments. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be combined with each other arbitrarily to form a feasible technical solution. Such combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0153] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A multimodal emotion understanding method with consistent hierarchical relationships, characterized in that: The following steps are involved: Step 1: Extract the original multimodal data features X that preserve the time dimension m ; where m∈{L,A,V}, X L , X A , X V Respectively represent text data features, voice data features and visual data features; Step 2: Use a common encoder ε shared To extract three modal data X m The modal common part X m irr =ε shared (X m );Use three modality-specific encoders to extract three modality data X respectively m The modality-specific part X m exc =ε m private (X m ); The shared encoder ε shared and three modality-specific encoders, all of which are trained encoders; During the training process, the common part and the unique part corresponding to each modality are connected, and then the reconstructed feature R is obtained through the decoder. m =D m ([X m exc ,X m irr ]), where [·,·] means connected, D m Represents the decoder corresponding to each modality; Cosine similarity is used as an indicator to measure the semantic consistency between feature vectors of different modalities, and the cosine similarity between feature vectors of different modalities is maximized; square difference is used as an indicator to measure the consistency of feature representation under different modalities in the same sample, and the square difference of feature representation under different modalities of the same sample is minimized; the square difference of inner product is used as an indicator to measure the geometric consistency between different modal sub-samples of different samples, and the square difference of inner product of different modal features of different samples is minimized; The reconstructed feature R m According to the above three indicators, consistency processing is performed, and then compared with the original multimodal data feature X m The consistency loss calculation of the three indicators is loaded together for training, and the training ends after reaching the preset convergence standard; Step 3: For the common parts of each modality, use the self-attention mechanism network Θ to obtain the intermediate result a C =[Θ(X L irr ),Θ(X A irr ),Θ(X V irr )]; For the unique parts of each modality, a cross-modal attention mechanism network Φ is used to calculate the cross-modal result a corresponding to each modality m =[Φ(X m exc ,X m1 exc ),Φ(X m exc ,X m2 exc )], where m1 and m2 represent two modes other than m; Final comprehensive sentiment estimate MLP is a multi-layer perceptron, a′ m =Sigmoid(a m ·W m )m∈{L,A,V,C}, Sigmoid() is the activation function, W m are the weights set for each group of modes.

2. The hierarchically consistent multimodal emotion understanding method according to claim 1, characterized in that: In step 1, one-dimensional temporal convolution is used to extract the original features of multimedia data while retaining the time dimension, and the data is further aligned.

3. The hierarchically consistent multimodal emotion understanding method according to claim 1, characterized in that: In step 2, a common encoder ε is used shared To extract three modal data X m The modal common part X m irr =ε shared (X m ), where m∈{L,A,V}, L,A,V represent text data, speech data and visual data respectively; The shared encoder ε shared , consisting of a multi-head self-attention layer and a feedforward neural network layer connected in sequence, wherein the feedforward neural network layer consists of a first convolutional layer, a first activation function layer, a second convolutional layer, a second activation function layer and a third convolutional layer connected in sequence.

4. The hierarchically consistent multimodal emotion understanding method according to claim 1, characterized in that: In step 2, three modality-specific encoders are used to extract the three modality data X respectively. m The modality-specific part X m exc =ε m private (X m ); The text modality-specific encoder is composed of a plurality of sequentially connected multi-layer self-attention mechanism layers and a convolutional layer; the multi-layer self-attention mechanism layer is composed of sequentially connected Transformer layers and convolutional layers, and is output after being connected through residuals; The speech modality-specific encoder is composed of a plurality of sequentially connected convolutional layers and a bidirectional long short-term memory network; The visual modality-specific encoder is composed of a plurality of sequentially connected convolutional layers, and each convolutional layer has a residual connection.

5. The hierarchically consistent multimodal emotion understanding method according to claim 1, characterized in that: In step 2, the decoder is composed of a plurality of sequentially connected hybrid layer structures and a deconvolution layer; the hybrid layer structure is composed of sequentially connected convolution layers, deconvolution layers and activation function layers, wherein the convolution layers and the deconvolution layers are connected with residual connections.

6. The hierarchically consistent multimodal emotion understanding method according to claim 1, characterized in that: In step 3, the self-attention mechanism network Θ sets a dedicated self-attention module for each modal data to perform data transposition and reconstruction processing; the processed data passes through the linear projection layer to further abstract and enhance the features from different modalities; the processed features of the three modalities are spliced ​​at the fusion point using the torch.cat function to generate a comprehensive feature representation a C ; The fusion process includes linear transformation and nonlinear activation operations.

7. The hierarchically consistent multimodal emotion understanding method according to claim 1, characterized in that In step 3, the cross-modal attention mechanism network Φ is composed of a sequentially connected cross-modal conversion network, a memory enhancement network, and a feature fusion and processing network; The cross-modal conversion network includes six cross-modal conversion modules, each of which is responsible for converting information of one modality into information formats of the other two modalities; The memory enhancement network, each mode is provided with a memory enhancement module having a three-layer network structure; The feature fusion and processing network consists of a feature splicing and fusion layer and a feature projection layer; the feature splicing and fusion layer is used to splice the modal features output by the memory enhancement network in dimension, and then output them after linear projection with the feature projection layer.

8. The hierarchically consistent multimodal emotion understanding method according to any one of claims 1 to 7, characterized in that: In step 2, during the training process, cosine similarity is used as an indicator to measure the semantic consistency between different features, and a similarity matrix S is constructed, where S[i,j] is the cosine similarity between feature i and feature j, i,j∈[0,N), N is the number of features of each modality of each sample, and S[i,i]=0; Get the loss function L sem (X m )for: During the training process, the square error is used to measure the consistency of representations between different modalities in the same sample, and the loss function L is obtained. rep (X m ): During the training process, the squared difference of the inner product is used as an indicator to measure the geometric consistency between different modalities of different samples, and the loss function L is obtained. geo (X m ): Where X m and They represent the modal features of two different samples respectively, and <·,·> refers to the matrix inner product.

9. A hierarchically consistent multimodal emotion understanding system, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement a multimodal emotion understanding method with consistent hierarchical relationships as described in any one of claims 1 to 8.

Images