A hierarchically consistent multimodal emotion understanding method and system

Abstract

The present invention discloses a multimodal emotion understanding method and system with consistent hierarchical relationships, designs a multi-level consistency method, and sets consistency at three different levels: between features, between modalities, and between samples to solve this problem. First, semantic consistency is set between features to enhance the aggregation of the same emotion within the modality, thereby strengthening the intra-modal interaction. Secondly, representation consistency is set between modalities to enhance modal synchronization by strengthening the consistency of representation. In order to further strengthen the synchronization of different modalities, a geometric consistency algorithm is designed between samples to enhance inter-modal synchronization by ensuring the geometric consistency between different modalities of different samples. Such a strategy alleviates the impact of the lack of intra-modal interaction and inter-modal synchronization. The present invention verifies the effectiveness and superiority of the multimodal emotion understanding method and system with consistent hierarchical relationships disclosed by the present invention in multimedia multimodal emotion understanding through extensive experiments.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology and relates to a multimodal emotion understanding method and system, and specifically to a multimodal emotion understanding method and system with consistent hierarchical relationships. Background Art

[0002] In the digital age, multimedia, particularly multimedia data with a human-centric theme, has become crucial for information dissemination and human interaction. Multimodal emotion understanding lies at the heart of multimedia interpretation. By integrating language, audio, and video modalities, multimodal emotion understanding aims to determine the polarity (positive or negative) and intensity (the degree of positivity or negativity) of emotions in multimodal data clips to better understand them. The integration of multimodal emotion understanding in multimedia applications enables content and services to be more tailored to users' emotional states and needs, thereby enhancing the naturalness and effectiveness of human-computer interactions.

[0003] For multimodal emotion understanding, the main challenge lies in the inherent heterogeneity between different modalities. Each modality uniquely conveys semantic information and emotional cues, which makes it difficult to form a complete and coherent emotion understanding. To address this problem, mainstream methods can be roughly divided into two categories: cross-modal attention-based methods and fusion strategy-based methods. Cross-modal attention-based methods focus on discovering and exploiting connections between various modalities. The attention mechanism in these methods explores and utilizes features in a single modality and obtains guidance from the context provided by other modalities. These methods strengthen the development of more integrated multimodal representations. On the other hand, fusion strategy-based methods aim to create a comprehensive representation strategy to generate unique multimodal representations.

[0004] While these methods excel at integrating information from multiple modalities to create richer emotional representations, they share a common shortcoming: a lack of depth in understanding emotion within each modality. This is primarily due to a neglect of intra-modal interactions. Emotions are conveyed differently within each modality. For example, in a language modality, positive emotions may be conveyed through direct expression or through subtle details embedded in the content. Without sufficient intra-modal interactions, this neglect leads to a shallow understanding within each modality, weakening the model's ability to thoroughly interpret the information presented by each modality. Furthermore, another key issue with existing methods is the discrepancy in predictions across modalities. This shortcoming stems from insufficient alignment of their representations across modalities. We refer to this phenomenon as loss of modal synchronization, which can lead to significant discrepancies in predictions across modalities, resulting in significant prediction errors and poor model robustness.

[0005] In summary, it is crucial to design a new method and system for multimodal emotion understanding to solve the above two problems. Summary of the Invention

[0006] In order to solve the above technical problems, the present invention proposes a multimodal emotion understanding method and system with consistent hierarchical relationships.

[0007] The technical solution adopted by the method of the present invention is: a multimodal emotion understanding method with consistent hierarchical relationships, comprising the following steps:

[0008] Step 1: Extract the original multimodal data features X that preserve the time dimension m ; where m∈{L,A,V},X L 、X A 、X V Represent text data features, voice data features and visual data features respectively;

[0009] Step 2: Use a common encoder ε shared To extract three modal data X m The modal common part X m irr =ε shared (X m ) ; Use three modality-specific encoders to extract three modality data X respectively m The modality-specific part X m exc =ε m private (X m );

[0010] The shared encoder ε shared and three modality-specific encoders, all of which are trained encoders; during the training process, the common part and the unique part corresponding to each modality are connected, and then the reconstructed feature R is obtained through the decoder m =D m ([X m exc , X m irr ]), where [·,·] means connected, D m Represents the decoder corresponding to each modality;

[0011] Cosine similarity is used as an indicator to measure the semantic consistency between feature vectors of different modalities, and the cosine similarity between feature vectors of different modalities is maximized; square difference is used as an indicator to measure the consistency of feature representation under different modalities within the same sample, and the square difference of feature representation under different modalities of the same sample is minimized; square difference of inner product is used as an indicator to measure the geometric consistency between different modal sub-samples of different samples, and the square difference of inner product of different modal features of different samples is minimized;

[0012] The reconstructed feature R mAccording to the above three indicators, consistency processing is performed, and then compared with the original multimodal data feature X m The consistency loss calculation of the three indicators is loaded together for training, and the training ends after reaching the preset convergence standard;

[0013] Step 3: For the common parts of each modality, use the self-attention mechanism network Θ to obtain the intermediate result a C =[Θ(X L irr ),Θ(X A irr ),Θ(X V irr )];

[0014] For the unique parts of each modality, a cross-modal attention mechanism network Φ is used to calculate the cross-modal result a corresponding to each modality m =[Φ(X m exc , X m1 exc ),Φ(X m exc , X m2 exc )], where m1 and m2 represent the other two modes except m;

[0015] Final comprehensive sentiment estimate MLP is a multi-layer perceptron, a′ m =Sigmoid(a m W m )m∈{L,A,V,C},Sigmoid() is the activation function, W m are the weights set for each group of modes.

[0016] Preferably, in step 1, one-dimensional time convolution is used to extract the original features of the multimedia data while retaining the time dimension, and the data is further aligned.

[0017] As a preference, in step 2, a common encoder ε is used. shared To extract three modal data X m The modal common part X m irr =ε shared (X m ), where m∈{L, A, V}, L, A, V represent text data, speech data, and visual data respectively;

[0018] The shared encoder ε shared, consisting of a multi-head self-attention layer and a feedforward neural network layer connected in sequence, wherein the feedforward neural network layer consists of a first convolutional layer, a first activation function layer, a second convolutional layer, a second activation function layer and a third convolutional layer connected in sequence.

[0019] As a preference, in step 2, three modality-specific encoders are used to extract the three modality data X respectively. m The modality-specific part X m exc =ε m private (X m );

[0020] The text modality-specific encoder is composed of several sequentially connected multi-layer self-attention mechanism layers and a convolutional layer; the multi-layer self-attention mechanism layer is composed of sequentially connected Transformer layers and convolutional layers, and the output is connected through residual connections;

[0021] The speech modality-specific encoder is composed of several sequentially connected convolutional layers and a bidirectional long short-term memory network;

[0022] The visual modality-specific encoder is composed of a number of sequentially connected convolutional layers, and each convolutional layer has a residual connection.

[0023] Preferably, in step 2, the decoder is composed of several sequentially connected hybrid layer structures and a deconvolution layer; the hybrid layer structure is composed of sequentially connected convolution layers, deconvolution layers and activation function layers, wherein the convolution layers and deconvolution layers are connected with residual connections.

[0024] Preferably, in step 3, the self-attention mechanism network Θ sets a dedicated self-attention module for each modal data to perform data transposition and reconstruction processing; the processed data are respectively passed through the linear projection layer to further abstract and enhance the features from different modalities; the features processed by the three modalities are spliced ​​at the fusion point using the torch.cat function to generate a comprehensive feature representation a C ; The fusion process includes linear transformation and nonlinear activation operations.

[0025] Preferably, in step 3, the cross-modal attention mechanism network Φ is composed of a sequentially connected cross-modal conversion network, a memory enhancement network, and a feature fusion and processing network;

[0026] The cross-modal conversion network includes six cross-modal conversion modules, each of which is responsible for converting information from one modality into the information formats of the other two modalities;

[0027] The memory enhancement network is provided with a memory enhancement module having a three-layer network structure for each mode;

[0028] The feature fusion and processing network consists of a feature splicing and fusion layer and a feature projection layer; the feature splicing and fusion layer is used to splice the modal features output by the memory enhancement network in dimension, and then output them after linear projection with the feature projection layer.

[0029] Preferably, in step 2, during the training process, cosine similarity is used as an indicator to measure the semantic consistency between different features, and a similarity matrix S is constructed, where S[i, j] is the cosine similarity between feature i and feature j, i, j∈[0, N), N is the number of features of each modality of each sample, and S[i, i]=0;

[0030] Get the loss function L sem (X m )for:

[0031]

[0032] During the training process, the squared error is used to measure the consistency of representations between different modalities in the same sample, and the loss function L is obtained. rep (X m ):

[0033]

[0034] During the training process, the squared difference of the inner product is used as an indicator to measure the geometric consistency between different modalities of different samples, and the loss function L is obtained. geo (X m ):

[0035]

[0036] where X m and represent the modal features of two different samples respectively, and <·,·> refers to the inner product of the matrix.

[0037] The technical solution adopted by the system of the present invention is: a multimodal emotion understanding system with consistent hierarchical relationships, including:

[0038] one or more processors;

[0039] A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the multimodal emotion understanding method with consistent hierarchical relationships.

[0040] Compared with the prior art, the beneficial effects of the present invention include:

[0041] (1) The present invention applies different levels of consistency algorithms to the original features and reconstructed features during the training process. This hierarchical approach makes emotion understanding across multiple modalities more accurate.

[0042] (2) We propose a semantic consistency strategy to aggregate the internal information of a single modality. By enforcing semantic consistency, we ensure a more accurate and nuanced interpretation of the emotional state in each modality.

[0043] (3) This paper designs a representation consistency algorithm at the modal level and a geometric consistency algorithm at the sample level to address the challenge of lack of modal synchronization and significantly improve the model performance.

[0044] (4) We conduct comprehensive experiments on public multimodal emotion understanding datasets and obtain superior or comparable results, which demonstrates the feasibility of our approach and the effectiveness of the hierarchical consistency we maintain in multimodal emotion understanding. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The technical solution of the present invention is further illustrated below using embodiments and specific implementation methods. In addition, in the process of illustrating the technical solution, some drawings are also used. For those skilled in the art, other drawings and the intention of the present invention can also be obtained based on these drawings without making any creative efforts.

[0046] Figure 1 Schematic diagram of the method according to an embodiment of the present invention;

[0047] Figure 2 The overall network framework diagram used in the embodiment of the present invention;

[0048] Figure 3 Schematic diagram of the intra-modal semantic consistency method in an embodiment of the present invention;

[0049] Figure 4 Schematic diagram of the inter-modal representation consistency and inter-sample geometric consistency method according to an embodiment of the present invention;

[0050] Figure 5 A schematic diagram of the structure of a shared encoder according to an embodiment of the present invention;

[0051] Figure 6 A schematic diagram of the structure of a text modality-specific encoder according to an embodiment of the present invention;

[0052] Figure 7 A schematic diagram of the structure of an audio modality-specific encoder according to an embodiment of the present invention;

[0053] Figure 8 A schematic diagram of the structure of a video modality-specific encoder according to an embodiment of the present invention;

[0054] Figure 9 A schematic diagram of the decoder structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to facilitate ordinary technicians in this field to understand and implement the present invention, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0056] Please see Figure 1 and Figure 2 This embodiment provides a multimodal emotion understanding method with consistent hierarchical relationships, including the following steps:

[0057] Step 1: Extract the original multimodal data features X that preserve the time dimension m ; where m∈{L,A,V},X L 、X A 、X V Represent text data features, voice data features and visual data features respectively;

[0058] In one embodiment, because multimedia data is a stream with a temporal dimension, preserving the temporal dimension when extracting raw features from the data is crucial for capturing the inherent dynamics and patterns in the data stream. To this end, this embodiment employs a 1D convolutional layer for each modality. By employing one-dimensional temporal convolution to derive these shallow encoding features, the temporal dimension of the input is effectively preserved, enabling processing of both aligned and unaligned data.

[0059] Step 2: Use a common encoder ε shared To extract three modal data X m The modal common part X m irr =ε shared (X m ) ; Use three modality-specific encoders to extract three modality data X respectively m The modality-specific part X m exc =ε m private (X m );

[0060] The shared encoder ε shared and three modality-specific encoders, all of which are trained encoders; during the training process, the common part and the unique part corresponding to each modality are connected, and then the reconstructed feature R is obtained through the decoder m =D m ([X m exc , X mirr ]), where [·,·] means connected, D m Represents the decoder corresponding to each modality;

[0061] Please see Figure 3 and Figure 4 In one embodiment, cosine similarity is used as an indicator to measure the semantic consistency between feature vectors of different modalities, and the cosine similarity between feature vectors of different modalities is maximized; square difference is used as an indicator to measure the consistency of feature representation under different modalities within the same sample, and the square difference of feature representation under different modalities of the same sample is minimized; square difference of inner product is used as an indicator to measure the geometric consistency between different modal sub-samples of different samples, and the square difference of inner product of different modal features of different samples is minimized;

[0062] Taking into account that the decoupling process may affect the consistency of the original feature processing settings, this embodiment processes the common parts and unique parts corresponding to each modality obtained by decoupling to obtain reconstructed features, and also performs consistency on them, and loads them together with the consistency loss calculation of the original features for training the entire model.

[0063] In this embodiment, the feature R m According to the above three indicators, consistency processing is performed, and then compared with the original multimodal data feature X m The consistency loss calculation of the three indicators is loaded together for training, and the training ends after reaching the preset convergence standard.

[0064] This embodiment uses cosine similarity to measure the semantic consistency between features of different modalities. By maximizing the cosine similarity between feature vectors of different modalities, the model can capture the semantic similarity between different features within a modality.

[0065] This example uses the mean square error (MSE) to measure the consistency of feature representations for the same sample across different modalities. By minimizing the squared difference in feature representations for the same sample across different modalities, the model's understanding of the same object's data across different modalities is enhanced.

[0066] This example uses the squared difference of inner products to measure the geometric consistency between different modalities of different samples. By minimizing the squared difference of the inner products of different modal features of different samples, the geometric relationship between mismatched modalities between samples is kept stable, enhancing the consistency of understanding between models.

[0067] In this embodiment, the model shows good performance indicators such as high precision and F1 score on the independent validation set. In addition, the model does not show overfitting on the validation set.

[0068] Please see Figure 5 In one embodiment, a common encoder ε is used.shared To extract three modal data X m The modal common part X m irr =ε shared (X m ), where m∈{L, A, V}, L, A, V represent text data, speech data, and visual data respectively;

[0069] The shared encoder ε shared , consisting of a multi-head self-attention layer and a feedforward neural network layer connected in sequence, wherein the feedforward neural network layer consists of a first convolutional layer, a first activation function layer, a second convolutional layer, a second activation function layer and a third convolutional layer connected in sequence.

[0070] The multi-head self-attention layer allows the model to consider information from different positions simultaneously during encoding, thereby more effectively processing time series data of various modalities. The feedforward neural network layer further processes the output of the self-attention layer to enhance feature expression capabilities.

[0071] The multi-head self-attention layer uses a similar mechanism to Transformers, but adjusts the focus to accommodate multimodal input. The weights of each head are initialized differently to focus on different features of text, speech, and visual data respectively.

[0072] The feedforward neural network layer uses the ReLU activation function to enhance nonlinear expression capabilities. In order to be common to the three modalities, the number of input and output channels of the feedforward neural network layer is set to 768, consistent with the text modality after BERT preprocessing, which has the richest number of features. The step size of the convolutional layer is kept at 1, and the convolution kernel size is adjusted to 3 to better capture the correlation and contextual information between the modalities. In the public encoder of the present invention, the feedforward neural network layer contains three convolutional layers.

[0073] In one embodiment, three modality-specific encoders are used to extract the three modality data X respectively. m The modality-specific part X m exc =ε m private (X m );

[0074] Please see Figure 6 In one embodiment, the text modality-specific encoder is composed of several sequentially connected multi-layer self-attention mechanism layers and a convolutional layer; the multi-layer self-attention mechanism layer is composed of sequentially connected Transformer layers and convolutional layers, and the output is connected through residual connections;

[0075] Multiple layers of self-attention mechanisms are implemented using multiple Transformer layers. Each layer incorporates a self-attention mechanism and a convolutional neural network, with residual connections at the end to further extract the semantics and context of the text data. In this example, three layers of self-attention mechanisms are implemented, with each layer having 768 input and output channels, consistent with the BERT preprocessed data.

[0076] The convolution layer sets the input and output channels to 768, which is consistent with the BERT preprocessed data; the convolution kernel size is set to 3 to better capture local context features.

[0077] Please see Figure 7 In one embodiment, the speech modality-specific encoder is composed of several sequentially connected convolutional layers and a bidirectional long short-term memory network;

[0078] This example uses multiple convolutional layers to extract the time-frequency features of speech signals. Each layer is followed by batch normalization and a ReLU activation function to enhance the stability of feature extraction and the ability to express nonlinearities. Three convolutional layers are used, with 128 input and output channels and a kernel size of 5, which is more suitable for capturing dynamic changes in speech.

[0079] Bidirectional Long Short-Term Memory (Bi-LSTM) network: A bidirectional LSTM is added after the convolutional layer to capture the front-end and back-end dependencies of time series data and adapt to complex speech changes.

[0080] Please see Figure 8 In one embodiment, the visual modality-specific encoder is composed of several sequentially connected convolutional layers, and residual connections are used between each convolutional layer.

[0081] This embodiment uses multiple levels of convolutional layers to gradually extract visual features. Each convolution layer is followed by maximum pooling and batch normalization to effectively extract key information from the image. The number of input and output channels of the convolutional layer is set to 128 to process richer image features, and the convolution kernel size is set to 3 to better capture local visual features. In this embodiment, a total of three convolutional layers are set to form a deep convolutional neural network. At the same time, residual connections are introduced in the convolutional layers to avoid the gradient vanishing problem during training, improving training efficiency and model depth.

[0082] Please see Figure 9 In one embodiment, the decoder is composed of several sequentially connected hybrid layer structures and a deconvolution layer; the hybrid layer structure is composed of sequentially connected convolution layers, deconvolution layers and activation function layers, wherein the convolution layers and deconvolution layers are connected with residual connections.

[0083] This example first uses a convolutional layer to further compress features and enhance inter-modal feature fusion. Because the input consists of the modal public and private components decoupled from the corresponding modality, the number of input channels in this convolutional layer is twice that of the preprocessed modality, with a convolution kernel size of 3 and a stride of 1.

[0084] This embodiment introduces a deconvolution layer after the convolution layer to upsample the data and restore it to its original size. The deconvolution layer can increase the model's generative capabilities, expand the feature map, and help restore the temporal characteristics of the data.

[0085] This embodiment adds a residual connection between each convolution and deconvolution layer, which helps the decoder maintain information flow in the deep network structure, reduces information loss, and improves decoding accuracy.

[0086] In this embodiment, the number of output channels of the last deconvolution layer is adjusted to match the original modal data to ensure that the dimension of the output data is consistent with the corresponding original modal data, which facilitates subsequent sentiment classification or sentiment regression processing.

[0087] This example adds batch normalization after each convolution and deconvolution layer, which helps improve model stability and convergence speed during training. It also uses the ReLU activation function to provide nonlinear processing capabilities and enhance the model's expressiveness.

[0088] Step 3: For the common parts of each modality, use the self-attention mechanism network Θ to obtain the intermediate result a C =[Θ(X L irr ),Θ(X A irr ),Θ(X V irr )];

[0089] For the unique parts of each modality, a cross-modal attention mechanism network Φ is used to calculate the cross-modal result a corresponding to each modality m =[Φ(X m exc , X m1 exc ),Φ(X m exc , X m2 exc )], where m1 and m2 represent the other two modes except m;

[0090] Final comprehensive sentiment estimate MLP is a multi-layer perceptron, a′ m =Sigmoid(a m Wm )m∈{L,A,V,C},Sigmoid() is the activation function, W m are the weights set for each group of modes.

[0091] In one embodiment, the self-attention mechanism network Θ sets a dedicated self-attention module for each modal data to perform data transposition and reconstruction processing. These modules focus on capturing the key features of each modality and its internal semantic relationships to improve the accuracy of sentiment analysis; the processed data are respectively passed through a linear projection layer to further abstract and enhance the features from different modalities; the processed features of the three modalities are spliced ​​at the fusion point using the torch.cat function to generate a comprehensive feature representation a C The fusion process includes linear transformation and nonlinear activation operations, as well as the dropout mechanism used during training to enhance the generalization ability of the model and prevent overfitting.

[0092] In one embodiment, the cross-modal attention mechanism network Φ is composed of a sequentially connected cross-modal conversion network, a memory enhancement network, and a feature fusion and processing network;

[0093] The cross-modal conversion network contains six cross-modal conversion modules, each of which is responsible for converting information from one modality into the information format of the other two modalities. The design of these modules ensures that information can be effectively converted and integrated between different modalities, enhancing the model's ability to process multiple data types.

[0094] The memory enhancement network is provided with a memory enhancement module with a three-layer network structure for each modality; these modules are used to deeply process and store information after cross-modal conversion, thereby improving the network's ability to remember and reproduce complex emotional features.

[0095] The feature fusion and processing network consists of a feature splicing and fusion layer and a feature projection layer. The feature splicing and fusion layer is used to splice the modal features output by the memory enhancement network in terms of dimensions, and then outputs them after linear projection with the feature projection layer. This enhances the model's ability to express emotions.

[0096] In one embodiment, during the training process, cosine similarity is used as an indicator to measure the semantic consistency between different features, and a similarity matrix S is constructed, where S[i,j] is the cosine similarity between feature i and feature j, i,j∈[0,N), N is the number of features of each modality of each sample, and S[i,i]=0;

[0097] Get the loss function L sem (X m )for:

[0098]

[0099] The element S[i,j] in the similarity matrix S represents the cosine similarity between feature i and feature j. Cosine similarity is calculated by taking the dot product of two feature vectors and dividing it by their respective modulos. This metric assesses the directional similarity between two feature vectors, indirectly reflecting their semantic similarity. In this method, S[i,i] is set to 0 to exclude comparisons between features themselves, ensuring that only the similarities between different features are compared.

[0100] The similarity matrix S is used to enhance the aggregation of similar emotions within a modality. By constructing the similarity matrix S, the model can identify and aggregate features with high similarity (i.e., similar emotions). For example, when processing text modalities, feature vectors generated by different words or sentence fragments will have a high cosine similarity if they are semantically close. This identification of similarities enables the model to cluster semantic features with similar emotions, thereby enhancing the representation of emotional consistency within that modality.

[0101] Within modalities, feature aggregation makes it easier for the model to identify and learn the features that are most representative of specific emotional expressions. For example, in visual modalities, similar expressions or colors may be clustered, enhancing the model's efficiency and accuracy in parsing and expressing the same emotion.

[0102] The loss function uses the similarity matrix S to optimize the model, ensuring that emotion-similar features that should be clustered are not mistakenly scattered during model training. This is achieved by minimizing the high cosine similarity between emotion-inconsistent features (i.e., different emotion features that are mistakenly identified as semantically similar).

[0103] In one embodiment, during the training process, the squared error is used to measure the consistency of representations between different modalities in the same sample, and the loss function L is obtained. rep (X m ):

[0104]

[0105] Squared difference is a common mathematical method used to calculate the difference between two variables. In multimodal sentiment understanding, representations generated by different modalities (such as text, vision, and audio) may differ due to inherent characteristics of the modalities. Therefore, each modality is first normalized before the squared difference calculation is performed. By calculating the squared difference between these representations, we can quantify the inconsistency between their representations.

[0106] The loss function optimizes the model by minimizing the squared difference between the representations of different modalities. This means that during the learning process, the model is driven to minimize the difference between the representations of each modality.

[0107] When representational differences between modalities are minimized, these modalities will be more consistent in expressing the sentiment of the same sample. For example, a positive sentiment description in text should closely match the corresponding visual expression (e.g., a smile) and audio representation (e.g., a pleasant tone of voice).

[0108] In one embodiment, during the training process, the squared difference of the inner product is used as an indicator to measure the geometric consistency between different modalities of different samples, and the loss function L is obtained. geo (X m ):

[0109]

[0110] where X m and They represent the modal features of two different samples respectively, and <·,·> refers to the inner product of the matrix.

[0111] The inner product is a mathematical operation typically used to calculate the dot product of two vectors and can characterize the angular relationship between them. In multimodal data processing, the inner product can be used to measure the similarity or consistency between feature vectors of different modalities. Here, this embodiment is used to quantify geometric consistency between modalities.

[0112] The loss function is designed to minimize the squared difference of the inner product of features between different modalities for different samples, so that the model can keep the features between the corresponding modalities geometrically consistent when processing different samples. For example, if the feature vectors of the textual modality and the visual modality can maintain similar inner product values ​​across different samples, this indicates that these modalities have consistent geometric structures when representing the sentiment of the sample.

[0113] Geometric consistency means that the feature spaces of different modalities have consistent directionality or alignment for the representation of similar or related emotions. This alignment ensures that emotional states observed in one modality are expressed in a similar manner in other modalities, thereby enhancing synchronization between different modalities. This synchronization between modalities makes the entire model's judgment of emotion more accurate and consistent, because each modality strengthens the other's emotional signal. During the model training process, by adjusting the model parameters to minimize the loss function, that is, reducing the squared difference of the inner products between different modalities of different samples, the model can be systematically guided to learn how to synchronize the feature representations of different modalities from a geometric perspective.

[0114] This embodiment also provides a multimodal emotion understanding system with consistent hierarchical relationships, including:

[0115] one or more processors;

[0116] A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the multimodal emotion understanding method with consistent hierarchical relationships.

[0117] The present invention is further described below through specific experiments.

[0118] This experiment uses the Pytorch deep learning framework. The hardware environment is an NVIDIA GeForce RTX3090*8 graphics card and an Intel(R) Xeon(R) Gold 6240 processor.

[0119] For the input multimodal multimedia data, this experiment took a series of preprocessing steps. For language data, the present invention uses the GloVe tool to convert the text into 300-dimensional word vectors. In addition, in order to obtain more complex word features, the present invention uses the BERT-base-uncased model, which generates a 768-dimensional hidden state representation for each word. For the video modality, this experiment uses the Facet tool to encode the video frames. This tool is good at capturing and understanding 35 unique facial action units, providing a comprehensive analysis of facial expressions. For the audio modality, this experiment uses the COVAREP tool to process the audio data and extract 74-dimensional feature vectors, making the audio expression more detailed. The code implementation of this experiment is built using the PyTorch framework version 1.10.1+cu111 and Transformer version 4.29.2. In all experiments, the batch size is set to 16 and trained for 30 epochs or until convergence to ensure the stability of the model performance. The results of the model trained for multiple epochs are selected to test the multimodal emotion understanding performance on the validation set.

[0120] The dataset for this experiment is:

[0121] CMU-MOSI: This dataset contains 2,199 short video clips from online stand-up videos of various subjects. The audio and video modalities are sampled at 12.5Hz and 15Hz, respectively. In the experiments, the CMU-MOSI dataset is divided into training, validation, and test sets, containing 1,284, 229, and 686 instances, respectively.

[0122] CMU-MOSEI: This dataset consists of 22,856 video segments extracted from YouTube movie reviews, with audio and video data sampled at 20Hz and 15Hz, respectively. Given its large size, the dataset is split into 16,326 training instances, 1,871 validation instances, and 4,659 test instances, providing comprehensive coverage for rigorous analysis.

[0123] CH-SIMS: Unlike the other two datasets, the CH-SIMS dataset contains 60 original videos and 2,281 refined video segments, each of which is at least one second and no longer than ten seconds. In our experiments, we split the CH-SIMS dataset into training, validation, and test sets in a 6:2:2 ratio, containing 1,368, 456, and 457 sentences, respectively.

[0124] Experimental evaluation indicators:

[0125] In the experiment, the performance evaluation indicators used include accuracy (ACC i (represents i-unit accuracy), F1 score, mean absolute error (MAE), and Pearson correlation coefficient (Corr). Consistent with previous work on multimodal emotion understanding, both the CMU-MOSI and CMU-MOSEI datasets are annotated with emotion intensity scores ranging from -3 to +3, covering the spectrum from extremely negative to extremely positive. Furthermore, following the work on the CH-SIMS dataset, this experiment annotated the CH-SIMS dataset with emotion intensity scores ranging from -1 to +1.

[0126] Comparison with existing methods

[0127] In order to verify the effectiveness of the present invention, the search results of the present invention are compared with the existing methods (the best result is marked in bold, the second best is marked in bold). Underline For details, please see Tables 1, 2, and 3 below.

[0128] Table 1: Comparison results with state-of-the-art methods on the CMU-MOSI dataset

[0129]

[0130]

[0131] Table 2: Results compared with state-of-the-art methods on the CMU-MOSEI dataset

[0132]

[0133]

[0134] Table 3: Comparison results with state-of-the-art methods on the CH-SIMS dataset

[0135]

[0136]

[0137] Tables 1, 2, and 3 show comparative results on the CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets, respectively. Our proposed method achieves superior or comparable performance across various settings compared to existing multimodal emotion understanding methods. The reasons for our method's significant outperformance over existing multimodal emotion understanding methods can be summarized as follows: First, our method utilizes feature decoupling techniques to process raw features, enhancing the model's discriminative power by reducing feature redundancy and focusing on significant emotion indicators. Furthermore, our method considers consistency between raw and reconstructed features at different levels during training. Our hierarchical structure consistency algorithm addresses the lack of intra-modal interaction in mainstream multimodal emotion understanding methods, further strengthening modal synchronization and significantly improving the performance of our model. Unfortunately, in experiments on the CMU-MOSI dataset in an unaligned setting, where features are limited and there is significant temporal discrepancy between modalities, our method slightly outperforms our method. This is primarily because our modality recalibration module (MRM) and unimodal label generation module (ULGM) excel at handling datasets with significant inter-modal discrepancies. However, our method excels on larger and more complex datasets such as CH-SIMS and CMU-MOSEI, and also shows the best performance on the aligned CMU-MOSI dataset, demonstrating its versatility and adaptability in multimodal emotion understanding.

[0138] To better validate the importance of the proposed module, we also conduct ablation experiments to demonstrate the effectiveness of each component, including the hierarchical structure consistency algorithm: semantic consistency, representation consistency, and geometric consistency. To more clearly demonstrate the impact of our hierarchical relationship consistency algorithm on feature reconstruction, we also analyze it separately as a component in the ablation experiment. This experiment selects binary accuracy, F1 score, and mean absolute error as the performance evaluation metrics. The results are shown in Table 4 (Sem-, Rep-, and Geo- represent semantic consistency, representation consistency, and geometric consistency applied to the original features, respectively, and Rec- represents our HSCL applied to the reconstructed features).

[0139] Table 4: Comparison of different configuration components

[0140]

[0141] Through experiments, we draw the following conclusions:

[0142] First, applying the proposed semantic consistency algorithm to the raw features significantly improved performance compared to using only the decoupled neural network for emotion understanding. This improvement demonstrates that the proposed semantic consistency algorithm effectively strengthens intra-modal interactions, enabling the model to gain a deeper understanding of the emotional cues conveyed by each modality.

[0143] Regarding algorithms for addressing the lack of modal synchronization, results show that using either the cross-modal representation consistency or geometric consistency between samples in this paper alone can effectively improve model performance on various datasets. When both are applied to the original features, model performance is further optimized, particularly in terms of improving the prediction accuracy of MOSI and MOSEI. This result validates the effectiveness of the proposed algorithm in addressing the synchronization challenge between different modalities and helps improve model performance. Furthermore, experiments show that when the hierarchical structure consistency algorithm is applied to the reconstructed features during training, model performance is significantly improved. For example, on the MOSI dataset, using the reconstruction consistency algorithm alone improves performance from the original ACC of 282.2, F1 score of 82.1, and MAE of 0.775 to ACC of 283.7, F1 of 83.7, and MAE of 0.726. When combined with the consistency algorithm for the original features, the F1 score improves from 84.2 to 85.2, the ACC2 score improves from 84.2 to 85.2, and the MAE decreases from 0.743 to 0.716. This finding further validates the proposed hierarchical structure consistency learning (HSCL) approach and decoupled neural network design, highlighting its applicability in enhancing the training process and extending the hierarchical consistency approach to reconstructed features.

[0144] To fully evaluate the effectiveness of our proposed method, we conducted a quantitative comparative experiment using a feature-decoupled neural network (FD). The results are shown in Table 5. While FD demonstrates efficient information extraction through feature decoupling and an attention mechanism, it fails to address the challenges of insufficient intra-modal interaction and ensuring modal synchronization. In contrast, our proposed HSCL method skillfully addresses these challenges. The results confirm the effectiveness of our proposed method and demonstrate that hierarchical structure consistency learning is beneficial for multimodal emotion understanding.

[0145] Table 5: Comparison of experimental results between the present invention and the simple decoupling neural network

[0146]

[0147]

[0148] In order to effectively evaluate the performance of the present invention under single-modal conditions and understand the performance of the hierarchical structure consistency algorithm of the present invention on a single modality and its contribution to the final result, a single-modal input comparison experiment with and without the use of the hierarchical structure consistency algorithm of the present invention is also conducted, and the results are shown in Table 6.

[0149] Table 6: Single-mode experimental results

[0150]

[0151] Experiments show that the language modality performs well when using only the decoupled neural network. The performance is further improved by applying the hierarchical structure consistency algorithm of the present invention, as evidenced by the improved metrics. In contrast, the audio and video modalities initially had limited performance when using only the decoupled neural network, but improved significantly after implementing the present invention. This improvement is attributed to better modal aggregation and interaction, especially the consistency algorithm of the present invention that addresses the lack of modal synchronization. These algorithms not only synchronize the different modalities, but also enhance the weaker modality, thereby improving the overall prediction performance and the robustness of the model.

[0152] It should be understood that the embodiments described above are only some of the embodiments of the present invention, rather than all of the embodiments. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention may be arbitrarily combined with each other to form a feasible technical solution. Such combination is not restricted by the order of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is mutually inconsistent or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0153] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of protection of the patent of the present invention. Under the guidance of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested by the present invention shall be based on the attached claims.

Claims

1. A multimodal emotion understanding method with consistent hierarchical relationships, characterized by: The following steps are involved: Step 1: Extract the original multimodal data features X that preserve the time dimension m ; where m∈{L,A,V}, X L 、X A 、X V Represent text data features, voice data features and visual data features respectively; Step 2: Use a common encoder ε shared To extract three modal data X m The modal common part X m irr =ε shared (X m ) ; Use three modality-specific encoders to extract three modality data X respectively m The modality-specific part X m exc =ε m private (X m ); The shared encoder ε shared and three modality-specific encoders, all of which are trained encoders; During the training process, the common part and the unique part corresponding to each modality are connected, and then the reconstructed feature R is obtained through the decoder. m =D m ([X m exc ,X m irr ]), where [·,·] means connected, D m Represents the decoder corresponding to each modality; Cosine similarity is used as an indicator to measure the semantic consistency between feature vectors of different modalities, and the cosine similarity between feature vectors of different modalities is maximized; square difference is used as an indicator to measure the consistency of feature representation under different modalities within the same sample, and the square difference of feature representation under different modalities of the same sample is minimized; square difference of inner product is used as an indicator to measure the geometric consistency between different modal sub-samples of different samples, and the square difference of inner product of different modal features of different samples is minimized; The reconstructed feature R m According to the above three indicators, consistency processing is performed, and then compared with the original multimodal data feature X m The consistency loss calculation of the three indicators is loaded together for training, and the training ends after reaching the preset convergence standard; Step 3: For the common parts of each modality, use the self-attention mechanism network Θ to obtain the intermediate result a C =[Θ(X L irr ),Θ(X A irr ),Θ(X V irr )]; For the unique parts of each modality, a cross-modal attention mechanism network Φ is used to calculate the cross-modal result a corresponding to each modality m =[Φ(X m exc ,X m1 exc ),Φ(X m exc ,X m2 exc )], where m1 and m2 represent the other two modes except m; Final comprehensive sentiment estimate MLP is a multi-layer perceptron, a′ m =Sigmoid(a m W m )m∈{L,A,V,C}, Sigmoid() is the activation function, W m are the weights set for each group of modes.

2. The hierarchically consistent multimodal emotion understanding method according to claim 1, characterized in that: In step 1, one-dimensional temporal convolution is used to extract the original features of multimedia data while retaining the time dimension, and the data is further aligned.

3. The hierarchically consistent multimodal emotion understanding method according to claim 1, characterized in that: In step 2, a common encoder ε is used shared To extract three modal data X m The modal common part X m irr =ε shared (X m ), where m∈{L,A,V}, L,A,V represent text data, speech data and visual data respectively; The shared encoder ε shared , consisting of a multi-head self-attention layer and a feedforward neural network layer connected in sequence, wherein the feedforward neural network layer consists of a first convolutional layer, a first activation function layer, a second convolutional layer, a second activation function layer and a third convolutional layer connected in sequence.

4. The hierarchically consistent multimodal emotion understanding method according to claim 1, characterized in that: In step 2, three modality-specific encoders are used to extract the three modality data X m The modality-specific part X m exc =ε m private (X m ); The text modality-specific encoder is composed of several sequentially connected multi-layer self-attention mechanism layers and a convolutional layer; the multi-layer self-attention mechanism layer is composed of sequentially connected Transformer layers and convolutional layers, and the output is connected through residual connections; The speech modality-specific encoder is composed of several sequentially connected convolutional layers and a bidirectional long short-term memory network; The visual modality-specific encoder is composed of a number of sequentially connected convolutional layers, and each convolutional layer has a residual connection.

5. The hierarchically consistent multimodal emotion understanding method according to claim 1, characterized in that: In step 2, the decoder is composed of several sequentially connected hybrid layer structures and a deconvolution layer; the hybrid layer structure is composed of sequentially connected convolution layers, deconvolution layers and activation function layers, wherein the convolution layers and deconvolution layers are connected with residual connections.

6. The hierarchically consistent multimodal emotion understanding method according to claim 1, characterized in that: In step 3, the self-attention mechanism network Θ sets a dedicated self-attention module for each modal data to perform data transposition and reconstruction processing; the processed data passes through the linear projection layer to further abstract and enhance the features from different modalities; the features processed by the three modalities are spliced ​​at the fusion point using the torch.cat function to generate a comprehensive feature representation a C ; The fusion process includes linear transformation and nonlinear activation operations.

7. The hierarchically consistent multimodal emotion understanding method according to claim 1 is characterized in that In step 3, the cross-modal attention mechanism network Φ is composed of a sequentially connected cross-modal conversion network, a memory enhancement network, and a feature fusion and processing network; The cross-modal conversion network includes six cross-modal conversion modules, each of which is responsible for converting information from one modality into the information formats of the other two modalities; The memory enhancement network is provided with a memory enhancement module having a three-layer network structure for each mode; The feature fusion and processing network consists of a feature splicing and fusion layer and a feature projection layer; the feature splicing and fusion layer is used to splice the modal features output by the memory enhancement network in dimension, and then output them after linear projection with the feature projection layer.

8. The hierarchically consistent multimodal emotion understanding method according to any one of claims 1 to 7, characterized in that: In step 2, during the training process, cosine similarity is used as an indicator to measure the semantic consistency between different features, and a similarity matrix S is constructed, where S[i,j] is the cosine similarity between feature i and feature j, i,j∈[0,N), N is the number of features of each modality of each sample, and S[i,i]=0; Get the loss function L sem (X m )for: During the training process, the squared error is used to measure the consistency of representations between different modalities in the same sample, and the loss function L is obtained. rep (X m ): During the training process, the squared difference of the inner product is used as an indicator to measure the geometric consistency between different modalities of different samples, and the loss function L is obtained. geo (X m ): where X m and They represent the modal features of two different samples respectively, and <·,·> refers to the inner product of the matrix.

9. A hierarchically consistent multimodal emotion understanding system, characterized by: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement a multimodal emotion understanding method with consistent hierarchical relationships as described in any one of claims 1 to 8.

Images