A multimodal sentiment classification method based on modality-enhanced convolutional graph

Abstract

The present invention discloses a multimodal sentiment classification method based on modality-enhanced convolutional graph. The present invention proposes a new modality-enhanced fusion framework, which is based on a graph convolutional neural network and provides an effective method for fusing unaligned multimodal sequences. With the help of text modality, a multimodal enhancement module is first used to enhance the visual and auditory modalities to obtain modal information with stronger sentiment discriminability, thereby helping the subsequent aggregation process. In addition, a text-driven multimodal feature graph is constructed to perform modal fusion, which can effectively deal with the imbalance problem between modalities in the graph convolution aggregation process. Finally, the fusion information extracted from the modality-enhanced convolutional graph is integrated into the text representation, thereby dynamically transforming the original text representation into the most accurate multimodal semantic space. Compared with existing multimodal fusion methods, the results of the present invention are better.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal emotion recognition at the intersection of natural language processing, speech, and vision, and specifically relates to a multimodal emotion classification method based on modality-enhanced convolutional graphs. Specifically, it is a method for judging the emotional state of a subject through a fusion network technology based on a text-driven graph convolutional network and an attention mechanism. Background Art

[0002] With the rapid development of multimedia technology, multimodal sentiment analysis has become a hot topic. Efficiently analyzing sentiment across data in different modalities is a major challenge in artificial intelligence. Compared to single-modality sentiment analysis, multimodal sentiment analysis can help more effectively and accurately understand the emotions underlying data, and is therefore widely used in sentiment analysis tasks. Generally speaking, different modalities can complement each other and better bridge the gap between semantics and sentiment.

[0003] However, previous studies have found that the fusion structure obtained in multimodal learning tasks is a three-way symmetric structure, in which the bidirectional cross-modal modeling method is the same. It is worth noting that many previous studies have found that the key information distributed among the three modalities is unbalanced, with the text modality containing more emotional information than the other modalities (visual and auditory). Therefore, without considering the relative importance of these three modalities, they cannot be properly integrated, which affects the final task performance. Summary of the Invention

[0004] The first purpose of the present invention is to propose a multimodal sentiment classification method based on modality-enhanced convolutional graphs in response to the shortcomings of the existing technology. The present invention proposes a new modality-enhanced fusion framework, which provides an effective method for fusing unaligned multimodal sequences based on graph convolutional neural networks. With the help of text modality, a multimodal enhancement module is first used to enhance the visual and auditory modalities to obtain modal information with stronger sentiment discriminability, thereby assisting the subsequent aggregation process. In addition, a text-driven multimodal feature graph is constructed for modal fusion, which can effectively deal with the imbalance problem between modalities in the graph convolution aggregation process. Finally, the fusion information extracted from the modality-enhanced convolutional graph is integrated into the text representation, thereby dynamically transforming the original text representation into the most accurate multimodal semantic space.

[0005] The technical solution adopted in the present invention is:

[0006] A multimodal sentiment classification method based on modality-enhanced convolutional graphs includes the following steps:

[0007] Step (1): Obtain multimodal data

[0008] The multimodal data includes voice modality, video modality, and text modality;

[0009] The purpose of multimodal fusion is to obtain the complementarity and consistency information between multiple modal data under the same task, thereby enhancing the task performance of sentiment analysis;

[0010] Step (2): Preprocessing of multimodal information data and extracting primary representations

[0011] In order to perform multimodal fusion operations on the feature level of multimodal data, different networks are used to extract the primary feature representation of each modal data, specifically:

[0012] 2-1 Use a one-dimensional convolutional network to convolve both the speech modality and the video modality data into the same time dimension, and obtain the visual modality M' after one-dimensional convolution processing v and speech mode M' a , and then use the bidirectional long short-term memory network (LSTM) to get the primary representation of the current modality, which is expressed as:

[0013]

[0014] Where a and v represent the voice mode and video mode respectively; M' s Represents the one-dimensional convolution data of mode s;

[0015] 2-2 Convert text modal data into primary feature form through BERT Embedding (a bidirectional encoder representation based on Transformer), expressed as:

[0016] X t =BERT Emedding(m0,m1,…,m n+1 )#(2)

[0017] Where t represents text mode; m n+1 Indicates the n+1th text data;

[0018] Step (3) utilizes multimodal enhancement to obtain complex emotional context-related information in the speech and video modalities that are most relevant to the text modality; specifically:

[0019] 3-1 Text modality primary representation data X t and other modal primary characterization data X s , s∈{a,v} are mapped together into a cross-modal emotion interaction space through matrix dot multiplication operation, that is, a joint cross-modal emotion representation space is constructed, which is expressed as:

[0020] X m =X t ⊙Xs ,s∈{a,v}#(3)

[0021] Where ⊙ represents the matrix dot multiplication operation.

[0022] 3-2 Since the information of other modal data is enriched together with the text modality in the cross-modal joint representation space, the L2 norm is used to normalize the joint representation data to calculate the impact of the text modality on the other two modalities, which is expressed as:

[0023] X′ m =||X m ||2#(4)

[0024] 3-3 After normalizing the joint representation data, it is used to change the other modal data X when inputting s The position in its original semantic space, and finally the other modal data X' after text enrichment processing s , expressed as:

[0025] X′ s =X′ m +X s ,s∈{a,v}#(5)

[0026] The above operations can enrich the emotional information within the speech modality and video modality to a certain extent, thereby obtaining a modality with more emotional state discrimination capabilities.

[0027] Step (4): Construct a multimodal convolutional graph for modal fusion

[0028] In order to capture cross-modal sentiment information, a spectral domain graph convolutional network is constructed to encode multimodal contextual information for text-driven cross-modal fusion.

[0029] The spectral domain graph convolutional network includes 2N multimodal undirected graphs, where N represents the number of text modalities, specifically including N text-speech modal graphs G ta =(V ta ,E ta ), and N text-visual graphs G tv =(V tv ,E tv ). V ts ,s∈{v,a} represents the discourse nodes of text modality, speech modality and video modality respectively. The number of nodes in the graph can be freely divided according to the time dimension of the modality. s∈{v,a} is a set of different modal relations, which represents the emotional context in the time domain and feature domain.

[0030] Each multimodal undirected graph is constructed as follows:

[0031] The nodes of text mode, voice mode, and video mode are represented as i represents the index of the data in the dataset, i∈{0,1,2……,n}.

[0032] In order to capture the correlation between different nodes, the cosine distance is used to calculate the edge weight between each multimodal undirected graph node, which is specifically expressed as follows:

[0033]

[0034] Based on the basic information of the above nodes and edge weights, multiple shallow bimodal undirected graphs are constructed to calculate the correlation between the text modality and the other two modalities. Specifically, the restatement graph Laplacian matrix of the undirected graph G = (V, E)

[0035]

[0036] in represents the adjacency matrix, represents the diagonal matrix of graph G, I represents the identity matrix, represents the sum of the adjacency matrix and the identity matrix.

[0037] The graph convolution iterations at different layers can be expressed as:

[0038]

[0039] Where α and β are two different hyperparameters greater than 0 and less than 1, σ represents the activation function, W(l) is the learnable weight matrix, and I represents the constant mapping; set Where η is also a hyperparameter. The residual connection of the first layer H(0) is added to the representation The constant map I is added to the weight matrix W(l).

[0040] After each image iteration is completed, the features of each image are spliced ​​according to the feature dimension to obtain the text-driven multimodal sentiment fusion information:

[0041]

[0042] Different from the existing parallel input, the present invention takes text as the most important modality and effectively integrates it to effectively deal with the inherent problem in multimodal learning - the problem of information imbalance between modalities.

[0043] Step (5), multimodal sentiment classification operation;

[0044] Use the linear transformation layer to transform h s Switch to text mode X tThe obtained data is then normalized using LayerNorm and Dropout to prevent overfitting during training.

[0045] h′ s =Dropout(LayerNorm(h s +X t ))#(10)

[0046] The output of the linear transformation layer is then further passed to BertEncoder, which is a bidirectional encoder with 12 layers of Transformer Layers:

[0047] h″ s =BERT(h′ s )#(11)

[0048] Please note that the first token of the output vector of the last layer refers to [CLS], which contains the information required for the classification task. Finally, a linear layer is used to analyze the obtained h″ s , and use the obtained y to achieve the final sentiment prediction.

[0049] y=Wh″ s +b#(12)

[0050] Where W is the weight matrix of the linear network and b is the bias vector of the linear network.

[0051] Preferably, the loss function uses mean square error (MSE).

[0052] A second object of the present invention is to provide a multimodal sentiment classification system based on modality-enhanced convolutional graphs for implementing the above method, comprising:

[0053] A multimodal data acquisition module is used to acquire multimodal data, including voice modality, video modality, and text modality;

[0054] A primary representation extraction module is used to preprocess multimodal information data and extract multimodal primary representation data;

[0055] A complex emotional context-related information extraction module is used to use multimodal enhancement to obtain complex emotional context-related information in the speech and video modalities that are most relevant to the text modality;

[0056] The multimodal fusion module uses the spectral domain graph convolutional network to encode multimodal context information and perform text-driven cross-modal fusion to obtain the text-driven multimodal emotion fusion information h s ;

[0057] Multimodal sentiment classification module is used to fusion the multimodal sentiment information h driven by text s Perform multimodal sentiment classification.

[0058] A third object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the above method.

[0059] A fourth object of the present invention is to provide a computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the above method is implemented.

[0060] The beneficial effects of the present invention are:

[0061] This paper proposes a text-driven multimodal fusion framework for multimodal sentiment analysis. The multimodal enhancement module enriches modal information and removes redundant information before constructing the multimodal graph and aggregating multimodal information, thereby improving multimodal information imbalance during graph convolution. The paper further constructs a multimodal transition word representation that dynamically captures changes in different non-verbal contexts, effectively complementing current multimodal fusion frameworks in the field of sentiment recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 Module flow chart of the method of the present invention;

[0063] Figure 2 This is a multimodal enhancement flow chart of the method of the present invention;

[0064] Figure 3 This is a multimodal fusion flow chart of the method of the present invention. DETAILED DESCRIPTION

[0065] The present invention will be further analyzed below with reference to specific embodiments and accompanying drawings.

[0066] The present invention provides a multimodal sentiment analysis method based on modality-enhanced convolutional graph, such as Figure 1-2 As shown in the figure, it can be used in different emotion classification scenarios, such as using video modality, audio modality and text modality to perform emotion classification on each sentence spoken by the subjects in the YouTube video dataset. Specifically:

[0067] Step (1): Obtain multimodal information data

[0068] When the subjects perform specific emotional tasks, the text modality data, voice modality data and video modality data of the subjects are recorded.

[0069] Step (2): Extracting the primary representation of multimodal information transformation

[0070] The task of multimodal sentiment analysis is to predict the sentiment intensity, polarity or sentiment label of a given multimodal input (video clip). Video includes three modalities: t (text), a (sound), v (visual): T m and d m represents the sequence length (e.g., number of frames) and the size of the feature vector for modality m.

[0071] Use BERT to encode the input sentence. The original sentence M t =(w1,…,w n ), first add [CLS] and [SEP] at the beginning and end of the sentence respectively, then embed the sentence, and the resulting text pattern is X t =(m0,m1,…,m n+1 ).

[0072] For speech modality data and video modality data, they are first convolved to the same time dimension:

[0073] M′ s =Conv1D(M s ), s∈{a,v}#(1)

[0074] Get the processed visual modality M' v and speech mode M' a Modality: A bidirectional long short-term memory (LSTM) network is used to encode emotional information in speech and video modalities.

[0075]

[0076] Step (3), multimodal enhancement module

[0077] The multimodal enhancement module can calculate the complex emotional context-related information in the speech modality and video modality that are most relevant to the text modality. Figure 2 As shown in the figure, first, in the multimodal enhancement module, the input text modality and speech modality (or video modality) are mapped together into a cross-modal emotion interaction space through matrix dot multiplication operation, that is, a joint cross-modal emotion representation space is constructed, which is expressed as:

[0078] X m =X t ⊙X s ,s∈{a,v}#(3)

[0079] The information of the speech modality (or video modality) is enriched together with the text modality in the cross-modal joint representation space. Secondly, L2 normalization is used to normalize the joint representation data in order to calculate the influence of the text modality on the other two modalities, which is expressed as:

[0080] X′ m =||X m ||2#(4)

[0081] After normalizing the joint representation data, it is used to change the other modal data X when inputting s The position in its original semantic space, and finally the other modal data X' after text enrichment processing s , expressed as:

[0082] X′ s =X′ m +X s ,s∈{a,v}#(5)

[0083] The above operations can enrich the emotional information within the speech modality and video modality to a certain extent, thereby obtaining a modality with more emotional state discrimination capabilities.

[0084] Step (4), the graph convolution fusion module constructs a multimodal convolution graph for modal fusion

[0085] In the graph convolution fusion module, 2N undirected graphs are constructed (N is the number of text / video / speech modalities in the training set), including N text-speech modality graphs G ta =(V ta ,E ta ), and N text-visual graphs G tv =(V tv ,E tv ). V ts ,s∈{v,a} represents the emotional context nodes of the two modalities, and the number of nodes in the graph can be freely divided according to the time dimension of the modality. s∈{v,a} is a set of relationships across different modalities, representing the sentiment context in both the time and feature domains. The method for constructing each multimodal undirected graph is as follows:

[0086] like Figure 3 As shown in the figure, the enhanced speech modality and video modality, as well as the text modality after primary transformation, are input into the graph convolution fusion module.

[0087] The nodes of the three modes are represented as i represents the index of the data in the dataset, i∈{0,1,2……,n}, which is determined by the input modal information X t , X' a , X'v , initialized. That is, the modal information corresponding to each sentence is taken as a node. The number of nodes can be divided according to the time dimension, and the number of nodes in each graph can be initialized to the number of modalities.

[0088] In order to capture the correlation between different nodes, cosine distance is used to calculate the weight between each graph node. The specific expression is as follows:

[0089]

[0090] Based on the basic information of the above nodes and edge weights, multiple shallow bimodal undirected graphs are constructed to calculate the correlation between the text modality and the other two modalities. Specifically, the restatement graph Laplacian matrix of the undirected graph G = (V, E) is first constructed. Then iterate each multimodal graph to calculate the multimodal fusion information H.

[0091]

[0092]

[0093] The present invention only needs to iterate each multimodal graph once, and then stitch the features obtained from each graph together according to the feature dimension to obtain the final multimodal fusion information:

[0094]

[0095] in Represents a splicing operation.

[0096] After each graph is iterated, we can get text-driven multimodal sentiment fusion information. Different from the previous parallel input, this invention takes text as the most important modality and effectively integrates it, thus effectively dealing with the inherent problem of multimodal learning - the problem of information imbalance between modalities.

[0097] Step (5), multimodal sentiment classification operation

[0098] Then, a linear transformation layer is used to transform h s Switch to text mode X t The obtained data is then normalized using LayerNorm and Dropout to prevent overfitting during training.

[0099] h′ s =Dropout(LayerNorm(h s +X t ))#(7)

[0100] The output of the linear transformation layer is then further passed to BertEncoder, which is a bidirectional encoder with 12 layers of Transformer Layers:

[0101] h″ s =BERT(h′ s )#(8)

[0102] Note that the first token of the output vector of the last layer refers to [CLS], which contains the information required for the classification task.

[0103] Then, a linear layer is used to analyze the obtained h″ s , and use the obtained y to achieve the final sentiment prediction.

[0104] y=Wh″ s +b#(9)

[0105] Where W is the weight matrix of the linear network and b is the bias vector of the linear network.

[0106] Mean squared error (MSE) is used as the loss function in this sentiment analysis task since it is a regression task.

[0107]

[0108] Figure 3 This is the multimodal fusion flowchart of three modalities A, V and T.

[0109] As shown in Table 1, the present invention and eight basic multimodal fusion methods simultaneously perform emotional state discrimination tasks on two multimodal emotion databases CMU-MOSI and CMU-MOSEI, where t is text data (text), v is visual data (video), and a is speech modality data (audio).

[0110] The metrics used are: Mean Absolute Error (MAE), the mean absolute difference between predicted and true values; Pearson Correlation (Corr), a measure of prediction bias, representing the proportion of predictions that correctly fall within seven ranges between -3 and +3; and binary classification accuracy (Acc-2) and F1 score calculated for positive / negative and non-negative / negative classification results. Comparisons with several existing multimodal fusion methods show that the results of our proposed method outperform existing fusion models, demonstrating the effectiveness of our method.

[0111] Table 1

[0112]

Claims

1. A multimodal sentiment classification method based on modality-enhanced convolutional graph, characterized by The following steps are involved: Step (1), obtaining multimodal data; the multimodal data includes voice modality, video modality, and text modality; Step (2), preprocessing of multimodal information data to extract primary representations; Step (3), using multimodal enhancement to obtain complex emotional context-related information in the speech modality and video modality that are most relevant to the text modality; Specifically: 3-1 Text modality primary feature data X t and other modal primary feature data X s , s∈{a,v} is mapped to a cross-modal emotion interaction space through matrix dot multiplication operation, that is, a joint cross-modal emotion representation space is constructed, which is expressed as: X m =X t ⊙X s ,s∈{a,v}#(3) Where ⊙ represents the matrix dot multiplication operation; a and v represent the voice modality and video modality respectively; 3-2 Since the information of other modal data is enriched together with the text modality in the cross-modal joint representation space, the L2 norm is used to normalize the joint representation data to calculate the impact of the text modality on the other two modalities, which is expressed as: X′ m =||X m ||2#(4) 3-3 Using normalized joint representation data to change other modal data X when input s The position in its original semantic space is used to obtain other modal data X' after text enrichment processing. s , expressed as: X' s =X' m +X s ,s∈{a,v}#(5) Step (4): Construct a multimodal convolutional graph for modal fusion In order to capture cross-modal emotional information, a spectral domain graph convolutional network is constructed to encode multimodal context information and perform text-driven cross-modal fusion to obtain the text-driven multimodal emotional fusion information h s ; The spectral domain graph convolutional network includes 2N multimodal undirected graphs, where N represents the number of text modalities, specifically including N text-speech modal graphs G ta =(V ta ,E ta ), and N text-visual graphs G tv =(V tv ,E tv );V ts ,s∈{v,a} represents the discourse nodes of text modality, speech modality and video modality respectively; It is a set of different modal relationships, representing the emotional context in the time domain and feature domain; Step (5) is to integrate the text-driven multimodal sentiment fusion information h s Perform multimodal sentiment classification.

2. The method according to claim 1, characterized in that Step (2) is specifically: 2-1 Use a one-dimensional convolutional network to convolve both the speech modality and the video modality data to the same time dimension, and obtain the visual modality M' after one-dimensional convolution processing v and speech mode M' a , and then use the bidirectional long short-term memory network LSTM to obtain the primary representation of the current modality, which is expressed as: Where a and v represent the voice mode and video mode respectively; M' s Represents the one-dimensional convolution data of mode s; 2-2 Convert text modal data into primary feature form through BERTEmbedding, expressed as: X t =BERT Emedding(m0,m1,…,m n+1 )#(2) Where t represents text mode; m n+1 Indicates the n+1th text data.

3. The method according to claim 1, characterized in that Step (4) Each multimodal undirected graph is constructed as follows: The nodes of text mode, voice mode, and video mode are represented as i represents the index of the data in the dataset, i∈{0,1,2……,n}; In order to capture the correlation between different nodes, the cosine distance is used to calculate the edge weight between each multimodal undirected graph node, which is specifically expressed as follows: Based on the basic information of the above nodes and edge weights, multiple shallow bimodal undirected graphs are constructed to calculate the correlation between the text modality and the other two modalities; specifically, the restatement graph Laplacian matrix of the undirected graph G = (V, E) in represents the adjacency matrix, represents the diagonal matrix of graph G, I represents the identity matrix, represents the sum of the adjacency matrix and the identity matrix; The graph convolution iterations at different layers are represented as follows: Where α and β are two different hyperparameters greater than 0 and less than 1, σ represents the activation function, W(l) is the learnable weight matrix, and I represents the constant mapping; set where η is also a hyperparameter; the residual connection of the first layer H(0) is added to the representation The constant map I is added to the weight matrix W(l); After each image iteration is completed, the features of each image are spliced ​​according to the feature dimension to obtain the text-driven multimodal sentiment fusion information: h s =h ta ⊕h tv #(9)。 4. The method according to claim 1, characterized in that Step (5) is specifically: 5-1 Use the linear transformation layer to transform h s Transfer to text modality primary feature data X t The semantic space of the dataset is then normalized using LayerNorm and Dropout to prevent overfitting during training. h′ s =Dropout(LayerNorm(h s +X t ))#(10) The output of the 5-2 linear transformation layer is further passed to the BertEncoder: h″ s =BERT(h′ s )#(11) Finally, h″ is obtained by a linear network analysis s , use the obtained y to achieve the final sentiment prediction; y=Wh″ s +b#(12) Where W is the weight matrix of the linear network and b is the bias vector of the linear network.

5. The method according to claim 4, characterized in that It is a bidirectional encoder with 12 Transformer Layers.

6. The method according to claim 4, characterized in that The BertEncoder loss function uses mean square error (MSE).

7. A multimodal sentiment classification system based on modality-enhanced convolutional graphs that implements the method of claims 1-6, characterized in that include: A multimodal data acquisition module is used to acquire multimodal data, including voice modality, video modality, and text modality; A primary representation extraction module is used to preprocess multimodal information data and extract multimodal primary representation data; A complex emotional context-related information extraction module is used to use multimodal enhancement to obtain complex emotional context-related information in the speech and video modalities that are most relevant to the text modality; The multimodal fusion module uses the spectral domain graph convolutional network to encode multimodal context information and perform text-driven cross-modal fusion to obtain the text-driven multimodal emotion fusion information h s ; Multimodal sentiment classification module is used to fusion the multimodal sentiment information h driven by text s Perform multimodal sentiment classification.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 6.

9. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 6 is implemented.

Images