A multimodal irony detection method based on knowledge graph and cross-modal attention

Abstract

The present invention discloses a multimodal irony detection method based on knowledge graph and cross-modal attention. The present invention constructs a cross-modal attention feature vector according to the Bert model, and uses a multimodal knowledge graph to obtain external information and screen the external information. Cross-modal attention constructs the relationship between audio information, text sentence information and sentiment value vector. Attention is calculated through the connection and influence between the audio information vector and the sentiment value vector, so as to pay attention to the contradictory expression in the sentence. The screening of external knowledge information is carried out by using the similarity between the information extracted from the text and the external information retrieved by the knowledge graph. Finally, the feature vectors of various types obtained are input into the classification layer for classification. The present invention can well solve the problem of fusion of different modal information, can pay attention to the contradictions in the sentence and can also extract the inconsistency features of different modalities, and can improve the accuracy of the irony detection task.

Description

Technical Field

[0001] The present invention relates to the technical field of irony detection, and specifically to a multimodal irony detection method based on knowledge graph and cross-modal attention. Background Art

[0002] Irony refers to the fact that the meaning of words when speaking or writing is often opposite to what is understood on the surface. It can conceal the speaker's intentions and add humor. It is ubiquitous in today's social media and daily life. When encountering ironic situations, sentiment analysis tasks are difficult to accurately analyze the emotional tendencies in the situation. In order to improve the recognition accuracy of sentiment analysis and accurately mine the accurate opinions in the situation, the irony detection task is of great research significance.

[0003] Text irony detection includes context-dependent irony and context-independent irony. Context-independent text irony detection is performed by detecting the "contradiction between the previous and next emotional expressions" in the target text sentence. Therefore, deep learning models are usually used to model text sentences containing irony, such as GRU, LSTM and other sequence modeling irony extraction contradiction features. However, this sequence model cannot capture the conflicting expressions in the sentence. Its learning process is linear and it is difficult to capture long-distance dependencies. Therefore, later generations proposed to use the principle of attention mechanism to construct intra-sentence attention, which not only improves the accuracy of the model, but also enhances the interpretability of the model. However, with the continuous deepening of research, detecting irony from a purely textual perspective is very limited, because irony in text sentences is often inseparable from its context.

[0004] The context of context-dependent irony detection usually refers to the context in a broad sense, which includes the voice, tone, expression, or situation of a person speaking, and of course it can also be the theme background of the article. Irony detection that combines text, audio, and image information is also called multimodal irony detection. It has been the focus of research in recent years, and social media data is often multimodal. Therefore, some people have proposed to use the current model method to encode video, audio, and text information, and then simply splice them and use SVM for classification. This can achieve certain results, but there are also some problems. For example, simple splicing is difficult to integrate the information features of each modality because the modalities are related. In addition, the extraction of modal features needs to be carried out in depth to meet the features required for irony. Summary of the invention

[0005] After comprehensively considering the above issues, the present invention proposes a multimodal irony detection method based on knowledge graph and cross-modal attention to address the problems existing in the prior art.

[0006] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0007] Step (1) constructs the initial word embedding vector Token Embedding and the position encoding vector Position Embedding according to the input text information T, and uses the word embedding vector and the position vector as the input of the Bert pre-training model. After training, the sentence vector H is obtained. t ;

[0008] Step (2) segment the input audio information V according to the text information T, and encode each segmented audio information using OpenSmile to obtain the audio information feature vector H v ;

[0009] Step (3) Encode the input video frame information G using ResNet and perform dimension transformation through the fully connected layer to obtain the video frame feature vector H g ;

[0010] Step (4) query the sentiment value of each word in the input text information T according to the external sentiment dictionary, and construct the sentiment value feature vector H s ;

[0011] Step (5) constructs the cross-modal attention according to the three vectors obtained in steps (1), (2), and (4) to obtain the cross-modal attention vector H att ;

[0012] Step (6) Input the video frame information G into the multimodal knowledge graph, obtain external knowledge information related to the video frame information G, construct an external knowledge vector and filter it to obtain an external knowledge vector H that is independent of the context. k ;

[0013] Step (7) converts the cross-modal attention vector H att , video frame information vector H g The external knowledge vector H k Connect and input to the classifier SoftMax to determine whether the input information is sarcastic information;

[0014] The beneficial effects of the present invention are as follows:

[0015] The present invention constructs its own word embedding vector based on the Bert model, with the purpose of utilizing the language model function that only utilizes Bert. In order to better pay attention to the information fusion between different modalities, this method constructs a cross-modal attention feature vector, and uses a multimodal knowledge graph to obtain external information and screen the external information. Cross-modal attention constructs the relationship between audio information, text information and sentiment value vector. The attention vector is calculated by the connection and influence between the audio information vector and the sentiment value vector, so as to focus on the prominent emotional expression in the sentence. The screening of external knowledge information is carried out by using the similarity between the information extracted from the text and the external information retrieved by the knowledge graph, and the external information that is inconsistent with the text information is retained as much as possible to capture the inconsistency between the modalities. Finally, the feature vectors of various types obtained are input into the classification layer for classification. The experimental results show that the present invention can well solve the problem of fusion of different modal information, can pay attention to the contradictions in the sentence, and can also extract the inconsistency features of different modalities, and can improve the accuracy of the irony detection task. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the overall implementation scheme of the present invention

[0017] Figure 2 The overall structure diagram of the model of the present invention

[0018] Figure 3 It is the sentence embedding graph constructed by BERT

[0019] Figure 4 This is a sample graph of a multimodal knowledge graph. DETAILED DESCRIPTION

[0020] The present invention will be further described below in conjunction with the accompanying drawings.

[0021] The present invention constructs an attention vector that can focus on the contradictory expressions within a sentence by modeling the relationship between audio features, emotional features, and text features. At the same time, the external knowledge graph is used to query the external knowledge information related to the video frame information, and the similarity between it and the text sentence information is calculated to retain the conflicting external information and further capture the conflicting information of different modalities, thereby effectively improving the accuracy of irony detection.

[0022] Reference Figure 1 is a flow chart of the overall implementation scheme of the present invention, a multimodal irony detection method based on knowledge graph and cross-modal attention, comprising the following steps:

[0023] Step (1) constructs the initial word embedding vector Token Embedding and the position encoding vector Position Embedding according to the input text information T, and uses the word embedding vector and the position vector as the input of the Bert pre-training model. After training, the sentence vector H is obtained. t ;

[0024] Step (2) segment the input audio information V according to the text information T, and encode each segmented audio information using OpenSmile to obtain the audio information feature vector H v ;

[0025] Step (3) Encode the input video frame information G using ResNet and perform dimension transformation through the fully connected layer to obtain the video frame feature vector H g ;

[0026] Step (4) query the sentiment value of each word in the input text information T according to the external sentiment dictionary, and construct the sentiment value feature vector H s ;

[0027] Step (5) constructs the cross-modal attention according to the three vectors obtained in steps (1), (2), and (4) to obtain the cross-modal attention vector H att ;

[0028] Step (6) Input the video frame information G into the multimodal knowledge graph, obtain external knowledge information related to the video frame information G, construct and filter the external knowledge vector, and obtain the external knowledge vector H related to the context. k ;

[0029] Step (7) connects the cross-modal attention vector, the video frame information vector and the relevant external knowledge vector and inputs them into the classifier SoftMax to predict whether the input information is sarcastic information. At the same time, the loss is calculated, and the trained model is retained after reaching the predetermined number of iterations. Finally, the stable model is used for the irony detection task.

[0030] Furthermore, the specific implementation process of step (1) is as follows:

[0031] 1-1 Construct Token Embedding for the BERT pre-training model input. Since this method uses BERT only to train the language model and obtain sentence vectors, the input is constructed in the form of [CLS] + text information + [SEP]; given the input text information T = {t1, t2, …, t n}, where n represents the length of the text information, and constructs the initialization embedding vector Token Embedding:

[0032] X=([CLS],t1,t2,…,t n ,[SEP]) (1)

[0033] 1-2 Construct the Position Embedding input of the BERT pre-training model. In order to obtain the sequential features of each word in the sentence, unlike the transformer, it does not use the sine-cosine encoding method, but is randomly initialized and learned; its expression is:

[0034] PE=(E0,E1,…,E n ,…,E n+1 ) (2)

[0035] Among them, E0, E1, …, E n+1 They represent [CLS], t1, t2, …, t in formula (1) respectively. n and [SEP] position code; X corresponds to PE, E0 is the position code corresponding to [CLS]; t1, t2, …, t n The sum is E1,…,E n The corresponding position code; E n+1 is the position code corresponding to [SEP];

[0036] 1-3 Concatenate the constructed Token Embedding and Position Embedding and input them into the BERT model, and take the feature vector between [CLS] and [SEP] as the hidden feature vector of the text information T, denoted as H t ∈R d×n , the process is summarized as follows:

[0037] H t = Bert(X; PE) = {h1,h2,…h n} (3).

[0038] Furthermore, the specific implementation process of step (2) is as follows:

[0039] 2-1 In order to better extract audio and video information, the audio and video are aligned according to the text information to obtain the audio segment and video frame corresponding to the text; the audio information is represented by V = {v1, v2, …, v n}, the video frame information is represented by G = {g1,g2,…,g n};

[0040] 2-2 The audio information V is extracted through OpenSmile to obtain the audio information feature vector. The process is as follows:

[0041]

[0042] Furthermore, the specific implementation process of step (3) is as follows:

[0043] 3-1 The aligned video frame information G is extracted through ResNet to obtain the feature vector of each frame information. The process is shown as follows:

[0044]

[0045] 3-2 H G The feature transformation is performed by the fully connected layer FC to obtain the video frame feature vector H g ∈R d×n :

[0046]

[0047] Furthermore, the specific implementation process of step (4) is as follows:

[0048] 4-1 For the input text information T = {t1, t2, ..., t n}, query the absolute value of the sentiment value of each word through the sentiment dictionary, record it as senti_vec and record the query result as: S = {s1, s2, ..., s n}.

[0049] 4-2 Use BERT to encode S and get the sentiment value feature vector H s ∈R d×n The expression is as follows:

[0050]

[0051] Furthermore, the specific implementation process of step (5) is as follows:

[0052] In order to integrate audio information and text information, a cross-modal attention vector is constructed, which is expressed as:

[0053]

[0054] Where W K , W Q , W V is the parameter that needs to be learned by the attention mechanism, d is the dimension of the word embedding vector Token Embedding, H s , H v With H t are the eigenvectors calculated above respectively;

[0055] Furthermore, the specific implementation process of step (6) is as follows:

[0056] 6-1 Input the video frame information into the external multimodal knowledge graph MMKG, and obtain multiple external knowledge after retrieval, that is, multiple triples {e1, e2, …, e n}, where e i Represents the i-th triple and uses BERT to encode it. The process is expressed as follows:

[0057]

[0058] in represents the encoding of the i-th triplet, and is the entity in the i-th triple, For the relationship between two entities;

[0059] 6-2 In order to improve the utilization rate of external knowledge information, it is necessary to screen external knowledge vectors. First, the current mainstream triple extraction model is used to extract triples from the text information T, and the triples of the text information are {e1, e2, …, e k}; After BERT encoding, the triple vector of text information is obtained, and its expression is as follows:

[0060]

[0061] in represents the encoding of the i-th triple, and is the entity in the i-th triple, For the relationship between two entities;

[0062] 6-3 Calculation The triple vector in The similarity of the triple vectors in is used to filter some external knowledge vectors. The process is as follows:

[0063]

[0064] in and Represents the triple vector and Cos_Sim ij is the cosine similarity value between them; as long as the similarity is within the specified threshold, the corresponding external knowledge vector will be retained, and the filtered knowledge vector will be recorded as H k ={[e h1 ,r1,e t1 ]1,…,[e hl ,r l ,e tl ] l}.

[0065] Furthermore, the specific implementation process of step (7) is as follows:

[0066] 7-1 The three types of information vectors obtained are concatenated and classified using the Softmax classifier. The process is as follows:

[0067]

[0068] The weight matrix W∈R 2×d , b is the bias term, d is the embedding dimension, It is the result of classification;

[0069] During 7-2 training, the standard gradient descent algorithm is used to update the parameters of the classifier Softmax and bert models, and the cross entropy loss function and L2 regularization are used. The calculation formula is as follows:

[0070]

[0071] in, is the prediction result of the i-th sample, y i is the true label value of sample i; N is the size of training data; R is L2 regularization, and λ is the weight coefficient of R.

[0072] Comparison between the present invention and the prior art:

[0073] Table 1 shows the results of irony detection related experiments in recent years and the experiments of this laboratory. The first column Modality in the table indicates the modality of the data, Method is the corresponding method model, and the other columns are evaluation indicators. Compared with the laboratory results of the single modality, the effect of the present invention is more significant. Compared with the text and visual modality method Bert+ResNet, it can be seen that the cross-modal attention of the present invention significantly improves the accuracy of the task. Judging from the performance of the baseline model in the trimodal data, the present invention uses an external knowledge graph to further expand the originally limited information, and well captures the inconsistency between the input information and the background information, thereby improving the detection effect of the task.

[0074] Table 1

[0075]

Claims

1. A multimodal irony detection method based on knowledge graph and cross-modal attention, characterized by The steps include: Step (1) constructs the initial word embedding vector Token Embedding and the position encoding vector Position Embedding according to the input text information T, and uses the word embedding vector and the position vector as the input of the Bert pre-training model. After training, the sentence vector H is obtained. t ; Step (2) segment the input audio information V according to the text information T, and encode each segmented audio information using OpenSmile to obtain the audio information feature vector H v ; Step (3) Encode the input video frame information G using ResNet and perform dimension transformation through the fully connected layer to obtain the video frame feature vector H g ; Step (4) query the sentiment value of each word in the input text information T according to the external sentiment dictionary, and construct the sentiment value feature vector H s ; Step (5) constructs the cross-modal attention according to the three vectors obtained in steps (1), (2), and (4) to obtain the cross-modal attention vector H att ; Step (6) Input the video frame information G into the multimodal knowledge graph, obtain external knowledge information related to the video frame information G, construct and filter the external knowledge vector, and obtain the external knowledge vector H related to the context. k ; Step (7) converts the cross-modal attention vector H att , video frame feature vector H g The external knowledge vector H k Connect and input to the classifier SoftMax to determine whether the input information is sarcastic information; The specific implementation process of step (5) is as follows: In order to integrate audio information with text information and sentiment value feature vectors, a cross-modal attention vector is constructed, which is expressed as: Where W K , W Q , W V is the parameter that needs to be learned by the attention mechanism, and d is the dimension of the word embedding vector Token Embedding; The specific implementation process of step (6) is as follows: 6-1 Input the video frame information into the external multimodal knowledge graph MMKG, and obtain multiple external knowledge after retrieval, that is, multiple triples {e1, e2, …, e n }, where e i Represents the i-th triple and uses BERT to encode it. The process is expressed as follows: in represents the encoding of the i-th triplet, and is the entity in the i-th triple, For the relationship between two entities; 6-2 Filter the external knowledge vectors; First, use the current mainstream triple extraction model to extract triples from the text information T, and get the triples of the text information as {e1, e2, …, e k }; After BERT encoding, the triple vector of text information is obtained, and its expression is as follows: in represents the encoding of the i-th triplet, and is the entity in the i-th triple, For the relationship between two entities; 6-3 Calculation The triple vector in The similarity of the triple vectors in is used to filter some external knowledge vectors. The process is as follows: in and Represents the triple vector and Cos_Sim ij is the cosine similarity value between them; as long as the similarity is within the specified threshold, the corresponding external knowledge vector will be retained, and the filtered knowledge vector will be recorded as H k ={[e h1 ,r1,e t1 ]1,…,[e hl ,r l ,e tl ] l }.

2. According to the multimodal irony detection method based on knowledge graph and cross-modal attention according to claim 1, the specific implementation process of step (1) is as follows: 1-1 Construct the Token Embedding of the BERT pre-training model input, so the input is constructed in the form of [CLS] + text information + [SEP]; given the input text information T = {t1, t2, …, t n }, where n represents the length of the text information, and constructs the initial embedding vector TokenEmbedding: X=([CLS],t1,t2,…,t n ,[SEP])(1) 1-2 Build the Position Embedding input of the BERT pre-training model, and randomly initialize the learning to obtain the sequential features of each word in the sentence. The expression is: OR=(E0,E1,…,E n ,…,BY n+1 ) (2) in, E0,E1,…,E n+1 They represent [CLS], t1, t2, …, t in formula (1) respectively. n and [SEP] position code; X corresponds to PE, E0 is the position code corresponding to [CLS]; t1, t2, …, t n The sum is E1,…,E n The corresponding position code; E n+1 is the position code corresponding to [SEP]; 1-3 Concatenate the constructed Token Embedding and Position Embedding and input them into the BERT model, and take the feature vector between [CLS] and [SEP] as the hidden feature vector of the text information T, denoted as H t ∈R d×n , the process is summarized as follows: H t =Bert(X;PE)={h1,h2,…h n } (3)。 3. According to the multimodal irony detection method based on knowledge graph and cross-modal attention according to claim 2, the specific implementation process of step (2) is as follows: 2-1 Align the audio and video according to the text information to obtain the audio segment and video frame corresponding to the text; the audio information is represented by V = {v1, v2, …, v n }, the video frame information is represented by G = {g1,g2,…,g n }; 2-2 The audio information V is extracted through OpenSmile to obtain the audio information feature vector. The process is as follows:

4. According to the multimodal irony detection method based on knowledge graph and cross-modal attention according to claim 3, the specific implementation process of step (3) is as follows: 3-1 The aligned video frame information G is extracted through ResNet to obtain the feature vector of each frame information. The process is shown as follows: 3-2 H G The feature transformation is performed by the fully connected layer FC to obtain the video frame feature vector H g ∈R d×n :

5. According to the multimodal irony detection method based on knowledge graph and cross-modal attention according to claim 4, the specific implementation process of step (4) is as follows: 4-1 For the input text information T = {t1, t2, ..., t n }, query the absolute value of the sentiment value of each word through the sentiment dictionary, record it as senti_vec and record the query result as: S = {s1, s2, ..., s n }; 4-2 Use BERT to encode S and get the sentiment value feature vector H s ∈R d×n The expression is as follows:

6. According to the multimodal irony detection method based on knowledge graph and cross-modal attention according to claim 1, the specific implementation process of step (7) is as follows: 7-1 The three types of information vectors obtained are concatenated and classified using the Softmax classifier. The process is as follows: The weight matrix W∈R 2×d , b is the bias term, d is the embedding dimension, It is the result of classification; During 7-2 training, the standard gradient descent algorithm is used to update the parameters of the classifier Softmax and bert models, and the cross entropy loss function and L2 regularization are used. The calculation formula is as follows: in, is the prediction result of the i-th sample, y i is the true label value of sample i; N is the size of training data; R is L2 regularization, and λ is the weight coefficient of R.

Images