A dialogue satire and emotion joint analysis method based on multi-task learning

Abstract

The application provides a dialogue satire and emotion joint analysis method based on multi-task learning, and the steps are as follows: a multi-modal dialogue emotion corpus is collected, text labels and multi-modal data of each dialogue are preprocessed, and features of text, images and audio are extracted respectively; the obtained features are fused by adopting a multi-head attention mechanism; the fused features are input into an interactive learning network, a multi-task learning paradigm based on soft parameter sharing is adopted to learn a satire task and an emotion task to obtain underlying representations of the satire task and the emotion task respectively; the satire recognition is taken as a main task, the emotion analysis is taken as a secondary task, a multi-task interaction method is used, emotion analysis knowledge is integrated into the satire detection task, and results of satire classification and emotion classification are obtained. The application can effectively capture global information of different modes, can simultaneously learn correlations and differences between different tasks by using a multi-task learning framework, and can improve the accuracy of satire and emotion task classification through interaction between the multi-task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of dialogue sentiment classification, and particularly relates to a dialogue sarcasm and sentiment joint analysis method based on multi-task learning. BACKGROUND

[0002] With the wide application of social networks, more and more people communicate with each other in the form of multi-modal dialogue records. In such a dialogue, people often express their emotions and sarcasm attitudes in various ways, such as language accompanied by facial expressions, gestures mixed with tone of voice, etc. Sarcasm is a metaphorical language to express human inner emotions, and the literal emotion of the speech and the real emotion expressed are often opposite. Given that sarcasm and emotion are interrelated, multi-modal sarcasm and emotion joint analysis in dialogue has always been a core research topic in artificial intelligence application. With the help of sarcasm analysis, we can more easily grasp the emotional state of the conversation party, which can help us adopt appropriate rhetoric, action, expression of appeasement and understanding, etc. in time. At the same time, sarcasm and emotion joint analysis can also be applied to social fields such as public opinion analysis, speech direction prediction, and communication theme tracking. In addition, the research of intelligent dialogue emotion recognition can also help to improve people's living standards, improve the ability of public political decision-making participation, and meet the growing pursuit of human life. The present application mainly studies the dialogue emotion analysis on social platforms, i.e. multi-modal dialogue sarcasm and emotion joint analysis based on multi-task learning.

[0003] Multi-modal dialogue emotion analysis based on multi-task learning mainly studies the trend of the speaker's emotional change in the dialogue process, and judges the real emotion expressed by the speaker with the help of different modalities (text, image, audio) and related task information. The main challenges it faces are: (1) how to utilize the complementary nature of cross-modal information to obtain effective multi-modal representation; (2) how to utilize the knowledge of related tasks to improve the classification results of the target emotion task.

[0004] At present, multi-modal emotion analysis technology based on multi-task learning has achieved many outstanding achievements and applications in the academic and industrial fields. They mainly focus on the fine classification of multiple emotion tasks, and utilize shared bottom-level representations to construct a deep learning architecture for multi-task emotion recognition. However, they have not systematically considered the uncertainty of emotion expression due to frequent interaction in dialogue, and have also less considered the differences between different emotion tasks. Therefore, there is still a lot of research space in the field of multi-modal dialogue emotion analysis based on multi-task learning. SUMMARY

[0005] In view of the technical problem that the frequent interaction in the existing multi-modal sentiment analysis does not consider the uncertainty of the expression of sentiment, the application provides a dialogue satire and sentiment joint analysis method based on multi-task learning, which comprehensively considers the feature information of multi-modal sentiment dialogue, and realizes more accurate classification of different emotional tasks (satire and sentiment).

[0006] In order to achieve the above-mentioned purpose, the technical scheme of the application is implemented as follows: a dialogue satire and sentiment joint analysis method based on multi-task learning, the steps of which are as follows:

[0007] Step one: collect a multi-modal dialogue sentiment corpus, the multi-modal dialogue sentiment corpus including N dialogue samples, each dialogue sample recording the text speech, expression image and sound of each speaker in sequence;

[0008] Step two: pre-process the text label of the multi-modal dialogue sentiment corpus, and randomly divide the processed multi-modal dialogue sentiment corpus into a training set and a test set;

[0009] Step three: pre-process the multi-modal data of each dialogue in the multi-modal dialogue sentiment corpus, and extract features of the text, image and audio three modes respectively;

[0010] Step four: adopt a multi-head attention mechanism method to fuse the text features, image features and audio features obtained in step three;

[0011] Step five: input the multi-modal fusion features into an interactive learning network composed of a double-gated network and multiple fully connected layers, learn the satire task and the sentiment task by adopting a multi-task learning paradigm based on soft parameter sharing, and obtain the bottom layer representation of the satire detection task and the sentiment analysis task respectively;

[0012] Step six: take satire recognition as the main task and sentiment analysis as the secondary task, use a multi-task interaction method, and integrate sentiment analysis knowledge into the satire detection task to obtain the results of satire classification and sentiment classification.

[0013] Preferably, the method for pre-processing the text label annotation of the multi-modal dialogue sentiment corpus is as follows:

[0014] Firstly, check whether the text statement and the corresponding video and audio are consistent, correct the text spelling errors and illegal characters, and check the context of the dialogue corresponding to the text and video;

[0015] Secondly, check the classification labels of the sentiment and satire of each standard dialogue, the sentiment label is annotated as positive, negative or neutral, and the satire label is summarized as satire and non-satire;

[0016] The number of samples in the training set: the number of samples in the test set is 7:3.

[0017] Preferably, the feature extraction method is:

[0018] For text features, input the text statements of each dialogue into the BERT pre-training model to obtain the word vector representation of each text statement, which is used to represent the text features H t of each statement.

[0019] For image features, the facial expression of the speaker at the end of the speech is extracted from the dialogue video as the image data for processing, and then the image is adjusted to the size of 224*224 pixels. The image pre-training model ResNet-152 is used to obtain the image features H v of 2048 dimensions.

[0020] For audio features, the audio is extracted from the dialogue video and saved in.wav format. The VGGish model pre-trained on YouTube AudioSet data is used for feature extraction. The.wav format audio file is resampled to 16kHz single-channel audio, and then the single-channel audio is subjected to short-time Fourier transform to obtain a spectrum graph. The spectrum graph is mapped and calculated to obtain a stable mel spectrum, which is framed with a fixed time length to obtain a 128-dimensional feature vector H a with audio semantics.

[0021] Preferably, the feature fusion method in step four is:

[0022] (1) Set a multi-head attention framework for text-image, take the text features H t as the query vector Query, and the image features H v as the key Key and value Value respectively, and use the multi-head attention mechanism method as follows:

[0023] Q ti = W ti H t

[0024] K vi = W vi H v

[0025] V vi = W si H v

[0026]

[0027] MHA(H t ,H v ) = Wm [Att(H t ,H v )1,...,Att(H t ,H v ) m ] T

[0028] where W m , W ti , W vi , W si denote parameter matrices, m denotes the number of multi-head attention mechanisms, Att(H t ,H v ) i is the attention score of the text feature H t and the image feature H v , softmax is a normalized exponential function, Q ti is a query vector, is a key-value vector, d k is a scaling factor; MHA(H t ,H v ) denotes a multi-head attention vector, which is composed of m attention Att(H t ,H v ) i vectors;

[0029] Then two residual connections and normalization layers and a feed-forward neural network sublayer are added after each multi-head attention mechanism in each layer to obtain the output of a single attention mechanism layer, and then M attention mechanism layers are stacked to obtain the fusion feature vector H (t,v) of the text and image, that is:

[0030] Z = LN(H t + MHA(H t ,H v )

[0031] O(t,v) = LN(H t + FFN(Z));

[0032] H (t,v) = O M (t,v)

[0033] where Z denotes the output of the first normalization layer and residual connection, O(t,v) denotes the output of a single text-visual attention mechanism layer; LN denotes the normalization layer operation, FFN(Z) denotes the feed-forward neural network sublayer operation, and O M (t,v) is the output obtained by splicing the M output O(t,v) vectors;

[0034] (2) Set the text-audio multi-head attention framework, and obtain the text feature H t as the query vector Query, the audio feature H a as the key Key and the value Value respectively, connect two residual connection and normalization layers and a feed-forward network sublayer after the multi-head attention mechanism to obtain the output of a single attention mechanism layer, and stack M attention mechanism sublayers to obtain the fusion feature vector H (t,a) of the text and audio.

[0035]

[0036] MHA(H t ,H a )=W m [Att(H t ,H a )1,…,Att(H t ,H a ) m ] T

[0037] Z=LN(H t +MHA(H t ,H a ));

[0038] O(t,a)=LN(H t +FFN(Z))

[0039] H (t,a) =O M (t,a)

[0040] wherein Q ti represents the query vector, represents the key vector, and V ai represents the value vector; W m represents the parameter matrix, and m represents the number of multi-head attention mechanisms; Z represents the output of the first layer normalization layer and residual connection, LN represents the normalization layer operation, FFN(Z) represents the feed-forward neural network sublayer operation, O(t,a) represents the output of a single text-audio attention mechanism sublayer, and O M (t,a) is a vector obtained by splicing M output O(t,a) vectors.

[0041] (3) The fusion feature vector H (t,v) of the text and image and the fusion feature vector H a of the text and audio are spliced to obtain the feature vector H c , and the feature vector H cAs the query vector (Query), key (Key), and value (Value), these are fused using a multi-head attention mechanism. A feedforward neural network and a two-layer canonical structure with residual connections are used to obtain the output of a single attention layer. N identical attention layers are stacked to finally obtain the multimodal fusion feature H. f The calculation method is as follows:

[0042]

[0043] MHA(H c ) = W n [Att(H c )1,…,Att(H c ) n ] T ;

[0044] O C =LN(H c +FNN(LN(H c +MHA(H c ))))

[0045] H f =O N (c)

[0046] Among them, W n Let n represent the parameter matrix, where n represents the number of multi-head self-attention mechanisms, and each self-attention mechanism is represented by a feature vector H. c Attention scores are calculated using the query, key, and value respectively. C Q represents the output of a single-layer multi-head self-attention mechanism; ci Represents the query vector. V represents the key vector. ci Represents a value vector, Att(H) c ) i This represents the output of the i-th head attention mechanism, used to calculate the feature vector H. c Attention score for each head, i = 1, ..., n, O N (c) From N types of output O C It is formed by concatenating vectors, where N represents the number of multi-head self-attention mechanism layers.

[0047] Preferably, the implementation method of the multi-task learning paradigm based on soft parameter sharing is as follows: a gating network is set up for the irony task and the emotion task respectively. All target tasks share the output results of l fully connected layers. The two gating networks are used to learn and select the signal weights of the output of each fully connected layer. The scores output by the gating networks are weighted and combined to obtain the underlying input representations of different tasks. The calculation method is as follows:

[0048]

[0049] F i (H f )=W i H f +b i

[0050]

[0051] where k denotes the sentiment analysis or sarcasm recognition task, i.e., k∈(sen, sar), where sen denotes the sentiment analysis task and sar denotes the sarcasm detection task, W i denote different parameter matrices, b i is a weight bias, F i (H f ), i = 1,..., l denotes the output of the ith fully connected layer, and l denotes the number of fully connected layers; G k (H f ) is used to assign weight scores to the outputs of the l fully connected layers,

[0052] Preferably, the multi-task interaction method is: using two gated recurrent unit combinations to learn the interaction of sarcasm recognition and sentiment analysis tasks, taking the sarcasm task as the main task and the sentiment analysis as the secondary task, obtaining the underlying input representation of the sentiment analysis task from step five As the input of the gated recurrent unit GRU sen of the sentiment task, the hidden vector containing the knowledge of the sentiment analysis task is obtained and the underlying input representation As the input of the gated recurrent unit GRU sar of the sarcasm task, the hidden vector is obtained Finally, the hidden vector and the hidden vector are input into the soft max regression function to calculate the classification probability respectively, and the results of sarcasm and sentiment classification are obtained.

[0053]

[0054]

[0055]

[0056]

[0057]

[0058] wherein, represents the output result of the sarcasm detection task, represents the output result of the sentiment analysis task, and GRU represents a gated recurrent unit, which is a recurrent neural network used to obtain hidden vectors in the sarcasm recognition and sentiment analysis tasks.

[0059] The present application has the beneficial effects that: on the basis of the existing multi-modal sentiment dialogue corpus, the feature information of text, image and audio is extracted respectively, the multi-head attention mechanism is used to effectively fuse the feature information of different modalities, the multi-task paradigm based on soft shared parameters is used to learn the differences and similarities between multi-tasks, and the multi-task interaction network is used to finally obtain more accurate classification results of different emotional tasks (sarcasm and sentiment) by integrating the knowledge of related tasks.

[0060] The present application proposes a multi-modal feature fusion method based on a multi-head attention mechanism, which can effectively capture global information of different modalities; the present application designs a novel multi-task learning framework that can simultaneously learn the correlation and difference between different tasks, and through the interaction between multi-tasks, improves the accuracy of sarcasm and sentiment task classification, and also provides a new idea for artificial intelligence systems and multi-modal dialogue sentiment analysis. BRIEF DESCRIPTION OF DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0062] Figure 1 The flowchart of the present application.

[0063] Figure 2 The structure diagram of the multi-modal feature fusion method of the present application.

[0064] Figure 3 The structure diagram of the multi-task learning framework.

[0065] Figure 4 The experimental comparison result diagram of the present application and other methods. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort fall within the protection scope of the present application.

[0067] As shown in the figure, a dialog sarcasm and emotion joint analysis method based on multi-task learning is as follows: Figure 1

[0068] Step one: collect a multi-modal dialog emotion corpus, the multi-modal dialog emotion corpus includes N dialog samples, and each dialog sample records the text speech, expression image and sound of each speaker in sequence.

[0069] In the disclosed corpus, multi-modal dialog emotion data is sorted and collected, wherein the collected dialog requirements are labeled with emotion and sarcasm labels, and the text label annotation of the corpus is preprocessed as follows:

[0070] Firstly, check whether the text statement and the corresponding video and audio are consistent, correct the text spelling errors and illegal characters, and check the context of the dialog corresponding to the text and video, to obtain N=690 standard dialogs.

[0071] Secondly, use Python design code to check the emotion and sarcasm classification labels of each standard dialog statement, and the emotion label only requires positive, negative or neutral, and the sarcasm label only summarizes two labels of sarcasm and non-sarcasm. If there are other labels, the dialog is deleted.

[0072] Step two: randomly select 70%*690 dialogs in the multi-modal dialog emotion corpus as a training set, and the remaining 30%*690 dialogs as a test set. The training set will be used for subsequent model method training, which is convenient for adjusting model parameters, and the test set is used for testing the performance of the test model. The purpose is to train a model method with the best performance on the training set, and apply it to the test set to calculate its performance.

[0073] Step three: pre-process the multi-modal data of each dialog in the multi-modal dialog emotion corpus, and extract features of the text, image and audio three modalities.

[0074] The data preprocessing method is as follows: the text, image (facial expression when speaking) and audio content of each speaker in each dialog are checked manually to see whether they are consistent and aligned, and some inconsistent dialog data is directly discarded.

[0075] The specific feature extraction method is as follows:​

[0076] First, for text features, the text sentences of each dialogue are input into the BERT pre-training model to obtain a 768-dimensional vector of the utterance, which represents the semantic understanding of each sentence. The BERT pre-training model obtains a word vector representation of each text sentence, which represents the text features H t .

[0077] Second, for image features, the facial expression of the speaker at the end of the utterance is extracted from the dialogue video as the image data for processing, and then the image is resized to 224*224 pixels. A large image pre-training model ResNet-152 is used to obtain a 2048-dimensional image feature H v .

[0078] Third, for audio features, first, the ffmpeg and MoviePy third-party libraries in python software are used to extract the audio from the dialogue video and save it in.wav format. The VGGish model pre-trained on YouTube's AudioSet data is used for feature extraction. The.wav format audio file is input, the audio is resampled to 16kHz single-channel audio, and the short-time Fourier transform is performed to obtain the frequency spectrum. The stable mel spectrum is obtained by mapping and calculating the frequency spectrum. The mel spectrum mainly uses the mel scale filter bank to map the original spectrum to obtain appropriate size audio features. These features are then framed with fixed time length, and finally a 128-dimensional high-dimensional feature vector H a .

[0079] Step four: the text features H t , image features H v and audio features H a are fused using the multi-head attention mechanism method.

[0080] As shown in Figure 2 , a text-centered multi-modal fusion subnetwork is designed, with text as the main branch (target) and image and speech as supplementary channels. The specific steps are as follows:

[0081] First step: set the text-image multi-head attention framework, with text as the center, and take the text features H t as the query vector Query, and the image features H v as the key Key and value Value, respectively. The multi-head attention mechanism method is as follows:

[0082] Q ti = Wti H t

[0083] K vi =W vi H v

[0084] V vi =W si H v

[0085]

[0086] MHA(H t H v ) = W m [Att(H t H v )1,…,Att(H t H v ) m ] T

[0087] Among them, W m W ti W vi W si The parameter matrix is ​​initially generated randomly in the model code and automatically adjusted to appropriate values ​​during training. m represents the number of multi-head attention mechanisms, which is set to 8 in this invention. Att(H t H v ) i It is to calculate text features H t and image features H v The attention score, softmax is the normalization exponential function, which normalizes the output value, Q. ti It is a query vector. It is a key-value vector, d k It is the scaling factor, representing The length of the vector dimension. MHA(H) t H v ) represents a multi-head attention vector, consisting of 8 attention types Att(H) t H v ) i It is formed by concatenating vectors.

[0088] Then, two residual connections, a normalization layer (LN), and a feedforward neural network (FNN) sublayer are added after each multi-head attention mechanism to obtain the output of a single attention mechanism layer. Finally, M attention mechanism layers are stacked to obtain the final fused feature vector H of the text image. (t,v) ,Right now:

[0089] Z = LN(H)t + MHA(H t , H v )

[0090] O(t,v)=LN(H t + MHA(H (t,v) , H M ))

[0091] H M (t,v)=O(t,v)

[0092] where Z denotes the output of the first layer of normalization layer and residual connection, O(t,v) denotes the output of a single text-visual attention mechanism layer. LN denotes the normalization layer operation, which normalizes the hidden layer in the neural network to the standard normal distribution, ensures the stability of the data feature distribution, and accelerates the convergence speed of the model. FFN(Z) denotes the feedforward neural network sublayer, which maps the data to a high latitude space and then maps it to a low latitude space, extracting deeper features. O t (t,v) is the output obtained by splicing M output O(t,v) vectors, and in the present application, M is set to 6.

[0093] Second step: similar to the text-visual attention mechanism, set the multi-head attention framework of text-audio, take the text feature H a as the query vector Query, and the audio feature H (t,a) as the key Key and value Value respectively, connect two residual connections and normalization layers and a feedforward network sublayer after the multi-head attention mechanism to obtain the output of a single attention mechanism layer, and finally stack M attention mechanism sublayers to obtain the fusion feature vector H t of text-audio, and the specific calculation method is:

[0094]

[0095] MHA(H a ,H m )=W t [Att(H a ,H t )1,…,Att(H a ,H m ) T

[0096] Z=LN(H t + MHA(H t ,H a ))

[0097] O(t,a)=LN(H t + FFN(Z))

[0098] H (t,a) =O M (t,a)

[0099] wherein Q ti represents a query vector, represents a key vector, V ai represents a value vector. W m represents a parameter matrix, the initial value is randomly generated by writing model code, and is adjusted to a proper value in the training process, m represents the number of multi-head attention mechanisms, which is set to 8 in the present application. Z represents the output of the first layer normalization layer and residual connection, LN represents the normalization layer operation, which is to normalize the hidden layer in the neural network to a standard normal distribution, to ensure the stability of the data feature distribution and accelerate the convergence speed of the model. FFN(Z) represents a feedforward neural network sublayer, which maps the data to a high latitude space and then maps it to a low latitude space, to extract deeper features. O(t,a) represents the output of a single text-audio attention mechanism sublayer, O M (t,a) is a vector obtained by splicing M output O(t,a) vectors, which is set to 6 in the present application.

[0100] Third step: splice the fusion feature vector H (t,v) of the text image and the fusion feature vector H a of the text-audio to obtain a feature vector H c , and take itself as a query vector Query, a key Key and a value Value, and fuse them through a multi-head attention mechanism method, to obtain the output result of a single attention mechanism layer by using a feedforward neural network and two layers of specifications with residual connection, and finally obtain a multi-modal fusion feature H f by stacking N same attention mechanism layers, and the calculation method is as follows:

[0101]

[0102] MHA(H c )=W n [Att(H c )1,…,Att(H c ) n ] T

[0103] O C =LN(H c +FNN(LN(H c +MHA(H c )))

[0104] H f =O N (c)

[0105] wherein Wn denotes the parameter matrix, the initial value is randomly generated in the model code, and is adjusted to the appropriate value during the training process, n denotes the number of multi-head self-attention mechanisms, each head of the self-attention mechanism takes the feature vector H c as Query, Key and Value respectively to calculate the attention score, and the application is set to 8. O C denotes the output of the single-layer multi-head self-attention mechanism. Q ci denotes the query vector, denotes the key vector, V ci denotes the value vector, Att(H c ) i denotes the output result of the i-th head of the attention mechanism, which is used to calculate the feature vector H c the attention score of each head, i = 1,..., n, O N (c) is spliced by N output vectors O C , N denotes the number of multi-head self-attention mechanism layers, and in the application, N is set to 6.

[0106] Step five: inputting the multi-modal fusion feature H f into an interactive learning network composed of a double-gating network and a plurality of fully connected layers, learning the satire task and the emotion task by adopting a multi-task learning paradigm based on soft parameter sharing, and respectively obtaining the underlying representation of the satire detection task and the emotion analysis task.

[0107] Step five: inputting the multi-modal fusion feature H f into a multi-task learning framework to learn the correlation and difference between the satire and emotion tasks. As Figure 3 shown, a multi-task learning paradigm based on soft parameter sharing is introduced in the satire detection and emotion analysis, the interactive learning network is a network set based on the soft parameter sharing paradigm, and an interactive learning network composed of a hybrid network and a plurality of fully connected layers (Fully Connected Layer, FCs) is developed. Specifically, a gating network is set for each target task, and all target tasks share the output results of n fully connected layers, wherein the gating networks Gate I and Gate II are used to learn the signal weight of each fully connected layer output. Each fully connected layer has its own learning direction, and finally the scores output by the gating network are weighted and combined to obtain the underlying input representation of different tasks, that is, The calculation method is:

[0108]

[0109] F i (H f)=W i H f +b i

[0110]

[0111] wherein k represents the sentiment analysis or sarcasm recognition task, i.e.k∈(sen,sar), wherein sen represents the sentiment analysis task, and sar represents the sarcasm detection task, W i represent different parameter matrices, b i is a weight offset, and the initial values of the parameter matrices and the weight offset are randomly generated by the model and are adjusted to appropriate values in the model training process. i (H f ), i=1,…,l, represents the output of the i-th full connection layer, and l represents the number of full connection layers, and in the present application, the value of l is set to 3. k (H f ) is used to assign a weight score to the output of the n full connection layers,

[0112] The present application adopts a multi-task learning paradigm based on soft parameter sharing, and constructs a learning network for the sarcasm detection and sentiment analysis tasks respectively, each task combines the outputs of different full connection layers through a gating network, and adjusts the weight score through model training to obtain a bottom layer input representation suitable for different tasks, which can simultaneously learn the correlation and difference between the sarcasm recognition and sentiment analysis tasks.

[0113] Step six: taking sarcasm recognition as the main task and sentiment analysis as the secondary task, using a multi-task interaction method, integrating sentiment analysis knowledge into the sarcasm detection task, first obtaining the hidden vector of the sentiment analysis task knowledge from the gating recurrent unit GRU sen containing the sentiment analysis task knowledge The hidden vector of the sentiment analysis task knowledge and the bottom layer input representation of the sentiment analysis task are input into the gating recurrent unit GRU sar of the sarcasm detection task, and the hidden vector is obtained.

[0114] Two gating recurrent units (Gated Recurrent Unit, GRU) are used to learn the interaction between sarcasm recognition and sentiment analysis tasks, and the classification effect of the main task is improved by learning the information of related tasks. Figure 3As shown, with satire as the primary task and sentiment analysis as the secondary task, the underlying input representation for the sentiment analysis task is first obtained from step five. GRU as a gated recurrent unit for emotional tasks sen The input is used to obtain the hidden vector. Incorporating information from emotional tasks, the hidden vectors will be... and Gated Loop Unit (GRU) as a satirical task sar The input is used to obtain the hidden vector. Finally, the hidden vector and hidden vectors The probabilities of each category are calculated by inputting them into a softmax regression function, yielding the results for sarcasm and sentiment classification. The specific method is as follows:

[0115]

[0116]

[0117]

[0118]

[0119] in, This represents the output of the satire detection task. This represents the output of a sentiment analysis task. GRU stands for Gated Recurrent Unit, a recurrent neural network used to obtain hidden vectors from different tasks.

[0120] The application improves the classification effect of the target task through the mode of multi-task interaction, and finally obtains the classification results of different tasks, compared with CNN, BiLSTM---from the literature [Zhou P, Shi W, Tian J, Qi Z, et al.Attention-based bidirectional long short-term memory networks for relation classification[C]. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 2016.], SVM+BERT---from the literature [Devlin J, Chang M.W, Lee K, and Toutanova K.Bert:Pre-training of deep bidirectional transformers for language understanding[C]. In NAACL-HLT 2019:Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2019, pp.4171-4186.], RCNN-RoBERTa---from the literature [Potamias R.A, Siolas G, and Stafylopatis A.G.A transformer-based approach to irony and sarcasm detection[J]. Neural Computing and Applications, 2020, pp.1-12.], EfficientNet---from the literature [Tan M and Le Q.V, Efficientnet:Rethinking model scaling for convolutional neural networks[C]. In International Conference on Machine Learning, 2019, pp.6105-6114.], UPB-MTL---from the literature [Vlad G.A, Zaharia G.E, Cercel D.C, Chiru C.G, and Matu S.T.Upb at semeval-2020task 8:Joint textual and visualmodeling in a multi-task learning architecture for memotion analysis[C].InProceedings of the Fourteenth Workshop on Semantic Evaluation,2020,pp.1208–1214.]、A-TML---from the literature[Chauhan D.S,Ekbal D.S R,A,and Bhattacharyya P,Sentiment and emotion help sarcasm?a multi-task learning framework for multi-modal sarcasm,sentiment and emotion analysis[C].In Proceedings of the 58thAnnual Meeting of the Association for Computational Linguistic,2020,pp.4351–4360.]Multiple advanced baseline results, such as the present invention can be very intuitive to observe the advancement, such as. Figure 4 As shown in the table, Figure 4 It can be seen that the classification probability of sarcasm classification and emotion analysis of the present application reaches 52.15 and 73.12 respectively, which is higher than that of other algorithms.

[0121] The application collects a multi-modal (image, text, sound) dialogue emotion corpus, and divides it into a training set and a test set; different pre-training models are used to extract features of three modal data in the corpus; a multi-head attention mechanism method is used to fuse three modal features, and then global information of different modalities is captured; the fused features are input into a multi-task learning framework to obtain better feature representation, wherein a fully connected layer neural network is used to perform different linear mapping on the input fused features through an internal hidden layer to output different dimensional feature representations, a double gating network (Gating Network) is used to calculate the weight score of each fully connected layer output, and then each task weights and sums the output of each fully connected layer according to the weight score to obtain the bottom representation; finally, two gated recurrent units (Gated Recurrent Unit, GRU) are combined to learn the interaction of sarcasm recognition and sentiment analysis tasks, the gated recurrent units GRU of two different tasks are integrated into the hidden information carried by the other task, and finally the sarcasm binary classification result (sarcasm and non-sarcasm) and the sentiment polarity three-classification result (positive, negative and neutral) are obtained through the Softmax regression function respectively. The application systematically considers the multi-modal feature fusion and multi-task learning in the dialogue process, proposes an advanced method, uses a multi-head attention mechanism to capture global information of different modalities, uses a multi-task learning framework to simultaneously learn the correlation and difference between different tasks, and through the interaction method between multiple tasks, the emotional state of the speaker in the dialogue process is calculated, the information interaction of multiple emotional tasks effectively improves the prediction accuracy of different tasks (sentiment analysis, sarcasm recognition), and provides a new idea for artificial intelligence systems and multi-modal dialogue emotion analysis.

[0122] The above only describes the preferred embodiments of the application and is not intended to limit the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A method for joint analysis of dialogue irony and sentiment based on multi-task learning, characterized in that, The steps are as follows: Step 1: Collect a multimodal dialogue sentiment corpus. The multimodal dialogue sentiment corpus includes N dialogue examples. Each dialogue example records the text, facial expressions, and voice of each speaker in chronological order. Step 2: Preprocess the text labels of the multimodal dialogue sentiment corpus, and randomly divide the preprocessed multimodal dialogue sentiment corpus into a training set and a test set; Step 3: Preprocess the multimodal data of each dialogue in the multimodal dialogue sentiment corpus, and extract features from the text, image and audio modalities respectively; Step 4: Use a multi-head attention mechanism to fuse the text features, image features, and audio features obtained in Step 3; The feature vector H is obtained by concatenating the fused feature vectors of the text image and the text audio. c The function takes itself as the query vector, key, and value, and fuses them through a multi-head attention mechanism. It then uses a feedforward neural network and a two-layer canonical structure with residual connections to obtain the output of a single attention mechanism layer. Stacking N identical attention mechanism layers finally yields the multimodal fusion feature H. f ; Step 5: Input the multimodal fusion features into an interactive learning network consisting of a dual-gated network and multiple fully connected layers. By adopting a multi-task learning paradigm based on soft parameter sharing, learn the irony task and sentiment task to obtain the underlying representations of the irony detection task and sentiment analysis task, respectively. Step Six: With irony recognition as the primary task and sentiment analysis as the secondary task, a multi-task interaction method is used to integrate sentiment analysis knowledge into the irony detection task, resulting in irony classification and sentiment classification. The implementation method of the multi-task learning paradigm based on soft parameter sharing is as follows: A gating network is set up for the irony task and the emotion task respectively. All target tasks share the output results of l fully connected layers. The two gating networks are used to learn and select the signal weights of the output of each fully connected layer. The scores output by the gating networks are weighted and combined to obtain the underlying input representations of different tasks. The calculation method is as follows: F i (H f )=W i H f +b i Among them, H f For multimodal fusion features, k represents the sentiment analysis or sarcasm detection task, i.e., k∈(sen,sar), where sen represents the sentiment analysis task and sar represents the sarcasm detection task. W i Representing different parameter matrices, b i It is the weight shift, F i (H f ), i = 1,...,l represents the output of the i-th fully connected layer, and l represents the number of fully connected layers; G k (H f ) is used to assign weight scores to the outputs of l fully connected layers. softmax is a normalized exponential function; The multi-task interaction method is as follows: Two gated recurrent units are combined to learn the interaction between the satire recognition and sentiment analysis tasks, with the satire task as the primary task and sentiment analysis as the secondary task. The underlying input representation of the sentiment analysis task is obtained from step five. GRU as a gated recurrent unit for emotional tasks sen The input yields a hidden vector containing knowledge of the sentiment analysis task. Incorporating information from emotional tasks, the hidden vectors will be... and underlying input representation Gated Loop Unit (GRU) as a satirical task sar The input is used to obtain the hidden vector. Finally, the hidden vector and hidden vectors The probabilities of the classifications are calculated by inputting them into the softmax regression function, yielding the results for sarcasm and sentiment classifications.

2. The method for joint analysis of dialogue irony and sentiment based on multi-task learning according to claim 1, characterized in that, The method for preprocessing the text tagging of the multimodal dialogue sentiment corpus is as follows: The first step is to check whether the text statements and the corresponding video and audio are consistent, correct spelling errors and illegal characters in the text, and check the text and video corresponding to the context of the dialogue; The second step is to check the emotional and ironic categories of the statements in each standard dialogue. The emotional tags are labeled as positive, negative, or neutral, and the ironic tags are categorized into two types: ironic and non-ironic. The ratio of the number of units in the training set to the number of units in the test set is 7:

3.

3. The method for joint analysis of dialogue irony and sentiment based on multi-task learning according to claim 1 or 2, characterized in that, The feature extraction method is as follows: For text features, the text sentences of each dialogue are input into the BERT pre-trained model to obtain the word vector representation of each text sentence, which is used to represent the text features H of each sentence. t ; For image features, the facial expression frame at the end of the speaker's speech in the dialogue video is extracted as the processed image data. The image is then resized to 224*224 pixels, and the 2048-dimensional image feature H is obtained using the ResNet-152 pre-trained image model. v ; For audio features, audio is extracted from the dialogue video and saved in .wav format. The .wav audio files are then used to extract features using a VGGish model pre-trained on YouTube's AudioSet data. The audio is resampled to 16kHz mono, and a short-time Fourier transform is performed on the mono audio to obtain a spectrogram. The spectrogram is then mapped and a stable mel spectrum is calculated. The mel spectrum is framed at fixed durations, ultimately yielding a 128-dimensional feature vector H with audio semantics. a .

4. The method for joint analysis of dialogue irony and sentiment based on multi-task learning according to claim 3, characterized in that, The feature fusion method in step four is as follows: (1) Set up a text-image multi-head attention framework, centered on the text, and integrate the text features H t As the query vector Query, image feature H v The multi-head attention mechanism is used as the key and value, respectively. Q ti =W ti H t K vi =W vi H v V vi =W si H v MHA(H t ,H v )=W m [That(H t ,H v )1,…,At(H t ,H v ) m ] T Among them, W m W ti W vi W si Let m represent the number of multi-head attention mechanisms, and Att(H) represent the parameter matrix. t H v ) i Text feature H t and image features H v The attention score, softmax is a normalized exponential function, Q ti It is a query vector. It is a key-value vector, d k It is a scaling factor; MHA(H) t H v ) represents a multi-head attention vector, consisting of m types of attention Att(H t H v ) i It is formed by concatenating vectors; Then, two residual connections, a normalization layer, and a feedforward neural network sublayer are added after the multi-head attention mechanism in each layer to obtain the output of a single attention mechanism layer. Finally, M attention mechanism layers are stacked to obtain the fused feature vector H of the text image. (t,v) ,Right now: Z=LN(H t +MHA(H t ,H v ) O(t,v)=LN(H t +FFN(Z); H (t,v) =O M (t,v) Where Z represents the output of the first normalization layer and residual connection, O(t,v) represents the output of a single text-visual attention mechanism layer; LN represents the normalization layer operation, FFN(Z) represents the feedforward neural network sub-layer operation, and O M (t,v) is the output formed by concatenating M output vectors O(t,v); (2) Set up a text-audio multi-head attention framework to integrate text features H t As a query vector Query, audio feature H a Using these as the key and value respectively, two residual connections and a normalization layer, along with a feedforward sublayer, are connected after the multi-head attention mechanism to obtain the output of a single attention mechanism layer. Stacking M attention mechanism sublayers yields the fused feature vector H of the text and audio. (t,a) The calculation method is as follows: Among them, Q ti Represents the query vector. V represents the key vector. ai W represents a value vector; m Let m represent the number of multi-head attention mechanisms; Z represent the output of the first normalization layer and residual connection; LN represent the normalization layer operation; FFN(Z) represent the feedforward neural network sub-layer operation; and O(t,a) represent the output of a single text-audio attention mechanism sub-layer. M (t,a) is a vector formed by concatenating M output vectors O(t,a); (3) The fusion feature vector H of the text image (t,v) The fusion feature vector H of text and audio a The feature vector H is obtained by concatenation. c and the feature vector H c As the query vector (Query), key (Key), and value (Value), these are fused using a multi-head attention mechanism. A feedforward neural network and a two-layer canonical structure with residual connections are used to obtain the output of a single attention mechanism layer. N identical attention mechanism layers are stacked to finally obtain the multimodal fusion feature H. f The calculation method is as follows: Among them, W n Let n represent the parameter matrix, where n represents the number of multi-head self-attention mechanisms, and each self-attention mechanism is represented by a feature vector H. c Attention scores are calculated using the query, key, and value respectively. C Q represents the output of a single-layer multi-head self-attention mechanism; ci Represents the query vector. V represents the key vector. ci Represents a value vector, Att(H) c ) i This represents the output of the i-th head attention mechanism, used to calculate the feature vector H. c Attention score for each head, i = 1, ..., n, O N (c) From N types of output O C It is formed by concatenating vectors, where N represents the number of multi-head self-attention mechanism layers.

5. The method for joint analysis of dialogue irony and sentiment based on multi-task learning according to claim 1, characterized in that, The method for obtaining the results of irony and sentiment classification is as follows: in, This represents the output of the satire detection task. This represents the output of the sentiment analysis task. GRU stands for Gated Recurrent Unit, a recurrent neural network used to acquire hidden vectors in the irony recognition and sentiment analysis tasks.

Images