A multi-label sentiment classification method and system based on label and text interaction

Abstract

The application discloses a multi-label sentiment classification method based on label and text interaction, comprising the following steps: obtaining a text to be analyzed, performing data cleaning processing on the obtained text to be analyzed, so as to obtain a preprocessed text with annotation marks, performing data format normalization processing on the preprocessed text with annotation marks, so as to obtain an id list and a mask template, inputting the id list and the mask template into a pre-trained multi-label sentiment classification model based on label and text interaction, and obtaining a sentiment label corresponding to the text. The application can solve the technical problems that the existing sentiment classification method based on deep learning ignores too much potential information in the specific field of sentiment classification, leading to insufficient deep sentence semantic learning, and that the coexistence and correlation of emotions are ignored due to independent classification processing of emotions, leading to low accuracy of the multi-label sentiment classification task.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of natural language processing, and more particularly relates to a multi-label emotion classification method and system based on label and text interaction. BACKGROUND

[0002] Emotion Classification refers to dividing texts into two or several types of praise or derogatory according to the meaning and emotion information expressed by the texts. Today is an information overload era, especially the rise of software platforms such as social, shopping and takeout, which leads to a large amount of text data. These texts contain a lot of information, including a variety of user emotion sets. Making full use of these information and performing emotion classification is very meaningful for both platform and user. For the platform, it can monitor user feedback to discover and solve user problems in time to promote the common progress of the platform and the user. For the user, on the one hand, it can avoid other users' negative incitement, and on the other hand, it can help users to reasonably restrain themselves on the network. Therefore, an accurate emotion classification method has great practical significance.

[0003] Since the features extracted by traditional machine learning methods are too shallow, the current mainstream emotion classification methods mainly tend to the field of deep learning. The first method is an encoder architecture based on transformer (a deep learning model using an encoder and decoder architecture based on self-attention mechanism). This method mainly uses the superior feature extraction ability of the transformer encoder to the recurrent neural network (RNN) and long short term memory (LSTM) network to achieve better results. In simple terms, the model is pre-trained using unsupervised data, and then used as an encoder to extract word features, followed by a classifier to fine-tune on the emotion classification task. The second method is a graph neural network-based method EmoGraph, which captures the dependency relationship between different emotions through a graph network to improve classification performance. The graph used is constructed by using the co-occurrence statistics between different emotion categories. The third method is a span prediction-based SpanEmo, which mainly converts the emotion classification problem into a span prediction problem and uses a pre-trained model based on transformer's bidirectional encoder representation (BERT) to achieve it.

[0004] However, the above methods all have some non-negligible defects: the method based on the transformer encoder architecture ignores too much potential information for this specific field of sentiment classification, thus leading to insufficient semantic learning of sentences; in addition, the method independently classifies emotions, ignoring the coexistence and correlation of emotions, thus leading to low accuracy of the multi-label sentiment classification task; the EmoGraph method mainly counts the co-occurrence data between class labels, but does not sufficiently mine the correlation between labels and words, thus leading to poor generalization of the sentiment classification model; as for the SpanEmo method, since there are multiple objects in a sentence that have an important influence on the label, the main part of SpanEmo is based on the BERT model, which encodes the label and the sentence together, so it cannot adaptively extract local discriminative features of different labels, thus failing to deeply learn the attention weights of each label and each part of the text to be analyzed. SUMMARY

[0005] In view of the above defects or improvement needs of the prior art, the present application provides a multi-label sentiment classification method based on label-text interaction. The purpose is to solve the technical problems that the existing deep learning-based sentiment classification method ignores too much potential information for this specific field of sentiment classification, leading to insufficient semantic learning of sentences; and the technical problems that the independent classification of emotions ignores the coexistence and correlation of emotions, leading to low accuracy of the multi-label sentiment classification task; and the technical problems that the correlation between labels and words is not sufficiently mined, leading to poor generalization of the sentiment classification model; and the technical problems that different labels cannot be adaptively extracted, leading to the inability to deeply learn the attention weights of each label and each part of the text to be analyzed.

[0006] To achieve the above purpose, according to one aspect of the present application, a multi-label sentiment classification method based on label-text interaction is provided, comprising the following steps:

[0007] (1) obtaining a text to be analyzed;

[0008] (2) performing data cleaning processing on the text to be analyzed obtained in step (1) to obtain a preprocessed text with annotation marks.

[0009] (3) performing data format normalization processing on the preprocessed text with annotation marks obtained in step (2) to obtain an id list and a mask template.

[0010] (4) inputting the id list and the mask template obtained in step (3) into a pre-trained multi-label sentiment classification model based on label-text interaction to obtain the sentiment label corresponding to the text.

[0011] Preferably, step (2) is to extract some special symbols in the text as words, to extract dates or users as unified representations, to extract elongated words as normal words with annotations, so as to obtain the preprocessed text with annotation marks by using ekphrasis.

[0012] Step (3) is specifically, first, construct a sentence pair, that is, first, with the preprocessed text with annotation marks obtained in step (2) as the source text, and the source text itself as the target text, to obtain the sentence pair. <cls>To start, all the emotion tags are concatenated into a single question ending in a question mark, which is the first sentence of the sentence pair, followed by <sep>which is followed by the preprocessed text with the annotation tags as the second sentence of the sentence pair, and finally appended with <sep>If the total length of the sentence is greater than 128, truncate to 128, and if less than 128, pad the end <pad>until the total length is 128; then a mask template is constructed, which has a length of 128, and is one-to-one corresponding to the words of the sentence pair constructed <pad>The position of 0, and the rest is 1; finally, the sentence pair is encoded, and the spliced final sentence pair is encoded into an id list using the XLNet wordpiece analyzer.

[0013] Preferably, the multi-label sentiment classification model comprises an encoding end based on an XLNet architecture, a decoding end based on label query interactive attention, and a linear classification layer connected in sequence;

[0014] The encoding end based on the XLNet structure comprises one embedding layer and c encoding layers, wherein c is a natural number greater than 1;

[0015] For the embedding layer, the input of the id list with a length of L is the sentence pair vector with a length of L*d and the position encoding with a length of L*d, L is the length of the id list, and d is the embedding dimension size;

[0016] For the encoding layer, the specific structure is:

[0017] The first layer is a multi-head self-attention layer, the input is the sentence pair vector with a length of L*d and the position encoding with a length of L*d obtained by the embedding layer, and the output is a feature vector with a length of L*d;

[0018] The second layer is a feedforward neural network layer, the input is the feature vector with a length of L*d obtained by the last layer, and the output is a feature vector with a length of L*d.

[0019] For the encoding end based on the XLNet structure, the input and output dimensions of each encoding layer are the same, and finally the final sentence pair feature vector with a length of L*d is output through one embedding layer and c encoding layers.

[0020] The decoding end based on label query interactive attention comprises one data processing layer and u decoding layers, wherein u is a natural number greater than 1;

[0021] For the data processing layer, the input is the sentence pair feature vector with a length of L*d output by the encoding end based on the XLNet structure, and the output is a label feature vector with a length of N*d, a label embedding with a length of N*d, a text feature vector with a length of M*d, and a position encoding with a length of M*d, wherein N is the number of sentiment labels, and M is the total length of the sentence pair minus the length of the first sentence in the sentence pair composed of the set of sentiment labels;

[0022] For the decoding layer, the specific structure is:

[0023] The first layer is a multi-head interactive attention layer, the input is the output N*d of the data processing layer, the label embedding with a length of N*d, the text feature vector with a length of M*d, and the position encoding with a length of M*d, and the output is a feature vector with a length of N*d;

[0024] The second layer is a feedforward neural network layer, an input of which is the N*d feature vector obtained by the previous layer, and an output of which is an N*d feature vector.

[0025] For the decoding end of the label query based interactive attention, the input and output dimensions of each decoding layer are the same, and finally an N*d sentiment label feature vector is output through one input processing layer and u decoding layers.

[0026] The linear classification layer has the following specific structure:

[0027] The first layer is a linear layer, an input of which is the N*d sentiment label feature vector output by the decoding end of the label query based interactive attention, and an output of which is an N*1 text correlation feature vector for each sentiment label;

[0028] The second layer is a probabilistic layer, an input of which is the N*1 text correlation feature vector for each sentiment label obtained by the previous layer, and an output of which is a text correlation probability for each label with a length of N, and the sentiment label corresponding to the text can be obtained according to the probability.

[0029] Preferably, the label-text interactive multi-label sentiment classification model is obtained by the following steps:

[0030] (4-1) Obtain an English training set data;

[0031] (4-2) The English training set data obtained in step (4-1) is subjected to data reading, cleaning and data format normalization processing in sequence to obtain an id list and a mask template;

[0032] (4-3) The id list and the mask template obtained in step (4-2) are input into an XLNet model for encoding, and the encoding result is segmented to obtain a plurality of label feature vectors and a plurality of text feature vectors to be analyzed;

[0033] (4-4) The label feature vectors and the text feature vectors to be analyzed obtained in step (4-3) are input into a decoding end for decoding to obtain a final sentiment label representation;

[0034] (4-5) The final sentiment label representation obtained in step (4-4) is input into a linear layer to obtain a final text correlation feature for each label, a loss value of a loss function of the label-text interactive multi-label sentiment classification model is calculated using the text correlation feature for each label and the actual sentiment label corresponding to the text in step (4-2-1), and each parameter of the XLNet model, the decoding end and the linear layer is iteratively trained using the loss value to obtain a trained label-text interactive multi-label sentiment classification model.

[0035] Preferably, step (4-2) specifically includes the following sub-steps:

[0036] (4-2-1) Use Pandas to read the English training set data obtained in step (4-1) to split each row into text and the actual emotional label corresponding to the text, clean the text to obtain preprocessed text with annotation marks, and directly read the first row header of the English training set data to obtain all emotional labels to obtain an emotional label set;

[0037] (4-2-2) constructing sentence pairs based on the pre-processed text with annotation marks and the sentiment label set obtained in step (4-2-1), constructing corresponding mask templates based on the sentence pairs, and encoding the sentence pairs to obtain an ID list;

[0038] Preferably, step (4-3) specifically includes the following sub-steps:

[0039] (4-3-1) Perform embedding and position encoding on the ID list obtained in step (4-2) to obtain a feature vector and position encoding of the sentence pair, whose dimension is L*d, where d is the embedding dimension size, which is generally 512, 768 or 1024, preferably 768, and L represents the length of the ID list;

[0040] (4-3-2) Input the sentence pair feature vector and position encoding obtained in step (4-3-1) and the mask template obtained in step (4-2) into the XLNet model to obtain the final feature vector of the sentence pair, whose dimension is L*d;

[0041] (4-3-3) The final feature vector of the sentence pair obtained in step (4-3-2) is split to obtain a label feature vector with a dimension of N*d and a feature vector of the text to be analyzed with a dimension of M*d, where N is the number of emotion tags in the emotion tag set obtained in step (4-2-1), and M is the remaining length after deducting the length of the first sentence in the sentence pair from the total length of the sentence pair.

[0042] Preferably, step (4-4) specifically includes the following sub-steps:

[0043] (4-4-1) Add the N*d-dimensional label feature vector obtained in step (4-3-3) to the N*d-dimensional learnable label embedding to obtain the query Q, take the M*d-dimensional text feature vector obtained in step (4-3-3) as the value V, and add the M*d-dimensional position code to the value V to obtain the key K;

[0044] Specifically, in this step, an embedding table with the same dimension N*d as the label feature vector is initialized, and the weight of the embedding table is a learnable label embedding of N*d, which corresponds to the label feature vector one by one, and the position encoding of M*d corresponds to the text feature vector to be analyzed one by one, which is calculated as follows:

[0045] PE(pos,2a) = sin(pos / 10000 2a / d )

[0046] PE(pos,2a+1) = cos(pos / 10000 2a / d )

[0047] Where pos represents the position of the word in the text feature vector, and pos∈[0,M-1], 2a represents the even index of the embedding dimension, 2a+1 represents the odd index of the embedding dimension, and a∈[0,d / 2-1], PE represents the position encoding result, sin represents the sine operation, and cos represents the cosine operation.

[0048] (4-4-2) Input the query Q, key K, value V obtained in step (4-4-1) and the label feature vector obtained in step (4-3) into the multi-head interaction attention layer to obtain the final output of the multi-head interaction attention layer, which has a dimension of N*d;

[0049] Specifically, the attention result of each head in the multi-head interaction attention layer of this step and the final output of the multi-head interaction attention layer are calculated as follows:

[0050]

[0051] MultiHead = Concat(head1, head2, …, head h )W O

[0052] x = LayerNorm(Q0+Dropout(MultiHead))

[0053] Where head i is the attention result of the i-th head in the multi-head interaction attention layer, with a dimension of N*d w , Q0 is a label feature vector with a dimension of N*d, Q EMB is a learnable label embedding with a dimension of N*d, K0 and V are both text feature vectors to be analyzed with a dimension of M*d, K POS is a non-learnable position encoding with a dimension of M*d, W i Q , W i K and W i V Q, K and V are projected into a learnable projection matrix with dimension d*d respectively w , d w =d / h, h represents the total number of heads of the multi-head interaction attention layer, W O represents the weight matrix of the linear layer with dimension d*d, x is the final output of the multi-head interaction attention layer with dimension N*d, wherein i∈[1, the total number of heads h of the multi-head interaction attention layer], Concat represents a function of concatenating input tensors according to the last dimension, MultiHead represents the attention result after multi-head calculation, LayerNorm represents normalizing all features of each sample, Dropout represents stopping the activation value of a neuron from working with a certain probability g during the forward propagation of the deep learning model, and is used to avoid model overfitting.

[0054] (4-4-3) inputting the final output of the multi-head interaction attention layer obtained in step (4-4-2) into a feedforward neural network to obtain the final output of the feedforward neural network layer with dimension N*d, and subtracting 1 from the total number u of decoding layers in the decoding end, if the value of u is 0, the obtained output is the final sentiment label representation, otherwise the obtained output is used as the input of the next decoding layer.

[0055] Specifically, the final output of the feedforward neural network layer in this step is obtained by the following formula:

[0056] FFN(x)=max(0,xW 1 +b 1 )W 2 +b 2

[0057] z=LayerNorm(x+Dropout(FFN(x)))

[0058] wherein W 1 is the projection weight of the first linear layer in the feedforward neural network with dimension d*4d, b 1 is the corresponding bias of W 1 , W 2 is the projection weight of the second linear layer in the feedforward neural network with dimension 4d*d, b 2 is the corresponding bias of W 2 , FFN represents the preliminary output of the feedforward neural network with dimension N*d, z is the final output of the feedforward neural network layer with dimension N*d, and max represents the maximum value operation.

[0059] Preferably, step (4-5) specifically comprises the following substeps:

[0060] (4-5-1) Linear layer projection and probabilistic processing are performed on the final emotion label representation with a dimension of N*d obtained in step (4-4) to obtain the probability of the text associated with each label, which is a one-dimensional tensor with a length of N;

[0061] Specifically, in this step, linear layer projection is to project the final emotion label representation with a dimension of N*d to N*1 to obtain the association feature of the text for each label. Then, the sigmoid function method is used to perform probabilistic processing on the association feature of the text for each label:

[0062]

[0063] wherein represents the association feature value of the text for the nth label, p n is the association probability of the text for the nth label, and e represents the base number of natural logarithm.

[0064] (4-5-2) According to the probability of the text associated with each label obtained in step (4-5-1) and the actual emotion label corresponding to the text in step (4-2-1), the loss value of the loss function of the multi-label emotion classification model based on the interaction between the label and the text is calculated, and the loss value is used to iteratively train the model to obtain the trained multi-label emotion classification model based on the interaction between the label and the text.

[0065] Specifically, in the training process of the model in this step, the loss function L mainly includes two loss functions L1 and L2, wherein the former is used to maximize the probability of the correct label, and the latter is used to maximize the distance between the positive and negative labels. The specific definitions are as follows:

[0066]

[0067]

[0068] L=(1-α)L1+αL2

[0069] wherein N represents the number of emotion labels, y n represents the actual value of the nth emotion label, p n is the association probability of the text for the nth emotion label, N + is the number of emotion labels with an actual value of 1, N - is the number of emotion labels with an actual value of 0, p j represents the jth element value of the vector p, and p k represents the kth element value of the vector p, and α is a weight that controls the contribution of L1 and L2 to the total loss L, and α∈[0,1].

[0070] According to another aspect of the present application, a multi-label sentiment classification system based on label-text interaction is provided, comprising:

[0071] A first module for obtaining text to be analyzed;

[0072] A second module for performing data cleaning processing on the text to be analyzed obtained by the first module to obtain preprocessed text with annotation marks.

[0073] A third module for performing data format normalization processing on the preprocessed text with annotation marks obtained by the second module to obtain an id list and a mask template.

[0074] A fourth module for inputting the id list and the mask template obtained by the third module into a pre-trained multi-label sentiment classification model based on label-text interaction to obtain sentiment labels corresponding to the text.

[0075] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0076] (1) The present application adopts step (2), which preprocesses the data clearly through the Twitter preprocessing tool ekpharasis to obtain general domain information, thereby mining more potential information of the text, and thus solves the technical problem of insufficient deep semantic learning of sentences;

[0077] (2) The present application adopts steps (4-2-2) to (4-3), which inputs all the sentiment label sets into the XLNet pre-training model by concatenating them into a sentence ending with a question mark and the text to be analyzed, so that the model can learn all the labels, thus solving the technical problem of low precision of multi-label sentiment classification task due to ignoring the coexistence and correlation of emotions;

[0078] (3) The present application adopts steps (4-3-3) to (4-4), which cuts the output encoding features of the XLNet pre-training model into label and sentence features to be analyzed and introduces them into the decoding end to learn from each other, thereby deeply mining the correlation between labels and words, thus solving the technical problem of weak generalization of the sentiment classification model;

[0079] (4) The application can adaptively extract local discriminant features of different labels by introducing a one-to-one learnable label embedding corresponding to the label at the decoding end, taking it as an important component of the query in the interactive attention module, and letting it detect and collect features related to the label from the features of the sentence to be analyzed, thereby solving the technical problem of being unable to deeply learn the attention weight of each label and each part of the text to be analyzed. BRIEF DESCRIPTION OF DRAWINGS

[0080] Figure 1 is a flowchart of the multi-label sentiment classification method based on the interaction between labels and text of the application;

[0081] Figure 2 is an architecture diagram of the multi-label sentiment classification model based on the interaction between labels and text of the application. DETAILED DESCRIPTION

[0082] In order to make the purpose, technical scheme and advantages of the application more clear, the application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as they do not conflict with each other.

[0083] The basic idea of the application is to improve the effect of sentiment classification from four aspects. First, the data is preprocessed and cleaned using the Twitter preprocessing tool ekphrasis to obtain general domain information, thereby mining more potential information of the text. Second, all class labels are constructed into a sentence ending with a question mark and a sentence pair with the sentence to be analyzed, which are input into the XLNet pre-training model, so that the model can learn all the labels and realize the coexistence and correlation of the labels. Then, the output encoding features of the XLNet pre-training model are divided into label and sentence to be analyzed features and introduced into the decoding end to learn from each other, thereby deeply mining the correlation between the labels and the words. Finally, a one-to-one learnable label embedding corresponding to the label is introduced at the decoding end, which is taken as an important component of the query in the interactive attention module, letting it detect and collect features related to the label from the features of the sentence to be analyzed, thereby adaptively extracting local discriminant features of different labels.

[0084] As shown in Figure 1 , the application provides a multi-label sentiment classification method based on the interaction between labels and text, comprising the following steps:

[0085] (1) obtaining a text to be analyzed;

[0086] (2) performing data cleaning processing on the text to be analyzed obtained in step (1) to obtain preprocessed text with annotation marks;

[0087] In particular, this step uses ekphrasis (a library for data preprocessing developed by the DataStories team) to extract some special symbols in the text as words, extract dates or users as uniform representations, and extract elongated words as annotated normal words, such as': ) ',':- )'to <happy>'gooooood' becomes good <elongated>and October 7th becomes <date>Thus, the preprocessed text with annotation marks is obtained.

[0088] The advantage of this step is that general domain information can be obtained, so that more potential information of the text can be mined.

[0089] (3) performing data format normalization processing on the preprocessed text with annotation marks obtained in step (2) to obtain an id list and a mask template;

[0090] Specifically, this step first constructs a sentence pair: first, the sentence pair is constructed by taking the first sentence in the preprocessed text with annotation marks as the first sentence of the sentence pair and the second sentence in the preprocessed text with annotation marks as the second sentence of the sentence pair. <cls>To begin, all of the sentiment labels (including anger, anticipation, disgust, fear, joy, love, optimism, pessimism, sadness, surprise, and trust) are concatenated into a single sentence ending in a question mark as the first sentence of a sentence pair, followed by <sep>which is followed by the preprocessed text with the annotation tags as the second sentence of the sentence pair, and finally appended with <sep>If the total length of the sentence is greater than 128, truncate to 128, otherwise pad the end <pad>until the total length is 128; then a mask template is constructed, which has a length of 128, and is one-to-one corresponding to the words of the sentence pair constructed <pad>The positions of the special tokens are 0, and the rest are 1; finally, the sentence pair is encoded, and the spliced final sentence pair is encoded into an id list using the XLNet wordpiece analyzer (from the transformers library of Hugging Face), that is, each word in the sentence pair is converted into the corresponding id in the word table.

[0091] (4) The id list and the mask template obtained in step (3) are input into a pre-trained multi-label sentiment classification model based on label-text interaction (as shown in Figure 2 ) to obtain the sentiment label corresponding to the text.

[0092] As shown in Figure 2 , the multi-label sentiment classification model of the application comprises an encoding end based on XLNet architecture, a decoding end based on label query interactive attention, and a linear classification layer connected in sequence.

[0093] The encoding end based on XLNet structure comprises one embedding layer and c encoding layers, wherein c is a natural number greater than 1, and is preferably 12;

[0094] For the embedding layer, the input of the id list with a length of L is the sentence pair vector with L*d and the position encoding with L*d, L is the length of the id list (which is equal to 128), and d is the embedding dimension size (which is equal to 768);

[0095] For the encoding layer, the specific structure is:

[0096] The first layer is a multi-head self-attention layer, the input is the sentence pair vector with L*d and the position encoding with L*d obtained by the embedding layer, and the output is a feature vector with L*d;

[0097] The second layer is a feedforward neural network layer, the input is the feature vector with L*d obtained by the last layer, and the output is a feature vector with L*d.

[0098] For the encoding end based on XLNet structure, since the input and output dimensions of each encoding layer are the same, the final L*d sentence pair feature vector is output through one embedding layer and c encoding layers.

[0099] The decoding end based on label query interactive attention comprises one data processing layer and u decoding layers, wherein u is a natural number greater than 1, and is preferably 3;

[0100] For the data processing layer, the input is the L*d sentence pair feature vector output by the encoding end based on the XLNet structure, and the output is an N*d label feature vector, an N*d label embedding, an M*d text feature vector and an M*d position encoding, where N is the number of sentiment labels (which is equal to 11), and M is the total length of the sentence pair minus the length of the first sentence in the sentence pair composed of the sentiment label set;

[0101] For the decoding layer, the specific structure is as follows:

[0102] The first layer is a multi-head interactive attention layer, which inputs the output N*d label feature vector, N*d label embedding, M*d text feature vector and M*d position encoding of the data processing layer, and outputs an N*d feature vector.

[0103] The second layer is a feedforward neural network layer, which inputs the N*d feature vector obtained from the previous layer and outputs an N*d feature vector.

[0104] For the decoding end based on label query interactive attention, since the input and output dimensions of each decoding layer are the same, the final N*d sentiment label feature vector is output through 1 input processing layer and u decoding layers.

[0105] The specific structure of the linear classification layer is as follows:

[0106] The first layer is a linear layer, which inputs the N*d sentiment label feature vector output by the decoding end based on label query interactive attention, and outputs an N*1 text association feature vector for each sentiment label.

[0107] The second layer is a probabilistic layer, which inputs the N*1 text association feature vector for each sentiment label obtained from the previous layer, and outputs an association probability of the text for each label with a length of N. According to the probability, the sentiment label corresponding to the text can be obtained.

[0108] Specifically, the multi-label sentiment classification model based on label and text interaction in step (4) is obtained by the following steps:

[0109] (4-1) Obtain English training set data;

[0110] Specifically, the English training set data obtained in this step is the SemEval 2018 Task 1 E-c data set.

[0111] (4-2) The English training set data obtained in step (4-1) is subjected to data reading, cleaning and data format normalization processing in sequence to obtain an id list and a mask template;

[0112] The step specifically includes the following sub-steps:

[0113] (4-2-1) Data reading is performed on the English training set data obtained in step (4-1) using Pandas (a data analysis package of Python) to split each row into text and the actual sentiment label corresponding to the text, data cleaning is performed on the text to obtain preprocessed text with annotation marks, and all sentiment labels are directly read from the table header of the first row of the English training set data to obtain a set of sentiment labels;

[0114] Specifically, each row of the English training set data includes a piece of text and the relationship between the text and 11 sentiment labels, 0 indicating no relationship and 1 indicating a relationship, wherein the 11 sentiment labels are anger, anticipation, disgust, fear, joy, love, optimism, pessimism, sadness, surprise and trust.

[0115] (4-2-2) Constructing a sentence pair according to the preprocessed text with annotation marks and the set of sentiment labels obtained in step (4-2-1), constructing a corresponding mask template according to the sentence pair, and encoding the sentence pair to obtain an id list;

[0116] Specifically, first, a sentence pair is constructed, and the specific construction process of the sentence pair is as follows: first, the preprocessed text with annotation marks obtained in step (4-2-1) is tokenized to obtain a token list, and then the token list is converted into a sentence pair, and the specific conversion process is as follows: the token list is converted into a sentence pair by taking the first token as the first sentence and the last token as the second sentence. <cls>To start, the set of sentiment tags is then concatenated into a question ending in a question mark as the first sentence of a sentence pair, followed by <sep>which is followed by the preprocessed text with the annotation tags as the second sentence of the sentence pair, and finally appended <sep>if the total length is greater than 128, truncate to 128, if less than 128, pad the back <pad>until the total length is 128; then a mask template is constructed, which has a length of 128, and is one-to-one corresponding to the words of the sentence pair constructed <pad>The positions of the special tokens are 0, and the rest are 1; finally, the spliced final sentence pair is encoded into an id list using the XLNet wordpiece analyzer (from the transformers library of Hugging Face), that is, each word in the sentence pair is converted into the corresponding id in the word table.

[0117] The advantage of this step is that the model can learn all the sentiment labels, thereby realizing the coexistence and correlation of emotions.

[0118] (4-3) The id list and the mask template obtained in step (4-2) are input into the XLNet model for encoding, and the encoding result is segmented to obtain a plurality of label feature vectors and a plurality of text feature vectors to be analyzed;

[0119] This step specifically includes the following sub-steps:

[0120] (4-3-1) The id list obtained in step (4-2) is subjected to embedding operation and position coding processing in sequence to obtain a feature vector of the sentence pair and a position code, and the dimension is L*d, wherein d is the embedding dimension size, which is generally 512, 768 or 1024, preferably 768, and L represents the length of the id list;

[0121] (4-3-2) The sentence pair feature vector and the position code obtained in step (4-3-1) and the mask template obtained in step (4-2) are input into the XLNet (which is an autoregressive model based on Bert and Transformer-XL) model to obtain a final feature vector of the sentence pair, and the dimension is L*d;

[0122] (4-3-3) The final feature vector of the sentence pair obtained in step (4-3-2) is subjected to segmentation processing to obtain a label feature vector with a dimension of N*d and a text feature vector to be analyzed with a dimension of M*d, wherein N is the number of emotion labels in the emotion label set obtained in step (4-2-1) (which is equal to 11), and M is the total length of the sentence pair minus the length of the first sentence in the sentence pair;

[0123] Specifically, the first sentence includes <cls>, all the emotional tags, a question mark ended sentence, and a divider <sep>.

[0124] The steps (4-3-1) to (4-3-3) have the advantage that the label contained in the encoding end output is decoupled from the text feature information, so that the association between the label and the word can be further mined in depth.

[0125] (4-4) input the label feature vector obtained in step (4-3) and the text feature vector to be analyzed into the decoding end for decoding to obtain the final sentiment label representation;

[0126] The decoding end of the present step comprises u decoding layers, wherein the initial value of u is a natural number greater than 1, and is preferably 3. The output of the previous decoding layer is taken as the input of the next decoding layer, and the output of the last decoding layer is the final sentiment label representation. The present step comprises the following sub-steps:

[0127] (4-4-1) add the label feature vector obtained in step (4-3-3) with a dimension of N*d to a learnable label embedding with a dimension of N*d to obtain a query Q, and take the text feature vector to be analyzed obtained in step (4-3-3) with a dimension of M*d as a value V, and add the value V to a position encoding with a dimension of M*d to obtain a key K;

[0128] Specifically, in the present step, an embedding table with the same dimension N*d as the label feature vector is initialized. The weights of the embedding table are the learnable label embedding with a dimension of N*d, which corresponds to the label feature vector one by one. The position encoding with a dimension of M*d corresponds to the text feature vector to be analyzed one by one, and is calculated as follows:

[0129] PE(pos,2a)=sin(pos / 10000 2a / d )

[0130] PE(pos,2a+1)=cos(pos / 10000 2a / d )

[0131] wherein pos represents the position of the word in the text feature vector, and pos∈[0,M-1], 2a represents the even index of the embedding dimension, 2a+1 represents the odd index of the embedding dimension, and a∈[0,d / 2-1], PE represents the position encoding result, sin represents the sine operation, and cos represents the cosine operation.

[0132] (4-4-2) input the query Q, the key K, the value V obtained in step (4-4-1) and the label feature vector obtained in step (4-3) into a multi-head interaction attention layer to obtain the final output of the multi-head interaction attention layer, which has a dimension of N*d;

[0133] Specifically, the attention result of each head in the multi-head interactive attention layer of the present step and the final output of the multi-head interactive attention layer are obtained by the following formula:

[0134]

[0135] MultiHead=Concat(head1,head2,……,head h )W O

[0136] x=LayerNorm(Q0+Dropout(MultiHead))

[0137] wherein head i is the attention result of the i-th head in the multi-head interactive attention layer, with a dimension of N*d w , Q0is a label feature vector with a dimension of N*d, Q EMB is a learnable label embedding with a dimension of N*d, K0and V are both text feature vectors to be analyzed with a dimension of M*d, K POS is an unlearnable position encoding with a dimension of M*d, W i Q , W i K and W i V are learnable projection matrices for projecting Q, K and V respectively to a single head dimension, each with a dimension of d*d w , d w =d / h, h represents the total number of heads of the multi-head interactive attention layer, W O represents the weight matrix of the linear layer, with a dimension of d*d, x is the final output of the multi-head interactive attention layer, with a dimension of N*d, wherein i∈[1, the total number of heads h of the multi-head interactive attention layer], Concat represents a function of concatenating input tensors according to the last dimension, MultiHead represents the attention result after multi-head calculation, LayerNorm represents normalizing all features of each sample, Dropout represents stopping the activation value of a neuron from working with a certain probability g (Bernoulli distribution) during the forward propagation of the deep learning model, which is used to avoid model overfitting.

[0138] The above steps (4-4-1) to (4-4-2) have the advantage that a learnable embedding corresponding to the label is introduced, the embedding is added to the label feature as a query, and the features related to the label are queried from the features of the sentence to be analyzed, so that the local discriminative features of different labels can be adaptively extracted.

[0139] (4-4-3) input the final output of the multi-head attention interaction layer obtained in step (4-4-2) into a feedforward neural network to obtain a final output of the feedforward neural network layer, which has a dimension of N*d, and subtract 1 from the total number u of decoding layers in the decoding end, if the value of u is 0, the obtained output is the final sentiment label representation, otherwise the obtained output is used as the input of the next decoding layer.

[0140] Specifically, the final output of the feedforward neural network layer in this step is obtained by the following formula:

[0141] FFN(x) = max(0, xW 1 +b 1 )W 2 +b 2

[0142] z = LayerNorm(x + Dropout(FFN(x)))

[0143] wherein W 1 is the projection weight of the first linear layer in the feedforward neural network, having a dimension of d*4d, b 1 is the corresponding bias of W 1 , and W 2 is the projection weight of the second linear layer in the feedforward neural network, having a dimension of 4d*d, b 2 is the corresponding bias of W 2 , FFN represents the preliminary output of the feedforward neural network, having a dimension of N*d, z is the final output of the feedforward neural network layer, having a dimension of N*d, and max represents the maximum value operation.

[0144] (4-5) input the final sentiment label representation obtained in step (4-4) into a linear layer to obtain the final associated features of the text for each label, use the associated features of the text for each label and the actual sentiment label corresponding to the text in step (4-2-1) to calculate the loss value of the loss function of the multi-label sentiment classification model based on the interaction between the label and the text, and use the loss value to iteratively train the parameters of the XLNet model, the decoding end and the linear layer to obtain the trained multi-label sentiment classification model based on the interaction between the label and the text.

[0145] This step specifically includes the following sub-steps:

[0146] (4-5-1) perform linear layer projection and probability processing on the final sentiment label representation obtained in step (4-4) and having a dimension of N*d to obtain the probability of the association of the text for each label, which is a one-dimensional tensor with a length of N;

[0147] Specifically, in this step, the linear layer projection is to project the final emotional label representation with dimension N*d to N*1 to obtain the association feature of the text for each label, and then the probability processing of the association feature of the text for each label adopts the method of sigmoid function:

[0148]

[0149] wherein represents the association feature value of the text for the nth label, p n is the association probability of the text for the nth label, e represents the base number of natural logarithm, which is an infinite non-cyclic decimal, and its value is 2.71828...

[0150] (4-5-2) Calculate the loss value of the loss function of the multi-label sentiment classification model based on the interaction between the label and the text according to the probability of the association of the text for each label obtained in step (4-5-1) and the actual emotional label corresponding to the text in step (4-2-1), and use the loss value to iteratively train the model to obtain the trained multi-label sentiment classification model based on the interaction between the label and the text.

[0151] Specifically, in the training process of the model in this step, the loss function L mainly includes two loss functions L1 and L2, wherein the former is used to maximize the probability of the correct label, and the latter is used to maximize the distance between the positive and negative labels, and the specific definitions are as follows:

[0152]

[0153]

[0154] L = (1-α)L1 + αL2

[0155] wherein N represents the number of emotional labels, y n represents the actual value of the nth emotional label (0 or 1), p n is the association probability of the text for the nth emotional label, N + is the number of emotional labels with actual value 1 (if N + = 0, then L2 = 0), N - is the number of emotional labels with actual value 0 (there is no sample containing all emotional labels, so N - cannot be 0), p j represents the jth element value of the vector p, and p k represents the kth element value of the vector p, and α is the weight controlling the contribution of L1 and L2 to the total loss L, and α ∈ [0, 1].

[0156] The step has the advantages that a weighted sum of two loss functions is used, on the one hand, the probability of a correct label is maximized, and on the other hand, the distance between positive and negative labels is maximized, so as to further improve the classification accuracy.

[0157] Test results

[0158] The test environment of the application: under the CentOS 7.9 operating system, the CPU is Intel(R) Xeon(R) E5-2678V3, the GPU is 1 piece of NVIDIA Tesla M40 24GB, the algorithm of the application is implemented by using Pytorch programming, and the specific settings of main hyperparameters are as shown in Table 1.

[0159] In order to illustrate the effectiveness of the method of the application and the improvement of the classification effect, the English test set of the data set SemEval2018Task 1E-c is tested, the statistical information of the test set of the data set is as shown in Table 2, the test results obtained by the application are compared with those of the current advanced method, and the evaluation results are as shown in Table 3.

[0160] According to the test results on the English test set of the data set SemEval 2018Task 1E-c recorded in Table 3, it can be seen that the multi-label sentiment classification method based on the interaction between labels and texts in the application is superior to the existing method in Macro-F1, Micro-F1 and Jaccard_Score, three multi-label classification indexes.

[0161] Table 1: Hyperparameter settings

[0162]

[0163]

[0164] Table 2: Statistical information of the data set (only the test set)

[0165]

[0166] Table 3: Comparison of test results

[0167]

[0168]

[0169] The multi-label sentiment classification method based on label and text interaction provided by the application firstly uses the preprocessing tool ekphrasis of Twitter to preprocess and clean the data, obtains general domain information, and thus mines more potential information of the text; secondly, all class labels are constructed into a sentence and a sentence to be analyzed are spliced into a sentence pair and input into an XLNet pre-training model, so that the model can learn all labels, and thus the coexistence and correlation of emotions are considered. Then, the output encoding features of the XLNet pre-training model are divided into label and sentence to be analyzed features and are introduced into the decoding end to let them learn from each other, so as to deeply mine the correlation between the label and the word; finally, the learnable label embedding corresponding to the label is introduced into the decoding end, and is taken as an important component of the query in the interactive attention module, so that it can detect and collect the features related to the label from the features of the sentence to be analyzed, and thus the local discriminant features of different labels can be adaptively extracted.

[0170] Those skilled in the art will easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.< / sep> < / cls> < / pad> < / pad> < / sep> < / sep> < / cls> < / pad> < / pad> < / sep> < / sep> < / cls> < / date> < / elongated> < / happy> < / pad> < / pad> < / sep> < / sep> < / cls>

Claims

1. A multi-label sentiment classification method based on the interaction between labels and texts, characterized in that: The following steps are involved: (1) Obtain the text to be analyzed; (2) performing data cleaning on the text to be analyzed obtained in step (1) to obtain a preprocessed text with annotation marks; (3) performing data format normalization on the pre-processed text with annotation marks obtained in step (2) to obtain an ID list and a mask template; (4) Inputting the ID list and mask template obtained in step (3) into a pre-trained multi-label sentiment classification model based on the interaction between labels and texts to obtain the sentiment label corresponding to the text; The multi-label sentiment classification model consists of three parts: an encoder based on the XLNet architecture, a decoder based on interactive attention for label query, and a linear classification layer. The encoding end based on the XLNet structure consists of 1 embedding layer and c encoding layers, where the value of c is a natural number greater than 1; For the embedding layer, its input is an ID list of length L, and its output is an L*d sentence pair vector and an L*d position encoding, where L is the length of the ID list and d is the embedding dimension size; For the coding layer, its specific structure is: The first layer is a multi-head self-attention layer, which takes as input the L*d sentence pair vector and L*d position encoding obtained by the embedding layer, and outputs the L*d feature vector; The second layer is a feedforward neural network layer, whose input is the L*d feature vector obtained in the previous layer, and the output is the L*d feature vector; For the encoding end based on the XLNet structure, since the input and output dimensions of each encoding layer are the same, the final L*d sentence pair feature vector is output through 1 embedding layer and c encoding layers; The decoding end of the interactive attention based on label query consists of 1 data processing layer and u decoding layers, where u is a natural number greater than 1; For the data processing layer, its input is the L*d sentence pair feature vector output by the encoding end based on the XLNet structure, and its output is the N*d label feature vector, N*d label embedding, M*d text feature vector and M*d position encoding, where N is the number of sentiment labels and M is the total length of the sentence pair minus the length of the first sentence in the sentence pair consisting of the sentiment label set.

2. The multi-label sentiment classification method based on label and text interaction according to claim 1 is characterized in that Step (2) is to use ekphrasis to extract some special symbols in the text into words, extract dates or users into unified representations, and extract elongated words into normal words with annotations, thereby obtaining preprocessed text with annotation marks; Step (3) is to first construct a sentence pair, that is, first <cls>Then all the emotion labels are concatenated into a sentence ending with a question mark as the first sentence of the sentence pair, followed by <sep>, followed by the preprocessed text with annotation tags as the second sentence of the sentence pair, and finally <sep>If the total length of the sentence pair is greater than 128, it will be truncated to 128. If it is less than 128, it will be padded to the end. <pad>Until the total length is 128; then construct a mask template with a length of 128, corresponding to the words of the constructed sentence pair, corresponding to <pad> The position of is 0, and the rest are 1; finally, encode the sentence pair, and use the XLNet word meta-analyzer to encode the concatenated final sentence pair into an id list.< / pad> < / pad> < / sep> < / sep> < / cls> 3. The multi-label sentiment classification method based on label and text interaction according to claim 1 or 2, characterized in that For the decoding layer, its specific structure is: The first layer is a multi-head interactive attention layer, whose input is the output of the data processing layer N*d label feature vector, N*d label embedding, M*d text feature vector and M*d position encoding, and the output is N*d feature vector; The second layer is a feedforward neural network layer, whose input is the N*d feature vector obtained in the previous layer, and the output is the N*d feature vector; For the decoding end of interactive attention based on label query, since the input and output dimensions of each decoding layer are the same, the final N*d emotion label feature vector is output through 1 input processing layer and u decoding layers; The specific structure of the linear classification layer is: The first layer is a linear layer, whose input is the N*d emotion tag feature vector output by the decoder of the interactive attention based on the label query, and the output is the N*1 associated feature vector of the text for each emotion tag; The second layer is the probabilistic layer. Its input is the associated feature vector of the N*1 text obtained in the previous layer for each emotional label, and its output is the association probability of the text with a length of N for each label. Based on this probability, the emotional label corresponding to the text can be obtained.

4. The multi-label sentiment classification method based on label and text interaction according to claim 3 is characterized in that The multi-label sentiment classification model based on the interaction between labels and text is trained through the following steps: (4-1) Obtain English training set data; (4-2) The English training set data obtained in step (4-1) is read, cleaned, and data format normalized to obtain an ID list and a mask template; (4-3) Inputting the ID list and mask template obtained in step (4-2) into the XLNet model for encoding, and segmenting the encoded result to obtain multiple label feature vectors and multiple text feature vectors to be analyzed; (4-4) Inputting the label feature vector obtained in step (4-3) and the feature vector of the text to be analyzed into the decoding end for decoding to obtain the final emotion label representation; (4-5) The final sentiment label representation obtained in step (4-4) is input into the linear layer to obtain the final associated features of the text for each label, and the associated features of the text for each label and the actual sentiment label corresponding to the text in step (4-2-1) are used to calculate the loss value of the loss function of the multi-label sentiment classification model based on the interaction between labels and texts, and the loss value is used to iteratively train the parameters of the XLNet model, the decoding end and the linear layer to obtain a trained multi-label sentiment classification model based on the interaction between labels and texts.

5. The multi-label sentiment classification method based on label and text interaction according to claim 4 is characterized in that Step (4-2) specifically includes the following sub-steps: (4-2-1) Use Pandas to read the English training set data obtained in step (4-1) to split each row into text and the actual emotional label corresponding to the text, clean the text to obtain preprocessed text with annotation marks, and directly read the first row header of the English training set data to obtain all emotional labels to obtain an emotional label set; (4-2-2) Construct sentence pairs based on the preprocessed text with annotation marks and the sentiment label set obtained in step (4-2-1), construct corresponding mask templates based on the sentence pairs, and encode the sentence pairs to obtain an ID list.

6. The multi-label sentiment classification method based on label and text interaction according to claim 5 is characterized in that Step (4-3) specifically includes the following sub-steps: (4-3-1) Perform embedding and position encoding on the ID list obtained in step (4-2) to obtain the feature vector and position encoding of the sentence pair, whose dimension is L*d, where d is the embedding dimension size and L is the length of the ID list; (4-3-2) Input the sentence pair feature vector and position encoding obtained in step (4-3-1) and the mask template obtained in step (4-2) into the XLNet model to obtain the final feature vector of the sentence pair, whose dimension is L*d; (4-3-3) The final feature vector of the sentence pair obtained in step (4-3-2) is split to obtain a label feature vector with a dimension of N*d and a feature vector of the text to be analyzed with a dimension of M*d, where N is the number of emotion tags in the emotion tag set obtained in step (4-2-1), and M is the remaining length after deducting the length of the first sentence in the sentence pair from the total length of the sentence pair.

7. The multi-label sentiment classification method based on label and text interaction according to claim 6 is characterized in that Step (4-4) specifically includes the following sub-steps: (4-4-1) Add the N*d-dimensional label feature vector obtained in step (4-3-3) to the N*d-dimensional learnable label embedding to obtain the query Q, take the M*d-dimensional text feature vector obtained in step (4-3-3) as the value V, and add the M*d-dimensional position code to the value V to obtain the key K; Specifically, in this step, an embedding table of the same dimension N*d is initialized according to the dimension of the label feature vector. The weight of the embedding table is the N*d learnable label embedding, which corresponds one-to-one with the label feature vector, and the M*d position encoding corresponds one-to-one with the feature vector of the text to be analyzed, which can be obtained as follows: PE(pos,2a)=sin(pos / 10000 2a / d ) PE(pos,2a+1)=cos(pos / 10000 2a / d ) Where pos represents the position of the word in the text feature vector, and pos∈[0,M-1], 2a represents the even index of the embedding dimension, 2a+1 represents the odd index of the embedding dimension, and a∈[0,d / 2-1], PE represents the position encoding result, sin represents the sine operation, and cos represents the cosine operation; (4-4-2) Input the query Q, key K, value V obtained in step (4-4-1) and the label feature vector obtained in step (4-3) into the multi-head interactive attention layer to obtain the final output of the multi-head interactive attention layer, whose dimension is N*d; Specifically, the attention results of each head in the multi-head interactive attention layer of this step and the final output of the multi-head interactive attention layer are obtained by the following formula: MultiHead=Concat(head1,head2,……,head h )W O x=LayerNorm(Q0+Dropout(MultiHead)) Among them head i is the attention result of the i-th head in the multi-head interactive attention layer, with a dimension of N*d w , Q0 is the label feature vector with dimension N*d, Q EMB is a learnable label embedding with dimension N*d, K0 and V are both feature vectors of the text to be analyzed with dimension M*d, K POS is a non-learnable positional encoding of dimension M*d, and is a learnable projection matrix that projects Q, K, and V to a single-head dimension, each with a dimension of d*d w , d w =d / h, where h represents the total number of heads in the multi-head interactive attention layer, and W O Represents the weight matrix of the linear layer, with dimension d*d, x is the final output of the multi-head interactive attention layer, with dimension N*d, where i∈[1, the total number of heads of the multi-head interactive attention layer h], Concat represents the function that concatenates the input tensors according to the last dimension, MultiHead represents the attention result after multi-head calculation, LayerNorm represents the normalization of all features of each sample, and Dropout means that during the forward propagation of the deep learning model, the activation value of a neuron is stopped with a certain probability g to avoid overfitting of the model; (4-4-3) The final output of the multi-head interactive attention layer obtained in step (4-4-2) is input into the feedforward neural network to obtain the final output of the feedforward neural network layer, whose dimension is N*d. At the same time, the total number of decoding layers u in the decoding end is subtracted by 1. If the value of u is 0, the obtained output is the final emotion label representation, otherwise the obtained output is used as the input of the next decoding layer; Specifically, the final output of the feedforward neural network layer in this step is obtained by the following formula: FFN(x)=max(0,xW 1 +b 1 )W 2 +b 2 z=LayerNorm(x+Dropout(FFN(x))) Where W 1 is the projection weight of the first linear layer in the feedforward neural network, with a dimension of d*4d, b 1 W 1 The corresponding bias, similarly, W 2 is the projection weight of the second linear layer in the feedforward neural network, with a dimension of 4d*d, b 2 W 2 Corresponding bias, FFN represents the preliminary output of the feedforward neural network, with a dimension of N*d, z is the final output of the feedforward neural network layer, with a dimension of N*d, and max represents the operation of taking the maximum value.

8. The multi-label sentiment classification method based on label and text interaction according to claim 7 is characterized in that: Step (4-5) specifically includes the following sub-steps: (4-5-1) Perform linear layer projection and probabilistic processing on the final sentiment label representation of dimension N*d obtained in step (4-4) to obtain the probability of the text being associated with each label, which is a one-dimensional tensor with a length of N; Specifically, in this step, the linear layer projection projects the final sentiment label representation of dimension N*d to N*1 to obtain the associated features of the text for each label. Then, the sigmoid function method is used to probabilistically process the associated features of the text for each label: in Represents the associated feature value of the text for the nth label, p n That is, the probability of the text being associated with the nth label, and e represents the base of the natural logarithm; (4-5-2) Calculating the loss value of the loss function of the multi-label sentiment classification model based on the interaction between labels and texts based on the probability of the text being associated with each label obtained in step (4-5-1) and the actual sentiment label corresponding to the text in step (4-2-1), and iteratively training the model using the loss value to obtain a trained multi-label sentiment classification model based on the interaction between labels and texts; Specifically, the loss function L in the model training process in this step mainly includes two loss functions L1 and L2, where the former is used to maximize the probability of the correct label and the latter is used to maximize the distance between positive and negative labels. The specific definitions are as follows: L=(1-α)L1+αL2 Where N represents the number of emotion labels, y n represents the actual value of the nth sentiment label, p n That is, the association probability of the text for the nth emotional label, N + is the number of sentiment labels with actual value 1, N - is the number of sentiment labels with actual value 0, p j Represents the jth element value of vector p, similarly p k Represents the kth element value of the vector p, α is the weight that controls the contribution of L1 and L2 to the total loss L, and α∈[0,1].

9. A multi-label sentiment classification system based on the interaction between labels and texts, characterized by: include: The first module is used to obtain the text to be analyzed; The second module is used to perform data cleaning on the article to be analyzed obtained by the first module to obtain pre-processed text with annotation marks; The third module is used to perform data format normalization processing on the pre-processed text with annotation marks obtained by the second module to obtain an ID list and a mask template; The fourth module is used to input the ID list and mask template obtained in the third module into a pre-trained multi-label sentiment classification model based on the interaction between labels and texts to obtain the sentiment label corresponding to the text; The multi-label sentiment classification model consists of three parts: an encoder based on the XLNet architecture, a decoder based on interactive attention for label query, and a linear classification layer. The encoding end based on the XLNet structure consists of 1 embedding layer and c encoding layers, where the value of c is a natural number greater than 1; For the embedding layer, its input is an ID list of length L, and its output is an L*d sentence pair vector and an L*d position encoding, where L is the length of the ID list and d is the embedding dimension size; For the coding layer, its specific structure is: The first layer is a multi-head self-attention layer, which takes as input the L*d sentence pair vector and L*d position encoding obtained by the embedding layer, and outputs the L*d feature vector; The second layer is a feedforward neural network layer, whose input is the L*d feature vector obtained in the previous layer, and the output is the L*d feature vector; For the encoding end based on the XLNet structure, since the input and output dimensions of each encoding layer are the same, the final L*d sentence pair feature vector is output through 1 embedding layer and c encoding layers; The decoding end of the interactive attention based on label query consists of 1 data processing layer and u decoding layers, where u is a natural number greater than 1; For the data processing layer, its input is the L*d sentence pair feature vector output by the encoding end based on the XLNet structure, and its output is the N*d label feature vector, N*d label embedding, M*d text feature vector and M*d position encoding, where N is the number of sentiment labels and M is the total length of the sentence pair minus the length of the first sentence in the sentence pair consisting of the sentiment label set.

Images