A multi-modal sentiment analysis method based on ERNIE and multi-feature fusion

Abstract

The application relates to the technical field of natural language processing, and discloses a multi-modal sentiment analysis method based on multi-feature fusion of ERNIE, which specifically comprises the following steps: preprocessing text data to obtain a data set, dividing the data set, constructing a sentiment analysis model, combining low-layer word features and high-layer semantic features, performing double-ambiguity embedding on an expression, performing word segmentation on the text data and using one-hot coding to obtain corresponding part-of-speech vectors, splicing the vectors to obtain a sentence vector, feeding the sentence vector into a CNN for processing and extraction to obtain a new representation, feeding the new representation into a segmented pooling layer, segmentally extracting maximum features in the sentence vector and splicing the maximum features, performing average pooling to obtain a final vector representation, and activating the feature vector to obtain a final classification result. The application uses weighted double-ambiguity embedding, i.e. embedding positive and negative meanings respectively to solve the expression ambiguity problem and further extract more important feature information so as to improve the sentiment classification accuracy.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a multimodal sentiment analysis method based on ERNIE combining emojis and word vectors. Background Technology

[0002] In recent years, the rapid development of deep learning technology has brought a new and convenient method to various tasks in the field of natural language processing. Deep learning-based classification models have achieved excellent results on datasets across multiple domains in sentiment analysis tasks. With the development of social networks, emojis have become widely used in people's evaluation texts. Users often prefer to use emojis to assist in expressing their emotional inclinations, replacing words; therefore, emojis also play an important role in sentiment analysis tasks. Traditional sentiment analysis methods typically remove emojis during preprocessing, which often leads to the loss of some important features. Furthermore, emojis often convey drastically different emotions in different contexts, making classification difficult. Existing sentiment analysis methods using pre-trained models typically use BERT for pre-training, but BERT often results in invalid masking in MLM pre-training tasks due to incomplete masking of entity words.

[0003] In conclusion, how to select appropriate methods to address the problems existing in Chinese texts and improve the accuracy of sentiment polarity classification in Chinese texts has become an urgent issue to be addressed. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a multimodal sentiment analysis method based on ERNIE and multi-feature fusion. ERNIE incorporates Chinese entity word information into the training task, and improves the vector representation capability of the language model through pre-training and fine-tuning. By preprocessing the ERNIE input and adding label embeddings to assist training, the sentence vectors formed by combining ERNIE, the emoji vectors formed by FastText, and the part-of-speech vectors formed by one-hot encoding are concatenated, and features are extracted using a convolutional neural network model. In the CNN, segmented pooling is used instead of traditional max pooling to ensure the extraction of potentially different polarity features, effectively improving the sentiment analysis results.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0006] This invention is a multimodal sentiment analysis method based on ERNIE and multi-feature fusion, constructing a sentiment analysis model that includes a word embedding layer, ERNIE, a CNN feature extraction layer with segmented pooling, and a FastText layer. The method is characterized by the following steps:

[0007] Step 1: Obtain text data containing emojis as training corpus. Clean the text, label it, label its sentiment polarity and emojis. In addition, record the proportion of each emoji appearing in positive / negative text and store the record table.

[0008] Step 2: Divide the dataset collected in Step 1 into a training set, a validation set, and a test set in a ratio of 8:1:1;

[0009] Step 3: Label different text types and build a multimodal sentiment analysis model;

[0010] Step 4: Convert the preprocessed data into vectors by looking up the corresponding characters in the ERNIE dictionary. Add label vectors to the end of the original sentence vectors and input them into the ERNIE model. This results in 12 sets of character vector representations from 12 encoder layers. The MultiHead-Attention calculation formula in the encoder is as follows:

[0011]

[0012]

[0013]

[0014] Where Q represents the query matrix; K represents the key matrix; and V represents the value matrix. The matrix representing the linear transformation of Q; The matrix representing the linear transformation of K; represents the matrix for the linear transformation of V; h represents the number of heads, each head can focus on information from different representations at different positions. Represents the vector dimension. This represents the weight matrix. It takes the vectors corresponding to the CLS identifier bits in layers 2-12 and all word vectors in layer 12.

[0015] Step 5: Use FastText to perform ambiguous embedding on the text containing emojis, and store the resulting emoji vectors in a vector table for easy retrieval.

[0016] Step 6: Segment the text data to obtain the part-of-speech features of each word and use one-hot encoding to obtain the corresponding part-of-speech vector.

[0017] Step 7: Concatenate the vectors obtained in Step 4 and Step 5, use part-of-speech encoding, and concatenate again according to the corresponding position in the text to obtain the final sentence vector of fused text, emoticon, part of speech and tag for each sentence;

[0018] Step 8: Feed the vector obtained in Step 7 into the CNN for convolutional operation to reduce dimensionality. Use segmented pooling to extract average features from the sentence vector in segments and concatenate them. The convolutional neural network calculates as follows:

[0019]

[0020] in The weights of the convolution kernel, For bias, the ReLU function is used as the activation function here. The size of the convolution kernel, matrix For the character , , ..., The resulting feature matrix is ​​formed by convolution. The feature vectors obtained after convolution are fed into a piecewise pooling layer to obtain the final vector representation.

[0021] Step 9: Activate the feature vector processed in Step 8 using the Tanh function, and then feed it into a fully connected layer with dropout and softmax for classification to obtain the final classification result.

[0022] Furthermore, in step 1, text data is acquired to form a document, and the document is preprocessed to ensure that only one emoji is retained within a single text data entry; the text and emoji are separated by a specific symbol for separate processing later.

[0023] Furthermore, in steps 1 and 2, the preprocessing includes: filtering out special characters and data annotations, filtering out stop words, assigning labels to different categories of text, labeling the sentiment polarity of text, and counting the proportion of the same emoji appearing in positive / negative contexts.

[0024] Further, in step 3, a category label is assigned to each piece of text data, such as "takeout reviews" or "product reviews". A corresponding number for each label is added to the vocab.txt file.

[0025] Further, in step 4, the pre-trained ERNIE model is loaded, specifically the three files ernie_config.json, pytorch_model.ckpt, and vocab.txt; the text data without facial expressions from the collected dataset is then processed using '...'. <sep>Identifiers connected to text type labels are used as training materials to fine-tune ERNIE. The word vectors output from the 12-layer ERNIE encoder are obtained. The word vectors corresponding to the CLS flag bits from layers 2 to 12 are concatenated, combining the low-level word features with the high-level semantic features to form the vector representation of the sentence features for each sentence. Finally, all word vectors from the last layer are used as the vector representation of the word features for each sentence.

[0026] Furthermore, in step 5, the emoji symbol part separated by a specific identifier is taken to obtain the emoji in the text. The emoji is regarded as a special word, and positive and negative polarity labels are assigned to the same text. The FastText model is used to perform bisemantic embedding on the emoji in the text under different labels, that is, to encode them with positive and negative meanings and store them in a vector storage table.

[0027] Furthermore, in step 6, the text data is first segmented to obtain words, and the part-of-speech tag corresponding to each word and its corresponding position is obtained through a part-of-speech tag lookup table. The part-of-speech tag is encoded into a part-of-speech tag vector using one-hot encoding and stored in the part-of-speech tag lookup table for later use.

[0028] Further, in step 7, the vectors from steps 4 and 5 are concatenated, specifically the sentence vector from the 12-layer vertical combination, the character vector from the last layer, and the positive and negative vectors corresponding to the emojis. Ont-hot encodings corresponding to different parts of speech are retrieved from the tag vector table and concatenated to the character vectors in the ERNIE according to their corresponding positions to obtain the final fused vector. The sentence vector is padded with all zeros to ensure sufficient length. A positive / negative emoji ratio table is consulted, and for each emoji with a high frequency of occurrence, part-of-speech vectors (adjectives, verbs, nouns, etc.) are used for labeling. Less frequent occurrences are padded with all zeros to assign different weights to different emotional polarities. The final vector is obtained by fusing tags, parts of speech, emojis, sentence vectors, and character vectors.

[0029] Further, in step 8, the final vector obtained in step 7 is fed into a convolutional neural network, and a new representation is obtained by using convolution operations to reduce dimensionality and extract features. The processed vector is then fed into a segmented pooling layer, where the maximum feature is captured segment by segment in the vector after convolution by a full convolution kernel. The obtained maximum feature is then averaged to obtain the final vector.

[0030] The beneficial effects of this invention are:

[0031] (1) This invention proposes to use the ERNIE pre-trained language model to train and generate character vectors. The pre-trained model adds Chinese entity information mask in the MASK task during pre-training, which can fully explore the semantic information of the text and fully extract the context information of the characters, thereby generating more efficient character vector representations on Chinese text.

[0032] (2) The ERNIE multi-layer sentence feature proposed in this invention can solve the problem of feature loss caused by convolutional neural network processing to a certain extent, and can extract more important feature information from different layers.

[0033] (3) The multimodal sentiment analysis method of the present invention incorporates emoji embedding. It uses bisemantic embedding to address the ambiguity of the same emoji in different contexts and assigns different weights to the bisemantic embedding according to the frequency of emoji occurrence. This solves the problem of emoji ambiguity to a certain extent, enhances the representation ability of sentences, solves the problem of insufficient sentiment features in short text contexts, and ultimately improves the recognition accuracy.

[0034] (4) The segmented pooling layer proposed in this invention can capture the maximum feature in a segmented manner when multiple keywords appear in the text, and then perform average pooling to capture as many potentially effective keyword features as possible.

[0035] (5) This invention proposes that adding text category label features can improve the sentiment classification effect of texts in different fields, and adding part-of-speech features can help the model pay more attention to keywords and key emoji information, thereby improving the accuracy of sentiment analysis. Attached Figure Description

[0036] Figure 1 This is an overall flowchart of the multimodal sentiment analysis method in this embodiment of the invention.

[0037] Figure 2 This is a schematic diagram of a convolutional neural network with segmented pooling in the multimodal sentiment analysis method of this invention.

[0038] Figure 3 This is a diagram illustrating the overall architecture of the multimodal sentiment analysis method in this embodiment of the invention.

[0039] Figure 4 This is a diagram illustrating the weighted ambiguous Emoji formation process of the multimodal sentiment analysis method in this embodiment of the invention.

[0040] Figure 5 This is a diagram illustrating the part-of-speech vector formation process of the multimodal sentiment analysis method in this embodiment of the invention. Detailed Implementation

[0041] The embodiments of the present invention will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the invention. That is, in some embodiments of the invention, these practical details are not essential.

[0042] like Figure 1 As shown, this invention is a multimodal sentiment analysis method based on ERNIE and multi-feature fusion, which includes the following steps:

[0043] Step 1: Obtain text data containing emoji symbols to form documents as training corpora. Perform data preprocessing on the documents to obtain a dataset. Record the proportion of each emoji appearing in positive or negative text and store the record table. The preprocessing includes: filtering out special characters and data annotations, filtering out stop words, assigning labels to different categories of text, labeling the sentiment polarity of text, and counting the proportion of the same emoji appearing in positive or negative data.

[0044] Step 2: Divide the dataset collected in Step 1 into a training set, a validation set, and a test set in a ratio of 8:1:1;

[0045] Step 3: Label different text types, such as Figure 3 As shown, a sentiment analysis model is constructed;

[0046] Step 4: Convert the preprocessed data from Step 1 into vectors by looking up the corresponding characters in the ERNIE dictionary. Add label vectors to the end of the original sentence vectors and input them into the ERNIE model. This means loading the pre-trained ERNIE model, loading the three files ernie_config.json, pytorch_model.ckpt, and vocab.txt, and using '...' to select the text data (excluding facial expressions) from the collected dataset. <sep>The identifier is concatenated with the text type label as a training parameter to fine-tune ERNIE, resulting in word vectors output by the 12 layers of the ERNIE encoder. The MultiHead-Attention calculation formula in the encoder is as follows:

[0047]

[0048]

[0049]

[0050] Where: Q represents the query matrix, K represents the key matrix, and V represents the value matrix. The matrix representing the linear transformation of Q. The matrix representing the linear transformation of K. Let V be the matrix representing the linear transformation of V, and h represent the number of heads, each of which can focus on information from different representations at different positions. Represents the vector dimension. This represents the weight matrix.

[0051] Step 5, as follows Figure 4 As shown, the emoji symbols separated by specific identifiers are taken to obtain the emoji expressions in the text. The emoji symbol is regarded as a special word. Different polarity labels with positive and negative values ​​are assigned to the same text. By using the FastText model, the emoji expressions appearing in the text are embedded in a bisemantic way under different labels. That is, the emoji expressions are encoded in a positive and negative way and stored in a vector table.

[0052] Step 6, as follows Figure 5 As shown, the text data is first segmented to obtain words. The part-of-speech tag corresponding to each word and its corresponding position is obtained through a part-of-speech tag lookup table. The part-of-speech tag is encoded into a part-of-speech tag vector using one-hot encoding and stored in the part-of-speech tag lookup table for later use.

[0053] Step 7: Concatenating the vectors obtained in Steps 4 and 5 involves vertically concatenating the sentence vector (combined with 12 layers), the character vector of the last layer, and the positive and negative vectors corresponding to the emoji. The ont-hot encoding corresponding to different parts of speech is retrieved from the tag vector table and concatenated to the character vector of ERNIE according to the corresponding positions to obtain the final fused vector. Sentence vectors without parts of speech are padded with all zeros to complete their length. The positive or negative emoji ratio table is consulted, and for individual emojis, the more frequent polarity is labeled with part-of-speech vectors, while the less frequent polarity is labeled with all zeros to assign different weights to different polarities. This yields the final vector that fuses the tags, parts of speech, emojis, sentence vectors, and character vectors.

[0054] Step 8, as follows Figure 2 As shown, the vector obtained in step 7 is fed into the CNN, and a new representation is obtained by using convolution operation to reduce dimensionality and extract features. The feature vector obtained after convolution is fed into the segmented pooling layer, and the maximum feature is extracted from the sentence vector in segments and concatenated. The final vector representation is obtained by average pooling.

[0055] The convolutional neural network calculates as follows:

[0056]

[0057] in The weights of the convolution kernel, For bias, the ReLU function is used as the activation function, h is the kernel size, and the matrix is... For the character , , ..., The characteristic matrix formed by this.

[0058] Step 9: Activate the feature vector processed in Step 8 using the Tanh function, and then feed it into a fully connected layer with dropout and softmax for classification to obtain the final classification result.

[0059] Example

[0060] The following detailed explanation of the multimodal sentiment analysis method of the present invention is based on embodiments:

[0061] Step 1: Obtain text data containing emojis and preprocess the data. The specific steps are as follows:

[0062] Step 11: Remove repeated emojis from the same text, keep only one emoji and move it to the end, use regular expressions to filter out special characters and non-Chinese characters such as "...", "---", "%", "abc", etc., convert traditional Chinese to simplified Chinese, import the stop word dictionary, customize the dictionary, remove stop words from the text, and save the processed document.

[0063] Step 12. Add labels to the different types of data collected, such as "food reviews" and "hotel reviews", and merge the different datasets into the same document.

[0064] Step 13. Separate the text, tags, emojis, and polarities of each data entry using the identifier "||" to obtain a list of each sentence segment, denoted as sen_list={sentence,tag,emoji,label}.

[0065] Step 14: Perform the operation described in the previous step on all sentences in the text data to obtain a list set S_list={sen_list1,.,sen_list} i ,..sen_list m }. Among them, sen_list i This represents the list of word segments for the i-th sentence.

[0066] Step 15: Calculate the frequency of positive and negative signs in the text data containing a single emoji and store it in a record table.

[0067] Step 2: Perform operations such as data set partitioning. The specific processing steps are as follows:

[0068] Step 21: Obtain the preprocessed text data, i.e., document a.txt. Divide it into training set, validation set, and test set in an 8:1:1 ratio, with the positive and negative sample ratios being consistent in each dataset.

[0069] Step 3: Label the text data with tags. Add different tags to the vocab.txt file and obtain the corresponding serial numbers for each tag. For example, process text like "The taste is very good, but the delivery is slow" to correspond to the tag "takeout review". <sep>The identifier connects the processed data of the text and label ("The taste is very good, but the delivery is slow"). <sep>"Food delivery reviews"

[0070] Step 4: Load the pre-trained ERNIE model from Baidu, including the files bert_config.json, bert_model.ckpt, and vocab.txt. Use the text content from Step 3 as training material to fine-tune ERNIE. The specific process is as follows:

[0071] Step 41: Set the maximum sentence length pad_size. Sentences exceeding this length will be truncated, and sentences shorter than this length will be padded.

[0072] Step 42: Segment the text into a list by character, and add a single quote (') to the beginning of the list. <cls>'Character identifier at the beginning of the sentence, added at the end of the set' <sep>The character marks the end of the sentence. Search vocab.txt for the corresponding number for each character and its marker, and replace the elements in the list to obtain the input vector.

[0073] Step 43: Repeat the above steps to process all the data and save it as dataset.pkl.

[0074] Step 44: Load dataset.pkl to fine-tune the ERNIE model, which includes the embedding layer, bidirectional Transformer encoder, and output layer. After training, obtain the ERNIE model and save it as ernie.ckpt.

[0075] Step 5: Load the pre-trained ERNIE model ernie.ckpt and vocab.txt configuration files. Input the pre-processed data into the ERNIE model to obtain the vector representation of each word in each sentence. Assume there exists a sentence S={ , ,..., The formula for ERNIE to form word vectors is as follows:

[0076]

[0077] in, Let be the i-th character in S.

[0078] The vector after sentence S is transformed is = { , ,..., The ERNIE 12-layer encoder obtains a total of 12 sets of word vectors, i.e., 12... The vector is obtained by selecting the output vector from the last eleven layers, taking the word vector corresponding to the "CLS" identifier, and concatenating them. :

[0079] in, This is the final concatenated vector; This is a vector concatenation operation; It is the word vector corresponding to the 'CLS' identifier bit output by the i-th layer encoder.

[0080] Pick The final vector is obtained by concatenating the vector with the last layer of character vectors. :

[0081] =

[0082] in This is the word vector corresponding to the 'CLS' flag bit output by the 11-layer encoder, i.e., the sentence vector. This is the input word vector for the last layer.

[0083] Step Six: Obtain the word vectors corresponding to the emojis using the FastText model. The process of forming the word vectors corresponding to the emoji characters is as follows:

[0084] Step 61: Assign two different symbols to each individual emoji vector, one for the specific symbol '1' used in a positive sentiment context and the other for the specific symbol '0' used in a negative sentiment context.

[0085] Step 62: Using FastText training, embed the identifiers representing different polarities into different vectors to obtain the positive and negative embeddings of each emoji. This is done given an input sequence { , ,…, }, and the current character Justice Sample Set and negative sample set The objective function is obtained through binary logistic loss:

[0086]

[0087] in Score function It is calculated by summing the scalar product between the n-gram embedding of the current word and the embeddings of the surrounding words.

[0088] Step 7: Concatenate the labeled text vector obtained in Step 5 and the emoji vector obtained in Step 6. ⊕ This yields the final vector representation of sentence S after transformation. = { , ,..., }

[0089] in represents the vector representation of the i-th character after fusion, and ⊕ represents vector concatenation.

[0090] Step 8: Segment the sentence, obtain the part-of-speech tag for each word, and incorporate it into the vector. The specific process is as follows:

[0091] Step 81: Use the Jieba word segmentation tool to segment the input sentence S. Select the precise segmentation mode, which will cut the sentence into the most accurate segments. The resulting set of all segmented words corresponding to sentence S is denoted as Set={w1,w2,...,w...}. k }

[0092] Step 82: Search the part-of-speech table to obtain the part-of-speech set P = {p1, p2, ..., p m} corresponding to the sentence S, where p i represents the part of speech of w i . For example, for the sentence "Roasted meat is very delicious", the word segmentation result is {"Roasted meat", "very", "delicious"}. The corresponding part-of-speech table is {"n", "adv", "adj"}. Here, "n" indicates a noun, "adv" represents an adverb, and "adj" represents an adjective. In the sentiment analysis task, adjectives have a relatively high weight.

[0093] Step 83: Use one-hot to encode the part of speech to obtain the part-of-speech embedding vector corresponding to each part of speech. Suppose there are k different parts of speech in the dataset, then the part-of-speech dimension is k-dimensional. For example, the part-of-speech vector corresponding to the adjective part of speech is [1, 0, 0, 0…, 0]. One-hot means that only one bit has a value of 1, and the rest are 0. In addition, add a vector of all 0s as the vector for no part of speech.

[0094] Step 84. Sequentially concatenate the word vectors i and the part-of-speech vectors i through vector concatenation operation to form ([[]] i , i ) = ⊕ . Finally, the sentence S is transformed into E pw ={ , ,....., } sequence. For example, the combination of "delicious" and "adj" forms (good, 'adj') and (eat, 'adj'). The output word vector dimension is 768. If the value of k is 32, then the dimension of the vector feature containing the part of speech after concatenation is 800. The sentence vector corresponding to the 'CLS' flag bit has no specific part of speech representation, so it is concatenated with a vector of all 0s. For the polarity of emoji vectors with a large difference in positive and negative frequencies, the corresponding part of speech of adjectives is used for concatenation. For the polarity with a slightly larger difference in frequencies, the corresponding part of speech of verbs is used for concatenation. For the polarity with a small difference in frequencies, the corresponding part of speech of nouns is used for concatenation. For the rest, a vector of all 0s is used for concatenation to increase the weight of the high-frequency polarity. Obtain the final vector that integrates the part of speech, label, emoji, sentence, and word vector.

[0095] Step Nine: Input the final vector after the fusion and concatenation in Step Eight into the convolutional layer and the segmented pooling layer of the CNN to extract features. The specific process is as follows:

[0096] Step 91: Combine the vectors from Step 8. The input to the CNN layer uses three different kernel sizes: {2, 3, 4}, with 256 kernels of each size, totaling 768 kernels. The selected kernel size is... Features are extracted using convolutional kernels, among which Define a matrix to represent the convolution kernel size. For the character , , ..., The feature matrix is ​​composed of features, and then the convolution kernel is used to process the input feature matrix. Perform convolution operation, i.e.

[0097]

[0098] in: The weights of the convolution kernel, For bias, The activation function is ReLU. Here, ReLU is used as the activation function, and its calculation formula is:

[0099]

[0100] For length of The sentence, after convolution, yields the feature vector. :

[0101]

[0102] Step 92: Feed the feature vectors after CNN convolution into a segmented pooling layer to extract the maximum features at different positions. This model uses two-segment max pooling, that is, dividing a sentence into two segments, extracting the maximum features separately, and then performing average pooling to obtain the final vector. .

[0103] Step 10: Activate the vector obtained in Step 9 using the Tanh function:

[0104]

[0105] It constrains inputs that are too large or too small to a range of -1 to 1. When the activation value of the input is low, matrix operations can be performed directly, making training easier.

[0106] The activated results are input into softmax to obtain the probability distribution of the labels corresponding to the statement polarity:

[0107] In the formula, For the probability distribution of emotional polarity, Let be the weight matrix of the softmax function, and b be the bias. Finally, we obtain the sentiment polarity label Y corresponding to the input sentence S, thus completing the multimodal sentiment analysis.

[0108] This invention's sentiment analysis method enhances the model's vector representation capability by combining feature vectors provided by different layers of the ERNIE encoder. It uses concatenated CLS vectors from different ERNIE levels as sentence feature vectors and the highest-level character vectors as character feature vectors. Furthermore, during text preprocessing, it embeds emojis, typically removed in traditional sentiment analysis methods, as text features. FastText is used for ambiguous embedding of emojis, assigning different weights to the ambiguous embeddings based on the frequency of emojis appearing in different sentiment polarity scenarios to maximize the acquisition of information from the text. Adding emojis effectively improves sentiment classification results. CNNs are used to capture sentence features from different ERNIE levels, such as low-level word features and high-level semantic features. Segmented pooling is used to fully mine the maximum features of different parts of the text containing emojis and multi-feature fusion. This addresses the feature loss problem caused by tailing a CNN when using a preprocessing model. For the issue of the same emoji potentially having positive or negative sentiment polarity in different contexts, weighted ambiguous embeddings are used, embedding positive and negative emojis separately to resolve emoji ambiguity and extract more important feature information to improve sentiment classification accuracy.

[0109] The above description is only a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. Any equivalent modifications or changes made by those skilled in the art based on the content disclosed in the present invention should be included within the scope of protection set forth in the claims.< / sep> < / cls> < / sep> < / sep> < / sep> < / sep>

Claims

1. A multi-modal sentiment analysis method based on ERNIE and multi-feature fusion, a sentiment analysis model is constructed, the sentiment analysis model comprises a word embedding layer, an ERNIE and a CNN feature extraction layer with segmented pooling and a FastText layer, characterized in that: The multimodal sentiment analysis method includes the following steps: Step 1: Obtain text data containing emojis to form documents as training corpus. Perform data preprocessing on the documents to obtain a dataset. Record the proportion of each emoji appearing in positive or negative text and store the record table. Step 2: Divide the dataset collected in Step 1 into a training set, a validation set, and a test set in a ratio of 8:1:1; Step 3: Label different text types and build a sentiment analysis model; Step 4: Convert the preprocessed data from Step 1 into vectors by looking up the corresponding characters in the ERNIE dictionary. Add a label vector to the end of the original sentence vector and input it into the ERNIE model to obtain 12 sets of character vector representations from the 12-layer encoder. Take the character vectors corresponding to the CLS identifiers from layers 2 to 12 and concatenate them. That is, combine the word features of the lower layer with the semantic features of the higher layer as the vector representation of the sentence features of each sentence. Take all the character vectors from the 12th layer as the vector representation of the character features of each sentence. Step 5: Use FastText to perform ambiguous embedding on the text containing emojis and store the resulting emoji vectors in a vector table. Specifically, extract the emoji symbol parts separated by specific identifiers to obtain the emoji expressions in the text. Treat each emoji as a special word and assign positive and negative polarity labels to the same text. Use the FastText model to perform ambiguous embedding on the emoji expressions appearing in the text under different labels, that is, perform positive and negative encoding on the emoji expressions and store them in the vector table. The process of forming the word vectors corresponding to emoji characters is as follows: Step 51: Assign two different symbols to each individual emoji vector, one for the specific symbol '1' used in a positive sentiment context and the other for the specific symbol '0' used in a negative sentiment context; Step 52, using FastText training to embed the labels marking different polarities into different vectors, i.e. to obtain the positive and negative embeddings of each emoji, by giving an input sequence , ,…, } and the positive sample set and the negative sample set of the current word , the objective function is obtained by binary logistic loss, and the word vector is obtained by maximizing the score of the positive sample and minimizing the score of the negative sample: ， wherein denotes the scoring function is computed by summing the scalar products between the n-gram embedding of the current word and the context word embeddings; Step 6: Segment the text data containing emojis to obtain the part-of-speech features of each word and use one-hot encoding to obtain the corresponding part-of-speech vector; Step 7: Concatenate the vectors obtained in Step 4 and Step 5, use part-of-speech encoding, concatenate again according to the corresponding position in the text, look up the positive or negative proportion table of emojis, use part-of-speech vectors to label the polarity of a single emoji that appears frequently, and use all 0s to label the polarity that appears infrequently in order to assign different weights to different polarities, and obtain the final sentence vector of fused text, emoji, part of speech and tags for each sentence. Step 8: Feed the vector obtained in Step 7 into the CNN, use convolution operation to reduce dimensionality and extract features to obtain a new representation, feed the feature vector obtained after convolution into the segmented pooling layer, extract the maximum feature in the sentence vector segment by segment and concatenate them, and obtain the final vector representation after average pooling. Step 9: Activate the feature vector processed in Step 8 using the Tanh function, and then feed it into a fully connected layer with dropout and softmax for classification to obtain the final classification result.

2. The multi-modal sentiment analysis method based on ERNIE and multi-feature fusion according to claim 1, characterized in that: Only one emoji in each text data in the data set is retained, and the text and the emoji are separated by a specific identifier.

3. The multi-modal sentiment analysis method based on ERNIE and multi-feature fusion according to claim 2, characterized in that: The specific identifier is || 4. The multi-modal sentiment analysis method based on ERNIE and multi-feature fusion according to claim 1, characterized in that: In step 4, the preprocessed data is converted into a vector by searching the ERNIE dictionary, the label vector is added to the tail of the original sentence vector, and the ERNIE model is inputted, which means that the pre-trained ERNIE model is loaded, the ernie_config.json, pytorch_model.ckpt, and vocab.txt files are loaded, and the text data without expressions in the collected data set is used to 'tokenize' the text data to obtain the tokenized data, and the tokenized data is converted into a vector by searching the ERNIE dictionary. <sep>The text type label is connected by the identifier to fine-tune ERNIE, and the word vectors of 12 layers of encoder output of ERNIE are obtained, wherein the MultiHead-Attention calculation formula in the encoder is as follows:< / sep> ， ， ， where: Q represents a query matrix, K represents a key matrix, V represents a value matrix, a matrix representing a linear transformation of Q, a matrix representing a linear transformation of K, a matrix representing a linear transformation of V, h represents a number of heads, each head can focus on information from different representations at different positions, represents a vector dimension, represents a weight matrix.

5. The multi-modal sentiment analysis method based on ERNIE and multi-feature fusion according to claim 1, characterized in that: The step 6 is specifically: firstly, the text data is segmented to obtain segmented words, the part-of-speech of each word corresponding to the corresponding position is obtained through a part-of-speech lookup table, the part-of-speech is encoded into a part-of-speech vector in a one-hot encoding mode, and the part-of-speech lookup table is stored for subsequent use.

6. The multi-modal sentiment analysis method based on ERNIE and multi-feature fusion according to claim 1, characterized in that: The step 7 is specifically: the vectors obtained in steps 4 and 5 are spliced, which means that the 12-layer combined sentence vector, the last layer word vector and the emoji corresponding positive and negative vector are spliced longitudinally, the one-hot code corresponding to different part-of-speech in the label vector table is taken and spliced into the word vector of ERNIE according to the corresponding position to obtain the final fusion vector, the sentence vector without part-of-speech is filled with 0 to make up the length, the positive or negative proportion table of the emoji is looked up, the part-of-speech vector is marked for the polarity of the emoji appearing more, and 0 is marked for the polarity of the emoji appearing less to give different weights to different polarities, and finally the final vector of the fusion label, part-of-speech, emoji, sentence vector and word vector is obtained.

7. The multi-modal sentiment analysis method based on ERNIE and multi-feature fusion according to claim 1, characterized in that: In step 8, the convolutional neural network is calculated as follows: ， wherein is a weight of a convolution kernel, is a bias, a Relu function is used as an activation function, h is a size of a convolution kernel, and a matrix is a word , , …, composes a feature matrix.

8. The multi-modal sentiment analysis method based on ERNIE and multi-feature fusion according to claim 1, characterized in that: The preprocessing includes: filtering out special characters and data labels, filtering out stop words, assigning labels to different categories of text, annotating the sentiment polarity of the text, and counting the proportion of the same emoji appearing in positive or negative data.

Images