A multimodal evaluation object sentiment classification method based on grammar-guided network

Abstract

The present invention discloses a grammar-guided network based on a pre-trained model for the task of multimodal evaluation object sentiment classification. The network can perform end-to-end fine-grained sentiment analysis and judge the sentiment polarity while extracting the evaluation object. The method first uses a pre-trained model to perform modal alignment and fusion on the multimodal social media corpus used to obtain multimodal features based on external information; secondly, based on the selected pre-trained model, the noise in the multimodal feature matrix is ​​filtered; then, the attention calculation is performed on the modal fusion sequence based on the syntactic dependency tree to capture the contextual attention representation based on grammatical information; finally, for the evaluation object extraction and evaluation object sentiment classification tasks, a decoding layer is constructed and the loss function is optimized. The model network proposed in the present invention has an outstanding performance in the multimodal fine-grained end-to-end sentiment analysis task. All aspects of performance in the evaluation object sentiment classification task have been improved compared with the baseline method.

Description

Technical Field

[0001] The present invention relates to a fine-grained sentiment classification task, and in particular to a method for sentiment classification of a multimodal evaluation object based on a grammar-oriented network. Background Art

[0002] At present, most methods of fine-grained sentiment analysis do not regard the sentiment analysis task as a whole, but model specific subtasks separately. However, solving each subtask separately cannot take advantage of the commonalities between the two subtasks, and training models separately will also consume a lot of additional resources. Especially for the task of fine-grained sentiment classification, most methods predict the sentiment polarity given the evaluation object. However, in actual situations, the evaluation object in the corpus is hidden in the text and is not given explicitly. Therefore, the sentiment classification method based on a given evaluation object does not have strong application value at this stage. Summary of the invention

[0003] Purpose of the invention: Based on the shortcomings of the prior art, the present invention proposes a method for sentiment classification of multimodal evaluation objects based on a grammar-oriented network. The model can effectively encode multimodal features and perform targeted attention calculations for grammatical features.

[0004] Technical solution: A method for multimodal evaluation object sentiment classification based on grammar-oriented network. The model framework of this method is divided into four parts: encoding layer, noise filtering layer, grammar attention layer and decoding layer.

[0005] The encoding layer is the input of the entire model. LXMERT is used in the encoding layer to encode the multimodal features of the corpus. In addition to the LXMERT layer, the encoding layer also has character-level word vectors and an independent BERT encoding block. For text, the LXMERT model first uniformly fills or trims the sentences to a maximum length of n, and then represents them as position-based word vectors through word encoding and position encoding. Finally, the model passes through R Transformer encoding layers and sends it to the cross-modal encoding layer. For the image part, the model first uses the Faster-RCNN model to identify the foreground objects in the image and obtains its 2048-dimensional feature matrix. In addition, the model takes the position information of the object into consideration and represents its vertex coordinates as a position vector. Finally, the image features and position features are layer-normalized and averaged as the feature representation of the object. This step can be strictly expressed as follows:

[0006]

[0007]

[0008]

[0009] Where W F , W P , b F and b P is the parameter, v j is the feature representation of the final object. Similarly, the model sends the obtained object representation to the stacked Transformer for encoding, and then sends the object feature representation with dependencies between objects to the cross-modal encoding layer.

[0010] Each cross-modal encoding layer of LXMERT has two modules: a bidirectional cross-attention sublayer and a feedforward attention sublayer. The bidirectional cross-attention sublayer also consists of two parts. First, for time step i in the k-th layer text sequence, it pays attention to all m objects in the k-1th layer; similarly, the j-th object pays attention to the entire text sequence:

[0011]

[0012]

[0013] To further establish internal connections, the model applies self-attention to text sequences and object features:

[0014]

[0015]

[0016] Finally, the two modal vectors are respectively passed through the feed-forward network of Transformer as the final output of LXMERT.

[0017] At the same time, the output layer also uses the average of the 12-layer Transformer output vectors in BERT to obtain the final output of BERT:

[0018]

[0019] We also used Char-CNN to encode the text at the character level, and fused it with the text encoding obtained by LXMERT and the BERT encoding. For the fusion with BERT, we directly concatenated the BERT output with the character level word vector:

[0020]

[0021] A different strategy is adopted for the fusion with LXMERT. The two vectors are compressed to the same dimension, and then the character-level word vector is weighted summed with the LXMERT output vector with a smaller weight:

[0022] w i =ac (W c c i +b c )+(1-a c )(W l l i +b l )

[0023] Where W c , W l , b c and b l is the parameter, a c is the weight of the character-level word vector.

[0024] In the noise filtering layer, this method uniformly encodes the association between the image and the text in the <text-image> pair, thereby filtering the noise in the encoding layer. The noise filtering layer obtains the LXMERT-based multimodal text encoding W = {w1, w2, ···, w n} and BERT-based unimodal text encoding After the first feature matrix of the multimodal text output is fully connected, the Softmax function outputs the weights of the two encoding methods, and finally the weight guides the fusion of the two encodings, namely:

[0025] α=Softmax(W α w CLS +b α )

[0026]

[0027] Where α is the weight, is the first feature matrix of the multimodal text output, d is the feature dimension, W α and b α is a parameter, is the multimodal feature encoding after filtering the noise, and N is the length of the text sequence.

[0028] In the grammatical attention layer, this model introduces a syntactic dependency tree to model the dependencies between sentences, thereby enhancing the grammatical interpretability of the model. There is a graph G(V,E), where each unit in the multimodal sequence is a fixed point in the graph G, that is Edge E is the set of all edges with dependencies in the adjacency matrix M, and each edge has a corresponding weight, which is calculated by the correlation coefficient between the vertices at both ends of the edge. Specifically, when updating node i in a multimodal sequence, the attention weight value of the unit with dependencies relative to i is calculated by the following formula:

[0029]

[0030] in are all neighbor nodes of i, LeakyReLU is the activation function, which is an improvement of the ReLU activation function. It avoids unit death by setting the activation units less than 0 to a small negative number. W and ω are trainable parameters, and || represents the vector concatenation operation.

[0031] After obtaining the attention weights of node i for its neighboring nodes, the model updates the multimodal feature representation of node i according to the attention weight set, which can be expressed by the following formula:

[0032]

[0033] Where W and b are parameters. To make this layer more stable, the model uses multiple attention heads, where the parameters of each attention head are independent of each other. When outputting, the average of multiple attention heads is used as the final representation, that is:

[0034]

[0035] Since abbreviations and grammatical errors often cause complete failure of syntactic analysis, in order to alleviate the large amount of losses caused by syntactic analysis failure, the model introduces a grammar gate to directly determine whether the input sequence passes through the grammar attention layer. Let the first CLS of the input unimodal text encoding represent the text sequence, and perform binary classification on it through the fully connected layer to determine whether the text sequence is grammatically chaotic:

[0036]

[0037] z ★ = argmax(z CLS )

[0038] Where W CLS , b CLS If it is classified as a chaotic sequence, the input multimodal sequence is directly output without passing through the grammatical attention layer, otherwise it passes through the grammatical attention layer.

[0039] The grammatical attention layer determines the dependency between words through syntactic analysis, and only performs attention calculations on words with mutual relationships through the graph attention network, thereby enhancing the grammatical interpretability of the model. In addition, this layer takes into account the possible failure of syntactic analysis caused by the characteristics of social media texts, and determines whether the input sequence passes through the text graph network through the grammatical gate, thereby enhancing the effectiveness and reliability of this layer.

[0040] In order to enable the model to maximize the output probability of the true label sequence and avoid interference caused by invalid sequences, the CRF model is used as the decoding layer, and the negative value of its log-likelihood function is used as the loss function of the model. The specific formula is as follows:

[0041]

[0042]

[0043] Where N is the size of the sample set, X is the input text sequence, Y is the true annotation sequence corresponding to the text, and YX is the set of all possible outputs that conform to the dependency relationship.

[0044] During training, the model uses the Xavier method to initialize parameters. This initialization initializes the bias of each layer of the network to a zero vector, and the initialization of the parameter matrix conforms to Where n is the number of parameters. This initialization method can solve the gradient vanishing problem by keeping the output value of each layer in Gaussian distribution to avoid the decay of the activation value variance. The model is optimized using the Adam optimizer, which can dynamically adjust the learning rate and, for different parameters, the optimizer allows frequently changing parameters to be updated with a smaller step size, while sparse parameters are updated with a larger step size.

[0045] Beneficial effects:

[0046] 1) The present invention proposes a GOPREM model for the task of classifying the attributes of evaluation objects for social media corpus, and the model does not require the evaluation objects to be provided in advance.

[0047] 2) The model introduces the LXMERT pre-trained model for multimodal feature encoding, thereby solving the problem of limited performance of shallow networks, and the pre-trained model contains rich prior knowledge.

[0048] 3) The model uses a noise filtering layer to remove noise from multimodal fusion features. In order to address the errors that still exist in the self-attention mechanism in the Transformer model, the model introduces a grammatical attention layer, which analyzes the input text to obtain a syntactic dependency tree, and then uses a graph attention network to calculate the attention of each word only with words that have a grammatical dependency relationship in the sequence, so as to minimize the model error.

[0049] 4) In view of the possible failure of syntactic analysis caused by the informality of text in social media, the model introduces a grammar gate to determine whether the input passes through the grammar attention layer. This model extracts the evaluation object and judges its sentiment polarity at the same time, which has high practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1is an example of the corpus used in the training of the present invention;

[0051] Figure 2 It is the model architecture diagram of the present invention;

[0052] Figure 3 This is the coding layer architecture diagram of the model of the present invention

[0053] Figure 4 This is the framework diagram of the noise filtering layer of the model of the present invention

[0054] Figure 5 is an example of a syntactic dependency tree and the corresponding adjacency matrix. DETAILED DESCRIPTION

[0055] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0056] The present invention discloses a model of a grammar-guided network based on a pre-trained model for the task of multimodal evaluation object sentiment classification. Evaluation object extraction (ATE) and evaluation object sentiment classification (ATP) are two important subtasks in fine-grained sentiment analysis. However, most of the current fine-grained sentiment analysis tasks focus on one of the tasks. This model will solve the two subtasks at the same time and define them as ATEP. That is, while identifying the evaluation object in the sentence, the sentiment polarity of the sentence with respect to the evaluation object is judged. In the definition of this task, the input of the ATEP task is a natural language sequence, and the output is a label sequence corresponding to each word in the sequence, that is, this article also defines this task as a sequence labeling task. For the input sequence X={w1,w2,···,w n}, n is the sequence length, corresponding to the output sequence Y = {y1,y2,···,y n The labels of the output sequences follow the modified BIO-2 standard, i.e. i ∈{B-POS,B-NEU,B-NEG,I-POS,I-NEU,I-NEG,O}. The suffixes of the labels B and I represent their sentiment polarities, POS for positive, NEG for negative, and NEU for neutral. Similarly, for the sentence “The size is ideal while the quality is bad”, its class label sequence is Y={O,B-POS,O,O,O,O,B-NEG,O,O}. Figure 2 Shows an example of triples from a corpus text.

[0057] The model framework of the present invention is divided into four parts: encoding layer, noise filtering layer, grammatical attention layer and decoding layer. Its overall structure is as follows Figure 1 shown.

[0058] In the encoding layer of the present invention, LXMERT is selected for multimodal feature encoding of the corpus. In addition to the LXMERT layer, the encoding layer also has character-level word vectors and an independent BERT encoding block, whose structure is as follows: Figure 3 As shown in the figure. In the LXMERT model, visual encoding, text encoding and multimodal encoding are stacked with 5, 9 and 5 layers of Transformer encoders respectively, and the output vector dimension is 768. BERT uses the BERT-base pre-trained model 2 provided by Google, which contains 12 Transformer layers, and the word vector dimension is 768. The dimension of the character-level word vector is set to 30, and its initialization follows the uniform distribution of (-0.25, 0.25). Since the word segmentation used by LXMERT and BERT will divide a word into multiple word fragments, when aligning, the model will concatenate the character-level word vector of the word with all the word fragments corresponding to the word. The syntactic generation tree uses the API3 provided by spaCy. The sentence length and word length are set to 40 and 30 respectively. The initial learning rate of the model is set to 0.001, and the learning rate for fine-tuning BERT and LXMERT is set to 0.0001. Considering the characteristics of social media corpus, pictures do not always play a positive role as a supplement to text, and in some cases pictures exist as noise. Therefore, we use a BERT model to encode the text as the input of the subsequent network. The average of the 12-layer Transformer output vector in BERT is the final output of BERT:

[0059]

[0060] At the same time, Char-CNN is also used to encode the text at the character level, and then it is fused with the text encoding obtained by LXMERT and the BERT encoding respectively. For the fusion with BERT, the BERT output is directly concatenated with the character level word vector:

[0061]

[0062] A different strategy is adopted for the fusion with LXMERT. The two vectors are compressed to the same dimension, and then the character-level word vector is weighted summed with the LXMERT output vector with a smaller weight:

[0063] w i =a c (W c c i +b c )+(1-a c )(W l l i +b l )

[0064] Where W c , W l , b c and b l is the parameter, a c is the weight of the character-level word vector.

[0065] The input from the encoding layer enters the noise filtering layer. In the noise filtering layer, we uniformly encode the degree of association between the image and the text in the <text-image> pair, thereby filtering the noise generated in the encoding layer. The structure of the noise filtering layer is as follows: Figure 3 The noise filtering layer obtains the LXMERT-based multimodal text encoding W = {w1,w2,···,w n} and BERT-based unimodal text encoding After the first feature matrix of the multimodal text output is fully connected, the Softmax function outputs the weights of the two encoding methods, and finally the weight guides the fusion of the two encodings, namely:

[0066] α=Softmax(W α w CLS +b α )

[0067]

[0068] Where α is the weight, is the first feature matrix of the multimodal text output, d is the feature dimension, W α and b α is a parameter, is the multimodal feature encoding after filtering the noise, and N is the length of the text sequence.

[0069] After filtering the noise in the noise filtering layer, the grammatical attention layer model introduces a syntactic dependency tree to model the dependencies between sentences, thereby enhancing the grammatical interpretability of the model. This also reduces the problem that the LXMERT and BERT models will have some residual errors at the grammatical level, thus affecting the accuracy of the evaluation object sentiment classification task. The grammatical attention layer performs grammatical-level analysis on the sentence based on the syntactic dependency tree and the graph attention network. Specifically, first for the sentence T = {t1, t2, ···, t n}}, generate the adjacency matrix according to its syntactic dependency tree D Where N is the length of the sentence. In the adjacency matrix M, the words with dependency relationships in the syntactic dependency tree D are set to 1 (the words themselves are also set to have dependencies), and the other cells are set to 0. Figure 4An example of a syntactic dependency tree and the corresponding adjacency matrix is ​​shown. The grammatical attention layer attempts to strengthen the mutual connections in the grammar in the final attention representation. However, the model always emphasizes the characteristics of social media corpus, namely multiple abbreviations, multiple grammatical errors, and multiple typos. This poses a great challenge to the accuracy of syntactic analysis. And we know that the syntactic analysis after abbreviations has serious errors in the judgment of part of speech and word relations, which will cause the subsequent graph attention network to fail and greatly increase the loss of the model. Therefore, in order to alleviate the large amount of loss caused by the failure of syntactic analysis, the model introduces a grammatical gate to directly determine whether the input sequence passes through the grammatical attention layer. Let the first CLS of the input unimodal text encoding represent the text sequence, and perform a binary classification on it through the fully connected layer to determine whether the text sequence is grammatically chaotic:

[0070]

[0071] z ★ = argmax(z CLS )

[0072] Where W CLS , b CLS If it is classified as a chaotic sequence, the input multimodal sequence is directly output without passing through the grammatical attention layer, otherwise it passes through the grammatical attention layer.

[0073] Finally, after the output of the decoding layer, we can obtain the real label sequence. In order to enable the model to maximize the output probability of the real label sequence and avoid interference caused by invalid sequences, the CRF model is used as the decoding layer, and the negative value of its log-likelihood function is used as the loss function of the model. The specific formula is as follows:

[0074]

[0075]

[0076] Where N is the size of the sample set, X is the input text sequence, Y is the true annotation sequence corresponding to the text, and YX is the set of all possible outputs that conform to the dependency relationship.

[0077] In order to verify the advantages of the present invention over other models, a series of comparative experiments were conducted. The computer CPU used in the experiment is an 8-core 16-thread Core TM i9-9900K, GPU is Gigabyte RTX 2080Ti with 11G video memory. The steps of the experiment mainly include three aspects: first, data preparation; then model training; and finally, testing with the trained model to show the effect of the model.

[0078] 1) Data preparation

[0079] The datasets used in the experiment are the Twitter datasets corresponding to the Adaptive Co-attention Network for Named Entity Recognition in Tweets published by Zhang et al. and the Twitter datasets corresponding to the Visual Attention Model for Name Tagging in Multimodal Social Media published by Lu et al. When screening the corpus, first remove the tweets with texts other than English and tweets without pictures in the dataset. In the remaining data, if there is more than one picture corresponding to the text, one is randomly selected as the representative. Finally, delete the entries that do not contain any evaluation objects, have a text length of less than 3, or have difficult to understand text. Corpus annotation follows the BIO-2 standard. This paper divides the entire corpus into three parts: training set, validation set, and test set. The statistical information of the corresponding corpus is shown in the following table:

[0080]

[0081] 2) Model training

[0082] During training, the model uses the Xavier method to initialize parameters. This initialization initializes the bias of each layer of the network to a zero vector, and the initialization of the parameter matrix conforms to Where n is the number of parameters. This initialization method can solve the gradient vanishing problem by keeping the output value of each layer Gaussian distribution to avoid the decay of the activation value variance. The sentence length and word length are set to 40 and 30 respectively. The initial learning rate of the model is set to 0.001, and the learning rate for fine-tuning BERT and LXMERT is set to 0.0001. The training batch is 8 samples.

[0083] In order to better evaluate the performance of the GOPREM model, the GOPREM model was compared with the following different models under the same experimental environment, model settings, and corpus content:

[0084] BERT+BiLSTM+CRF This method uses the BERT pre-trained model for text encoding, and then uses the bidirectional LSTM model to extract the contextual semantic relationship of the text sequence. Finally, the CRF model is used for decoding. It is the basic architecture of many text sequence annotation models. When decoding, the evaluation object and sentiment polarity are jointly annotated.

[0085] GRACE is an end-to-end sentiment analysis model that proposes a gradient balance loss function and virtual adversarial training. This model is only for unimodal corpus.

[0086] GOPREM-LXMERT This method removes the LXMERT model and the subsequent noise filtering layer in GOPREM, that is, it does not perform multimodal fusion and only uses text to predict the sentiment polarity of the evaluation object.

[0087] ·RAN This method is a multimodal evaluation object extraction method.

[0088] GOPREM-FG This method removes the noise filtering layer in the GOPREM model and directly uses the multimodal encoding as the input of the grammatical attention layer.

[0089] GOPREM-GAt This method removes the grammatical attention layer in the GOPREM model and directly uses the output of the noise filtering layer as the input of the decoding layer.

[0090] GOPREM-F TBERT This method does not perform any fine-tuning on the BERT model in GOPREM and only uses its encoding as network input.

[0091] GOPREM-F TLXMERT This method does not perform any fine-tuning on the LXMERT model in GOPREM and only uses its encoding as the network input.

[0092] 3) Experimental results

[0093] Applying the prepared data to the above model, we can get the results shown in Table 1. The results show the precision, recall and F1-measure of the trained model on the test set. The larger the values ​​of these evaluation indicators, the better the performance of the model.

[0094] Table 1

[0095]

[0096] It can be seen from Table 1 that the GOPREM model proposed in the present invention obtains the best results. First, the first three unimodal models are compared, and it is found that BERT+BiLSTM+CRF performs the worst among them. This shows that although the structure already has strong semantic capture, temporal modeling and rule judgment capabilities, it is a general architecture and there is still a lot of room for optimization in specific fields. As one of the most outstanding end-to-end sentiment analysis models at present, the GRACE model has a performance that greatly exceeds the simple BERT+BiLSTM+CRF model, and the GOPREM model without LXMERT multimodal interaction is closer to it in performance. The GRACE model enhances model performance through innovative gradient balanced loss functions and virtual adversarial training, and innovates mainly in model training. The GOPREM model proposed in the present invention conducts targeted modeling by analyzing the characteristics of the corpus, and achieves similar or even better performance.

[0097] Secondly, ablation experiments were conducted on the GOPREM model, and it was found that: (a) the model performed poorly after removing the noise filter layer, with the F1-measure on the ATE task decreasing by 9.4% and the F1-measure on the ATP task decreasing by 9.6%. The performance of this method is even worse than that of the unimodal GOPREM-LXMERT model. It is speculated that this is because noise has a great impact on the fusion of text and images. The LXMERT model deeply fuses text and images through multiple layers of Transformer encoders. If noise filtering is not performed, it will have a considerable impact on the model performance. The noise filtering layer in GOPREM can effectively filter the noise in the image; (b) After removing the grammatical attention layer, the F1-measure of the model on the ATE task is reduced by 0.83%, and the F1-measure on the ATP task is reduced by 0.74%, indicating that the grammatical attention layer can effectively analyze the grammatical structure of the sentence and pay more attention to the grammatically related word pairs, thereby improving the model performance; (c) If the pre-trained model is not fine-tuned, the model effect will be greatly reduced. Not fine-tuning BERT makes the F1-measure of the ATE and ATP tasks decrease by 2.5% and 3.3% respectively, and not fine-tuning LXMERT makes the F1-measure of the ATE and ATP tasks decrease by 8.9% and 9.8% respectively. This shows that it is important to fine-tune the pre-trained model for a task, especially since the CLS features of the model are used extensively in this model, and it is even more necessary to fine-tune it on the task to achieve good results. Through the ablation experiment, it can be seen that each part of the GOPREM model has a great help to its performance. The GOPREM model proposed in this invention has excellent performance as can be seen from the experimental results.

Claims

1. A multimodal evaluation object sentiment classification method based on grammar-oriented network, characterized in that: The method includes four parts: encoding layer, noise filtering layer, grammatical attention layer and decoding layer. First, the multimodal fusion text vector matrix is ​​obtained through the encoding layer. The unimodal text vector matrix W is used as the input of the subsequent model. The noise of the encoding layer is filtered through the noise filtering layer. In the grammatical attention layer, the syntactic dependency tree is introduced to model the dependency relationship between sentences, thereby enhancing the grammatical interpretability of the model. Finally, the label sequence is obtained through the decoding layer. The grammatical attention layer determines the dependency between words through syntactic analysis, and performs attention calculation only on words with mutual relationship through the graph attention network, thereby enhancing the grammatical interpretability of the model; In the grammatical attention layer, the syntactic dependency tree is introduced to model the dependencies between sentences, thereby enhancing the grammatical interpretability of the model and improving the grammatical interpretability of the input multimodal sequence. There is a graph G(V,E), where each unit in the multimodal sequence is a fixed point in the graph G, that is Edge E is the set of all edges with dependencies in the adjacency matrix M, and each edge has a corresponding weight, which is calculated by the correlation coefficient between the vertices at both ends of the edge. Specifically, when updating node i in a multimodal sequence, the attention weight value of the unit with dependencies relative to i is calculated by the following formula: in are all neighbor nodes of i, LeakyReLU is the activation function, which is an improvement of the ReLU activation function. The unit death is avoided by setting the activation unit less than 0 to a small negative number. W and ω are trainable parameters, and || represents the vector concatenation operation.

2. According to claim 1, a multimodal evaluation object sentiment classification method based on grammar-oriented network is characterized in that: LXMERT is used in the encoding layer to encode the multimodal features of the corpus. In addition to the LXMERT layer, the encoding layer also has character-level word vectors and an independent BERT encoding block. The LXMERT must first process the unimodal input before performing cross-modal encoding. The BERT uses the average of the 12-layer Transformer output vectors as the final output of BERT. The character-level word vector uses Char-CNN to encode the text at the character level, alleviating the negative impact of its many abbreviations, grammatical errors, and typos.

3. According to claim 2, a multimodal evaluation object sentiment classification method based on grammar-oriented network is characterized in that: The LXMERT model used for multimodal encoding comes from Huggingface1, in which visual encoding, text encoding, and multimodal encoding are stacked using 5, 9, and 5 layers of Transformer encoders, respectively, and the output vector dimension is 768.

4. According to claim 1, a multimodal evaluation object sentiment classification method based on grammar-oriented network is characterized in that: BERT uses the BERT-base pre-trained model officially provided by Google, which contains 12 Transformer layers. The dimension of the word vector obtained is 768, the dimension of the character-level word vector is set to 30, and its initialization follows a uniform distribution of (-0.25, 0.25).

5. According to claim 1, a multimodal evaluation object sentiment classification method based on grammar-oriented network is characterized in that: During the model training phase, the initial learning rate of the model was set to 0.001, the learning rate for fine-tuning BERT and LXMERT was set to 0.0001, and the training batch size was 8 samples.

6. The method for multimodal evaluation object sentiment classification based on grammar-oriented network according to claim 1 is characterized in that: The noise filtering layer uniformly encodes the correlation between the image and the text in the <text-image> pair, thereby filtering the noise in the encoding layer. After training, the noise filtering layer can more accurately judge the relationship between the image and the text, thereby reasonably allocating weights to determine the proportion of the image in the output features, thereby alleviating the noise problem in the social media corpus.

7. The method for multimodal evaluation object sentiment classification based on grammar-oriented network according to claim 6 is characterized in that: The noise filtering layer obtains the LXMERT-based multimodal text encoding W = {w1, w2, ···, wn} and the BERT-based unimodal text encoding from the encoding layer. After the first feature matrix of the multimodal text output is fully connected, the Softmax function outputs the weights of the two encoding methods, and finally the weight guides the fusion of the two encodings, namely: α=Softumax(W.wcLs+b) Where α is the weight, is the first feature matrix of the multimodal text output, d is the feature dimension, Wα and bα are parameters, is the multimodal feature encoding after filtering the noise, and N is the length of the text sequence.

8. The method for multimodal evaluation object sentiment classification based on grammar-oriented network according to claim 1 is characterized in that: The CRF model is used as the decoding layer, and the negative value of its log-likelihood function is used as the loss function of the model. The specific formula is as follows: Where N is the size of the sample set, X is the input text sequence, Y is the true annotation sequence corresponding to the text, and YX is the set of all possible outputs that conform to the dependency relationship.

Images