A multi-granularity sentiment analysis method based on joint modeling

Abstract

The application provides a multi-granularity sentiment analysis method based on joint modeling, and belongs to the field of deep learning. The method uses BERT as a public encoding layer to provide text embedding expression for different sentiment analysis. Since different aspect category information is scattered in the comment R, in order to dynamically aggregate the embedding expression of the relevant word symbol of each aspect category, the model adds an attention pooling layer. The attention pooling layer can help the model focus on the word symbol with the highest relevance to the target aspect category. The [CLS] expression representing the overall comment content is used as the prediction layer input of the overall sentiment prediction task; the aspect-oriented sentiment polarity judgment depends on the information aspect text expression re-aggregated through the attention pooling layer; and the overall aspect-oriented sentiment polarity judgment result is used as the user portrait task through the dimension reduction matrix composed of prior expert knowledge.

Description

Technical Field

[0001] The present invention provides a multi-granularity sentiment analysis method based on joint modeling, which belongs to the field of deep learning. Background Art

[0002] In today's digital and information-driven society, the internet has become an integral part of people's daily lives. With the rapid development of e-commerce, the massive amount of user feedback on e-commerce platforms has become a valuable data resource for both consumers and businesses. Online businesses organize and analyze this feedback to accurately understand user experiences and guide future improvements. Furthermore, they can conduct precise, fine-grained sentiment analysis on users and tailor recommendations to them.

[0003] In the field of natural language processing, sentiment analysis, also known as opinion mining, involves classifying the overall sentiment polarity of input text. The goal of sentiment analysis is to classify text as positive, negative, or sometimes even neutral, based on its emotional state and subjective information. Document-level reviews often contain sentiments on multiple aspects. The coarse-grained sentiment analysis techniques described above struggle to capture opinions on each granular aspect, thereby achieving more accurate user profiles. In response, some work has proposed aspect-based sentiment analysis methods to determine sentiment trends toward specific aspects. Most of the currently proposed methods model separately at two analysis granularities, without any interaction between them. However, in real-world user profiling scenarios, the two granularities are often highly correlated, and ignoring the semantic information between them can further bottleneck model performance. Therefore, multi-granular analysis methods are both consistent with human cognitive paradigms and can produce more accurate profiles.

[0004] Current fine-grained sentiment analysis methods often adopt a pre-training followed by fine-tuning paradigm. They leverage the BERT pre-trained language model to embed aspect words, supplemented by an Attention mechanism to capture contextual terms highly relevant to the aspect words. Softmax is then input into a pooling layer to produce predictions. Typical examples include sentiment classification using a joint attention mechanism, which extends the LSTM to concatenate aspect word representations and contextual embeddings as network inputs; sentiment mining based on a multi-head attention mechanism, which utilizes global and local attention modules to capture interactive information of varying granularity between aspect words and context; and a combination of the attention mechanism and BERT's multiple interactive attention networks, which utilizes parallel computation of some Transformers to obtain hidden states and capture more semantic details.

[0005] While these efforts have innovated data representation and training networks, their training methods overlook the relationship between coarse-grained and fine-grained data, failing to achieve the goal of creating detailed user profiles. Furthermore, general expression calculations often overlook key sentiment terms relevant to the target domain. Therefore, to achieve user profiles that integrate multi-granularity sentiment analysis, prior knowledge is needed to further enhance the role of key information in user profiles. Summary of the Invention

[0006] In order to overcome the challenges mentioned above, the present invention proposes a multi-granularity sentiment analysis method based on joint modeling, and relies on a priori matrices to capture key information to achieve accurate user portraits. This patent relates to a method for constructing user portraits, which uses BERT as a common encoding layer to provide text embedding expressions for different sentiment analyses. Since different aspect category information is scattered in the review R, in order to dynamically aggregate the embedding expressions of relevant tokens of each aspect category, the model adds an attention pooling layer. The attention pooling layer can help the model focus on the tokens with the highest correlation with the target aspect category. The [CLS] expression representing the entire review content is used as the prediction layer input for the overall sentiment prediction task; the aspect-oriented sentiment polarity judgment depends on the information aspect text expression re-aggregated through the attention pooling layer; and the whole formed by them is also calculated through a dimensionality reduction matrix composed of prior expert knowledge and used as the user portrait task.

[0007] Step 1: Load the review text dataset of the object to be analyzed and process the dataset.

[0008] Step 1-1: Remove English and punctuation marks from the comment dataset and convert traditional Chinese characters to simplified Chinese.

[0009] Step 1-2: Use word segmentation tools to segment the comment text;

[0010] Step 1-3: Load the stop word dictionary and remove stop words.

[0011] Step 2: Input the processed dataset into BERT to obtain text representation.

[0012] Step 2-1: Load the BERT model, and set the dimension to the default value of the pre-trained model;

[0013] Step 2-1: Add [CLS] to the beginning of the word segmentation result. This symbol is used for the whole sentence and the output vector corresponding to this symbol is used as the semantic representation h of the entire text. [] , used for downstream text classification tasks, and the prior matrix is ​​also formed by this representation;

[0014] Step 2-2: Input the text representation of each word into the context representation layer, set the default dimension of the representation vector, and the resulting text representation is h i ;

[0015] Step 3: Place h [] and h i Processing for different downstream tasks.

[0016] Step 3-1: The text representation layer embeds the obtained word segmentation expression vector {h １ ,h2,…,h z}Constitute the matrix Where d represents the dimension of the vector, and Z represents the number of words in the sentence;

[0017] Step 4: Pass the text representation through the attention pooling layer to aggregate contextual information for each aspect category related token representation;

[0018] Step 4-1: The input of the attention pooling layer is the expression matrix The attention mechanism focuses on the tokens that are most relevant to the target aspect category, thereby converting the text expression into an expression targeting a specific aspect.

[0019] Step 4-1-1: Input the expression into the attention pooling layer with parameters, learn the parameters through the network, and use the activation function to map the result to the required space. The process is shown in the following formula:

[0020]

[0021] in It is the intermediate parameter after being mapped by the activation function, W i a are the parameters that need to be learned by the trained network;

[0022] Step 4-1-2: Calculate the attention weight. After inputting the intermediate parameters of the previous step into the softmax activation function, the attention weight of each token can be obtained. The purpose of this step is to more accurately focus on the parts with high relevance to the aspect category words. The calculation process is shown in the following formula:

[0023]

[0024] where α i is through the attention weight, are training parameters;

[0025] Step 4-1-3: Based on the attention weight, re-weight the results obtained by the context representation layer. Through the activation function, the representation results of a certain aspect after information aggregation can be obtained. The calculation process is shown in the following formula:

[0026]

[0027] where r i is the review text representation related to the i-th aspect a i , g i is a gating function to filter out the value without mentioning the i-th aspect a i ;

[0028] Step 5: Construct a priori matrix based on priori knowledge, and then transform the aspect category related text representation using the priori matrix.

[0029] Step 5-1: For the priori matrix of the user portrait task, the portrait class label in the special scene is L k h [cls] represents the sum average, and this step is repeated for k labels to construct a priori matrix P with a dimension of d*k.

[0030] Step 5-2: Calculate the matrix M i composed of r r obtained in step 4-1-3 and the priori matrix P, and pass it through the hyperbolic tangent activation function tanh to obtain M k :

[0031] M k = tanh(P T *M r )

[0032] Step 6: Use the calculated representations h [cls] , r i , M k as inputs for downstream prediction tasks, as shown in FIG. 6, which can be input to three downstream tasks, namely overall evaluation prediction task, multi-aspect sentiment prediction task, and user portrait task based on the two. Figure 1

[0033] Step 6-1: Provide input for aspect-oriented sentiment prediction task and obtain the required prediction value:

[0034]

[0035] is the required prediction value, W i q and are parameters, is a perturbation to enhance the generalization ability of the model;

[0036] Step 6-2: Perform user portrait prediction on the representation results calculated with priori knowledge in special scenes: ​

[0037]

[0038] is the prediction parameter, is a perturbation that enhances generalization ability, It is the full-dimensional text expression obtained in step 5-2.

[0039] Step 6-3: The goal of this task is to use the h in step 2-1 [cls] To perform overall sentiment analysis prediction, the process is as follows:

[0040]

[0041] represents the overall score of the prediction, β T Learning parameters for model training, W r represents the parameter about the comment r, b r It is about the review r deviation. This classification task relies on the additional hidden layer of BERT and realizes sentiment classification prediction through the activation function tanh.

[0042] Beneficial effects:

[0043] The present invention can be used on Chinese Internet business platforms and social media platforms. The present invention proposes a multi-granularity sentiment joint modeling and analysis method, which optimizes the text representation effect by combining a pre-trained language model and an attention mechanism. The calculation results are gained through a multi-task learning network, and the relevance of aspect words relative to the context is improved, thereby significantly improving the prediction accuracy of multi-aspect sentiment analysis and overall sentiment analysis. The present invention applies the above innovations to user portrait tasks based on social media, and uses the prior matrix constructed by the overall sentiment expression word symbols to greatly improve the prediction accuracy of the user portrait prediction task. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a hierarchical diagram of the joint model. DETAILED DESCRIPTION

[0045] The present invention will be further described below with reference to the accompanying drawings and examples.

[0046] Step 1: Load the annotated internet business platform review dataset and process the dataset.

[0047] Step 1-1: Remove English and punctuation marks from the comments in the dataset and convert them into simplified Chinese;

[0048] Step 1-2: Use Jieba word segmentation tool to segment the comment text;

[0049] Step 1-3: Load the stop word dictionary and remove stop words.

[0050] Step 2: Input the processed dataset into BERT to obtain text representation.

[0051] Step 2-1: Load the BERT model bert-base-Chinese, indicating that the dimension is the default value of 768 for the pre-trained model;

[0052] Step 2-1: Add [CLS] to the beginning of the word segmentation result. This symbol is used for the overall and the output vector corresponding to this symbol is used as the semantic representation h of the entire text. [] , used for downstream text classification tasks, and the prior matrix is ​​also formed by this representation;

[0053] Step 2-2: Input the text representation of each word into the context representation layer. The dimension of the representation vector is 768, and the resulting text representation is h i ;

[0054] Step 3: Place h [] and h i Processing for different downstream tasks.

[0055] Step 3-1: The text representation layer embeds the obtained word segmentation expression vector {h1,h2,…,h z}Constitute the matrix Where d represents the dimension of the vector, and Z represents the number of words in the sentence;

[0056] Step 4: Pass the text representation through the attention pooling layer to aggregate contextual information for each aspect category related token representation;

[0057] Step 4-1: The input of the attention pooling layer is the expression matrix The attention mechanism focuses on the tokens that are most relevant to the target aspect category, thereby converting the text expression into an expression targeting a specific aspect.

[0058] Step 4-1-1: Input the expression into the attention pooling layer with parameters, learn the parameters through the network, and use the activation function to map the result to the required space. The process is shown in the following formula:

[0059]

[0060] in It is the intermediate parameter after being mapped by the activation function, W i a are the parameters that need to be learned by the trained network;

[0061] Step 4-1-2: Calculate the attention weight. After inputting the intermediate parameters of the previous step into the softmax activation function, the attention weight of each token can be obtained. The purpose of this step is to more accurately focus on the parts with high relevance to the aspect category words. The calculation process is shown in the following formula:

[0062]

[0063] where α i is through the attention weight, are training parameters;

[0064] Step 4-1-3: Based on the attention weight, re-weight the results obtained by the context representation layer. Through the activation function, the representation results of a certain aspect after information aggregation can be obtained. The calculation process is shown in the following formula:

[0065]

[0066] where r i is related to the i-th aspect a i Related comment text representation, g i is a gating function used to filter out the information that does not mention the i-th aspect a i The value of

[0067] Step 5: Construct a priori matrix based on prior knowledge, and then use the prior matrix to transform the text representation related to the aspect category.

[0068] Step 5-1: For the prior matrix of the user portrait task, label the portrait class in the special scenario as L k Comments h [cls] Indicates summing and averaging, repeating this step for k labels to construct the prior matrix P d*k For example, in domain A, there are 6 categories of labels, which are determined by expert knowledge. A comment labeled A1 is passed through BERT to obtain its h [cls] Indicates that h for all comments marked as A1 [cls] Indicates that the average calculation is performed to obtain the first column in the prior matrix, which represents the influence of the label on the full dimension, and finally constructs the prior matrix of P with a dimension of d*6;

[0069] Step 5-2: Substitute the r obtained in step 4-1-3 i The matrix M formed r Calculate with the prior matrix P and activate the tanh function through hyperbolic tangent to get the text expression M k :

[0070] M k =tanh(P T *Mr )

[0071] Step 6: Convert the calculated expression h [cls] 、r i 、M k are used as input for downstream prediction tasks, such as Figure 1 As shown in the figure, it can be input into three downstream tasks, namely the overall evaluation prediction task, the multi-faceted sentiment prediction task, and the user profiling task based on the two.

[0072] Step 6-1: Provide input to the aspect-oriented sentiment prediction task and obtain the required prediction value:

[0073]

[0074] is the predicted value to be obtained, W i q and They are all parameters. It is a perturbation that enhances the generalization ability of the model;

[0075] Step 6-2: Predict the user profile based on the representation results of the prior knowledge involved in the calculation in special scenarios:

[0076]

[0077] is the prediction parameter, is a perturbation that enhances generalization ability, It is the full-dimensional text expression obtained in step 5-2.

[0078] Step 6-3: The goal of this task is to use the h in step 2-1 [cls] To perform overall sentiment analysis prediction, the process is as follows:

[0079]

[0080] W r Represents the parameters about the review r, b r It is about the review r deviation. This classification task relies on the additional hidden layer of BERT and realizes sentiment classification prediction through the activation function tanh.

Claims

1. A multi-granularity sentiment analysis method based on joint modeling, characterized in that: include: Step 1: Load the annotated internet business platform review dataset and process the dataset; Step 1-1: Remove English and punctuation marks from the comments in the dataset and convert them into simplified Chinese; Step 1-2: Use Jieba word segmentation tool to segment the comment text; Step 1-3: Load the stop word dictionary and remove stop words; Step 2: Input the processed dataset into BERT to obtain text representation; Step 2-1: Load the BERT model bert-base-Chinese, and set the dimension to the default value of 768 for the pre-trained model; Step 2-2: Add the segmentation result to the beginning of the sentence , this symbol is used as a whole and the output vector corresponding to this symbol is used as the semantic representation of the entire text , used for downstream text classification tasks, and the prior matrix is ​​also formed by this representation; Step 2-3: Input the text representation of each word into the context representation layer, set the dimension of the representation vector to 768, and the resulting text representation is ; Step 3: and Process different downstream tasks; Step 3-1: The text representation layer embeds the obtained word segmentation expression vector Composition matrix ,in, represents the dimension of the vector, Represents the number of words in the sentence; Step 4: Pass the text representation through the attention pooling layer to aggregate contextual information for each aspect category related token representation; Step 4-1: The input of the attention pooling layer is the expression matrix ,Relying on the attention mechanism, we can focus on the tokens with the highest relevance to the target aspect category, thereby converting the text expression into an expression targeting a specific aspect; Step 4-1-1: Input the expression into the attention pooling layer with parameters, learn the parameters through the network, and use the activation function to map the result to the required space. The process is shown in the following formula: ； in, It is the intermediate parameter after mapping through the activation function. are the parameters that need to be learned by the trained network; Step 4-1-2: Calculate the attention weight. After inputting the intermediate parameters of the previous step into the softmax activation function, the attention weight of each token can be obtained. The purpose of this step is to more accurately focus on the parts with high relevance to the aspect category words. The calculation process is shown in the following formula: ； in, is through the attention weight, are training parameters; Step 4-1-3: Based on the attention weight, re-weight the results obtained by the context representation layer. Through the activation function, the representation results of a certain aspect after information aggregation can be obtained. The calculation process is shown in the following formula: ； in, It is with Aspects The relevant comment text indicates that Is a gating function used to filter out Aspects The value of Step 5: Construct a priori matrix based on prior knowledge, and then use the priori matrix to transform the text representation related to the aspect category; Step 5-1: For the prior matrix of the user portrait task, label the portrait class in the special scenario as Comments Indicates adding and averaging. Repeat this step for each label to construct the prior matrix , whose dimensions are ; Among them, portraits in the field There are 6 categories of labels in total, determined by expert knowledge, which will be labeled The comments are obtained through BERT Indicates that for all the Comments Indicates the average calculation to obtain the first column in the prior matrix, representing The impact of labels on all dimensions; Step 5-2: Get the The matrix formed With the prior matrix Calculate and activate the hyperbolic tangent function ,get : ； Step 6: Convert the calculated expression 、 、 are used as input for downstream prediction tasks, 、 、 They are input into three downstream tasks, namely the overall evaluation prediction task, the multi-faceted sentiment prediction task, and the user profiling task based on the two. Step 6-1: Provide input to the aspect-oriented sentiment prediction task and obtain the required prediction value: ； in, is the predicted value that needs to be obtained, and They are all parameters. It is a perturbation that enhances the generalization ability of the model; Step 6-2: Predict the user profile based on the representation results of the calculation involving prior knowledge in special scenarios: ； in, is the prediction parameter, is a perturbation that enhances generalization ability, It is the full-dimensional textual expression obtained in step 5-2; Step 6-3: The goal of this task is to use the To perform overall sentiment analysis prediction, the process is as follows: ； in, represents the overall score of the prediction, Learning parameters for model training, Comments Parameters, About comments Bias, this classification task relies on BERT's additional hidden layer through the activation function Implement sentiment classification prediction.

Images