A sentiment analysis method based on retrieval and contrastive learning
By employing a sentiment analysis method based on retrieval and contrastive learning, this approach addresses the issue of insufficient cross-sentence knowledge learning and improves sentiment classification performance by acquiring and preprocessing sentiment text data, using Elasticsearch to retrieve similar samples, and training a sentiment classification model.
Patent Information
- Application Number
- CN202211609151.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-14
- Publication Date
- 2026-02-06
- Estimated Expiration
- 2042-12-14
AI Technical Summary
Existing sentiment analysis methods fail to effectively learn general knowledge across sentences, resulting in a deficiency in the model's ability to represent sentiment.
We employ a sentiment analysis method based on retrieval and contrastive learning. We preprocess sentiment text data, extract and label entities, construct samples, use Elasticsearch for inverted indexing, retrieve similar samples, train a sentiment classification model using a neural network, and use contrastive learning techniques to narrow the vector distance between samples and similar samples.
It improves the representational ability of the sentiment classification model, enhances the sentiment representation ability, and improves the sentiment classification performance.
Smart Images
Figure CN116227486B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of natural language processing, and particularly relates to a sentiment analysis method based on retrieval and contrast learning. BACKGROUND
[0002] With the popularity of the Internet, the number of Internet users is growing exponentially, accompanied by the fact that users are increasingly inclined to express their own opinions on the Internet, such as sharing their real-time dynamics on social media websites such as micro blogs, Facebook, or commenting on the dynamics of other users, or evaluating the use experience of commodities on shopping websites. By analyzing these texts with strong emotional color, scientific and effective help can be provided in social public opinion detection, commodity personalized recommendation, chat robot design, etc.
[0003] With the refinement of sentiment analysis tasks, the most common and widely used is the attribute-level sentiment analysis task, which studies all sentiment objects in a text and identifies the sentiment polarity of each sentiment object. Traditionally, sentiment analysis mainly uses deep learning-based methods to calculate the semantics of the text, mine deep semantic meanings, extract the essential features of the text, obtain the abstract representation of the text, and then perform sentiment polarity classification. Although traditional methods have achieved great success, most of them calculate each sentence independently, and do not model the relationships that may exist between different sentences, so they cannot learn cross-sentence general knowledge, which makes the model lack in sentiment representation ability.
[0004] Therefore, how to provide a sentiment analysis method based on retrieval and contrast learning to improve the sentiment representation ability of the model and thus improve the sentiment classification performance has become a technical problem to be solved. SUMMARY
[0005] The technical problem to be solved by the present application is to provide a sentiment analysis method based on retrieval and contrast learning to improve the sentiment representation ability of the model and thus improve the sentiment classification performance.
[0006] The present application is implemented as follows: a sentiment analysis method based on retrieval and contrast learning, comprising the following steps:
[0007] Step S10, a large amount of sentiment text data is obtained, and each of the sentiment text data is preprocessed;
[0008] Step S20, entities in each of the preprocessed sentiment text data are extracted, each of the entities is labeled to construct a sample, and a sentiment dataset is generated;
[0009] Step S30, indexing the samples in the sentiment dataset by Elasticsearch, retrieving similar samples for each sample;
[0010] Step S40, creating a sentiment classification model based on a neural network, training the sentiment classification model using the sentiment dataset, and using contrastive learning to narrow the vector distance between each sample and similar samples during training;
[0011] Step S50, using the trained sentiment classification model to perform sentiment analysis.
[0012] Further, the step S10 specifically comprises:
[0013] Automatically obtain a large amount of sentiment text data, and preprocess each of the sentiment text data by removing hyperlinks, special characters, and repeated characters.
[0014] Further, the step S20 specifically comprises:
[0015] Using named entity recognition technology to extract entities from each of the preprocessed sentiment text data, selecting some of the entities as aspects, and annotating each of the entities with sentiment polarity to construct samples and generate a sentiment dataset.
[0016] Further, the step S30 specifically comprises:
[0017] Divide the sentiment dataset into a training set, a validation set, and a test set based on a predetermined proportion;
[0018] Index the samples in the training set by Elasticsearch to construct a key-value list, with the key being the aspect and polarity of the sample and the value being the corresponding sample;
[0019] Retrieve similar samples for each sample based on the aspect and polarity.
[0020] Further, in the step S40, the sentiment classification model is used to perform word segmentation on the samples, convert the segmented input into a sample vector based on BERT, use contrastive learning to narrow the vector distance between each sample and similar samples, input the sample vector into a linear layer and a softmax function to obtain a predicted probability distribution, and then output the predicted result of the sentiment.
[0021] Further, in the step S40, the loss function for using contrastive learning to narrow the vector distance between each sample and similar samples is expressed as:
[0022]
[0023] wherein, loss SCL represents a contrast loss; sim() represents a similarity measure function; n represents a batch size during training of the sentiment classification model; h i represents the i-th sample; h j represents the j-th sample; represents a similar sample of the i-th sample; e represents a base number of a natural logarithm.
[0024] Further, the sample vector is input into a linear layer and a softmax function to obtain a predicted probability distribution, and then a predicted result of the sentiment is output, and the specific process is as follows:
[0025] After dimension reduction of the sample vector by using the linear layer, the probability distribution of the corresponding sample vector is predicted by using the softmax function.
[0026]
[0027] wherein, represents a predicted probability distribution of the sentiment classification model; the softmax() function is used to convert the calculation result into a probability distribution with a range of [0, 1] and a sum of 1; W and b represent model parameters of the linear layer;
[0028] The maximum value of the probability distribution is output as a predicted result of the sentiment.
[0029] Further, the training of the sentiment classification model by using the sentiment data set is specifically as follows:
[0030] In order to reduce the joint loss of loss SCL and loss CE , the sentiment classification model is trained by using the training set, and the trained sentiment classification model is verified and tested by using the validation set and the test set.
[0031] wherein, loss SCL represents a contrast loss, and loss CE represents a classification loss.
[0032] Further, the calculation formula of the classification loss is as follows:
[0033]
[0034] wherein, represents a predicted probability distribution; y represents a real probability distribution; -log() represents a negative log-likelihood function.
[0035] The present application has the following advantages:
[0036] By preprocessing the obtained emotional text data, extracting entities in each preprocessed emotional text data, and labeling each entity to construct samples, an emotional data set is generated, then the samples in the emotional data set are inverted indexed, each sample is retrieved with similar samples, the emotional data set is used to train the emotional classification model based on neural network, while training, the contrast learning technology is used to narrow the vector distance between each sample and its similar sample, finally the trained emotional classification model is used for emotional analysis, that is, through retrieval and contrast learning, the emotional classification model learns general knowledge from similar samples, enhances the representation ability of the emotional classification model, and obtains more robust sentence representation ability, which greatly improves the model emotional representation ability, and further greatly improves the emotional classification performance. BRIEF DESCRIPTION OF DRAWINGS
[0037] The application will be further described below with reference to the accompanying drawings and embodiments.
[0038] Figure 1 is a flowchart of the emotional analysis method based on retrieval and contrast learning of the application.
[0039] Figure 2 is a flowchart of the emotional analysis method based on retrieval and contrast learning of the application.
[0040] Figure 3 is a schematic diagram of the emotional classification model training of the application.
[0041] Figure 4 is a schematic diagram of the input module of the emotional classification model of the application.
[0042] Figure 5 is a schematic diagram of the BERT module of the emotional classification model of the application.
[0043] Figure 6 is a schematic diagram of the contrast learning module of the emotional classification model of the application.
[0044] Figure 7 is a schematic diagram of the linear classification module of the emotional classification model of the application. DETAILED DESCRIPTION
[0045] The technical solution in the embodiment of the application has the following general idea: through retrieval and contrast learning, the emotional classification model learns general knowledge from similar samples to improve the model emotional representation ability and further improve the emotional classification performance.
[0046] Please refer to Figures 1 to 7 Fig. 1 shows one of the preferred embodiments of the emotional analysis method based on retrieval and contrast learning of the application, which includes the following steps:
[0047] Step S10, a large amount of emotional text data is obtained, and each of the emotional text data is preprocessed;
[0048] Step S20, entities in each of the preprocessed emotional text data are extracted, each of the entities is labeled to construct a sample, and then an emotional data set is generated;
[0049] Step S30, the samples in the emotional data set are inverted indexed by Elasticsearch, and similar samples are retrieved for each of the samples; that is, similar sample retrieval is performed by using Elasticsearch;
[0050] Step S40, a sentiment classification model is created based on a neural network, the sentiment classification model is trained using the emotional data set, and at the same time, the vector distance between each of the samples and similar samples is narrowed using a contrast learning technique;
[0051] Step S50, the trained sentiment classification model is used for sentiment analysis.
[0052] The step S10 specifically comprises:
[0053] A large amount of emotional text data is automatically obtained, and each of the emotional text data is preprocessed by removing hyperlinks, special characters and repeated characters in a regular manner; the emotional text data can be an evaluation of a certain digital product, and the evaluation opinion text of the user for the product can be obtained from platforms such as Jingdong and Taobao.
[0054] The step S20 specifically comprises:
[0055] The entities in each of the preprocessed emotional text data are extracted using a named entity recognition technology, part of the entities are selected as aspects, each of the entities is labeled for sentiment polarity (including positive, neutral and negative) to construct a sample, and then a high-quality emotional data set is generated.
[0056] The step S30 specifically comprises:
[0057] The emotional data set is divided into a training set, a validation set and a test set based on a preset ratio; the preset ratio is preferably 8:1:1;
[0058] The samples in the training set are inverted indexed by Elasticsearch to construct a key-value list, that is, a {key, value} list, the key is the aspect and polarity of the sample, and the value is the corresponding sample; the inverted indexing mechanism in Elasticsearch can quickly search and extract the matched content;
[0059] Based on the aspect and polarity of the sample, similar samples are retrieved for each sample.
[0060] For a given target sample, its similar samples are retrieved from the database by exact matching, that is, samples with the same sample key as the target sample key are extracted from the database, and k samples are randomly selected as the similar samples of the target sample. It is worth noting that the number of retrieved samples may be less than k, in which case different dropout methods are used to process the target sample to create new similar samples to replace them.
[0061] In step S40, the sentiment classification model is used to segment the sample, convert the segmented input into a sample vector based on BERT, use contrastive learning technology to narrow the vector distance between each sample and similar samples, enable the sentiment classification model to learn cross-sentence general knowledge, and then input the sample vector into a linear layer and a softmax function to obtain a prediction probability distribution, and then output the prediction result of the sentiment.
[0062] In specific implementation, for Chinese text data, the Jieba tool can be used to segment the sample sentence; for English text data, the space is used for segmentation. The conversion process of the sample vector is as follows: the aspect of the sample is concatenated with the sample sentence after the [SEP] symbol as input, the vectorization representation of each token of the input is obtained by using the respective word vector method, and the sample representation vector about the specified aspect is obtained after BERT encoding calculation.
[0063] The sentiment classification model includes an input module, a BERT encoding module, a contrastive learning module, and a linear classification module.
[0064] Input module: for Chinese text, encode the Chinese characters segmented by Jieba according to double-byte encoding to obtain word vectors; for English text, obtain word vectors segmented by BPE according to Tokenizer in the BERT model. The sample word vector representation is input into the BERT model for semantic calculation.
[0065] BERT encoding module: the multi-head attention mechanism in BERT is used to extract semantic features from the input word vector representation, so that each word vector can aggregate context information to obtain more essential feature information. The multi-head attention mechanism enhances the model's ability to extract sentence features in terms of breadth; stacking multiple multi-head attention mechanisms ensures the model's representation ability for sentences in terms of depth, so that the model can learn deeper meanings of sentences.
[0066] The contrastive learning module: using contrastive learning to narrow the distance between the target sample representation and its similar sample representation, so that the model can perceive the general knowledge between the target sample and its similar sample, achieving the goal of enhancing the representation ability of the model.
[0067] The linear classification module: first purify the vector representation of the target sample using a linear model, then normalize the purified features using the softmax function to get the probability distribution of sample classification, and take the class with the maximum probability as the sentiment polarity prediction result of the sample.
[0068] In step S40, the loss formula for narrowing the vector distance between each sample and similar samples using contrastive learning technology is:
[0069]
[0070] Where, loss SCL represents the contrastive loss; sim() represents the similarity measurement function; n represents the batch size during sentiment classification model training, meaning that other samples in the same batch during training are considered as negative samples for contrastive learning; hi i represents the i-th sample; hj j represents the j-th sample; represents the similar sample of the i-th sample; e represents the base of natural logarithm.
[0071] The sample vector is input into the linear layer and the softmax function to obtain the predicted probability distribution, and then the predicted result of the sentiment is output, which is specifically:
[0072] After using the linear layer to reduce the dimension of the sample vector (dimension is the number of polarity classification), the softmax function is used to predict the probability distribution of the corresponding sample vector:
[0073]
[0074] Where, represents the probability distribution predicted by the sentiment classification model; the softmax() function is used to convert the calculation result into a probability distribution with a range of [0, 1] and a sum of 1; W and b represent the model parameters of the linear layer;
[0075] The maximum value of the probability distribution is output as the predicted result of the sentiment.
[0076] The training of the sentiment classification model using the sentiment dataset is specifically:
[0077] To reduce the loss SCL and lossCE The joint loss is used as the direction, and the sentiment classification model is trained using the training set. That is, the parameters of the sentiment classification model are continuously updated through error backpropagation. Then, the trained sentiment classification model is verified and tested using the validation set and the test set.
[0078] Where, loss SCL Represents the comparative loss. CE This represents the classification loss, also known as the cross-entropy loss.
[0079] The formula for calculating the classification loss is:
[0080]
[0081] in, y represents the predicted probability distribution; y represents the true probability distribution; -log() represents the negative log-likelihood function.
[0082] A second preferred embodiment of the sentiment analysis method based on retrieval and contrastive learning of the present invention includes the following steps:
[0083] 1. Construct a sentiment dataset
[0084] First, unprocessed text data is collected from relevant internet platforms. This text data may contain a large amount of invalid content, requiring data cleaning. Taking Chinese text data collected from Weibo as an example: “finally finally managed to grab 5 masks, feeling as happy as if I'd won the lottery, I'm so stressed, masks are so hard to find @Survive”, 1) from a human perspective, “finally finally” expresses strong emotion, but this description can confuse computers' understanding of the text; 2) “@Survive” is a link operation, intended to push the post to a specific user, and is meaningless for sentiment analysis. In the data cleaning stage, the most direct and effective method is to design regular expressions to delete invalid content. Therefore, the cleaned text data becomes “finally managed to grab 5 masks, feeling as happy as if I'd won the lottery, I'm so stressed, masks are so hard to find”. For each cleaned text, named entity recognition technology is used to extract entities present in the text. For example, the entities that might be extracted from the above text data include “mask”, “mood”, “lottery”, and “I”. Professionals are hired to select relevant entities as sample aspects and label their sentiment polarity (including positive, neutral, and negative). In the example above, "mask" is selected as the aspect, and the sentiment polarity is "positive." Sentences with defined aspects and polarities are then constructed into a sample.
[0085] After obtaining a large number of samples, the samples are randomly divided into a training set, a validation set and a test set in a ratio of 8:1:1.
[0086] 2. Retrieving similar samples
[0087] In order to quickly obtain similar samples, Elasticsearch is used to perform inverted indexing on the samples in the training set, and each sample is expanded into a {key, value} pair, where the key is the sample aspect and its polarity, and the value is the sample itself. Taking a sample in the Laptops dataset as an example, the sample "The battery life was shorter than expected." is presented as {battery life: negative, "The battery life was shorter than expected."} in the Elasticsearch database. For a given target sample, similar samples are obtained by using exact matching, that is, the key of the target sample is compared with the keys of other samples in the Elasticsearch database, and samples with the same key are selected. Taking the above example as an example, the extracted similar samples are {battery life: negative, "Has a 2-3 hours battery life."}, {battery life: negative, "The battery life is probably an hour as best."}. We equip each sample with k similar samples, when the number of similar samples in the database is greater than k, we can randomly select k samples; when the number of similar samples is less than k, the target sample itself is used as a similar sample to supplement, but in the coding stage, the target sample will be represented differently by using different dropout methods.
[0088] 3. Designing a sentiment classification model based on contrastive learning
[0089] As shown in Figure 3 , the sentiment classification model is based on BERT, and specifically includes an input module, a BERT encoding module, a contrastive learning module and a linear classification module.
[0090] Input module: as shown in Figure 4 , token encoding E token , segment encoding E segment and position encoding E position jointly constitute the model input E input = E token +E segment +E position . Among them, token encoding xt is the word vector representation obtained after the sentence is segmented, and the use of pre-trained word vectors such as Glove, Bert, etc. can effectively improve the performance of the model; segment encoding E segment is to distinguish different components of the input, in this example, the input is a sentence and aspect splicing, so it contains two parts of the component; position encoding E position is the essential feature of text serialization, which enables the model to recognize the position relationship information between tokens.
[0091] BERT encoding module: as shown in Figure 5 , the BERT encoding module contains three components, which are multi-head attention mechanism, feedforward neural network and residual connection and normalization. The most important one is the multi-head attention mechanism, which is composed of multiple self-attention mechanisms to aggregate context information from different angles. Assuming that the input matrix is E input , the output matrix of the self-attention mechanism is:
[0092]
[0093] In the formula: Q, K, V are input matrices E input obtained through different mapping matrices, d k is the vector dimension, and the softmax() function obtains the normalized result of the input.
[0094] The outputs of multiple self-attention mechanisms are spliced to obtain the result of the current input after multi-head attention mechanism:
[0095]
[0096] In the formula: W is a mapping matrix that maps the spliced matrix to a result E input with the same dimension as E head .
[0097] Add the original input E input to the output E head of the multi-head attention mechanism to form a residual connection, so that the multi-head attention mechanism learns the change of the input rather than the input itself. Then, normalization processing is performed to prevent gradient disappearance / gradient explosion problem. In addition, after another feedforward neural network and residual connection, the deep sentence meaning is further extracted.
[0098] The above process is a one-layer BERT encoding process. In order to dig deep semantic information of text, 6 layers are stacked in the BERT-Base pre-training model, and 12 layers are stacked in the BERT-Large pre-training model. The final output of the BERT encoding module is a vector representation E output that can fully represent the rich connotation of the text.
[0099] Contrastive learning module: as shown in Figure 6 After the above search step, each target sample representation is represented by k similar sample representations These similar samples constitute the positive samples of the target sample. In addition, other samples in a batch are negative samples of the target sample. Therefore, in a batch, there are b original samples, k x b search samples, and a total of (k + 1) x b samples. For a non-search sample , it has k positive samples and k x (b - 1) negative samples. Calculate the similarity distance of positive and negative samples, and define the contrastive loss as increasing the similarity of positive samples and decreasing the similarity of negative samples:
[0100]
[0101] In the formula: sim() is a similarity measure function, and b is the batch size during model training.
[0102] During model training, the contrastive loss is optimized to enable the model to learn cross-sentence general knowledge and improve the representation ability of the model.
[0103] Linear classification module: during model training, after obtaining the target sample vector representation E output , on the one hand, the vector is input into the contrastive learning module to calculate the contrastive loss with the similar sample representation; on the other hand, the vector is input into the linear classification module to obtain the predicted classification loss. As shown in Figure 7 , E output will first undergo a linear transformation to convert the dimension to the same as the number of classifications, and then use the softmax() function to obtain the probability distribution of classification prediction:
[0104]
[0105] In the formula: W L , b L are the training parameters of the linear classification module.
[0106] After obtaining the predicted probability distribution, the cross-entropy is used to calculate the classification loss:
[0107]
[0108] 4. Text sentiment prediction
[0109] During the training phase, the model is trained using the training set, and the optimization loss is the joint loss of the contrastive loss and the classification loss: loss = loss CE + λlossSCL ;
[0110] wherein, lambda is a hyper-parameter, controlling the degree of influence of the contrastive loss on the model training.
[0111] Subsequently, the model performing best on the validation set is output as the final text sentiment prediction classifier for sentiment prediction of new input texts. To further observe the performance of the output model, the test set is used to evaluate the accuracy of the classifier. Specifically, the samples in the test set are input into the model, and after BERT encoding, the vector representation of the samples is obtained, and then the linear classification module is used to obtain the prediction probability distribution of the samples. The classification result with the maximum probability is taken as the sentiment polarity prediction result, and whether the classification is accurate is judged, and then the classification accuracy of the classifier is obtained.
[0112] In summary, the advantages of the present application are:
[0113] By preprocessing the obtained sentiment text data, extracting entities in each preprocessed sentiment text data, and labeling each entity to construct samples, a sentiment data set is generated, and then the samples in the sentiment data set are inverted indexed, each sample is retrieved with similar samples, the sentiment classification model based on neural network is trained using the sentiment data set, and at the same time, the contrastive learning technology is used to narrow the vector distance between each sample and its similar sample, finally, the trained sentiment classification model is used for sentiment analysis, that is, through retrieval and contrastive learning, the sentiment classification model learns general knowledge from similar samples, enhances the representation ability of the sentiment classification model, and obtains more robust sentence representation ability, which greatly improves the model sentiment representation ability, and further greatly improves the sentiment classification performance.
[0114] Although the specific embodiments of the present application are described above, those skilled in the art should understand that the specific examples described are only illustrative, and are not intended to limit the scope of the present application, and equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present application should be covered within the scope of the claims of the present application.
Claims
1. A method of sentiment analysis based on retrieval and contrastive learning, characterized in that: The method comprises the following steps: Step S10, obtaining a large amount of emotional text data, and preprocessing each of the emotional text data; Step S20, extracting entities in each of the preprocessed emotional text data, labeling each of the entities to construct samples, and further generating an emotional data set; Step S30, performing inverted indexing on the samples in the emotional data set by Elasticsearch, and retrieving similar samples for each of the samples; Step S40, creating an emotional classification model based on a neural network, training the emotional classification model using the emotional data set, and simultaneously narrowing the vector distance between each of the samples and similar samples using a contrast learning technique; the emotional classification model is used to perform word segmentation on the samples, converts the segmented input into a sample vector based on BERT, narrows the vector distance between each of the samples and similar samples using the contrast learning technique, inputs the sample vector into a linear layer and a softmax function to obtain a predicted probability distribution, and further outputs a predicted result of emotion; the loss formula for narrowing the vector distance between each of the samples and similar samples using the contrast learning technique is: ； wherein, represents a contrastive loss; represents a similarity measure function; n represents a batch size during training of the sentiment classification model; represents the i-th sample; represents the j-th sample; represents a similar sample of the i-th sample; e represents the base of the natural logarithm; Step S50, performing emotional analysis using the trained emotional classification model.
2. The sentiment analysis method based on retrieval and contrastive learning of claim 1, wherein: The step S10 specifically comprises: automatically obtaining a large amount of emotional text data, and preprocessing each of the emotional text data by removing hyperlinks, special characters, and repeated characters. 3.The method of claim 1, wherein: The step S20 specifically comprises: extracting entities in each of the preprocessed emotional text data using a named entity recognition technique, selecting part of the entities as aspect, labeling each of the entities for emotion polarity to construct samples, and further generating an emotional data set.
4. The sentiment analysis method based on retrieval and contrastive learning of claim 3, wherein: The step S30 specifically comprises: dividing the emotional data set into a training set, a validation set, and a test set based on a preset proportion; performing inverted indexing on the samples in the training set by Elasticsearch to construct a key-value list, with the key being the aspect and polarity of the sample and the value being the corresponding sample; retrieving similar samples for each sample based on the aspect and polarity.
5. The method of claim 1, wherein: The step of inputting the sample vector into a linear layer and a softmax function to obtain a predicted probability distribution and further output a predicted result of emotion specifically comprises: after dimension reduction of the sample vector by a linear layer, predicting the probability distribution of the corresponding sample vector using a softmax function: ； wherein, represents a probability distribution predicted by the sentiment classification model; The function is used to convert the calculation result into a probability distribution with a range of [0, 1] and a sum of 1; W and b represent model parameters of the linear layer. probability distribution The category corresponding to the maximum value is output as the sentiment prediction result.
6. The sentiment analysis method based on retrieval and contrastive learning of claim 4, wherein: The step of training the emotional classification model using the emotional data set specifically comprises: to reduce and The training set is used to train the emotion classification model, and the validation set and the test set are used to verify and test the trained emotion classification model. wherein, denotes the contrastive loss, denotes the classification loss.
7. The method of claim 6, wherein: the calculation formula of the classification loss is: ； where, denotes the predicted probability distribution; denotes the true probability distribution; denotes the negative log-likelihood function.