Training method of aspect-level sentiment analysis model based on self-supervised contrastive learning

Abstract

The application provides a kind of based on self-supervised contrast learning's aspect-level sentiment analysis model training method, system, storage medium and electronic equipment, it is related to natural language processing technical field.Compared with complex downstream structure, the framework only contains pooling operation and fully connected dimension reduction operation, realizes with less additional parameter fine-tuning BERT model.Different from single task fine-tuning mode, the framework takes self-supervised contrast learning as auxiliary task, aspect-level sentiment analysis task as main task, constructs the way of joint fine-tuning to fine-tune pre-training BERT model.Thereby, with the simple downstream structure, more accurate mining of each aspect sentiment in review text is realized.In addition, by means of the characteristics of maximizing similar entity distance and minimizing non-similar entity distance, the BERT model fine-tuned as downstream task can generate more representative features, and effectively assist the aspect-level sentiment analysis task to judge the sentiment conveyed by aspect word entity.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a training method and system of an aspect-level sentiment analysis model based on self-supervised contrast learning, a storage medium and an electronic device. BACKGROUND

[0002] With the rapid development of social media, platforms for Internet users to post personal opinions are gradually increasing, and a large amount of comment data begins to be generated and accumulated. How to use natural language processing (NLP) technology to mine valuable information from these data has always been the focus of the relevant field. Sentiment analysis, as an important task in the field of natural language processing, aims to mine the emotions expressed by users from these massive comment data.

[0003] Aspect-level sentiment analysis is a kind of fine-grained sentiment analysis, which aims to identify the sentiment tendency of the reviewer towards the review object from different aspects. For example, in a comment about a car, "The configuration style is very good, and I personally think the pedal is redundant and easy to dirty the pants legs", "configuration style" and "pedal" are two aspects that the reviewer is concerned about. Aspect-level sentiment analysis can accurately analyze whether the user expresses positive or negative emotions from "configuration style" and "pedal" by combining the context comment information. In the prior art, some people use BERT model for aspect-level sentiment analysis, for example:

[0004] BERT-AEN model (BERT attentional encoder network, BERT-AEN) designs a downstream structure based on attentional encoder network, which improves the classification effect of the model on the sentiment tendency of a specific object by associating aspect word features and text features. And BERT-TD (BERT Target-dependent, BERT-TD) designs an aspect-level sentiment analysis model based on BERT pre-training model and temporal dynamics (Temporal Dynamics).

[0005] However, no matter which model is used, a complex and diverse downstream structure is designed. Although the diverse downstream structure can effectively construct text features, it often coincides with part of the hidden layer of the BERT model, so that the overall structure of the model still needs to be further improved. In addition, considering that the structure of the BERT model itself is already relatively complete and has strong feature extraction capability, the effect gain produced by designing a complex downstream structure is not obvious. SUMMARY

[0006] (1) Technical problems solved

[0007] To overcome the shortcomings of the prior art, the application provides a training method and system of an aspect-level sentiment analysis model based on self-supervised contrast learning, a storage medium and an electronic device, and solves the technical problem of high downstream structure complexity.

[0008] (II) Technical solutions

[0009] To achieve the above object, the application is implemented by the following technical solutions:

[0010] A training method of an aspect-level sentiment analysis model based on self-supervised contrast learning, the aspect-level sentiment analysis model comprising a BERT encoding layer and a joint learning layer; the training method comprising:

[0011] S1, obtaining a review text and corresponding aspect words and aspect word sentiment labels;

[0012] S2, in the BERT encoding layer, comprising:

[0013] concatenating the review text as segment one and the aspect words as segment two, taking the concatenation result as the input of the BERT encoder, and obtaining the encoding features of each word;

[0014] performing a pooling operation on the encoding features of each word to obtain a text feature representation fused with overall context semantics and aspect word high-level feature representations with similar semantics in segment one and segment two, respectively;

[0015] S3, in the joint learning layer, comprising:

[0016] According to the predefined sentiment type, the text feature representation is sequentially input into a fully connected layer and a softmax classification layer to obtain the probability distribution of the aspect words, and the sentiment corresponding to the maximum probability value is taken as the prediction result; and according to the aspect word sentiment label and the prediction result, a first loss function is constructed;

[0017] According to the aspect word high-level feature representations with similar semantics in segment one and segment two, a second loss function is constructed;

[0018] According to the first loss function and the second loss function, a total loss function is constructed for jointly training the aspect-level sentiment analysis model until convergence.

[0019] Preferably, the process of obtaining the encoding features of each word specifically comprises:

[0020] defining the review text as {content1, content2,..., content N}, where N represents the number of input samples in the review text; for each input sample in the review text, two placeholders [CLS] and [SEP] are added at the beginning and end of the input sample, respectively, and the review text and aspect word are segmented by the [SEP] separator to form two segments of the input BERT model, represented as where the aspect word a τ+1 ,...,a τ+n is an aspect word entity in the review text;

[0021] After the input sample is encoded by BERT, the feature representation corresponding to each word in the context and aspect word two segments is generated

[0022]

[0023] where c corresponds to segment one, a corresponds to segment two, content i is the i-th input sample, n represents the number of words in the aspect word, m represents the number of words in the input sample, d bert represents the dimension of the word feature representation after encoding.

[0024] Preferably, the encoded features of the words are subjected to average pooling operation to obtain a text feature representation h c :

[0025]

[0026] and according to the position index of the aspect word in the input sample, the aspect word features in the encoded segment one and segment two are subjected to average pooling to obtain aspect word high-level feature representations with similar semantics, respectively and

[0027]

[0028]

[0029] where avg_pooling represents average pooling; [:] represents feature sequence slicing; left_index represents the number of words to the left of the aspect word in segment one.

[0030] Preferably, the first loss function is constructed using a cross-entropy loss function based on an L2 regularization term:

[0031]

[0032]

[0033] where LossABSA represents a cross-entropy loss function based on an L2 regularization term; y i represents an aspect word sentiment label, expressed in a one-hot vector; is the sentiment polarity predicted by the model, λ is a regularization parameter, and θ is a parameter to be trained in the model; w o represents a parameter matrix to be trained in the full connection layer, b o represents a bias term.

[0034] Preferably, the second loss function is constructed by minimizing the InfoNCE loss:

[0035]

[0036] where Loss SSCL represents minimizing the InfoNCE loss; represents and the similarity between the aspect word representations, and τ is a temperature coefficient for scaling the similarity values between feature representations; represents the similarity between the aspect word representations of other input samples in the current batch of review texts, i≠j.

[0037] Preferably, the total loss function is represented as:

[0038] Loss = Loss ABSA + Loss SSCL (8).

[0039] A training system of an aspect-level sentiment analysis model based on self-supervised contrastive learning, the aspect-level sentiment analysis model comprising a BERT encoding layer and a joint learning layer; the training system comprising:

[0040] An acquisition module for acquiring review texts and corresponding aspect words and aspect word sentiment labels;

[0041] An encoding module for the BERT encoding layer, comprising:

[0042] The review text is taken as a segment one, and the aspect word is taken as a segment two, the concatenation result is taken as an input of the BERT encoder, and the encoding features of each word are acquired;

[0043] The encoding features of each word are subjected to a pooling operation, and text feature representations fused with overall context semantics, aspect word high-level feature representations with similar semantics in the segment one and the segment two are respectively acquired;

[0044] A learning module for the joint learning layer, comprising:

[0045] According to the predefined emotion type, the text feature representation is sequentially input into a full connection layer and a softmax classification layer to obtain a probability distribution of the aspect word, and a sentiment corresponding to a maximum probability value is taken as a prediction result; and according to the aspect word sentiment label and the prediction result, a first loss function is constructed;

[0046] According to the high-level feature representation of the aspect words with similar semantics in the first segment and the second segment, a second loss function is constructed.

[0047] According to the first loss function and the second loss function, a total loss function is constructed for jointly training the aspect-level sentiment analysis model until convergence.

[0048] A storage medium storing a computer program for training an aspect-level sentiment analysis model based on self-supervised contrast learning, wherein the computer program causes a computer to perform the training method as described above.

[0049] An electronic device comprising:

[0050] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs include instructions for performing the training method as described above.

[0051] (III) Beneficial Effects

[0052] The present application provides a training method, system, storage medium and electronic device for an aspect-level sentiment analysis model based on self-supervised contrast learning.

[0053] Beneficial Effects:

[0054] In the present application, the aspect-level sentiment analysis model includes a BERT encoding layer and a joint learning layer. Compared with a complex downstream structure, the framework only includes a pooling operation and a full connection dimension reduction operation, and realizes fine-tuning of the BERT model with fewer additional parameters. Unlike the single-task fine-tuning method, the framework takes self-supervised contrast learning as an auxiliary task and aspect-level sentiment analysis as a main task, and constructs a joint fine-tuning method to fine-tune the pre-trained BERT model. Thus, a more accurate mining of each aspect sentiment in the review text is realized with a simple downstream structure. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and other accompanying drawings can be obtained by those of ordinary skill in the art without creative labor on the basis of these accompanying drawings.

[0056] Figure 1 A SSCL-ABSA model structure diagram provided for the embodiments of the present application;

[0057] Figure 2 A block diagram of a training method of an aspect-level sentiment analysis model based on self-supervised contrast learning provided for the embodiments of the present application;

[0058] Figure 3 A structural schematic diagram of a BERT encoder provided for the embodiments of the present application. DETAILED DESCRIPTION

[0059] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and other accompanying drawings can be obtained by those of ordinary skill in the art without creative labor on the basis of these accompanying drawings.

[0060] The embodiments of the present application provide a training method, system, storage medium and electronic device of an aspect-level sentiment analysis model based on self-supervised contrast learning, which solve the technical problem of high complexity of downstream structure.

[0061] The technical solutions in the embodiments of the present application are as follows to solve the above technical problems:

[0062] Mining comment information in the analysis background technology can help users, enterprises and the like to make business decisions. On the one hand, users analyze the evaluation indicators that they are concerned about through the existing evaluation information, and make commodity selection. On the other hand, enterprises find the problems existing in the existing services by mining the event features that users are concerned about, so as to improve user satisfaction.

[0063] In order to fully exert the advantages of the pre-trained BERT model and reduce the complexity of the downstream structure, for example, Figure 1As shown, an embodiment of the present application designs a joint fine-tuning framework (Self-Supervised Contrastive Learning Aspect-based Sentiment Analysis, SSCL-ABSA) combining self-supervised contrastive learning and aspect-level sentiment analysis tasks, which realizes more accurate mining of each aspect sentiment in review text with a simple downstream structure. The framework includes two modules, namely the BERT encoding layer and the joint learning layer.

[0064] Task definition: As a subtask of sentiment analysis, aspect-level sentiment analysis includes aspect term extraction (Aspect Term Extract, ATE) and aspect sentiment classification (Aspect Sentiment Classification, ASC). The purpose of aspect term extraction is to extract aspect entity words from review text. The current mainstream methods can be divided into three categories: unsupervised, weakly supervised and supervised. Among them, the unsupervised model is based on frequent pattern mining, syntactic rules, topic model and deep learning method. The weakly supervised model is based on word alignment and label propagation method. The supervised model regards the aspect term extraction task as a word-level sequence labeling problem. Early based on traditional feature engineering method to train sequence labeling classifier. Aspect sentiment classification is to analyze the sentiment expressed by the reviewer for a specific aspect through the context information of the review under the premise of given review text and aspect word entity.

[0065] The main purpose of the embodiment of the present application is to study aspect sentiment analysis, and a joint fine-tuning framework combining self-supervised contrastive learning and aspect-level sentiment analysis task is designed to complete the task, and the specific process is as follows.

[0066] In the BERT encoding stage, first, the review text and the aspect word are spliced into two segments to input the BERT encoder, and the feature representation of each word can be encoded. Then, the following three kinds of pooling operations are performed on the word features, including overall word feature pooling, segment one aspect word feature pooling and segment two aspect word feature pooling. Respectively, the text feature representation fused with the overall context semantics, and the high-level aspect word feature representation with similar semantics in segment one and segment two are obtained.

[0067] In the joint learning stage, on the one hand, the overall text feature is input into the classification layer to complete the sentiment classification task, and on the other hand, based on the aspect word feature pair, a self-supervised contrastive learning method is designed to complete the similarity matching task. The core purpose of the joint learning stage is to combine the two learning tasks, constantly fine-tune the BERT encoder, and promote the generation of more accurate feature representation in the encoding stage.

[0068] For better understanding of the above technical solutions, the above technical solutions will be described in detail below in combination with the drawings of the specification and specific embodiments.

[0069] Embodiments:

[0070] As Figure 2 shown, the embodiment of the application provides a training method of an aspect-level sentiment analysis model based on self-supervised contrastive learning, the aspect-level sentiment analysis model comprising a BERT encoding layer and a joint learning layer; the training method comprising:

[0071] S1, obtaining a comment text and corresponding aspect words and aspect word sentiment labels;

[0072] S2, in the BERT encoding layer, comprising:

[0073] S21, concatenating the comment text as segment one and the aspect words as segment two, taking the concatenation result as the input of the BERT encoder, and obtaining the encoding features of each word;

[0074] S22, performing a pooling operation on the encoding features of each word to obtain a text feature representation fused with overall context semantics, and aspect word high-level feature representations with similar semantics in segment one and segment two, respectively;

[0075] S3, in the joint learning layer, comprising:

[0076] S31, according to a predefined sentiment type, inputting the text feature representation into a fully connected layer and a softmax classification layer in turn to obtain a probability distribution of the aspect words, taking the sentiment corresponding to the maximum probability value as a prediction result, and constructing a first loss function according to the aspect word sentiment label and the prediction result;

[0077] S32, constructing a second loss function according to the aspect word high-level feature representations with similar semantics in segment one and segment two;

[0078] S33, constructing a total loss function according to the first loss function and the second loss function, for jointly training the aspect-level sentiment analysis model until convergence.

[0079] In the embodiment of the application, the aspect-level sentiment analysis model comprises a BERT encoding layer and a joint learning layer, compared with a complex downstream structure, the framework only contains a pooling operation and a fully connected dimension reduction operation, and the BERT model is fine-tuned with fewer additional parameters. Unlike the single-task fine-tuning method, the framework takes self-supervised contrastive learning as an auxiliary task and aspect-level sentiment analysis as a main task, and constructs a joint fine-tuning method to fine-tune the pre-trained BERT model. Thus, a more accurate mining of each aspect sentiment in the comment text is realized with a simple downstream structure.

[0080] Next, the steps of the above technical solutions will be described in detail:

[0081] In step S1, the review text and the corresponding aspect word and aspect word sentiment label are obtained.

[0082] In step S2, the BERT encoding layer is used.

[0083] First of all, it should be pointed out that compared with the traditional neural network model constructed word library table, the feature representation generated by the pre-training language model as an encoding tool in the NLP field has further improved effect. The pre-training language model represented by the BERT model often includes two stages of pre-training and fine-tuning. The BERT obtained by pre-training has achieved remarkable results in classification, question answering, translation and other tasks. If the pre-training language model is used based on the feature and fine-tuning strategy, the effect of the BERT model in the specific downstream task can be further improved. After the SSCL-ABSA uses the BERT model to encode the review text and aspect word to obtain the feature representation of each word, the average pooling operation is adopted to generate the text sentence vector representation and aspect word feature pair.

[0084] The BERT model uses the encoder of the bidirectional Transformer network as the hidden layer feature extractor, and designs two pre-training tasks of masked language model (Masked Language Model, MLM) and next sentence prediction (Next Sentence Prediction, NSP) to improve the overall semantic representation ability of the model. Among them, the MLM task is to predict the missing words in the text, and the NSP task is to jointly predict the sentence sequence relationship of the input text segment. Pre-training experiments are carried out on the model under a large amount of corpus, and the BERT pre-training model obtained has strong semantic information representation ability.

[0085] The structure of the BERT model is shown in Figure 3 , wherein [E1, E2,..., E N ] represents the word embedding vector input into the model, and after passing through the multi-layer bidirectional Transformer encoder (Trm) in turn, the feature representation [T1, T2,..., T NFirst, regarding the model input, for single-segment text input, placeholders [CLS] and [SEP] need to be added at the beginning and end of the text respectively, for example, <[CLS]segment[SEP]>. Meanwhile, "[CLS]segment1[SEP]segment2[SEP]" indicates two input text segments, with [SEP] as the input text segment separator. Second, the hidden layers of the BERT model mainly consist of the Transformer encoder, which is composed of two main modules: a multi-head self-attention mechanism and a feedforward neural network. The former fuses word feature information, helping to solve the problem of long-distance information loss. The latter helps improve the overall fitting ability of the model. Finally, the top-level output of the encoder generates word features in the text, which provide important support for downstream tasks. It is worth mentioning that the vector labeled with [CLS] after encoding incorporates the semantic representation of the overall context.

[0086] In existing technologies, the BERT model provides different BERT values ​​depending on the number of layers in the Transformer encoder. base and BERT large Two versions are available, and their specific parameter structures are shown in Table 1. This embodiment of the invention uses BERT. base As an encoder.

[0087] Table 1. Structural parameter settings for the BERT model

[0088]

[0089] Accordingly, S2 specifically includes:

[0090] S21. The comment text is concatenated as segment one and the aspect words are concatenated as segment two. The concatenation result is used as the input of the BERT encoder to obtain the encoding features of each word.

[0091] like Figure 1 As shown, the comment text is in the format {content1,content2,...,content...} N The batch input model is in the form of}, where N represents the number of input samples in the comment text. Referring to the previous section, for each input sample in the comment text, placeholders [CLS] and [SEP] are added at the beginning and end of the input sample, respectively. The [SEP] separator is used to separate the comment text and the aspect terms, forming two segments that are input into the BERT model, represented as follows: Among them, aspect word a τ+1 ,...,a τ+n It is an aspect-related entity in the comment text.

[0092] After the input sample is encoded by BERT, the feature representation corresponding to each word in the two segments of context and aspect word is generated

[0093]

[0094] wherein c corresponds to segment one, a corresponds to segment two, content i is the i-th input sample, n represents the number of words in the aspect word, m represents the number of words in the input sample, d bert represents the dimension of the word feature representation after encoding.

[0095] S22, the encoded features of the words are pooled to obtain text feature representation that integrates overall context semantics, aspect word high-level feature representation that integrates similar semantics in segment one and segment two, respectively.

[0096] In the prior art, there are various ways to obtain the overall vector representation of the text based on the word feature representation generated by BERT encoding. Among them, the three widely used methods are as follows: the vector represented by the [CLS] placeholder; the last layer word feature is pooled; the first layer and the last layer word feature is pooled. The pooling operation includes maximum pooling and average pooling.

[0097] For example, the average pooling operation is performed on the encoded features of the words to obtain the text feature representation g c :

[0098]

[0099] and according to the position index of the aspect word in the input sample, the average pooling is performed on the aspect word features in the encoded segment one and segment two to obtain the aspect word high-level feature representation with similar semantics, which are and

[0100]

[0101]

[0102] wherein avg_pooling represents average pooling; [:] represents feature sequence slicing; left_index represents the number of words to the left of the aspect word in segment one; and constitute the aspect word feature representation sample pair <h a_seg1 , h a_seg2 >.

[0103] In step S3, the joint learning layer is used.

[0104] Pre-training language models trained on large-scale corpus can achieve good results in general tasks. In order to further improve the effect of the pre-training model on specific tasks in a specific field, it is often necessary to add a downstream task to fine-tune it. As described earlier, the pre-training process of the BERT model uses the MLM and NSP two tasks to jointly learn, which realizes the complementary effect in the model parameter adjustment process. In the fine-tuning stage, multi-task joint learning is used, which helps the model learn more specific domain knowledge and helps improve the effect of the downstream main task.

[0105] The embodiment of the application takes the aspect-level sentiment analysis task as the main task, and jointly fine-tunes the BERT pre-training model parameters in a self-supervised contrast learning manner. This kind of joint learning method helps to improve the effect of BERT model encoding and generating word representation, and further promotes the improvement of aspect-level sentiment analysis task effect.

[0106] Specifically, the S3 specifically comprises:

[0107] S31, according to the pre-defined sentiment type, the text feature representation is sequentially input into the full connection layer, the softmax classification layer, and the probability distribution of the aspect word is obtained. As shown in formula (6), wherein w o is the parameter matrix that needs to be trained for the full connection layer, b o is the bias term, and the sentiment corresponding to the maximum probability is taken as the prediction result.

[0108] Step S31 is essentially an aspect-level sentiment analysis process: as shown in formula (5), the first loss function is constructed by using a cross-entropy loss function based on an L2 regularization term according to the aspect word sentiment label and the prediction result in this step:

[0109]

[0110]

[0111] Wherein, Loss ABSA represents a cross-entropy loss function based on an L2 regularization term; y i represents the aspect word sentiment label, represented by a one-hot vector; is the sentiment polarity predicted by the model, and λ is the regularization parameter and θ is the parameter to be trained in the model.

[0112] S32, according to the high-level feature representation of the aspect words with similar semantic in the segment one and the segment two, a second loss function is constructed.

[0113] Step S32 is essentially a self-supervised contrastive learning process: the core purpose of fine-tuning the pre-trained language model in a contrastive learning manner is to minimize the distance between similar features and maximize the distance between dissimilar features when the model generates features. The effect of fine-tuning the model in a contrastive learning manner depends largely on the quality of constructing positive and negative samples. One optional way is to input a batch of samples into the encoder after data augmentation, and the features obtained from the same picture are used as positive samples, and the features obtained from other pictures are used as negative samples. Finally, the encoder is required to minimize the InfoNCE loss during fine-tuning, so as to continuously learn the ability to verify similar features and distinguish dissimilar features. Following the above idea, an embodiment of the present application proposes a way to construct positive and negative samples for fine-tuning model parameters in a contrastive learning manner.

[0114] In the current batch of review texts, the input sample content i is composed of a fragment one and a fragment two spliced by text and aspect words, wherein both fragments contain aspect words. After the sample content i is encoded by BERT and the aspect word features are pooled, the aspect word feature representations with similar semantics are obtained, i.e., pair which are used as similar features. In addition, the aspect word representations obtained from other samples in the same batch are used as dissimilar features wherein i≠j.

[0115] Similarly, an embodiment of the present application uses the minimization of InfoNCE loss as the objective function, which is specifically shown in formula (7), wherein sim(i,j) represents the similarity value between features i and j, τ is called a temperature coefficient and is used to scale the similarity value between feature representations. N samples are used in each batch of data, and N pairs of aspect word representations are obtained. If the InfoNCE loss is to be minimized, the model is required to minimize the similarity distance between aspect word representations in the same sample, and to maximize the similarity distance between aspect word representation i and other 2*(N-1) aspect word representations in each sample.

[0116]

[0117] wherein Loss SSCL represents the minimization of InfoNCE loss; represents the similarity between and represents the similarity between the aspect word representations obtained from other input samples in the current batch of review texts.

[0118] S33, constructing a total loss function according to the first loss function and the second loss function, and using the total loss function to jointly train the aspect-level sentiment analysis model until convergence.

[0119] ​Step S33 is essentially a joint multi-task learning process: the aspect-level sentiment analysis task and the contrastive learning task share the word feature representation generated by the BERT encoder, while the two tasks jointly fine-tune the BERT model parameters. The specific fine-tuning process is reflected in the two task training processes, which are fed back through the sum of the training losses to achieve the effect of parameter updating. Formula (8) shows the joint target loss function adopted by the SSCL-ABSA model, which includes two parts: the cross-entropy loss function for completing aspect-level sentiment analysis and the InfoNCE loss function adopted by the similarity matching task based on self-supervised contrastive learning.

[0120] Loss=Loss ABSA +Loss SSCL (8)。

[0121] The embodiment of the application takes advantage of the characteristics of contrastive learning of maximizing the distance between similar entities and minimizing the distance between non-similar entities, and the BERT model fine-tuned as a downstream task can generate more representative features, and effectively assist the aspect-level sentiment analysis task to judge the sentiment conveyed by the aspect word entity.

[0122] In order to verify the effectiveness of the SSCL-ABSA model provided by the embodiment of the application, three public aspect-level sentiment analysis data sets are used for experiments.

[0123] (1) Experimental data

[0124] In order to verify the effectiveness of the SSCL-ABSA model, the embodiment of the application uses three public aspect-level sentiment analysis data sets for experiments. Including the Restaurant and Laptop data sets provided by the international semantic evaluation competition (SemEval 2014), and the Twitter data set collected by Dong et al. in the paper "Adaptive recursive neural network for target-dependent twitter sentiment classification". The sentiment polarity of the aspect words in the three data sets includes negative, neutral and positive. After the training set and the test set are divided, Table 2 shows the number of samples used to train and test the model in different data sets, and more detailedly describes the number of samples contained by different sentiment polarities.

[0125] Table 2 Data Statistics

[0126]

[0127] (2) Parameter settings

[0128] The encoder adopted by the SSCL-ABSA model is BERTbase The encoding generated feature dimension d bert 768. During model training, 32 samples were used per batch, AdamW was used as the optimizer, and the model parameters were constantly optimized at a learning rate of 2e-5. In addition, in order to reduce the risk of overfitting during model training, the random inactivation rate was set to 0.1, and the early stopping strategy was adopted. In the model verification stage, the classification accuracy (accuracy) and macro-F1 (macro-f1) were used as the model evaluation indicators.

[0129] (3) Experimental environment

[0130] The experimental environment of the embodiment of the application is shown in Table 3:

[0131] Table 3 Experimental environment

[0132]

[0133] (4) Verify the classification effect of the model

[0134] The experiment uses two evaluation indicators, accuracy and macro-F1, to represent the classification effect of the model. As shown in formula 12, accuracy represents the ratio of the number of samples predicted correctly to the total number of samples, where T represents the number of samples predicted correctly, and N represents the total number of samples. As shown in formula 13, Macro-F1 represents the harmonic mean of precision and recall of each class, where TP represents the number of predicted positive samples, FP represents the number of predicted false positive samples, TN represents the number of predicted correct negative samples, and FN represents the number of predicted false negative samples.

[0135]

[0136]

[0137] In order to verify the effectiveness of the classification effect of the model provided by the embodiment of the application, the following aspect-level sentiment analysis model is selected for analysis and comparison. The experimental results are shown in Table 4, and the various indicators of the application on the two data sets are optimal, which can verify the effectiveness of the application in this task.

[0138] Table 4 Model classification effect

[0139]

[0140] Note: The bolded data indicates the best effect

[0141] The embodiment of the application provides a training system for an aspect-level sentiment analysis model based on self-supervised contrast learning, which includes a BERT encoding layer and a joint learning layer. The training system comprises:

[0142] an acquisition module configured to acquire a review text and corresponding aspect words and aspect word sentiment labels;

[0143] an encoding module configured to encode the review text in a BERT encoding layer, including:

[0144] concatenate the review text as a first segment and the aspect words as a second segment, and take the concatenation result as an input of a BERT encoder to obtain encoding features of each word;

[0145] perform a pooling operation on the encoding features of each word to obtain a text feature representation fused with overall context semantics and aspect word high-level feature representations with similar semantics in the first segment and the second segment, respectively;

[0146] a learning module configured to learn in a joint learning layer, including:

[0147] input the text feature representation into a fully connected layer and a softmax classification layer in sequence to obtain a probability distribution of the aspect words, take a sentiment corresponding to a maximum probability value as a prediction result, and construct a first loss function according to the aspect word sentiment labels and the prediction result;

[0148] construct a second loss function according to the aspect word high-level feature representations with similar semantics in the first segment and the second segment;

[0149] construct a total loss function according to the first loss function and the second loss function, and use the total loss function to jointly train the aspect-level sentiment analysis model until convergence.

[0150] An embodiment of the present application provides a storage medium storing a computer program for training an aspect-level sentiment analysis model based on self-supervised contrast learning, wherein the computer program causes a computer to execute the training method.

[0151] An embodiment of the present application provides an electronic device, including:

[0152] one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs include a program for executing the training method.

[0153] It can be understood that the training system, the storage medium and the electronic device based on self-supervised contrast learning of the aspect-level sentiment analysis model provided by the embodiments of the present application correspond to the training method of the aspect-level sentiment analysis model based on self-supervised contrast learning provided by the embodiments of the present application, and the explanation, examples and beneficial effects of the related content can refer to the corresponding part in the training method, which will not be repeated here.

[0154] Compared with the prior art, the following beneficial effects are achieved:

[0155] 1、In the embodiment of the present application, the aspect-level sentiment analysis model includes a BERT encoding layer and a joint learning layer, compared with a complex downstream structure, the framework only contains a pooling operation and a full connection dimension reduction operation, and the BERT model is fine-tuned with fewer additional parameters. Unlike the single-task fine-tuning method, the framework takes self-supervised contrastive learning as an auxiliary task and aspect-level sentiment analysis as a main task, and constructs a joint fine-tuning method to fine-tune the pre-trained BERT model. Thus, more accurate mining of aspects and sentiments in the review text is realized with a simple downstream structure.

[0156] 2、In the embodiment of the present application, the characteristics of maximizing the distance of similar entities and minimizing the distance of non-similar entities are used, and the fine-tuned BERT model after the downstream task can generate more representative features, and effectively assist the aspect-level sentiment analysis task to judge the sentiment conveyed by the aspect entity.

[0157] It should be noted that in the embodiments of the present application, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. Without more limitations, the element defined by the sentence "including a" does not exclude the presence of another identical element in the process, method, article or equipment including the element.

[0158] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit it; although the foregoing embodiments of the present application have been described in detail, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training an aspect-level sentiment analysis model based on self-supervised contrastive learning, characterized in that, The aspect-level sentiment analysis model comprises a BERT encoding layer and a joint learning layer; the training method comprises: S1, obtaining a review text and corresponding aspect words and aspect word sentiment labels; S2, in the BERT encoding layer, comprising: concatenating the review text as segment one and the aspect words as segment two, taking the concatenation result as the input of the BERT encoder to obtain the encoding features of each word; performing a pooling operation on the encoding features of each word to obtain text feature representation fused with overall context semantics, aspect word high-level feature representation with similar semantics in segment one and segment two, respectively; S3, in the joint learning layer, comprising: according to the predefined sentiment types, inputting the text feature representation into a fully connected layer and a softmax classification layer in sequence to obtain the probability distribution of the aspect words, taking the sentiment corresponding to the maximum probability value as the prediction result, and constructing a first loss function according to the aspect word sentiment labels and the prediction result; constructing a second loss function according to the aspect word high-level feature representation with similar semantics in segment one and segment two; constructing a total loss function according to the first loss function and the second loss function, and jointly training the aspect-level sentiment analysis model until convergence.

2. The training method of an aspect-level sentiment analysis model based on self-supervised contrastive learning according to claim 1, wherein, The process of obtaining the encoding features of each word specifically comprises: The comment text is defined as {content1, content2,..., content N}, where N represents the number of input samples in the comment text; for each input sample in the comment text, two placeholders [CLS] and [SEP] are added at the beginning and end of the input sample, respectively, and the comment text and aspect words are separated by the [SEP] separator to form two segments of the input BERT model, represented as [CLS], [SEP], [SEP], wherein the aspect words a τ+1 ,...,a τ+n are aspect word entities in the comment text; After the input sample is encoded by BERT, a feature representation corresponding to each word in the two segments of context and aspect words is generated wherein c corresponds to the first segment, a corresponds to the second segment, content i For the i-th input sample, n represents the number of words in the aspect word, m represents the number of words in the input sample, d bert represents the dimension of the word feature representation after encoding.

3. The training method of an aspect-level sentiment analysis model based on self-supervised contrastive learning according to claim 2, wherein, An average pooling operation is performed on the encoding features of each word to obtain a text feature representation h that integrates the overall contextual semantics c : and according to the position index of the aspect word in the input sample, average pooling is performed on the aspect word features in the encoded segment one and segment two to obtain high-level feature representations of the semantically similar aspect words, respectively and wherein avg_pooling represents average pooling, [:] represents feature sequence slicing, and left_index represents the number of words to the left of the aspect words in segment one.

4. The aspect-level sentiment analysis model training method based on self-supervised contrastive learning according to claim 3, wherein, The first loss function is constructed by using a cross-entropy loss function based on an L2 regularization term: wherein, Loss ABSA represents a cross-entropy loss function based on an L2 regularization term; y i represents an aspect word sentiment label, represented in a one-hot vector; is a model predicted sentiment polarity, λ is a regularization parameter, θ is a parameter to be trained in the model; w o represents a parameter matrix to be trained in the fully connected layer, b o represents a bias term.

5. The method of claim 4, wherein the aspect-level sentiment analysis model is trained based on self-supervised contrastive learning. The second loss function is constructed by using a minimum InfoNCE loss: where Loss SSCL represents minimizing the InfoNCE loss; represents and the similarity between the representations, and τ is a temperature coefficient used to scale the similarity values between representations; represents the similarity between the aspect word representations of other input samples in the current batch of review texts, i≠j.

6. The training method of the aspect-level sentiment analysis model based on self-supervised contrast learning according to claim 5, wherein the total loss function is represented as: Loss = Loss ABSA + Loss SSCL (8).

7. A training system of an aspect-level sentiment analysis model based on self-supervised contrastive learning, characterized in that, The aspect-level sentiment analysis model comprises a BERT encoding layer and a joint learning layer; the training system comprises: an acquisition module configured to acquire a review text and corresponding aspect words and aspect word sentiment labels; an encoding module configured to, in the BERT encoding layer, comprising: concatenating the review text as segment one and the aspect words as segment two, taking the concatenation result as the input of the BERT encoder to obtain the encoding features of each word; performing a pooling operation on the encoding features of each word to obtain text feature representation fused with overall context semantics, aspect word high-level feature representation with similar semantics in segment one and segment two, respectively; a learning module configured to, in the joint learning layer, comprising: according to the predefined sentiment types, inputting the text feature representation into a fully connected layer and a softmax classification layer in sequence to obtain the probability distribution of the aspect words, taking the sentiment corresponding to the maximum probability value as the prediction result, and constructing a first loss function according to the aspect word sentiment labels and the prediction result; constructing a second loss function according to the aspect word high-level feature representation with similar semantics in segment one and segment two; constructing a total loss function according to the first loss function and the second loss function, and jointly training the aspect-level sentiment analysis model until convergence.

8. A storage medium, characterized by The storage has a computer program for training an aspect-level sentiment analysis model based on self-supervised contrastive learning, wherein the computer program causes a computer to perform the training method according to any one of claims 1-6.

9. An electronic device, comprising: Comprise: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs comprise a program for executing the training method according to any one of claims 1-6.

Images