News event sentiment analysis system based on NLP technology

Abstract

The invention describes a news event sentiment analysis system based on an NLP technology, which can perform sentiment analysis on chapter-level long news, and is mainly composed of the following parts: S100: news text segmentation; s200, a news corpus is constructed, a NewsBert model suitable for news text information acquisition is obtained by performing fine tuning on a Bert model, segmented segments are respectively input into the trained model, each segmented segment obtains a feature vector, and an original text is expressed as a feature sequence; s300, inputting the feature sequence into a BiLSTM network for classification prediction to obtain emotion tendency prediction of the current segmentation number, and using a news data set with emotion annotation to perform fine adjustment on the BiLSTM network to improve classification accuracy; and S400, prediction result fusion: fusing the emotion tendency prediction results under different segmentation numbers, and obtaining the final news event emotion tendency through a voting method.

Description

Technical Field

[0001] The present invention relates to the technical field of computer technology, and in particular to a news event sentiment analysis system based on NLP technology. Background Art

[0002] Analyzing the emotions of news events is important in many aspects. For example, through sentiment analysis, we can better understand the public's emotional attitude towards specific events or topics, which helps governments, enterprises, media, etc. understand social public opinion and adjust strategies or take measures in a timely manner; sentiment analysis can be used to monitor public opinion trends. Understanding the public's emotional tendencies can help quickly discover and respond to potential negative public opinion to mitigate possible reputation damage; sentiment analysis can be used to conduct market research. Sentiment analysis of news reports on products or services can help companies understand market feedback. Positive news reports may increase the popularity of products or services, while negative reports may require companies to take measures to improve their image; sentiment analysis can be used to make investment decisions. According to the emotional tendencies of news events, possible market trends can be predicted to maximize investment benefits; sentiment analysis can be used to assist policy making and public decision-making. The government can use sentiment analysis to more comprehensively understand the public's emotional attitudes towards policies and decisions, which helps to formulate policies that are more in line with social expectations. In general, news event sentiment analysis provides a way to gain an in-depth understanding of public emotions, public opinion, and social dynamics. It is of great value to decision-making, management, and public relations. This analysis helps to better understand and respond to social reactions in the era of information explosion;

[0003] In recent years, natural language processing (NLP) technology has also made great progress. Especially since the advent of the Transformers model based on the self-attention mechanism, large language models have made breakthrough progress in many fields and have also been put into practice. In text classification and sentiment analysis, many models can achieve good results on short texts. However, the length of many news event texts reaches several thousand words, which is much higher than the input length limit of most models. Even models without hard restrictions will cause the classification effect to drop rapidly due to the difficulty in extracting information from long texts. On the other hand, for large language models, long texts will increase the computational burden, causing the reasoning time to increase rapidly with the length of the text. The truncation method commonly used to process long texts will lose a lot of useful information and reduce the accuracy of prediction. In order to solve the above problems, a news event sentiment analysis system based on NLP technology is proposed. Summary of the invention

[0004] In order to solve the problem of low accuracy of sentiment classification of long news, the present invention provides a news event sentiment analysis system based on NLP technology, which can perform sentiment analysis on long news at the chapter level, and the sentiment is divided into positive, negative and neutral;

[0005] S100: Segment the long news text. Since the model has a limit on the length of the input sequence, and too long input will affect the model performance, and simple truncation processing will cause information loss, affecting the accuracy of model prediction, the present invention divides the long text into short texts of similar length and inputs them into the model in segments. Different numbers of segmentation can obtain different granularity information. Longer segmentation segments are convenient for obtaining comprehensive information in a single segment, while shorter segments can better extract local information. The present invention uses different numbers of segmentation and fuses the prediction values ​​of different segmentation lengths to more effectively extract the information of the long news text and improve the prediction accuracy.

[0006] S200: We use our own collected news corpus to perform secondary pre-training on the pre-trained language model to obtain the NewsBert model suitable for news text information acquisition. We use the pre-trained large language model to extract text features. The pre-trained model uses the Transformers architecture based on the self-attention mechanism. Transformers allows the model to directly view all positions in the input sequence and process global information in a single step. For an input text x, its self-attention output can be defined as formula (1):

[0007]

[0008] Among them, Q, K and V are query, key and value matrices respectively, which are usually obtained by multiplying the input x with the weight matrix W, such as formula (2), formula (3) and formula (4):

[0009] Q = xW Q (2)

[0010] K=xW K (3)

[0011] V=xW v (4)

[0012] d_k is the dimension of the key, and the attention output gives a weighted representation of the input, which is able to capture long-range dependencies within the sequence.

[0013] In order to enable the model to focus on different parts of the input, multi-head attention is introduced. For h heads, multi-head attention can be defined as formula (5):

[0014] MulitHead(Q,K,V)=Concat(head1,...,head h )W O (5)

[0015] Among them, each head is formula (6):

[0016] head i =Attention(QW Qi ,KW Ki ,VW Vi ) (6),

[0017] After calculation through the self-attention mechanism, it is output to the feedforward neural network. The feedforward neural network contains two layers of linear transformation and a ReLU activation function. The ReLU function form is formula (7):

[0018] ReLU(x)=max(x,0) (7),

[0019] The Transformers architecture includes two parts, encoding and decoding, wherein the decoding part is mostly used for text generation tasks. The present invention only uses the encoding part for feature extraction, and uses the Bert model as a pre-training model for secondary pre-training. The news corpus used in the secondary training is unlabeled news data crawled from the Internet. After cleaning and filtering, special characters and sensitive information are removed. The secondary training adopts an unsupervised training method of continuous mask prediction to replace continuous fragments in the original text with special characters [MASK]. The goal of model training is to restore the replaced characters. After the training is completed, the NewsBert model is obtained, which can effectively extract features from news texts.

[0020] After the secondary training is completed, NewsBert is used to extract features from the text. For each short text segment segmented from the long news text, the news title and the short text are concatenated and separated using a special character [SEP]. A special character [CLS] is added before the input sequence as a representative of the entire sequence. After each text is input into the NewsBert model, the vector corresponding to [CLS] will be used as the feature vector of the input short text segment. For the original news text, after each segmentation, the model is input to obtain a feature vector sequence with a length equal to the number of segmented segments.

[0021] S300: After the long text is segmented, the model in S100 outputs a feature vector sequence, and a recurrent network is used to classify the sequence. The unidirectional recurrent network belongs to an autoregressive model. Each step can only obtain the input information of the current step and the previous step, but the information of the subsequent step cannot be obtained. Since the classification task does not require the generation of new text, there is no information leakage problem. Therefore, the present invention uses a bidirectional recurrent network BiLSTM to classify the sequence. The network output is three units, corresponding to three emotional tendencies. The output is used to obtain the probability distribution using the softmax function. The model predicts the category with the highest probability. The softmax function form is formula (8):

[0022]

[0023] Where i is the current output category, x is the value of the neural network output layer, and n is the number of output categories. In the present invention, n is 3.

[0024] Multiple parallel NewsBert+BiLSTMs constitute the model architecture used in the present invention. In order to classify more accurately, the model needs to be trained and the parameters adjusted. The present invention fixes the parameters of the NewsBert part and only updates the parameters of the BiLSTM model. After the model training is completed, the emotional tendency can be obtained by inputting a piece of news;

[0025] S400: For the original news text, different segmentation lengths are used to obtain different segmentation results. Each segmentation result obtains a sentiment prediction through the NewsBert+BiLSTM model. After obtaining multiple sentiment prediction results of the original news text, the present invention improves the prediction accuracy by fusion of the results. The fusion method used by the present invention is to obtain a group of predictions with low correlation coefficients through correlation analysis, and then predict the final result through voting. The sentiment tendency with the most votes is used as the final output. When the scores of different tendencies are the same, the prediction result is neutral.

[0026] Beneficial effects of the present invention:

[0027] The present invention uses different numbers of segmentations and fuses the prediction values ​​of different segmentation lengths to more effectively extract information from long news texts and improve prediction accuracy. After obtaining multiple emotion prediction results of the original news text, the present invention improves the prediction accuracy by fusing the results. The fusion method used by the present invention is to obtain a group of predictions with low correlation coefficients through correlation analysis, and then predict the final result through voting. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flow chart of the news event sentiment analysis system based on NLP technology of the present invention;

[0029] Figure 2 It is a schematic diagram of the overall architecture of the usage model of the present invention;

[0030] Figure 3 It is a schematic diagram of the training and prediction process of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] Example 1

[0033] In order to solve the problem of low accuracy of sentiment classification of long news, the present invention provides a news event sentiment analysis system based on NLP technology, which can perform sentiment analysis on long news at the chapter level. The sentiment is divided into positive, negative and neutral. The system process is as follows: Figure 1 As shown:

[0034] S100: First, the long news text is segmented into shorter text segments of similar length. The number of segments used in this example is 5-15, that is, the original text is segmented into 5, 6, 7, 14, and 15 segments by length. To avoid dividing a complete sentence into two segments, if the breakpoint is not at the end of the sentence when segmenting by length, it is postponed to the end of the sentence. Due to the limitation of segmenting the whole sentence, the length of each segment after segmentation is not exactly the same, but the difference is not large. For each segmented short text segment, the news title and the short text are concatenated and separated by a special character [SEP], and a special character [CLS] is added before the input sequence;

[0035] S200: Use the news corpus collected by the user to perform secondary pre-training on the pre-trained language model to obtain the NewsBert model suitable for obtaining news text information. Use the Bert model as the pre-training model for secondary pre-training. The news corpus used in the secondary training is unlabeled news data crawled from the Internet, covering news from multiple fields such as finance, technology, entertainment, and current affairs from 2000 to 2023. After cleaning and filtering, the data is cleaned and filtered to remove special characters and sensitive information. The secondary training first uses BPE (Byte Pair Encoding, an unsupervised word segmentation algorithm based on statistics) to train word segmentation, and merges the obtained word segmentation results with the vocabulary of the pre-trained Bert model to obtain the vocabulary of NewsBert. Then, the unsupervised training method of continuous mask prediction is used to replace continuous fragments in the original text with special characters [MASK]. The goal of model training is to restore the replaced characters, and the cross entropy classification loss is used as the objective function to minimize it. The cross entropy function form is formula (9):

[0036]

[0037] Where p is the predicted probability, q is the true probability, and C is the number of predicted categories. In this example, C is the number of words in the model vocabulary.

[0038] After the second training, NewsBert is obtained. The model structure is the same as the pre-trained Bert, and the output vector dimension is 768. Each text segmented in S100 is input into the NewsBert model, and the vector corresponding to [CLS] is obtained as the feature vector of the input short text segment. For the original news text, after each segmentation, the short text is input into the model respectively to obtain a feature vector sequence with a length of the number of segmented segments. In this example, each news corresponds to 11 vector sequences with lengths of 5-15 respectively.

[0039] S300: After the long text is segmented, the parallel NewsBert model outputs a feature vector sequence, and a recurrent network is used to classify the sequence. In this example, BiLSTM is used to classify the sequence. The network structure is as follows:

[0040] The input layer has a dimension of 768 and accepts the output from NewsBert;

[0041] The number of hidden layers is 3, the dimension of each layer is 256, and ReLU is used as the activation function.

[0042] The output layer dimension is 3, corresponding to three emotional tendencies.

[0043] A dropout layer is added between different layers of the neural network to randomly discard some neurons to avoid overfitting. In this example, the dropout rate is set to 0, 2,

[0044] Multiple parallel NewsBert+BiLSTM constitute the model architecture used in the present invention, such as Figure 2 As shown, in order to classify more accurately, the model needs to be trained and the parameters adjusted. In this example, the parameters of the NewsBert part are fixed, and only the parameters of the BiLSTM model are updated. The training process uses the strategy of gradient descent combined with back propagation, and the loss function uses the cross entropy function to minimize the classification loss.

[0045] S400: Each segmentation of the original news text is passed through the NewsBert+BiLSTM model to obtain a sentiment prediction. In this example, 11 prediction results are obtained for each news item. The prediction accuracy is improved by fusion of results. This example uses the voting method to fuse the results. Specifically, the correlation coefficients of the prediction results of different segmentations on the labeled data set are calculated. Correlation coefficients lower than 0 and 95 are considered to have low correlation. The number of segmentations obtained in this example is 6, 7, 8, 10, 11, 12, and 14. These segmentation numbers are used as the results of the model, that is, each news item obtains 7 prediction results. These 7 prediction results are used to predict the final news sentiment tendency through voting. When the scores of different tendencies are the same, the sentiment tendency predicted in this example is neutral. Finally, the method used in this example can achieve a prediction accuracy of more than 85% on the test data. Figure 3Describes the overall process of training and prediction in this example.

[0046] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present invention, all of which belong to the protection of the present invention.

Claims

1. A news event sentiment analysis system based on NLP technology, characterized by: The following steps are involved: S100: News text segmentation, which segments long news into shorter text segments. Different segmentation numbers are used to obtain different segmentation results, so as to better obtain information of different granularities. S200: Build a news corpus by using Bert (Bidirectional Encoder The Representations from Transformers (Bidirectional Encoding Representations based on Transformers architecture) model is fine-tuned to obtain the NewsBert model suitable for news text information acquisition. The segmented fragments are input into the NewsBert model, and each segmented fragment is obtained as a feature vector. The original text is represented as a feature sequence. S300: Input the feature sequence into the BiLSTM (Bidirectional Long Short Term Memory) network for classification, predict the sentiment tendency, use the sentiment-annotated news data set to fine-tune the BiLSTM network, fix the parameters of the NewsBert part, and after the training is completed, cut the long text into pieces and input them into the NewsBert+BiLSTM network to obtain a sentiment prediction; S400: Prediction result fusion: each segmentation number of the original text obtains a sentiment prediction, and the sentiment tendency prediction results under different segmentation numbers are fused, and the final sentiment tendency of the news event is obtained through voting.

2. The news event sentiment analysis system based on NLP technology according to claim 1 is characterized in that: Step S100 uses different segmentation numbers to obtain multiple segmentation results, each segmentation result is composed of shorter text segments of similar length. Different segmentation numbers can obtain different granularity information. A smaller number of segmentation will obtain longer segmented segments, which is convenient for obtaining comprehensive information in a single segment; while a larger number of segmentation will obtain shorter segmented segments, which can better extract local information. By using different segmentation numbers and fusing the predicted values, the prediction accuracy of the model is improved.

3. The news event sentiment analysis system based on NLP technology according to claim 2 is characterized in that: Step S200 uses the collected news corpus to perform secondary pre-training on the pre-trained Bert model to obtain a NewsBert model suitable for obtaining news text information. The news corpus is news data crawled from the Internet, covering news from multiple fields from 2000 to 2023, and sensitive information is removed after data cleaning and filtering. The secondary training uses an unsupervised training method of continuous mask prediction to obtain a NewsBert model.

4. The news event sentiment analysis system based on NLP technology according to claim 3 is characterized in that: Step S200 concatenates the news title with the text segmented in step S100, separates them with special characters [SEP], and adds special characters [CLS] in front. The concatenated text is input into the NewsBert model trained in step S200. Each text segment is passed through the model to obtain a feature vector. The original text can be represented as a feature vector sequence. For each segmentation of the original long text, a feature vector sequence with a length equal to the number of segmented segments is obtained.

5. The news event sentiment analysis system based on NLP technology according to claim 4 is characterized in that: Step S300 uses the BiLSTM network to classify the feature vector sequence and train the BiLSTM network. NewsBert is integrated with the BiLSTM network. The overall network input is text and outputs three units, corresponding to three emotional tendencies. The data set used for training is a public data set with emotional annotations. During the training process, the parameters of NewsBert are fixed, and only the parameters of the BiLSTM network are adjusted.

6. The news event sentiment analysis system based on NLP technology according to claim 5 is characterized in that: Step S400 fuses the prediction values ​​under different numbers of segmentations. According to the NewsBert+BiLSTM network, each segmentation of the original text has a sentiment prediction. The prediction with a lower correlation coefficient is obtained through correlation analysis. Then, through the voting method, the sentiment tendency with the most votes is used as the final output. When the scores of different sentiment tendencies are the same, the prediction is neutral.

Images