Public opinion analysis system based on text hotspots and emotions

Abstract

The invention discloses a public opinion analysis system based on text hotspots and emotions, and the system comprises a data obtaining module which is used for obtaining text data on a social platform and a corresponding user ID; the data cleaning module is used for cleaning the acquired text data; the topic analysis module is used for performing feature extraction and clustering on the text data; the sentiment analysis module is used for extracting sentiment information and reflecting the sentiment tendency and change of the social platform user; and the comprehensive analysis module is used for comprehensively analyzing the topic display result and the negative emotion new classification result. According to the method, topic analysis and sentiment analysis are included at the same time, and the results of topic analysis and sentiment analysis are further fused; and according to a fusion result of topic analysis and sentiment analysis, performing comprehensive analysis to obtain whether a person performs public opinion guidance or not so as to fully and comprehensively use text data to perform public opinion analysis.

Description

Technical Field

[0001] The present invention belongs to the technical field of public opinion analysis, and specifically is a public opinion analysis system based on text hot spots and emotions. Background Art

[0002] After entering the Internet era, a large amount of text data is generated every day on the Internet. Social platforms such as Weibo and game chat have attracted more and more users due to their simple operation, short and casual content, and strong real-time performance. While massive information brings people a huge amount of information, it also brings the negative impact of information explosion. How to use these real-time texts to quickly and effectively grasp information and understand public opinion orientation has become one of the hot research directions in recent years.

[0003] At present, the main public opinion analysis methods are sentiment analysis and topic analysis, each with its own advantages and disadvantages. Sentiment analysis uses supervised learning for classification. Topic analysis uses supervised learning and unsupervised learning. Most of them only perform sentiment analysis or topic analysis, which is relatively simple and not comprehensive enough. Topic analysis also has the problems of low accuracy and long time, and it is also necessary to specify the number of clusters in advance, which has certain limitations on topic analysis.

[0004] The patent "A cognitive domain big data public opinion early warning system based on deep learning CN117807185A" introduces a cognitive domain big data public opinion early warning system based on deep learning, which provides public opinion early warning services by real-time monitoring and analysis of large-scale online public data. Based on deep learning algorithms, it combines modules such as real-time data collection, data filtering, sentiment analysis and public opinion early warning to achieve the assessment and early warning of public opinion risks. Its characteristics are that it cannot predict the evolution trend of public opinion, but only relies on sentiment analysis, without further integrating the results of topic analysis and sentiment analysis, and extracting the center of the topic.

[0005] Combining the above methods, the existing methods of public opinion analysis rarely combine sentiment analysis and topic analysis, and cannot more widely and accurately judge the public opinion situation of new topics in the text. Summary of the invention

[0006] In view of the above problems, the present invention is proposed to provide a public opinion analysis system based on text hot spots and emotions that overcomes the above problems or at least partially solves the above problems.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A public opinion analysis system based on text hot spots and emotions, the system comprising:

[0009] A data acquisition module is used to obtain text data and corresponding user IDs on social platforms;

[0010] A data cleaning module is used to clean the acquired text data;

[0011] Topic analysis module, used to extract features and cluster text data;

[0012] Sentiment analysis module, which is used to extract sentiment information and reflect the sentiment trends and changes of users on social platforms;

[0013] The comprehensive analysis module is used to conduct a comprehensive analysis of the results of topic display and the results of the new classification of negative emotions.

[0014] Optionally, the system further includes a topic application module, and the topic application module includes:

[0015] Topic display unit, which selects hot topics from the clustering results for display;

[0016] The keyword extraction and display unit extracts keywords in the topic analysis module and displays the results;

[0017] The discussion volume analysis and display unit uses the overall discussion volume of the topic to form a trend chart and display it.

[0018] Optionally, the system further includes a sentiment analysis application module, and the sentiment analysis application module includes:

[0019] Sentiment distribution unit, which displays the overall sentiment distribution of user discussion content on the social platform;

[0020] The emotional trend distribution unit displays the emotional content of user discussions on the social platform at fixed time intervals and forms a trend chart;

[0021] The negative emotion sub-classification unit classifies the obtained negative emotion classification text data and subdivides the negative emotions.

[0022] Optionally, the data cleaning module includes:

[0023] Normal cleaning unit: in the input corpus, Chinese punctuation mixed with English punctuation and full-width and half-width characters are cleaned, and invalid garbled characters are removed;

[0024] Sentence length cleaning unit, which cleans short text;

[0025] The word segmentation unit performs word segmentation on the text data;

[0026] Remove stop word units and remove stop words from the word segmentation results;

[0027] Get the word segmentation unit and obtain the part of speech of the word segmentation result through the word segmentation tool;

[0028] A part-of-speech filtering unit, which filters the parts of speech;

[0029] The whitelist filtering unit removes text data that contains special terms for social platforms from the text data.

[0030] Optional topic analysis modules include:

[0031] Model fine-tuning unit, which uses text data to fine-tune the model;

[0032] The feature extraction unit compares and selects features and uses principal component analysis to simplify features and improve clustering speed;

[0033] The clustering unit uses an unsupervised process to generate cluster centers based on judgment conditions.

[0034] Optionally, the model fine-tuning unit includes:

[0035] The word segmentation result x 1 ,x 2 ,…x m Input into the large language model to get the output h of the last moment of the top layer l m , then enter the fully connected layer, and then add a softmax layer for classification, and finally use the cross entropy loss function to calculate the loss;

[0036] Among them, the softmax function:

[0037]

[0038] z i is the i-th output of the fully connected layer, z k is the kth output of the fully connected layer, a i Represents the i-th output value of the softmax function;

[0039] Among them, the cross entropy loss function is:

[0040]

[0041] y i represents the actual classification result, a i is the result of formula (1).

[0042] Optionally, the feature extraction unit includes:

[0043] Input the text data to be clustered, and obtain the sentence vector representation of the text data based on the large language model of the model fine-tuning unit;

[0044] Select the word vector representation and perform average pooling operation for feature extraction;

[0045] Using variable parameter weights, the neural network on the subsequent side of the word embedding continues to be trained. Both word vector representation and character vector representation are used, and weights are adjusted to further improve the sentence vector representation.

[0046] An alternative smaller set of variables is created to combine the essence of the attributes, and the original data can be projected onto this smaller set.

[0047] Optionally, sentiment analysis modules include:

[0048] The word vector training unit uses text data from the same social platform to train the word2vec method;

[0049] Sentiment classification training unit: After jiaba or hanlp performs word segmentation, the base word vector model is used to obtain the word vector representation, which is then input into the TextCNN model for three-category training of sentiment classification.

[0050] The sentiment classification unit inputs new text data into the model based on the model obtained by the sentiment classification training unit to obtain the sentiment classification of the text data.

[0051] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0052] 1. The present invention includes both topic analysis and sentiment analysis, and further integrates the results of topic analysis and sentiment analysis; based on the integrated results of topic analysis and sentiment analysis, a comprehensive analysis is performed to determine whether anyone is guiding public opinion, so as to fully and comprehensively use text data for public opinion analysis.

[0053] 2. The present invention includes both topic analysis and sentiment analysis, and then performs topic display, keyword extraction and display, discussion volume analysis and display, sentiment distribution, sentiment distribution trend, etc., which greatly alleviates the problem of single public opinion analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A flowchart of a public opinion analysis system based on text hot spots and emotions provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0056] See also Figure 1 This embodiment provides a public opinion analysis system based on text hot spots and emotions, and the system includes:

[0057] The data acquisition module is used to obtain text data and corresponding user IDs on the social platform.

[0058] Users on social platforms are always communicating. In order to consider real-time, the text data and corresponding user IDs on the social platforms are obtained through the feature platform at regular intervals, such as obtaining data every 20 minutes. The user IDs used are used to merge the results of topic analysis and sentiment analysis.

[0059] The data cleaning module is used to clean the acquired text data.

[0060] The data cleaning module includes:

[0061] Normal cleaning unit, in the input corpus, cleans Chinese punctuation mixed with English punctuation and full and half-width characters, and removes invalid garbled characters; then removes some special punctuation marks in the sentence, as well as repeated use of punctuation marks. The above mainly uses regular expressions.

[0062] The sentence length cleaning unit cleans the short text.

[0063] The word segmentation unit is then used to segment the text data using word segmentation tools such as jieba or hanlp.

[0064] Remove stop word units and remove stop words from the word segmentation results. If a piece of text data has no other content after removing the stop words, the piece of text data will be discarded.

[0065] Get the word segmentation unit. Use word segmentation tools such as jieba or hanlp to get the part of speech of the word segmentation result.

[0066] The part-of-speech filtering unit filters the parts of speech and can effectively remove text data without substantial content.

[0067] The whitelist filtering unit removes text data that contains special terms for social platforms from the text data.

[0068] The topic analysis module is used to extract features and cluster text data.

[0069] The topic analysis module includes:

[0070] The model fine-tuning unit uses text data to fine-tune the model, which can effectively improve the representation of sentence vectors and effectively improve the clustering accuracy of topic analysis.

[0071] The feature extraction unit compares and selects features and uses principal component analysis to simplify features and improve clustering speed;

[0072] The clustering unit uses an unsupervised process to generate cluster centers based on the judgment conditions, solving the problem of needing to specify the number of clusters in advance. The clustering process takes into account the similarity between the current text and the cluster center and the last sentence (the last sentence of the cluster), making the clustering process not only short in time, but also with good accuracy of clustering results.

[0073] Specifically, the model fine-tuning unit includes:

[0074] The word segmentation result x 1 ,x 2 ,…x m Input into the large language model to get the output h of the last moment of the top layer l m , then enter the fully connected layer, and then add a softmax layer for classification, and finally use the cross entropy loss function to calculate the loss.

[0075] Among them, the softmax function:

[0076]

[0077] z i is the i-th output of the fully connected layer, z k is the kth output of the fully connected layer, a i Represents the i-th output value of the softmax function;

[0078] Among them, the cross entropy loss function is:

[0079]

[0080] y i represents the actual classification result, a i is the result of formula (1).

[0081] Fine-tuning the model on your own data can achieve better results than directly training with your own data, and it uses less data and takes less time. In general, data sets related to subsequent tasks are used to make the obtained word vector representation more meaningful and improve the results of subsequent tasks.

[0082] The feature extraction unit includes:

[0083] Input the text data to be clustered, and obtain the sentence vector representation of the text data based on the large language model of the model fine-tuning unit;

[0084] Select the word vector representation to perform average pooling operation for feature extraction.

[0085] Variable parameter weights are used to allow word embedding and subsequent neural networks to continue training. Both word vector representation and character vector representation are used, and weights are adjusted to further improve sentence vector representation.

[0086] An alternative smaller set of variables is created to combine the essence of the attributes, and the original data can be projected onto this smaller set.

[0087] The topic application module further analyzes and applies the results of topic analysis, helping us to pay attention to current topics and changing trends more conveniently and quickly.

[0088] Topic application modules include:

[0089] The topic display unit selects hot topics from the clustering results in the topic analysis module for display, so as to facilitate early warning and early processing of public opinions that may be formed.

[0090] The keyword extraction and display unit extracts keywords in the topic analysis module and displays the results. Here, this embodiment uses TextRank to extract keywords. TextRank is a keyword extraction algorithm, and then the results are displayed. TextRank is used for keyword extraction because it is an unsupervised method, does not require the construction of a data set for training, and makes more full use of the relationship between text elements, which has good results. Keyword extraction and display can extract the core of public opinion and directly display the core content of hot topics discussed from time to time, which is clearer and more clear at a glance.

[0091] The discussion volume analysis and display unit displays the overall discussion volume of the topic by all users of the social platform at fixed time intervals (such as 10 minutes) to form a trend chart and display it. The discussion volume can show the changes in the user's attention to a topic over time.

[0092] The sentiment analysis module is used to extract sentiment information and reflect the sentiment trends and changes of users on social platforms. Sentiment analysis is the attitude of users towards an object, whether they support, oppose or are neutral. Here, there are three types of sentiment: positive sentiment, negative sentiment and neutral sentiment. Sentiment analysis accurately extracts sentiment information and constantly reflects the sentiment trends and changes of users on social platforms.

[0093] The sentiment analysis module includes:

[0094] The word vector training unit uses text data from the same social platform to train the word2vec method;

[0095] The sentiment classification training unit first performs word segmentation based on jiaba or hanlp, obtains the word vector representation based on the word vector model trained above, and inputs it into the TextCNN model for three-category training of sentiment classification;

[0096] The sentiment classification unit, based on the model obtained through sentiment classification training, obtains the sentiment classification of the text data when new text data is input into the model.

[0097] Sentiment analysis application module, the sentiment analysis application module comprises:

[0098] Sentiment distribution unit, which displays the overall sentiment distribution of user discussion content on the social platform;

[0099] The emotional trend distribution unit displays the emotional content of users' discussion content on the social platform at a fixed time interval (such as 10 minutes) and forms a trend chart. The emotional trend distribution can show the changes in users' emotional tendencies over time.

[0100] The negative sentiment sub-classification unit classifies the obtained negative sentiment classified text data and subdivides the negative sentiment. The sub-classification model used here is the same as the sentiment classification model, which is also the TextCNN model. The negative sentiment sub-classification is used for user spot checks when the results of topic analysis and sentiment analysis are integrated.

[0101] The comprehensive analysis module is used to conduct a comprehensive analysis of the results of topic display and the results of the new classification of negative emotions.

[0102] Combined with the results of topic display in topic analysis and the results of new classification of negative emotions in sentiment analysis, we randomly check user IDs to see if there are users who have been expressing certain negative emotions and clustered together, so as to judge the influence of the user. If there are, the user ID will be fed back for subsequent processing. This can effectively prevent users from guiding public opinion.

[0103] This embodiment includes both topic analysis and sentiment analysis, and further integrates the results of topic analysis and sentiment analysis; based on the integrated results of topic analysis and sentiment analysis, a comprehensive analysis is performed to determine whether anyone is guiding public opinion, so as to fully and comprehensively use text data for public opinion analysis.

[0104] This embodiment includes both topic analysis and sentiment analysis, and then performs topic display, keyword extraction and display, discussion volume analysis and display, sentiment distribution, sentiment distribution trend, etc., alleviating the problem of single public opinion analysis.

[0105] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A public opinion analysis system based on text hot spots and emotions, characterized in that: The system comprises: A data acquisition module is used to obtain text data and corresponding user IDs on social platforms; A data cleaning module is used to clean the acquired text data; Topic analysis module, used to extract features and cluster text data; Sentiment analysis module, which is used to extract sentiment information and reflect the sentiment trends and changes of users on social platforms; The comprehensive analysis module is used to conduct a comprehensive analysis of the results of topic display and the results of the new classification of negative emotions.

2. A public opinion analysis system based on text hot spots and emotions as claimed in claim 1, characterized in that: The system further includes a topic application module, which includes: Topic display unit, which selects hot topics from the clustering results for display; The keyword extraction and display unit extracts keywords in the topic analysis module and displays the results; The discussion volume analysis and display unit uses the overall discussion volume of the topic to form a trend chart and display it.

3. A public opinion analysis system based on text hot spots and emotions as claimed in claim 1, characterized in that: The system also includes a sentiment analysis application module, which includes: Sentiment distribution unit, which displays the overall sentiment distribution of user discussion content on the social platform; The emotional trend distribution unit displays the emotional content of user discussions on the social platform at fixed time intervals and forms a trend chart; The negative emotion sub-classification unit classifies the obtained negative emotion classification text data and subdivides the negative emotions.

4. The public opinion analysis system based on text hot spots and emotions as claimed in claim 1, characterized in that: The data cleaning module includes: Normal cleaning unit: in the input corpus, Chinese punctuation mixed with English punctuation and full-width and half-width characters are cleaned, and invalid garbled characters are removed; Sentence length cleaning unit, which cleans short text; The word segmentation unit performs word segmentation on text data; Remove stop word units and remove stop words from the word segmentation results; Get the word segmentation unit and obtain the part of speech of the word segmentation result through the word segmentation tool; A part-of-speech filtering unit filters the parts of speech; The whitelist filtering unit removes text data that contains special terms for social platforms from the text data.

5. The public opinion analysis system based on text hot spots and emotions as claimed in claim 1, characterized in that: The topic analysis module includes: Model fine-tuning unit, which uses text data to fine-tune the model; The feature extraction unit compares and selects features and uses principal component analysis to simplify features and improve clustering speed; The clustering unit uses an unsupervised process to generate cluster centers based on judgment conditions.

6. A public opinion analysis system based on text hot spots and emotions as claimed in claim 5, characterized in that: The model fine-tuning unit comprises: The word segmentation result x 1 ,x 2 ,…x m Input into the large language model to get the output h of the last moment of the top layer l m , then enter the fully connected layer, and then add a softmax layer for classification, and finally use the cross entropy loss function to calculate the loss; Among them, the softmax function: z i is the i-th output of the fully connected layer, z k is the kth output of the fully connected layer, a i Represents the i-th output value of the softmax function; Among them, the cross entropy loss function is: y i represents the actual classification result, a i is the result of formula (1).

7. The public opinion analysis system based on text hot spots and emotions as claimed in claim 5, characterized in that: The feature extraction unit includes: Input the text data to be clustered, and obtain the sentence vector representation of the text data based on the large language model of the model fine-tuning unit; Select the word vector representation and perform average pooling operation for feature extraction; Using variable parameter weights, the neural network on the subsequent side of the word embedding continues to be trained. Both word vector representation and character vector representation are used, and weights are adjusted to further improve the sentence vector representation. An alternative smaller set of variables is created to combine the essence of the attributes, and the original data can be projected onto this smaller set.

8. The public opinion analysis system based on text hot spots and emotions as claimed in claim 1, characterized in that: The sentiment analysis module includes: The word vector training unit uses text data from the same social platform to train the word2vec method; Sentiment classification training unit: After jiaba or hanlp performs word segmentation, the base word vector model is used to obtain the word vector representation, which is then input into the TextCNN model for three-category training of sentiment classification. The sentiment classification unit inputs new text data into the model based on the model obtained by the sentiment classification training unit to obtain the sentiment classification of the text data.

Images