Sentiment early warning method based on public opinion knowledge graph and XGBoost multi-feature fusion

Abstract

The present invention discloses a sentiment warning method based on a public figure public opinion knowledge graph and XGBoost multi-feature fusion, and adopts a combination of XGBoost gradient boosting decision tree and knowledge graph to analyze the public opinion warning value from a multi-feature perspective. First, microblog texts are obtained in real time, and each microblog text is processed. Then, topic extraction is achieved through LDA, XGBoost performs microblog sentiment analysis, public figure public opinion knowledge graph is constructed, and the heat of public opinion events is calculated. Finally, the sentiment warning value is calculated and graded. The method of combining XGBoost gradient boosting decision tree and knowledge graph is adopted, and the heat value and sentiment warning value calculation formula of topics and events are designed and proposed at the same time. The public opinion warning value is calculated from the perspective of multi-dimensional communication features such as forwarding, commenting, likes and sentiment polarity, and different grade division mechanisms are adopted for topics and events, so that the public opinion analysis is more accurate and faster.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and relates to knowledge graph construction and public opinion warning technology, and specifically is a sentiment warning method based on a public figure public opinion knowledge graph and XGBoost multi-feature fusion. Background Art

[0002] Public figures refer to people with wide social visibility and influence. The openness and immediacy of the Internet have made the Internet a source of rumors. It often takes only a few minutes for a piece of negative public opinion to trigger a crisis. For public figures, it is very important to obtain negative public opinion in a timely manner. Only in this way can they deal with it in time and curb the outbreak of the crisis. Public opinion monitoring can quickly monitor negative news that harms one's reputation, and make timely response measures to prevent the further spread and dissemination of malicious information, and nip negative information in the bud as soon as possible. It can also play a role in timely guiding online sentiment and shaping credibility.

[0003] XGBoost stands for eXtreme Gradient Boosting. It is an optimized distributed gradient boosting library designed to be efficient, flexible and portable. XGBoost is also a tool for large-scale parallel boosting trees. It is currently the fastest and best open source boosting tree toolkit, more than 10 times faster than common toolkits. It has a good effect on sentiment classification.

[0004] In the big data environment, the application of traditional public opinion analysis technology is far from meeting the actual needs of network public opinion management, so it is necessary to innovate and study more scientific knowledge organization technology and intelligent knowledge processing technology. The knowledge graph has standardized structure, rich semantics, and supports efficient query and complex knowledge calculation, which can provide strong support for public opinion topic discovery, hot spot tracking, and sentiment warning analysis. Summary of the invention

[0005] The present invention adopts the combination of XGBoost gradient boosting decision tree and knowledge graph to analyze the public opinion warning value from a multi-feature perspective. First, the microblog text is obtained in real time and each microblog text is processed. Then, the topic extraction is realized through LDA, XGBoost is used to analyze the microblog sentiment, the public figure public opinion knowledge graph is constructed, and the heat of public opinion events is calculated. Finally, the sentiment warning value is calculated and graded.

[0006] The purpose of the present invention can be achieved through the following technical solutions:

[0007] S1 is a sentiment early warning method based on the public opinion knowledge graph of public figures and XGBoost multi-feature fusion. The method includes the following processes: public opinion data collection and preprocessing: enter the name of the designated public figure in the Weibo search box, obtain the corresponding content web link, store the crawled data in the MySQL database through web crawler technology, and clean the obtained data.

[0008] S2 uses the LDA model to extract topics: the topic of each document in the document set is given in the form of probability distribution; by analyzing the document set, the topic distribution is extracted, and topic clustering or text classification is performed based on the topic distribution; during training, no manually annotated training set is required, only the document set and the number of specified topics k are required. The core formula of LDA is as follows:

[0009] P(word|doc)=P(word|topic)*P(topic|doc)

[0010] Among them, doc represents the entire document content, word represents the words in the document, and topic represents the topic of the article. From the formula, it is calculated by dividing the number of times the word word appears in the document doc by the total number of words in the document.

[0011] S3 uses XGBoost to perform sentiment analysis on Weibo: load data from the database, separate variable X and label Y; divide the data set into a training set and a test set, use the training set to train the model, and use the test set to test the model accuracy; directly use the classifier and regressor encapsulated by XGBoost, and use XGBClassifier to build the model.

[0012] S4 builds a public figure public opinion knowledge graph: defines five entity categories: topic, user, event, public figure, event sentiment polarity. Defines five entity relationships: <event, belongs to, topic>, <user, publish, event>, <public figure, participate, event>, <public figure, contains, topic>, <event, express, emotion>. Finally, the constructed knowledge graph is stored in the Neo4j graph database.

[0013] S5 calculates the heat of hot topics through the graph: According to the topic heat calculation formula WI(Z) = w(label(Z))(retweet(Z)+comment(Z)+likes(Z))

[0014] The influence of each original microblog in the topic is calculated by combining the relationship between topics, events, and event attributes with Cypher query statistics. Among them, Z represents a text cluster composed of multiple documents, w(label(Z)) represents the weight corresponding to the topic category in the Z text cluster, retweet(Z) represents the sum of the number of reposts of all microblog texts in the Z text cluster, comment(Z) represents the sum of the number of comments of all microblog texts in the Z text cluster, and likes(Z) represents the sum of the number of likes of all microblog texts in the Z text cluster.

[0015] S6 calculates the heat of hot events through the graph:

[0016] WI(e)=retweet(e)+comment(e)+likes(e) and Cypher use the graph query to statistically analyze the relationship between events and event attributes to calculate the influence of each blog post. Among them, e represents a hot microblog event, retweet(e) represents the sum of the number of reposts of the microblog text, comment(e) represents the sum of the number of comments on the microblog text, and likes(e) represents the sum of the number of likes on the microblog text.

[0017] S7 multi-dimensional "propagation characteristics" public opinion warning sentiment value calculation: topic sentiment value

[0018] TS(Z)=WI(Z)*a+WI(-Z)*a, WI(Z) represents the positive topic heat value, and WI(-Z) represents the negative topic heat value. Where a is a constant, the corresponding negative sentiment polarity is -1, and the positive sentiment polarity is 1. Event sentiment value ES(e)=WI(e)*c, WI(e) represents the event heat value, where c is a constant, the corresponding negative sentiment polarity is -1, and the positive sentiment polarity is 1;

[0019] S8 public opinion level classification steps:

[0020] (1) The topic public opinion level is divided into three parts according to different public opinion warning sentiment values.

[0021] -100,000 orange warning (general); when the public opinion value is less than -200,000 yellow warning (serious); when the public opinion value is less than -400,000 red warning (more serious); when the warning level is reached, a text message or email will be sent to the relevant person in charge to handle the public opinion.

[0022] (2) According to different public opinion warning sentiment values, the public opinion level of the event is divided into three parts. When the public opinion value is less than

[0023] -10,000 orange warning (general), when the public opinion value is less than -30,000 yellow warning (serious), when the public opinion value is less than

[0024] -50,000 red warning (more serious). When the warning level is reached, a text message or email will be sent to the relevant person in charge to handle public opinion.

[0025] The creativity of the present invention is mainly reflected in:

[0026] In view of the low accuracy and slow speed of sentiment analysis in public opinion warning, only sentiment analysis and simple event statistics are performed. This paper adopts the method of combining XGBoost gradient boosting decision tree and knowledge graph, and designs and proposes the calculation formula of the heat value and sentiment warning value of topics and events. The public opinion warning value is calculated from the perspective of multi-dimensional communication characteristics such as forwarding, commenting, likes and sentiment polarity, and different level division mechanisms are adopted for topics and events, making public opinion analysis more accurate and faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is the overall structure diagram of the sentiment warning research method based on the public figure public opinion knowledge graph and XGBoost multi-feature fusion of the present invention.

[0028] Figure 2 It is an entity relationship diagram of the public figure public opinion knowledge graph of the present invention.

[0029] Figure 3 It is the entity attribute graph of the public figure public opinion knowledge graph of the present invention.

[0030] Figure 4 This is the multi-feature fused emotional warning message diagram of the present invention. DETAILED DESCRIPTION

[0031] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0032] The technical solution adopted by the present invention is a sentiment warning method based on the public opinion knowledge graph of public figures and XGBoost multi-feature fusion.

[0033] Specific steps for data collection:

[0034] Use scrapy framework to obtain data: find the advanced search page of Weibo, enter keywords of public figure related events, analyze the page, deconstruct the URL composition, sort out the crawled fields, and use xpath to locate and extract, construct multi-level pages, and use yield requests to pass in new URLs for the next round of crawling and parsing. The crawled fields include publishing user information, blog posts, blog post links, publishing time, blog post sources, number of comments, number of likes, number of reposts, event pictures and other data. Finally, save the crawled data in mysql database.

[0035] Specific steps of data preprocessing:

[0036] Introduce the jieba package in Python, connect to the database, store the data in the database into a list, remove frequently appearing stop words, perform Chinese word segmentation on the above list text data through the precise mode of jiaba word segmentation, establish a corpus dictionary for the segmented text data, calculate tfidf on the corpus, and convert the dictionary vector to the batch of text data to be tested for similarity.

[0037] Specific steps for topic extraction through LDA model:

[0038] Introduce the gensim package in Python, perform model training through LdaModel under the models package, set the number of topics of lda, save the trained LDA model, use the model to obtain the topic probability of the document, find the topic distribution with a topic probability greater than 0.6, and save the results in the MySQL database.

[0039] Steps for Weibo sentiment analysis using XGBoost:

[0040] Import the XGBoost package in the python package, load data from the database, separate the variable X and label Y; divide the data set into a training set and a test set, use the training set to train the model, and use the test set to test the model accuracy; use XGBClassifier to build and save the model, load the model for sentiment classification. Finally, save the result data in the MySQL database.

[0041] Steps to build a public opinion knowledge graph of public figures:

[0042] Install the neo4j graph database, create a java project, add spring-data-neo4j and related dependencies in Maven, create five types of NodeEntity (node ​​entity): topic, user, event, public figure, event emotion polarity. Create five types of RelationshipEntity (relationship entity): <event, belongs to, topic>, <user, publish, event>, <public figure, participate, event>, <public figure, include, topic>, <event, expression, emotion>, and finally store the graph in the repository.

[0043] Steps to calculate the heat of hot topics through graphs:

[0044] Calculation formula based on topic popularity

[0045] WI(Z)=w(label(Z))(retweet(Z)+comment(Z)+likes(Z))

[0046] By combining the relationship among topics, events, and event attributes with Cypher's graph query statistics, the influence of each original microblog in the topic can be calculated. Where Z represents a text cluster consisting of multiple documents, w(label(Z)) represents the weight corresponding to the topic category in the Z text cluster, retweet(Z) represents the sum of the number of reposts of all microblog texts in the Z text cluster, comment(Z) represents the sum of the number of comments of all microblog texts in the Z text cluster, and likes(Z) represents the sum of the number of likes of all microblog texts in the Z text cluster.

[0047] Steps to calculate the heat of hot events through the graph:

[0048] According to WI(e) = retweet(e) + comment(e) + likes(e) and Cypher's combination of graph query statistics and analysis of events and event attributes, the influence of each blog post can be calculated. Among them, retweet(e) represents the sum of the number of reposts of the microblog text, comment(e) represents the sum of the number of comments on the microblog text, and likes(e) represents the sum of the number of likes on the microblog text.

[0049] Steps for calculating the sentiment value of multi-dimensional "propagation characteristics" public opinion warning:

[0050] Topic sentiment warning value TS(Z) = WI(Z)*a + WI(-Z)*a, WI(Z) represents the positive topic heat value, WI(-Z) represents the negative topic heat value. Where a is a constant, the corresponding negative sentiment polarity is -1, and the positive sentiment polarity is 1; event sentiment warning value ES(e) = WI(e)*c, WI(e) represents the event heat value, where c is a constant, the corresponding negative sentiment polarity is -1, and the positive sentiment polarity is 1;

[0051] Steps for classifying public opinion levels:

[0052] (1) The topic public opinion level is divided into three parts according to different public opinion warning sentiment values.

[0053] -100,000 orange warning (general); when the public opinion value is less than -200,000 yellow warning (serious); when the public opinion value is less than -400,000 red warning (more serious); when the warning level is reached, a text message or email will be sent to the relevant person in charge to handle the public opinion.

[0054] (2) According to different public opinion warning sentiment values, the public opinion level of the event is divided into three parts. When the public opinion value is less than

[0055] -10,000 orange warning (general), when the public opinion value is less than -30,000 yellow warning (serious), when the public opinion value is less than

[0056] -50,000 red warning (more serious). When the warning level is reached, a text message or email will be sent to the relevant person in charge to handle public opinion.

Claims

1. Based on the public opinion knowledge graph of public figures and XGBoost multi-feature fusion sentiment warning method, the characteristics are: The method comprises the following steps, S1 Public opinion data collection and preprocessing: Enter the name of a designated public figure in the Weibo search box, obtain the corresponding content web link, store the crawled data in the MySQL database through web crawler technology, and clean the obtained data; S2 uses the LDA model to extract topics: the topic of each document in the document set is given in the form of probability distribution; the topic distribution is extracted by analyzing the document set, and topic clustering or text classification is performed based on the topic distribution; during training, no manually annotated training set is required, only the document set and the number of specified topics k are required; S3 uses XGBoost to perform sentiment analysis on Weibo: load data from the database, separate variable X and label Y; divide the data set into a training set and a test set, use the training set to train the model, and use the test set to test the model accuracy; directly use the classifier and regressor encapsulated by XGBoost, and use XGBClassifier to build the model; Import XGBoost in the python package, load the XGBClassifier model for sentiment classification, and finally save the sentiment classification result data in the MySQL database; S4 builds a public figure public opinion knowledge graph: defines five entity categories: topic, user, event, public figure, event sentiment polarity; defines five entity relationships: <event, belongs to, topic>, <user, publish, event>, <public figure, participate, event>, <public figure, includes, topic>, <event, expression, emotion>; finally, stores the constructed knowledge graph in the Neo4j graph database; steps to build a public figure public opinion knowledge graph: Install the neo4j graph database, create a Java project, and add spring-data-neo4j and related dependencies in Maven; S5 uses the graph to calculate the heat of hot topics: Calculation formula based on topic popularity WI(Z)=w(label(Z))(retweet(Z)+comment(Z)+likes(Z)) and Cypher perform statistical analysis on the graph query to analyze the relationship among topics, events, and event attributes, and calculate the influence of each original microblog in the topic; where Z represents a text cluster consisting of multiple documents, w(label(Z)) represents the weight corresponding to the topic category in the Z text cluster, retweet(Z) represents the sum of the number of reposts of all microblog texts in the Z text cluster, comment(Z) represents the sum of the number of comments of all microblog texts in the Z text cluster, and likes(Z) represents the sum of the number of likes of all microblog texts in the Zth text cluster; S6 calculates the heat of hot events through the graph: WI(e)=retweet(e)+comment(e)+likes(e) and Cypher query the graph to analyze the relationship between events and event attributes, and calculate the influence of each blog post; where e represents a hot microblog event, retweet(e) represents the sum of the number of reposts of the microblog text, comment(e) represents the sum of the number of comments on the microblog text, and likes(e) represents the sum of the number of likes on the microblog text; S7 Multi-dimensional Communication Feature Public Opinion Warning Sentiment Value Calculation: Topic Sentiment Value TS(Z)=WI(Z)*a+WI(-Z)*a, WI(Z) represents the positive topic heat value, WI(-Z) represents the negative topic heat value; a is a constant, the corresponding negative sentiment polarity is -1, and the positive sentiment polarity is 1; event sentiment value ES(e)=WI(e)*c, WI(e) represents the event heat value, c is a constant, the corresponding negative sentiment polarity is -1, and the positive sentiment polarity is 1; S8 public opinion level classification steps: (1) The topic public opinion level is divided into three parts according to different public opinion warning sentiment values. When the public opinion value is less than -100,000, it is an orange warning; when the public opinion value is less than -200,000, it is a yellow warning; when the public opinion value is less than -400,000, it is a red warning. When the warning level is reached, a text message or email will be sent to the relevant person in charge to handle the public opinion; (2) The public opinion level of an event is divided into three parts according to different public opinion warning sentiment values. When the public opinion value is less than -10,000, it is an orange warning; when the public opinion value is less than -30,000, it is a yellow warning; when the public opinion value is less than -50,000, it is a red warning. When the warning level is reached, a text message or email will be sent to the relevant person in charge to handle the public opinion.

2. The method for early warning based on public opinion knowledge graph and XGBoost multi-feature fusion emotion according to claim 1 is characterized by: The core formula of LDA is as follows: P(word|doc)=P(word|topic)*P(topic|doc) Among them, doc represents the content of the entire document, word represents the words in the document, and topic represents the subject of the article.

Images