Distributed news public opinion monitoring system

Abstract

The invention relates to a news public opinion monitoring system based on distribution, which comprises a control server and a plurality of monitoring points distributed and operated, and is characterized in that each monitoring point is provided with a crawler module, a weight module and an analysis module; the crawler module monitors and crawls the public opinion space, collects data from different social platforms, and outputs the data to the weight module; the weight module carries out cleaning and keyword matching on the data, and then carries out keyword-based weight sorting and data output; the analysis module calculates a threat index and a predicted threat index according to the input data, reports the data with the high threat index to the control server, and performs LDA extraction on the data with the high threat index to supplement a keyword database; and the control server performs manual emergency response on the data which may cause the emergent public public opinion event, and monitors and controls the keyword database of the weight module. The system can efficiently and accurately monitor news public opinions.

Description

Technical Field

[0001] The present invention relates to the technical field of news and public opinion monitoring, and in particular to a distributed news and public opinion monitoring system. Background Art

[0002] News and public opinion monitoring systems primarily use news representation and subject association to enhance the effectiveness of public opinion risk prediction for target content. However, in the context of capital chasing the market, the association rules between subjects are complex. Existing news and public opinion monitoring systems primarily serve businesses, government agencies, and universities. Therefore, the risk status of a given subject at a given moment is the result of the interaction and superposition of related news events within the entity and its associated subjects over a period of time, as well as multi-dimensional associations and dynamic changes. To address this need, existing public opinion monitoring systems primarily include:

[0003] 1. Design of a public opinion dissemination and control system based on learning to rank. By introducing labeled data, we analyze the correlation between public opinion events. We construct an evaluation index for public opinion evolution and use learning to rank methods to mine the mixed features between labeled data to achieve public opinion dissemination control.

[0004] 2. Based on the number of reposts and comments, we explore the impact mechanism of Weibo text content and quantifiable indicators in Weibo on the communication and interactive capabilities based on Weibo. Usually, the number of reposts and comments is used as the dependent variable to study the communication effect of government news.

[0005] 3. Public opinion dissemination and diffusion model simulation system designed based on infectious disease model. By defining the conditional relationships between multiple subjects, an ideal information dissemination, transition, diffusion and guidance system is established.

[0006] 4. Design a monitoring system based on the principle of systematization, including establishing an online public opinion indicator system, establishing an online public opinion early warning system, and building a mechanism for handling public opinion incidents. Design a news and public opinion feedback system based on the execution system of government agencies such as market supervision.

[0007] The above technical methods demonstrate the ability of news and public opinion monitoring systems to identify connections and transfer paths between entities, but they still fail to address the problem of interpreting complex information. For example, in the case of health public opinion, insufficient explanation of specialized terminology or intellectual panic caused by capitalism can trigger public opinion. Currently, there is no solution to this problem. Due to the enormous workload and scope, it is difficult to predict netizens' knowledge gaps in practical solutions.

[0008] Therefore, how to introduce professional related knowledge other than news to learn how different types of news express target subjects, and how to consider the time factor to model the transmission of public opinion risk monitoring between related subjects are issues that need to be addressed urgently. Summary of the Invention

[0009] The purpose of the present invention is to provide a distributed news and public opinion monitoring system that can monitor news and public opinion efficiently and accurately.

[0010] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a news public opinion monitoring system based on distribution, including a control server and multiple monitoring points that are distributed and operated, and each monitoring point is provided with a crawler module, a weight module and an analysis module; the crawler module collects data from different social platforms by monitoring and crawling the public opinion space, and outputs the acquired data to the weight module; the weight module cleans and matches the data with keywords, and then performs weight sorting and data output based on keywords; the analysis module judges the threat index and predicts the threat index of the data according to the data input by the weight module, and reports the data with high threat index to the control server, performs LDA extraction on the data with high threat index to supplement the keyword database of the weight module; the control server performs manual emergency response to data that may cause sudden public opinion events, and monitors and controls the keyword database of the weight module.

[0011] Furthermore, the crawler module collects APP data by: using the Xposed framework to simulate real users' operations on the social platform;

[0012] The crawler module collects website data by passively accepting website push data based on Requests to simulate the public opinion influence on real social users under objective circumstances;

[0013] The method for the crawler module to collect short video platform data is: extracting video content and comments on the short video platform based on Xposed&whisper; crawling APP data through Xposed, and recognizing video content through whisper voice.

[0014] Furthermore, the weight module cleans and matches the data based on keywords, and then performs keyword-based weight sorting and data output, specifically including:

[0015] 1) Clean the data input by the crawler module;

[0016] 2) Matching the cleaned valid data with the keywords in the keyword database;

[0017] 3) According to the matching results between valid data and keywords, the valid data is sorted based on the keyword weight;

[0018] 4) Output the valid data to the analysis module in sequence according to the weight order.

[0019] Furthermore, the keywords in the keyword database are derived from LDA extraction of data with high threat index analyzed by the analysis module.

[0020] Furthermore, the weighted sorting is based on the number divided by time. The more times it appears within the effective time, the higher the ranking, so as to effectively filter out non-emergency public opinions.

[0021] Furthermore, the analysis module determines the threat index of the data and predicts the threat index based on the data input by the weight module, specifically including:

[0022] Perform sentiment analysis: Use NLP algorithms to analyze sentence sentiment and obtain sentiment analysis scores. Positive sentiment scores range from 0 to 1, and negative sentiment scores range from -1 to 0. Assume that for a given text t, the sentiment score output by the sentiment classifier is s. t \in[-1,1], where s t >0 indicates positive sentiment, s t <0 indicates negative emotions;

[0023] Conduct spreadability analysis: calculate the spread probability and spread speed of the text by analyzing the text's spread path and network structure; assume that for a given text t, the keyword set is K t , which contains the main stem of the text, and the spreadability index is c t , the calculation of the spreadability index is as follows:

[0024]

[0025] Among them, [K t ] represents the number of keywords in text t, and [t] represents the total number of words in text t;

[0026] Perform empathy analysis: comprehensively consider the emotional tendency, emotional intensity and text similarity, calculate the empathy degree, and obtain the empathy degree index; assume that for a given text t, the emotional score output by the sentiment classifier is s t , the emotional intensity is i t , the similarity between text t and known emotional text T is sim(t,T), then the empathy index is e t , expressed as:

[0027]

[0028] Conduct influence analysis: Use the noun phrase extraction algorithm to analyze the sentence, extract the news body, perform part-of-speech judgment, and obtain the influence index; assume that for a given text t, the extracted noun phrase set is N t , which contains the main noun phrases of the text, and the influence index is pt , expressed as:

[0029]

[0030] Among them, [N t ] represents the number of noun phrases in text t, [t] represents the total number of words in text t, and f(n) represents the part-of-speech weight of noun phrase n, which is set according to the specific part of speech;

[0031] Based on the above indicators, calculate the threat index: Assume that for a given text t, the sentiment score is s t , the spreadability index is c t , the empathy index is e t , the influence index is p t , then the threat index is r t , expressed as:

[0032] r t =s t ×c t ×e t ×p t

[0033] Based on the above indicators, calculate the predicted threat index r' t , expressed as:

[0034] r t ′=s t ×c t ×e t

[0035] The analysis module generates a danger intelligence report based on the threat index and the predicted threat index analysis. The content of the danger intelligence report includes relevant data and analysis results, and is reported to the control server; at the same time, the analysis module also performs LDA analysis on data with a high threat index, extracts the keywords analyzed by the LDA, and puts them into the keyword database of the weight module to optimize the weight ranking.

[0036] Furthermore, the control server performs manual emergency response to the data reported by the distributed monitoring points, takes precautions for traffic with a high predicted threat index, and promptly processes traffic with a high threat index.

[0037] Compared with the existing technology, the present invention has the following beneficial effects: it provides a distributed news and public opinion monitoring system, which establishes a news and public opinion responsive monitoring framework based on monitoring points, and monitors the public opinion spaces of different social platforms such as WeChat group chats, Weibo, Zhihu, and Douyin through distributed deployment of monitoring points. Then, through analysis and processing, a dangerous situation report based on the threat index and the predicted threat index is obtained, so that the staff can deal with the early warning in time, thereby improving the efficiency, accuracy and comprehensiveness of public opinion monitoring, and has strong practicality and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a block diagram of the system implementation principle of an embodiment of the present invention;

[0039] Figure 2 This is a diagram showing the working principle of the crawler module in an embodiment of the present invention;

[0040] Figure 3 is a workflow diagram of the weight module in an embodiment of the present invention;

[0041] Figure 4 1 is a diagram showing the working principle of the analysis module in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0043] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0044] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0045] like Figure 1As shown, this embodiment provides a news public opinion monitoring system based on distribution, including a control server and a plurality of monitoring points that are distributed and run, and each monitoring point is provided with a crawler module, a weight module and an analysis module. The crawler module collects data from different social platforms such as WeChat group chats, Weibo, Zhihu, and Douyin by monitoring and crawling the public opinion space, and outputs the acquired data to the weight module. The weight module cleans and matches the data with keywords, and then performs weight sorting and data output based on keywords. The analysis module judges the threat index and predicted threat index of the data based on the data input by the weight module, and reports the data with high threat index to the control server, and performs LDA extraction on the data with high threat index to supplement the keyword database of the weight module. The control server performs manual emergency response to data that may cause sudden public opinion events, and monitors and controls the keyword database of the weight module.

[0046] The working principle of the crawler module is as follows Figure 2 As shown in the figure, the specific method of collecting data from various platforms is:

[0047] 1. Collect APP data: Use the Xposed framework to simulate real users' operations on social platforms.

[0048] 2. Collect website data: Passively accept website push data based on Requests to simulate the impact of public opinion on real social users under objective circumstances.

[0049] 3. Collect short video platform data: Extract video content and comments from short video platforms using Xposed and Whisper. Use Xposed to capture app data and Whisper to recognize video content.

[0050] The workflow of the weight module is as follows Figure 3 As shown, it mainly realizes data cleaning and keyword-based weight ranking, including:

[0051] 1) Clean the data input by the crawler module.

[0052] 2) Match the cleaned valid data with the keywords in the keyword database.

[0053] 3) According to the matching results between valid data and keywords, the valid data is sorted based on the keyword weight.

[0054] The keywords in the keyword database are derived from LDA extraction of data with high threat index analyzed by the analysis module.

[0055] 4) Output the valid data to the analysis module in sequence according to the weight order.

[0056] Among them, the weighted sorting is based on the quantity divided by time. The more times it appears within the effective time, the higher the ranking, which can effectively filter out non-emergency public opinion.

[0057] The working principle of the analysis module is as follows Figure 4 As shown, it mainly judges the threat index and predicted threat index based on the data input by the weight module and generates a report, which specifically includes:

[0058] Perform sentiment analysis: Use NLP algorithms to analyze sentence sentiment and obtain sentiment analysis scores. Positive sentiment scores range from 0 to 1, and negative sentiment scores range from -1 to 0. Assume that for a given text t, the sentiment score output by the sentiment classifier is s. t \in[-1,1], where s t >0 indicates positive sentiment, s t <0 indicates negative sentiment.

[0059] Conduct spreadability analysis: calculate the spread probability and spread speed of the text by analyzing the text's spread path and network structure; assume that for a given text t, the keyword set is K t , which contains the main stem of the text, and the spreadability index is c t , the calculation of the spreadability index is as follows:

[0060]

[0061] Among them, [K t ] represents the number of keywords in text t, and [t] represents the total number of words in text t.

[0062] Perform empathy analysis: comprehensively consider the emotional tendency, emotional intensity and text similarity, calculate the empathy degree, and obtain the empathy degree index; assume that for a given text t, the emotional score output by the sentiment classifier is s t , the emotional intensity is i t , the similarity between text t and known emotional text T is sim(t,T), then the empathy index is e t , expressed as:

[0063]

[0064] Conduct influence analysis: Use the noun phrase extraction algorithm to analyze the sentence, extract the news body, perform part-of-speech judgment, and obtain the influence index; assume that for a given text t, the extracted noun phrase set is N t , which contains the main noun phrases of the text, and the influence index is p t , expressed as:

[0065]

[0066] Among them, [N t ] represents the number of noun phrases in text t, [t] represents the total number of words in text t, and f(n) represents the part-of-speech weight of noun phrase n, which is set according to the specific part of speech.

[0067] Based on the above indicators, calculate the threat index: Assume that for a given text t, the sentiment score is s t , the spreadability index is c t , the empathy index is e t , the influence index is p t , then the threat index is r t , expressed as:

[0068] r t =s t ×c t ×e t ×p t

[0069] Based on the above indicators, calculate the predicted threat index r' t , expressed as:

[0070] r t ′=s t ×c t ×e t

[0071] The analysis module generates a threat intelligence report based on the threat index and predicted threat index, including relevant data and analysis results, and reports it to the control server. The analysis module also performs LDA analysis on data with high threat indices, extracting keywords from the LDA analysis and inserting them into the keyword database of the weighting module to optimize weight ranking.

[0072] Based on the data acquired by each monitoring point being processed and reported, the control server performs manual emergency response to the data reported by the distributed monitoring points, taking precautions against traffic with a high predicted threat index and promptly processing traffic with a high threat index.

[0073] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0074] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0075] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0077] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. A distributed news and public opinion monitoring system, characterized by: The system includes a control server and multiple monitoring points that are distributed and run, and each monitoring point is equipped with a crawler module, a weight module and an analysis module; the crawler module collects data from different social platforms by monitoring and crawling the public opinion space, and outputs the acquired data to the weight module; the weight module cleans the data and matches keywords, and then performs keyword-based weight sorting and data output; the analysis module judges the threat index and predicts the threat index of the data based on the data input by the weight module, and reports the data with a high threat index to the control server, performs LDA extraction on the data with a high threat index to supplement the keyword database of the weight module; the control server performs manual emergency response to data that may cause sudden public opinion events, and monitors and controls the keyword database of the weight module.

2. A distributed news and public opinion monitoring system according to claim 1, characterized in that: The crawler module collects APP data by using the Xposed framework to simulate real users' operations on the social platform; The crawler module collects website data by passively accepting website push data based on Requests to simulate the public opinion influence on real social users under objective circumstances; The method for the crawler module to collect short video platform data is: extracting video content and comments on the short video platform based on Xposed&whisper; crawling APP data through Xposed, and recognizing video content through whisper voice.

3. A distributed news and public opinion monitoring system according to claim 1, characterized in that: The weight module cleans the data and matches keywords, and then performs keyword-based weight sorting and data output, specifically including: 1) Clean the data input by the crawler module; 2) Matching the cleaned valid data with the keywords in the keyword database; 3) According to the matching results between valid data and keywords, the valid data is sorted based on the keyword weight; 4) Output the valid data to the analysis module in sequence according to the weight order.

4. A distributed news and public opinion monitoring system according to claim 3, characterized in that: The keywords in the keyword database are derived from LDA extraction of data with high threat index analyzed by the analysis module.

5. A distributed news and public opinion monitoring system according to claim 3, characterized in that: The weighted sorting is based on the quantity divided by time. The more times it appears within the effective time, the higher the ranking, so as to effectively filter out non-emergency public opinion.

6. A distributed news and public opinion monitoring system according to claim 1, characterized in that: The analysis module determines the threat index of the data and predicts the threat index based on the data input by the weight module, specifically including: Perform sentiment analysis: Use NLP algorithms to analyze sentence sentiment and obtain sentiment analysis scores. Positive sentiment scores range from 0 to 1, and negative sentiment scores range from -1 to 0. Assume that for a given text t, the sentiment score output by the sentiment classifier is s. t \in[-1,1], where s t >0 indicates positive sentiment, s t <0 indicates negative emotions; Conduct spreadability analysis: calculate the spread probability and spread speed of the text by analyzing the text's spread path and network structure; assume that for a given text t, the keyword set is K t , which contains the main stem of the text, and the spreadability index is c t , the calculation of the spreadability index is as follows: Among them, [K t ] represents the number of keywords in text t, and [t] represents the total number of words in text t; Perform empathy analysis: comprehensively consider the emotional tendency, emotional intensity and text similarity, calculate the empathy degree, and obtain the empathy degree index; assume that for a given text t, the emotional score output by the sentiment classifier is s t , the emotional intensity is i t , the similarity between text t and known emotional text T is sim(t,T), then the empathy index is e t , expressed as: Conduct influence analysis: Use the noun phrase extraction algorithm to analyze the sentence, extract the news body, perform part-of-speech judgment, and obtain the influence index; assume that for a given text t, the extracted noun phrase set is N t , which contains the main noun phrases of the text, and the influence index is p t , expressed as: Among them, [N t ] represents the number of noun phrases in text t, [t] represents the total number of words in text t, and f(n) represents the part-of-speech weight of noun phrase n, which is set according to the specific part of speech; Based on the above indicators, calculate the threat index: Assume that for a given text t, the sentiment score is s t , the spreadability index is c t , the empathy index is e t , the influence index is p t , then the threat index is r t , expressed as: r t =s t ×c t ×e t ×p t Based on the above indicators, calculate the predicted threat index r' t , expressed as: r t ′=s t ×c t ×e t The analysis module generates a danger intelligence report based on the threat index and the predicted threat index analysis. The content of the danger intelligence report includes relevant data and analysis results, and is reported to the control server; at the same time, the analysis module also performs LDA analysis on data with a high threat index, extracts the keywords analyzed by the LDA, and puts them into the keyword database of the weight module to optimize the weight ranking.

7. A distributed news and public opinion monitoring system according to claim 1, characterized in that: The control server performs manual emergency response to the data reported by the distributed monitoring points, takes precautions against traffic with a high predicted threat index, and promptly processes traffic with a high threat index.

Images