A Method for Air Pollution Public Opinion Analysis Based on Topic Algorithm and Network Theory

Abstract

This invention belongs to the field of air pollution online public opinion analysis technology, and relates to an air pollution public opinion analysis method based on topic algorithms and network theory. The method mines air pollution public opinion data to obtain hot topics, uses keyword co-occurrence networks to abstract the relationship between topics and keywords, and calculates the degree distribution, clustering coefficient, node betweenness number, and network robustness of the network to obtain the connection between topics and keywords, as well as the distribution of key topics, providing a scientific basis for decision-making related to public opinion governance.

Description

Technical Field

[0001] This invention belongs to the field of air pollution online public opinion analysis technology, and relates to an air pollution public opinion analysis method based on topic algorithms and network theory. Background Technology

[0002] Online air pollution public opinion analysis based on social media data can provide a scientific basis for relevant decision-making departments to understand the public's demands for environmental pollution control, build a multi-dimensional environmental governance system and institutional framework, and improve the modernization level of the ecological environment governance system and governance capacity, which has important application significance.

[0003] The relevant academic literature 1 (Kay, S., Zhao, B., Sui, D., 2015. Can Social Media Clear the Air? A Case Study of the Air Pollution Problem in Chinese Cities. The Professional Geographer 67(3), 351-363.) collected representative cases of air pollution topics on Weibo from October 2012 to June 2013. Combined with public opinion data from other sources, the study used case analysis to find that citizens' opinions on Weibo can promote environmental protection, and relevant departments can understand the development of social media public opinion related to environmental pollution from Weibo. However, this method relies on a lot of manual data collection and qualitative analysis, and can only achieve post-event attribution and analysis of related topics and events, but cannot achieve real-time monitoring and feedback of public opinion on air pollution.

[0004] Academic paper 2 (Zheng, S., Wang, J., Sun, C., Zhang, X., Kahn, ME, 2019. Airpollution lowers Chinese urbanites' expressed happiness on social media. Nature Human Behaviour 3(3), 237-243.) uses daily pollutant concentration data and social media sentiment data from 144 Chinese cities in 2014 to analyze the impact of pollutant concentration changes on public happiness using baseline linear regression, instrumental variable regression, and heterogeneity analysis. Based on this method, real-time feedback on public quality of life can be obtained using social media sentiment data, which has certain practical significance. However, this method has limitations. It only analyzes the emotional tendencies contained in the sentiment data, neglecting the analysis and utilization of specific sentiment content. Furthermore, this approach has high data requirements, assuming the data conforms to a specific distribution to utilize the method in this study, thus limiting its application in air pollution sentiment analysis using social media data.

[0005] The relevant existing patent 1 (CN202210539480.3, A method for analyzing online public opinion based on BERT) discloses a method for analyzing online public opinion based on BERT. This invention mainly addresses the problem of high complexity in traditional methods and proposes a method that can improve analysis efficiency. The BERT in this invention is a Transformer with only an encoder, which maps the input text sequence to three spaces: query, key, and value, to obtain query vector, key vector, and value vector. Then, for each query vector, the Softmax exponential normalization function is applied to the inner product of the query vector and the key vector to obtain the encoded vector. The encoded vector is then input into a trained classifier to obtain the sentiment analysis tendency of online public opinion.

[0006] Existing patent 2 (CN201910853147.8, Environmental Public Opinion System Based on B / S Architecture) discloses an environmental public opinion system based on a B / S architecture. This system includes a public opinion map submodule and a public opinion management submodule. The public opinion map submodule is used to statistically analyze and filter environmental public opinion data from different regions; the public opinion management submodule provides users with full-text fuzzy search and precise search functions, allowing them to search for relevant environmental public opinion information based on input key information from the public opinion information in the public opinion map submodule. This invention achieves digital management of environmental public opinion information, providing a technical platform for environmental regulators to conduct digital and information-based office work.

[0007] The two existing patent technologies mentioned above have the following problems: Patent 1's analysis of online public opinion data is limited to sentiment analysis, ignoring the analysis of specific text content and themes of public opinion, which has strong limitations and biases; Although Patent 2 has realized the management and storage of environmental public opinion data, it lacks fine-grained analysis of public opinion data and cannot obtain the focus of attention and key governance directions of air pollution public opinion based on relevant data, so its practical application value is relatively limited.

[0008] Existing technologies have the following shortcomings:

[0009] 1. Current analysis of online public opinion on air pollution is still relatively coarse-grained, lacking specific analysis of public opinion themes and content, and cannot meet the requirements of refined environmental management and governance decision-making.

[0010] 2. The models used (such as deep learning models) are highly complex and require a lot of computing power, such as dedicated GPUs, resulting in high model training costs.

[0011] 3. The model requires pre-defined assumptions, and the data must meet the model's basic assumptions before the model can be used. The applicability of existing models still needs to be improved.

[0012] 4. Existing models ignore the inherent connections between different themes and keywords in air pollution public opinion, and lack quantitative analysis of the connections between themes and keywords, which makes it impossible to accurately evaluate the effectiveness of public opinion governance. Summary of the Invention

[0013] The problem this invention aims to solve is to overcome the shortcomings of existing technologies and provide a new framework and analysis method for air pollution public opinion analysis using social media data. By combining topic algorithms with complex network modeling methods, fine-grained and quantitative analysis of air pollution public opinion data is achieved, thereby improving the scientific rigor and reliability of public opinion analysis.

[0014] The technical solution of the present invention:

[0015] A method for analyzing public opinion on air pollution based on topic algorithms and network theory, comprising the following steps:

[0016] Step 000: Based on the collected social media sentiment data, obtain detailed text content of daily public concerns about air pollution and store the data in the database;

[0017] Step 100: Calculate keyword weights based on the Term Frequency-Inverse Document Frequency (TF-IDF) algorithm to extract important keywords from social media sentiment texts. The TF-IDF calculation formula is as follows:

[0018]

[0019] Where TF represents the probability of a keyword appearing in the text, m represents the number of times a keyword t appears in a single document, and M represents the total number of occurrences of all words in the document; IDF reflects the importance of a specific word, and N represents the number of words in the document, n t This represents the number of documents containing the specific keyword t.

[0020] Step 200: Use the text processed by the TF-IDF algorithm as input to the Latent Dirichlet Distribution Topic Model (LDA) to form the TF-IDF-LDA model and output the final topics.

[0021] Step 300: Based on the topic results, construct a topic keyword co-occurrence network.

[0022] Step 400: Analyze the topological characteristics of the constructed co-occurrence network and summarize the results.

[0023] Preferably, step 100 includes the following steps:

[0024] Step 110: Segment the original text into keywords.

[0025] Step 120: Calculate keyword weights based on the TF-IDF algorithm.

[0026] Step 130: Output the keyword weights.

[0027] Preferably, step 200 includes the following steps:

[0028] Step 210: Select an appropriate number of topics and an appropriate hyperparameter vector.

[0029] Step 220: For each word in each document in the corpus, randomly assign a topic number.

[0030] Step 230: Rescan the corpus and update the topic number for each word using the Gibbs sampling formula.

[0031] Step 240: Repeat Gibbs sampling until Gibbs sampling converges.

[0032] Step 250: Analyze the topics of each word in each document in the corpus to obtain the document topic distribution.

[0033] Preferably, step 300 includes the following steps:

[0034] Step 310: Select the N keywords with the highest frequency for each topic as the basic nodes in the network, and number the nodes in ascending order.

[0035] Step 320: If two keywords appear in the same topic and their frequencies are both in the top M, then the two keywords are said to have a co-occurrence relationship, and an edge is generated between the nodes corresponding to the two keywords.

[0036] Step 330: Traverse all nodes and repeat step 320 until the entire keyword co-occurrence network is generated.

[0037] Preferably, step 400 includes the following steps:

[0038] Step 410: Calculate the degree distribution of the keyword co-occurrence network, count the degree of each node, and draw the degree distribution diagram.

[0039] Step 420: Calculate the clustering coefficient of the keyword co-occurrence network. The formula for the clustering coefficient is defined as:

[0040]

[0041] Where T(u) represents the number of triples passing through node u, deg(u) represents the degree of node u, and the clustering coefficient reflects the clustering of nodes in the network.

[0042] Step 430: Calculate the node betweenness number of the keyword co-occurrence network. The formula for the node betweenness number is defined as:

[0043]

[0044] Where V represents the set of nodes, σ(s,t) is the number of shortest paths through node (s,t), σ(s,t│v) represents the number of shortest paths between nodes s and t that pass through node v, and the node betweenness measure the importance of nodes in the network.

[0045] Step 440: Implement two different removal strategies for nodes in the network to verify the robustness of the network. Use the relative size of the largest subgraph of the remaining network after removing nodes as the metric for the robustness of the network.

[0046] Step 450: Summarize the results to obtain the themes and core keywords with high clustering coefficients, and implement key governance on the online public opinion content corresponding to these keywords.

[0047] The beneficial effects of this invention are:

[0048] 1. It enables more granular and quantitative analysis of public opinion themes and keywords, which is of greater reference value for precise control of public opinion governance;

[0049] 2. By constructing a keyword co-occurrence network, public opinion keywords are abstracted into nodes in the network. The intrinsic relationship between public opinion topics and keywords, and between keywords, is analyzed. The importance of nodes in the network is evaluated by using the node degree distribution, clustering coefficient, node betweenness, and network robustness index. This reflects the influence of different keywords in public opinion events and helps in the joint prevention and control of public opinion events.

[0050] 3. Low computational complexity, no need for high-performance servers to participate in model training. Attached Figure Description

[0051] Figure 1 This is an overall schematic diagram of the present invention;

[0052] Figure 2 This is a schematic diagram of the TF-IDF-LDA algorithm;

[0053] Figure 3 This is a schematic diagram of the keyword co-occurrence network generated in the embodiment;

[0054] Figure 4 This is a degree distribution diagram from the embodiment;

[0055] Figure 5 This is a schematic diagram of the clustering results in the embodiment;

[0056] Figure 6 This is a diagram illustrating the betweenness of the twelve nodes with the highest betweenness in the network in this embodiment;

[0057] Figure 7 This is a schematic diagram comparing the experimental results in the embodiments. Detailed Implementation

[0058] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0059] like Figure 1 As shown, the present invention provides a method for analyzing social media sentiment regarding air pollution based on the TF-IDF-LDA algorithm and network theory, comprising the following steps:

[0060] Step 000: Collect public opinion content related to air pollution from social media from June 1, 2019 to May 31, 2021. Based on the collected social media public opinion data, obtain detailed text content of daily public concern about air pollution and store the data in the database.

[0061] Step 100: Calculate keyword weights based on the TF-IDF algorithm to extract important keywords from social media sentiment texts. The TF-IDF calculation formula is as follows:

[0062]

[0063] Where TF represents the probability of a keyword appearing in the text, m represents the number of times a keyword t appears in a single document, and M represents the total number of occurrences of all words in the document; IDF reflects the importance of a specific word, and N represents the number of words in the document, n t This represents the number of documents containing the specific keyword t.

[0064] Step 100 specifically includes the following steps:

[0065] Step 110: Segment the original text into keywords.

[0066] Step 120: Calculate keyword weights based on the TF-IDF algorithm.

[0067] Step 130: Output the keyword weights.

[0068] Step 200: Use the text processed by the TF-IDF algorithm as input to the LDA model, and output the final topic. The structure of the TF-IDF-LDA algorithm is as follows: Figure 2 As shown. Specifically, it includes the following steps:

[0069] Step 210: Select an appropriate number of topics and an appropriate hyperparameter vector.

[0070] Step 220: For each word in each document in the corpus, randomly assign a topic number.

[0071] Step 230: Rescan the corpus and update the topic number for each word using the Gibbs sampling formula.

[0072] Step 240: Repeat Gibbs sampling until Gibbs sampling converges.

[0073] Step 250: Analyze the topics of each word in each document in the corpus to obtain the document topic distribution.

[0074] After calculation, a total of 15 topics were obtained, numbered from topic 1 to topic 15. Each topic contains several keywords. Taking topic 1 as an example, the ten most frequently occurring words are "air pollution", "every year", "organization", "protect", "guarantee", "haze", "fog", "monitoring", "indoor", and "Beijing".

[0075] Step 300: Based on the topic results, construct a topic keyword co-occurrence network. This includes the following steps:

[0076] Step 310: Select the 5 most frequent keywords for each topic as the basic nodes in the network, and number the nodes in ascending order.

[0077] Step 320: If two keywords appear in the same topic and both appear in the top 20, then the two keywords are said to have a co-occurrence relationship, and an edge is generated between the nodes corresponding to the two keywords.

[0078] Step 330: Traverse all nodes and repeat step 320 until the entire keyword co-occurrence network is generated. The final generated keyword co-occurrence network is as follows: Figure 3 As shown. Figure 3 The co-occurrence network has a total of 75 nodes, corresponding to keywords under 75 different topics, and generates a total of 350 edges. The average degree of the network is 14.133, indicating that each node is connected to an average of 14 other nodes.

[0079] Step 400: Analyze the topological characteristics of the constructed co-occurrence network and summarize the results. This includes the following steps:

[0080] Step 410: Calculate the degree distribution of the keyword co-occurrence network, count the degree of each node, and draw the degree distribution graph, such as... Figure 4 As shown. Figure 4 The distribution of nodes in the network shows that y = 1.233*x -1.537 The function form, and the correlation coefficient R 2 The degree distribution of the constructed keyword co-occurrence network, reaching 0.839, indicates that the degree distribution follows a power-law distribution, and the entire network is scale-free. This reflects that the importance of nodes in the network varies, with nodes having higher degrees contributing the vast majority of edge connections. In the context of actual air pollution public opinion governance, where certain core keywords attract the majority of public attention, it is crucial to focus on keywords corresponding to nodes with higher degrees.

[0081] Step 420: Calculate the clustering coefficients of the keyword co-occurrence network.

[0082] The formula for the clustering coefficient is defined as follows:

[0083]

[0084] Where T(u) represents the number of triples passing through node u, deg(u) represents the degree of node u, and the clustering coefficient reflects the clustering of nodes in the network.

[0085] The clustering results are as follows Figure 5 As shown.

[0086] Step 430: Calculate the node betweenness numbers of the keyword co-occurrence network.

[0087] The formula for node betweenness is defined as:

[0088]

[0089] Where V represents the set of nodes, σ(s,t) is the number of shortest paths through node (s,t), σ(s,t│v) represents the number of shortest paths between nodes s and t that pass through node v, and the node betweenness measure the importance of nodes in the network.

[0090] The betweennesses of the twelve nodes with the highest betweenness in the network are as follows: Figure 6 As shown.

[0091] Step 440: Implement two different removal strategies for nodes in the network to verify the network's robustness. The two removal strategies are: Strategy 1: Randomly remove nodes from the network; Strategy 2: Prioritize removing nodes with high degree in the network, using the relative size of the largest subgraph remaining after node removal as a metric for network robustness. The comparison results are as follows: Figure 7 As shown, the robustness of the network is significantly reduced when the strategy of prioritizing the removal of nodes with high degrees is adopted. Two different strategies simulated the changes in public opinion after the governance of key keywords in public opinion. The results of strategy two show that targeted governance of key keywords can effectively control public opinion.

[0092] Step 450: Summarize the results to obtain the themes and core keywords with high clustering coefficients. The online public opinion content corresponding to these keywords can be subject to key governance.

[0093] This invention mines public opinion data on air pollution based on the TF-IDF-LDA topic algorithm to obtain the distribution of topics of public concern about air pollution. It uses keyword co-occurrence networks to abstract the relationship between topics and keywords. By calculating the degree distribution, clustering coefficient, node betweenness number, and network robustness of the network, it obtains the connection between topics and keywords, the connection between keywords, and the distribution of key topics. By constructing networks, it abstracts the topics and keywords that appear in real public opinion, quantitatively assesses the influence of different keywords, and analyzes the correlation of keywords, providing a scientific basis for decision-making related to public opinion governance.

Claims

1. A method for analyzing public opinion on air pollution based on topic algorithms and network theory, characterized in that, The steps are as follows: Step 000: Based on the collected social media sentiment data, obtain detailed text content of daily public concerns about air pollution and store the data in the database; Step 100: Calculate keyword weights based on the TF-IDF algorithm to extract important keywords from social media sentiment texts. The TF-IDF calculation formula is as follows: Where TF represents the probability of a keyword appearing in the text, m represents the number of times a keyword t appears in a single document, and M represents the total number of occurrences of all words in the document; IDF reflects the importance of a specific word, and N represents the number of words in the document, n t This represents the number of documents containing the specific keyword t. Step 200: Use the text processed by the TF-IDF algorithm as input to the Latent Dirichlet Distribution (LDA) topic model to form the TF-IDF-LDA model and output the final topics; Step 300: Based on the topic results, construct a topic keyword co-occurrence network; Step 400: Analyze the topological characteristics of the constructed co-occurrence network and summarize the results; Specifically, step 100 includes the following steps: Step 110: Segment the original text into keywords; Step 120: Calculate keyword weights based on the TF-IDF algorithm; Step 130: Output the keyword weights; Specifically, step 200 includes the following steps: Step 210: Select an appropriate number of topics and an appropriate hyperparameter vector; Step 220: For each word in each document in the corpus, randomly assign a topic number; Step 230: Rescan the corpus and update the topic number for each word using the Gibbs sampling formula; Step 240: Repeat Gibbs sampling until Gibbs sampling converges; Step 250: Analyze the topics of each word in each document in the corpus to obtain the document topic distribution; Specifically, step 300 includes the following steps: Step 310: Select the N keywords with the highest frequency for each topic as the basic nodes in the network, and number the nodes in ascending order; Step 320: If two keywords appear under the same topic and their frequencies are both ranked in the top M, then the two keywords are said to have a co-occurrence relationship, and an edge is generated between the nodes corresponding to the two keywords. Step 330: Traverse all nodes and repeat step 320 until the entire keyword co-occurrence network is generated; Specifically, step 400 includes the following steps: Step 410: Calculate the degree distribution of the keyword co-occurrence network, count the degree of each node, and draw the degree distribution diagram; Step 420: Calculate the clustering coefficient of the keyword co-occurrence network. The formula for the clustering coefficient is defined as: Where T(u) represents the number of triples passing through node u, deg(u) represents the degree of node u, and the clustering coefficient reflects the clustering of nodes in the network. Step 430: Calculate the node betweenness number of the keyword co-occurrence network. The formula for the node betweenness number is defined as: Where V represents the set of nodes, σ(s,t) is the number of shortest paths through node (s,t), σ(s,t│v) represents the number of shortest paths between nodes s and t that pass through node v, and the node betweenness measures the importance of nodes in the network. Step 440: Implement two different removal strategies for nodes in the network to verify the robustness of the network. Use the relative size of the largest subgraph of the remaining network after removing nodes as the metric for the robustness of the network. Step 450: Summarize the results to obtain the themes and core keywords with high clustering coefficients, and implement key governance on the online public opinion content corresponding to these keywords.

Images