Network public opinion analysis method and system based on big data and storage medium

Abstract

The invention relates to a network public opinion analysis method based on big data. The method comprises the following steps: S1, collecting large-scale network data from a data source; s2, preprocessing the collected large-scale network data, including parallel cleaning and standardized processing, and converting the large-scale network data into mass text data; s3, performing sentiment analysis on the preprocessed mass text data; s4, performing clustering analysis on the sentiment analysis result to identify and classify similar or related public opinion topics; s5, on the basis of the large-scale historical public opinion data, predicting a future trend through a deep learning model; and S6, displaying analysis and prediction results in a graphical mode, wherein the analysis and prediction results comprise an emotional distribution map, a public opinion popularity map and a clustering result display and trend prediction map. The invention also provides a big data-based network public opinion analysis system, which not only can improve the precision and efficiency of public opinion monitoring, but also can provide powerful technical support for the fields of public management, enterprise brand maintenance, crisis public customs and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of network security, and in particular relates to a network public opinion analysis system based on big data. Background Art

[0002] With the rapid development of the Internet and the widespread popularity of social media, massive amounts of network data are constantly generated and spread rapidly, forming a complex and ever-changing network public opinion environment. The frequency of occurrence, speed of spread and scope of influence of network public opinion have reached unprecedented heights, making governments, enterprises and the public pay more and more attention to the monitoring and analysis of network public opinion. However, in the face of massive data, traditional public opinion analysis methods are difficult to cope with, and there are problems such as low efficiency, insufficient data processing capabilities, and low analysis accuracy. Therefore, the network public opinion analysis system based on big data technology has gradually become a research hotspot and practical demand.

[0003] The core of big data technology lies in the rapid processing, analysis and mining of massive data. It can process multiple types of data (text, images, audio, video, etc.) from multiple data sources, and achieve efficient data storage, management and analysis through distributed computing frameworks (Hadoop, Spark). The application of big data technology in online public opinion analysis can not only greatly improve the data processing speed, but also discover deep information and potential trends that are difficult to capture with traditional analysis methods.

[0004] As one of the key technologies for big data public opinion analysis, natural language processing can deeply understand and process a large amount of unstructured text data. Natural language processing technology includes data cleaning, normalization processing, sentiment analysis, topic identification and other links. By removing noise, extracting key content and understanding sentiment tendencies, it provides a solid data foundation and precise analysis capabilities for public opinion analysis. In addition, machine learning and deep learning models are widely used in tasks such as sentiment classification and topic modeling, which greatly improves the accuracy and efficiency of analysis.

[0005] In the context of big data, the speed of public opinion transmission and the scope of influence are difficult to predict, and traditional static analysis methods can no longer meet the needs. To this end, technologies such as time series analysis and deep learning models (LSTM) are introduced into the trend prediction module to accurately predict the future direction of public opinion. These prediction technologies can predict future changes in public opinion based on historical data and external influencing factors (such as emergencies and policy changes), providing scientific basis for decision makers.

[0006] In addition, as the amount of data and the complexity of analysis increase, the need for visualization of public opinion analysis becomes increasingly important. Big data visualization technology can present the analysis results in an intuitive graphical form, such as sentiment distribution maps, heat change maps, clustering result displays, etc., providing users with a multi-dimensional and multi-level public opinion analysis perspective. Through the interactive interface, users can quickly understand the information behind the data and make timely and accurate decisions.

[0007] In summary, the Internet public opinion analysis system based on big data provides a complete and efficient solution for public opinion analysis through large-scale data collection, natural language processing, sentiment analysis, topic clustering, trend prediction and visualization display and other technical means. The application of this system can not only improve the accuracy and efficiency of public opinion monitoring, but also provide strong technical support for public management, corporate brand maintenance and crisis public relations.

[0008] The information disclosed in this background technology section is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as acknowledging or suggesting in any form that the information constitutes the prior art already known to those skilled in the art. Summary of the invention

[0009] The purpose of the present invention is to provide a network public opinion analysis method, system and storage medium based on big data to solve the problems existing in the prior art.

[0010] In order to achieve the above object, the present invention provides a method for analyzing network public opinion based on big data, comprising:

[0011] S1: Collect large-scale network data from data sources;

[0012] S2: Preprocessing the collected large-scale network data, including parallel cleaning and normalization, and converting it into massive text data;

[0013] S3: Perform sentiment analysis on the pre-processed massive text data;

[0014] S4: Perform cluster analysis on sentiment analysis results to identify and classify similar or related public opinion topics;

[0015] S5: Based on large-scale historical public opinion data, predict future trends through deep learning models;

[0016] S6: Display the analysis and prediction results in a graphical way, including sentiment distribution map, public opinion heat map, clustering result display and trend prediction map.

[0017] Furthermore, the data sources described in S1 include, but are not limited to: social media platforms, news websites, blogs, and forums.

[0018] Furthermore, the large-scale network data collection described in S1 is to capture text, image, audio and video multimedia data through distributed crawler technology and real-time stream processing technology, and then s , to determine its collection priority.

[0019] Furthermore, the weight W of the data source s Calculation formula:

[0020] W s =γ 1 F s +γ 2 V s +γ 3 R s +γ 4 U s

[0021] Among them, F s Indicates the update frequency of the data source, V s Indicates the access speed of the data source, R s Indicates the reliability of the data source, U s represents the user engagement of the data source, γ 1 ,γ 2 ,γ 3 ,γ 4 is the weight coefficient, and finally a hash verification mechanism and a distributed storage system are used to ensure data integrity and efficient storage.

[0022] Furthermore, the parallel cleaning and normalization processing described in S2 includes: removing HTML tags, removing special characters, filtering stop words, extracting stems, restoring morphology, correcting spellings, and replacing synonyms. The preprocessing adopts a big data processing framework and performs word frequency normalization processing through the following formula:

[0023]

[0024] Among them, F(ω,d) is the standardized frequency, f(ω,d) represents the number of occurrences of word ω in document d, and max{f(ω′,d)} represents the number of occurrences of word ω′ with the highest frequency in document d.

[0025] Furthermore, the stem extraction is performed using a feature extraction algorithm based on TF-IDF and processed through a distributed computing framework. The TF-IDF is term frequency minus inverse document frequency. The calculation formula of the TF-IDF value is as follows:

[0026]

[0027] Among them, f(ω,d) represents the frequency of occurrence of word ω in document d, ∑ω′∈d f(ω′,d) represents the sum of the occurrence times of all words in document d, D represents the document set, |D| represents the total number of documents, and |{d′∈D:ω∈d′}| represents the number of documents d′ containing the word ω in the document set D.

[0028] Furthermore, the sentiment analysis described in S3 is based on a bidirectional long short-term memory network algorithm, and the sentiment score S of the text is calculated by parallel computing using the following formula:

[0029]

[0030] Among them, ω i represents the weight of sentiment word i, p i represents the probability of word i appearing in the text; s i It represents the sentiment polarity score of word i, i is -1 for negative, i is 0 for neutral, and i is 1 for positive; n represents the total number of sentiment words.

[0031] Furthermore, the public opinion clustering described in S4 uses the K-means algorithm and parallelizes the clustering task through a big data processing framework to minimize the following objective function:

[0032]

[0033] Where J is the value of the objective function, also called the mismatch error, K represents the number of clusters, and C k represents the kth cluster, and x represents the cluster in C k The sample points of the cluster, μ k Represents C k The center point of the cluster.

[0034] Furthermore, the deep learning model described in S5 is LSTM, which is a long short-term memory network, and the future public opinion value T(t) is predicted by the following formula:

[0035] T(t)=α 1 T(t-1)+α 2 T(t-2)+…+α p T(tp)+βX(t)+ε

[0036] Where T(t-1) represents the public opinion value of the previous time step, T(t-2) represents the public opinion value of the previous time step, X(t) is the external influencing factor of the current time step, p is the order of the model, and α 1 ,α 2 ,α p is the model parameter, β is the influence coefficient of external factors, and ε is the error term.

[0037] The present invention also provides a network public opinion analysis system based on big data, the analysis system comprising:

[0038] Big data collection module: used to collect large-scale network data from data sources;

[0039] Big data preprocessing module: used to preprocess the collected large-scale network data, including parallel cleaning and normalization processing, and convert it into massive text data;

[0040] Large-scale sentiment analysis module: used to perform sentiment analysis on massive pre-processed text data;

[0041] Big data public opinion clustering module: used to perform cluster analysis on sentiment analysis results to identify and classify similar or related public opinion topics;

[0042] Big data trend prediction module: Based on large-scale historical public opinion data, future trends are predicted through deep learning models;

[0043] Big data visualization module: used to display the results of analysis and prediction in a graphical way, including sentiment distribution map, public opinion heat map, clustering result display and trend prediction map.

[0044] The present invention also provides a computer-readable storage medium, which stores computer program instructions. When the computer program instructions are executed by a computer, the computer executes the steps of any of the above-mentioned methods for analyzing network public opinion based on big data.

[0045] By adopting the above technical solution, the present invention has the following beneficial effects:

[0046] 1. The network public opinion analysis system based on big data provided by the present invention has the ability to efficiently process massive data, and realizes real-time collection and parallel processing of multiple data sources through distributed computing frameworks (Hadoop and Spark). Compared with traditional public opinion analysis methods, the system can quickly capture, clean and analyze massive unstructured data, such as text data in social media, news websites, forums and blogs, significantly improving data processing speed and analysis accuracy. Combined with natural language processing technology, the system can automatically remove irrelevant information, extract key content, and perform deep processing such as sentiment analysis and topic clustering on the text, so as to accurately reflect the emotional tendency and topic distribution of public opinion. This efficient processing and analysis capability enables the system to promptly detect network public opinion hotspots, abnormal emotional fluctuations and potential crises, and provide decision makers with fast and accurate information support.

[0047] 2. The system of the present invention integrates advanced time series analysis and deep learning models (LSTM), and can accurately predict the development trend of public opinion based on large-scale historical data. Through the mining and modeling of massive data, the system can identify the key factors that affect the trend of public opinion and realize the intelligent prediction of future changes in public opinion. This prediction capability provides forward-looking support for the public opinion management of governments and enterprises, and helps to take countermeasures in advance and reduce public opinion risks. At the same time, the visualization module of the system can present complex analysis results in an intuitive graphical form, helping users to quickly understand and grasp the dynamics of public opinion. Through multi-dimensional public opinion display, users can gain insight into changes in public opinion from multiple angles such as emotion, time, and region, and improve the scientificity and accuracy of decision-making. In general, the system of the present invention significantly improves the comprehensiveness, real-time and predictability of public opinion analysis, and has important application value for the efficient management of online public opinion. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A flowchart of the network public opinion analysis method based on big data provided by the present invention;

[0050] Figure 2 Schematic diagram of the network public opinion analysis system based on big data provided by the present invention. DETAILED DESCRIPTION

[0051] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Combination Figure 1 As shown, a network public opinion analysis method based on big data includes:

[0053] S1: Collect large-scale network data from data sources;

[0054] S2: Preprocessing the collected large-scale network data, including parallel cleaning and normalization, and converting it into massive text data;

[0055] S3: Perform sentiment analysis on the pre-processed massive text data;

[0056] S4: Perform cluster analysis on sentiment analysis results to identify and classify similar or related public opinion topics;

[0057] S5: Based on large-scale historical public opinion data, predict future trends through deep learning models;

[0058] S6: Display the analysis and prediction results in a graphical way, including sentiment distribution map, public opinion heat map, clustering result display and trend prediction map.

[0059] Specifically, the data sources described in S1 include, but are not limited to: social media platforms, news websites, blogs, and forums.

[0060] Specifically, the large-scale network data collection described in S1 is to capture text, image, audio and video multimedia data through distributed crawler technology and real-time stream processing technology, and then calculate the weight W of the data source. s , to determine its collection priority.

[0061] The distributed crawler technology is abbreviated as Colly in English, and the real-time stream processing technology is abbreviated as Apache Kafka in English.

[0062] Specifically, the weight W of the data source is s Calculation formula:

[0063] W s =γ 1 F s +γ 2 V s +γ 3 R s +γ 4 U s

[0064] Among them, F s Indicates the update frequency of the data source, V s Indicates the access speed of the data source, R s Indicates the reliability of the data source, U s represents the user engagement of the data source, γ 1 ,γ 2 ,γ 3 ,γ 4 is the weight coefficient, and finally a hash verification mechanism and a distributed storage system are used to ensure data integrity and efficient storage.

[0065] The English abbreviation of distributed storage system is HDFS.

[0066] Specifically, the parallel cleaning and normalization processing described in S2 includes: removing HTML tags, removing special characters, filtering stop words, extracting stems, restoring morphology, correcting spellings, and replacing synonyms. The preprocessing adopts a big data processing framework and performs word frequency normalization processing through the following formula:

[0067]

[0068] Among them, F(ω,d) is the standardized frequency, f(ω,d) represents the number of occurrences of word ω in document d, and max{f(ω′,d)} represents the number of occurrences of word ω′ with the highest frequency in document d.

[0069] The big data processing framework is abbreviated as Apache Spark in English.

[0070] Specifically, the stem extraction is performed using a feature extraction algorithm based on TF-IDF and processed through a distributed computing framework. The TF-IDF is term frequency minus inverse document frequency. The calculation formula for the TF-IDF value is as follows:

[0071]

[0072] Among them, f(ω,d) represents the frequency of occurrence of word ω in document d, ∑ ω′∈d f(ω′,d) represents the sum of the occurrence times of all words in document d, D represents the document set, |D| represents the total number of documents, and |{d′∈D:ω∈d′}| represents the number of documents d′ containing the word ω in the document set D.

[0073] Specifically, the sentiment analysis described in S3 is based on a bidirectional long short-term memory network algorithm, and the sentiment score S of the text is calculated by parallel computing using the following formula:

[0074]

[0075] Among them, ω i represents the weight of sentiment word i, p i represents the probability of word i appearing in the text; s i It represents the sentiment polarity score of word i, i is -1 for negative, i is 0 for neutral, and i is 1 for positive; n represents the total number of sentiment words.

[0076] The bidirectional long short-term memory network is abbreviated as BiLSTM in English.

[0077] Specifically, the public opinion clustering described in S4 uses the K-means algorithm and parallelizes the clustering task through a big data processing framework to minimize the following objective function:

[0078]

[0079] Where J is the value of the objective function, also called the mismatch error, K represents the number of clusters, and C k represents the kth cluster, and x represents the cluster in C k The sample points of the cluster, μ k Represents C k The center point of the cluster.

[0080] The big data processing framework is abbreviated as Apache Hadoop in English.

[0081] Specifically, the deep learning model described in S5 is LSTM, which is a long short-term memory network, and the future public opinion value T(t) is predicted by the following formula:

[0082] T(t)=α 1 T(t-1)+α 2 T(t-2)+…+α p T(tp)+βX(t)+ε

[0083] Where T(t-1) represents the public opinion value of the previous time step, T(t-2) represents the public opinion value of the previous time step, X(t) is the external influencing factor of the current time step, p is the order of the model, and α 1 ,α 2 ,α p is the model parameter, β is the influence coefficient of external factors, and ε is the error term.

[0084] In another embodiment, see Figure 2 As shown, the present invention also provides a network public opinion analysis system based on big data, and the analysis system includes:

[0085] Big data collection module: used to collect large-scale network data from data sources;

[0086] Big data preprocessing module: used to preprocess the collected large-scale network data, including parallel cleaning and normalization processing, and convert it into massive text data;

[0087] Large-scale sentiment analysis module: used to perform sentiment analysis on massive pre-processed text data;

[0088] Big data public opinion clustering module: used to perform cluster analysis on sentiment analysis results to identify and classify similar or related public opinion topics;

[0089] Big data trend prediction module: Based on large-scale historical public opinion data, future trends are predicted through deep learning models;

[0090] Big data visualization module: used to display the results of analysis and prediction in a graphical way, including sentiment distribution map, public opinion heat map, clustering result display and trend prediction map.

[0091] In another embodiment, the present invention further provides a computer-readable storage medium, which stores computer program instructions. When the computer program instructions are executed by a computer, the computer executes the steps of any of the above-described methods for analyzing online public opinion based on big data.

[0092] It should be noted that the storage medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any storage medium other than a computer-readable storage medium that can send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, system, or device. The program code contained on the storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0093] The units involved in the embodiments of the present invention may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, limit the units themselves.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for analyzing network public opinion based on big data, characterized in that: The method comprises: S1: Collect large-scale network data from data sources; S2: Preprocessing the collected large-scale network data, including parallel cleaning and normalization, and converting it into massive text data; S3: Perform sentiment analysis on the pre-processed massive text data; S4: Perform cluster analysis on sentiment analysis results to identify and classify similar or related public opinion topics; S5: Based on large-scale historical public opinion data, predict future trends through deep learning models; S6: Display the analysis and prediction results in a graphical way, including sentiment distribution map, public opinion heat map, clustering result display and trend prediction map.

2. According to the big data-based network public opinion analysis method of claim 1, it is characterized in that: The data sources described in S1 include, but are not limited to: social media platforms, news websites, blogs, and forums.

3. According to the big data-based network public opinion analysis method of claim 1, it is characterized in that: The large-scale network data collection described in S1 is to capture text, image, audio and video multimedia data through distributed crawler technology and real-time stream processing technology, and then calculate the weight W of the data source. s , to determine its collection priority.

4. According to the big data-based network public opinion analysis method of claim 3, it is characterized in that: The weight W of the data source s Calculation formula: W s =γ1F s +γ2V s +γ3R s +γ4U s Among them, F s Indicates the update frequency of the data source, V s Indicates the access speed of the data source, R s Indicates the reliability of the data source, U s It represents the user participation of the data source, γ1, γ2, γ3, and γ4 are weight coefficients, and finally the hash verification mechanism and distributed storage system are used to ensure data integrity and efficient storage.

5. According to the big data-based network public opinion analysis method of claim 1, it is characterized in that: The parallel cleaning and normalization processing described in S2 includes: removing HTML tags, removing special characters, filtering stop words, extracting stems, restoring morphology, correcting spellings, and replacing synonyms. The preprocessing adopts a big data processing framework and performs word frequency normalization processing through the following formula: Among them, F(ω,d) is the standardized frequency, f(ω,d) represents the number of occurrences of word ω in document d, and max{f(ω′,d)} represents the number of occurrences of word ω′ with the highest frequency in document d.

6. According to the big data-based network public opinion analysis method of claim 5, it is characterized in that: The stem extraction is performed using a TF-IDF-based feature extraction algorithm and processed through a distributed computing framework. The TF-IDF is term frequency-inverse document frequency. The calculation formula of the TF-IDF value is as follows: Among them, f(ω,d) represents the frequency of occurrence of word ω in document d, ∑ ω′∈d f(ω′,d) represents the sum of the occurrence times of all words in document d, D represents the document set, |D| represents the total number of documents, and |{d′∈D:ω∈d′}| represents the number of documents d′ containing the word ω in the document set D.

7. According to the big data-based network public opinion analysis method of claim 1, it is characterized in that: The sentiment analysis described in S3 is based on the bidirectional long short-term memory network algorithm, and the sentiment score S of the text is calculated by parallel computing using the following formula: Among them, ω i represents the weight of sentiment word i, p i represents the probability of word i appearing in the text; s i It represents the sentiment polarity score of word i, i is -1 for negative, i is 0 for neutral, and i is 1 for positive; n represents the total number of sentiment words.

8. According to the big data-based network public opinion analysis method of claim 1, it is characterized in that: The public opinion clustering described in S4 uses the K-means algorithm and parallelizes the clustering task through the big data processing framework to minimize the following objective function: Where J is the value of the objective function, also called the mismatch error, K represents the number of clusters, and C k represents the kth cluster, and x represents the cluster in C k The sample points of the cluster, μ k Represents C k The center point of the cluster.

9. The method for analyzing network public opinion based on big data according to claim 1, characterized in that: The deep learning model described in S5 is LSTM, which is a long short-term memory network. The future public opinion value T(t) is predicted by the following formula: T(t)=α1T(t-1)+α2T(t-2)+…+α p T(tp)+βX(t)+ε Where T(t-1) represents the public opinion value of the previous time step, T(t-2) represents the public opinion value of the previous time step, X(t) is the external influencing factor of the current time step, p is the order of the model, α1, α2, α p is the model parameter, β is the influence coefficient of external factors, and ε is the error term.

10. A network public opinion analysis system based on big data, characterized in that: The analysis system comprises: Big data collection module: used to collect large-scale network data from data sources; Big data preprocessing module: used to preprocess the collected large-scale network data, including parallel cleaning and normalization processing, and convert it into massive text data; Large-scale sentiment analysis module: used to perform sentiment analysis on massive pre-processed text data; Big data public opinion clustering module: used to perform cluster analysis on sentiment analysis results to identify and classify similar or related public opinion topics; Big data trend prediction module: Based on large-scale historical public opinion data, future trends are predicted through deep learning models; Big data visualization module: used to display the results of analysis and prediction in a graphical way, including sentiment distribution map, public opinion heat map, clustering result display and trend prediction map.

11. A computer-readable storage medium storing computer program instructions. When the computer program instructions are executed by a computer, the computer executes the steps of the network public opinion analysis method based on big data as described in any one of claims 1 to 9.

Images