Public opinion analysis method and system based on data acquisition

Abstract

The invention particularly relates to a public opinion analysis method and system based on data acquisition. The public opinion analysis method based on data acquisition comprises the following steps: acquiring various public opinion information on the Internet in real time, preprocessing data, and converting the data into structured text data; carrying out word segmentation processing and part-of-speech tagging on the text, carrying out sentiment analysis by utilizing a natural language processing technology, and carrying out site and character feature extraction and classification on entities; similar event aggregation is carried out based on article classification and article content similarity, and the abstract extraction effect is improved by utilizing similarity and a graph center theory thought; and deeply analyzing the public opinion information subjected to sentiment analysis to form a public opinion analysis report. According to the public opinion analysis method and system based on data collection, intelligent analysis of mass public opinion data is achieved, the public opinion monitoring and early warning efficiency is improved, real-time public opinion information service can be provided for users, and high practicability and market value are achieved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a public opinion analysis method and system based on data collection. Background Art

[0002] With the rapid development of Internet technology, people have more and more channels to obtain information. Social media, news websites, blogs and other platforms have become important ways for people to obtain information. The amount of information on these platforms is huge, including a large amount of public opinion information. Public opinion information is of great reference value to enterprises, organizations, media and other institutions. It can help them understand the public's attitude and views on a certain topic and make corresponding decisions.

[0003] However, there are some problems with traditional public opinion analysis methods. For example, public opinion analysis requires a lot of manpower and material resources, and the accuracy and real-time nature of the analysis results cannot be guaranteed. In addition, traditional public opinion analysis methods often only focus on a specific topic or field, while ignoring other information, resulting in limitations in the analysis results.

[0004] In order to realize intelligent analysis and effective utilization of massive public opinion data and improve the efficiency of public opinion monitoring and early warning, the present invention proposes a public opinion analysis method and system based on data collection. Summary of the invention

[0005] In order to remedy the deficiencies of the prior art, the present invention provides a simple and efficient public opinion analysis method and system based on data collection.

[0006] The present invention is achieved through the following technical solutions:

[0007] A public opinion analysis method based on data collection includes the following steps:

[0008] Step S1: Data collection

[0009] Collect all kinds of public opinion information on the Internet in real time through batch data collection technology, and the collection sources include but are not limited to websites, forums and social media;

[0010] Step S2: Data preprocessing

[0011] Optical character recognition (OCR) technology is used to process the collected unstructured data and convert it into structured text data for subsequent analysis and utilization. At the same time, irrelevant characters, advertisements and duplicate content are removed through data cleaning and data deduplication, which improves data quality and reduces the error rate in the data processing process.

[0012] Use natural language processing technology to segment the text and tag the text after segmentation to prepare for subsequent sentiment analysis and entity recognition;

[0013] Step S3: Sentiment Analysis

[0014] Based on the part-of-speech tagging of the text after word segmentation, sentiment analysis is performed using natural language processing technology;

[0015] Sentiment analysis uses a simplified BERT model and a bidirectional attention mechanism to reform the input layer. It adds encoding of positive and negative sentiment words based on position vectors, word vectors, and sentence vectors to solve timing problems, enhance semantic dependence on context, and judge whether public opinion information is positive, negative, or neutral, providing basic data for subsequent public opinion analysis.

[0016] Step S4: Entity Identification

[0017] Search and collect encyclopedia information about people and global locations, and use the named entity recognition (NER) algorithm, the BERT model, or a combination of the bidirectional long short-term memory network (BiLSTM) and the conditional random field (CRF) to extract and classify the location and person features of the entities;

[0018] Step S5: Event aggregation

[0019] Based on article classification and article content similarity, similar events are aggregated, graph-centric theory is applied to keyword extraction, and the similarity and graph-centric theory are used to improve the effect of abstract extraction;

[0020] Comprehensively consider sentence similarity and word contribution to extract the summary and calculate sentence similarity;

[0021] Step S6: Public Opinion Report

[0022] Based on the ElasticSearch aggregation analysis function, the public opinion information that has undergone sentiment analysis is deeply analyzed to explore the information value behind it, including the development trend of events, the people involved and their opinions, and form a public opinion analysis report; the public opinion analysis results are displayed in the form of charts, lists and / or maps to facilitate users to intuitively understand and grasp the public opinion dynamics; at the same time, various public opinion analysis reports are generated according to user needs to provide support for decision-making.

[0023] In the step S1, the web crawler Scrapy technology framework is used to write crawler scripts and set crawler rules to ensure compliance with data collection and avoid infringing user privacy; the collected data is stored in a database.

[0024] In step S3, the sentiment analysis model is first trained using a machine learning or deep learning algorithm BERT, and then the preprocessed text is input into the sentiment analysis model; the sentiment analysis model calculates based on the part-of-speech tags of the text and outputs sentiment tags, including positive, negative, and neutral; finally, the sentiment tags are associated with the original data for subsequent analysis.

[0025] In the step S4, firstly, a named entity recognition NER algorithm, a BERT model or a combination of a bidirectional long short-term memory network BiLSTM and a conditional random field CRF is used to train a place and person entity recognition model; then, entity recognition is performed on the preprocessed text to extract place and person entities, and the recognized place and person entities are associated with the original data.

[0026] In step S4, the entity recognition algorithm is calculated using the following formula:

[0027]

[0028] Among them, y is the entity label sequence, x is the input text, and y i is the label of the i-th word, n is the total number of word labels in the entity label sequence y; P(y i |x,y i-1 ) means that given the input x and the previous element y i-1 Under the condition that the current element y i probability.

[0029] In step S5, a machine learning classifier is used to implement article classification, and the machine learning classifier uses a support vector machine (SVM) or a neural network;

[0030] Calculate the similarity of article content through cosine similarity;

[0031] Keywords are identified by calculating the centrality of nodes in the graph. The iterative formula of the centrality measurement PageRank is as follows:

[0032]

[0033] Where: PR(v i ) is the node v i PageRank value, d is the damping coefficient, usually set to 0.85; In(v i ) is pointing to node v i The node set, L(v i ) is the node v i The number of outgoing links.

[0034] Combining graph-centricity and text similarity, we define a summary extraction formula as follows:

[0035] score(s i )=α·centrality(s i )+(1-α)·similarity(s i ,document)

[0036] Where: s i is a sentence in the document, centrality(s i ) is the sentence s i The centrality score, similarity(s i , document) is the sentence s i The similarity with the whole document, α is the weight coefficient used to balance the influence of centrality and similarity.

[0037] In step S6, the processed data is imported into the ElasticSearch search engine, and the aggregation analysis function of the ElasticSearch search engine is used to group, count and sort the data, and a public opinion analysis report is generated according to the sentiment tags, location and person entity information;

[0038] The public opinion analysis report includes hot topics, sentiment distribution and regional distribution;

[0039] In addition, users can customize monitoring thresholds according to their needs, monitor public opinion trends in real time, and issue timely warnings when abnormal situations are discovered.

[0040] A public opinion analysis system based on data collection, comprising:

[0041] The data collection module is responsible for collecting various public opinion information on the Internet in real time through batch data collection technology. The collection sources include but are not limited to websites, forums and social media;

[0042] The data preprocessing module is responsible for processing the collected unstructured data using optical character recognition (OCR) technology and converting it into structured text data for subsequent analysis and utilization. At the same time, it removes irrelevant characters, advertisements and duplicate content through data cleaning and data deduplication to improve data quality and reduce the error rate in the data processing process. It uses natural language processing technology to segment the text and perform part-of-speech tagging on the segmented text to prepare for subsequent sentiment analysis and entity recognition.

[0043] The sentiment analysis module is responsible for part-of-speech tagging of text after word segmentation and uses natural language processing technology to perform sentiment analysis. The sentiment analysis module adopts a simplified BERT model and uses a bidirectional attention mechanism to reform the input layer. It adds encoding of positive and negative sentiment words based on position vectors, word vectors, and sentence vectors to effectively solve the timing problem, make the semantic dependence on the context more accurate, and judge whether public opinion information is positive, negative, or neutral, providing basic data for subsequent public opinion analysis.

[0044] The entity recognition module is responsible for searching and collecting encyclopedia information about people and global locations, and extracting and classifying location and person features of entities using the named entity recognition (NER) algorithm, the BERT model, or a combination of the bidirectional long short-term memory network (BiLSTM) and the conditional random field (CRF);

[0045] The event aggregation module is responsible for aggregating similar events based on article classification and article content similarity, applying graph-centric thinking to keyword extraction, and using similarity and graph-centric thinking to improve the effect of abstract extraction;

[0046] The public opinion report generation module is responsible for conducting in-depth analysis of public opinion information that has undergone sentiment analysis based on the ElasticSearch aggregation analysis function, exploring the value of the information behind it, including the development trend of events, the people involved and their opinions, and forming a public opinion analysis report; the public opinion analysis results are displayed in the form of charts, lists and / or maps to facilitate users to intuitively understand and grasp the public opinion dynamics; at the same time, various public opinion analysis reports are generated according to user needs to provide support for decision-making.

[0047] A public opinion analysis device based on data collection, characterized in that it includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above-mentioned method steps when executing the computer program.

[0048] A readable storage medium, characterized in that: a computer program is stored on the readable storage medium, and the computer program implements the above method steps when executed by a processor.

[0049] The beneficial effects of the present invention are: the public opinion analysis method and system based on data collection realize the intelligent analysis of massive public opinion data, improve the efficiency of public opinion monitoring and early warning, can provide users with real-time public opinion information services, and have high practicality and market value. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Attached Figure 1 It is a schematic diagram of the public opinion analysis method based on data collection of the present invention. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0053] The public opinion analysis method based on data collection includes the following steps:

[0054] Step S1: Data collection

[0055] Through batch data collection technology, various types of public opinion information on the Internet are collected in real time. The collection sources include but are not limited to websites, forums and social media (Weibo, WeChat public accounts, blogs, forums, news, Twitter, Facebook, etc.), with a wide coverage to ensure that the most timely and comprehensive public opinion information is obtained.

[0056] Step S2: Data preprocessing

[0057] Optical character recognition (OCR) technology is used to process the collected unstructured data and convert it into structured text data for subsequent analysis and utilization. At the same time, irrelevant characters, advertisements and duplicate content are removed through data cleaning and data deduplication, which improves data quality and reduces the error rate in the data processing process.

[0058] Use natural language processing technology to segment the text and tag the text after segmentation to prepare for subsequent sentiment analysis and entity recognition;

[0059] Step S3: Sentiment Analysis

[0060] Based on the part-of-speech tagging of the text after word segmentation, sentiment analysis is performed using natural language processing technology;

[0061] Sentiment analysis uses a simplified BERT model and a bidirectional attention mechanism to reform the input layer. It adds encoding of positive and negative sentiment words based on position vectors, word vectors, and sentence vectors, effectively solving the timing problem and making the semantic dependence on context more accurate. It determines whether public opinion information is positive, negative, or neutral, and provides basic data for subsequent public opinion analysis.

[0062] The BERT (Bidirectional Encoder Representations from Transformers) model is a natural language processing model based on deep learning. It understands and generates natural language text through pre-training and fine-tuning.

[0063] The BERT model uses the Transformer architecture and is pre-trained on a large-scale corpus. The core of the BERT model is a bidirectional encoder, which means that it takes into account the contextual information of each word when processing the input text. This bidirectionality makes BERT perform well in tasks such as text classification, question answering systems, and named entity recognition.

[0064] Step S4: Entity Identification

[0065] Search and collect encyclopedia information about people and global locations, and use the named entity recognition (NER) algorithm, the BERT model, or a combination of the bidirectional long short-term memory network (BiLSTM) and the conditional random field (CRF) to extract and classify the location and person features of the entities;

[0066] Step S5: Event aggregation

[0067] Based on article classification and article content similarity, similar events are aggregated, graph-centric theory is applied to keyword extraction, and the similarity and graph-centric theory are used to improve the effect of abstract extraction;

[0068] Taking into account sentence similarity and word contribution, the summary is extracted, and the comprehensive evaluation is improved by about 6% compared with the best open source products on the market;

[0069] Step S6: Public Opinion Report

[0070] Based on the ElasticSearch aggregation analysis function, the public opinion information that has undergone sentiment analysis is deeply analyzed to explore the information value behind it, including the development trend of events, the people involved and their opinions, and form a public opinion analysis report; the public opinion analysis results are displayed in the form of charts, lists and / or maps to facilitate users to intuitively understand and grasp the public opinion dynamics; at the same time, various public opinion analysis reports are generated according to user needs to provide support for decision-making.

[0071] In step S1, the web crawler Scrapy technology framework is used to write crawler scripts and set crawler rules to ensure compliance with data collection and avoid infringing user privacy; the collected data is stored in a database, such as MySQL, MongoDB, etc.

[0072] In step S3, the sentiment analysis model is first trained using a machine learning or deep learning algorithm BERT, and then the preprocessed text is input into the sentiment analysis model; the sentiment analysis model calculates based on the part-of-speech tags of the text and outputs sentiment tags, including positive, negative, and neutral; finally, the sentiment tags are associated with the original data for subsequent analysis.

[0073] Sentiment analysis algorithms are well known to those skilled in the art and will not be described in detail here.

[0074] In the step S4, firstly, a named entity recognition NER algorithm, a BERT model or a combination of a bidirectional long short-term memory network BiLSTM and a conditional random field CRF is used to train a place and person entity recognition model; then, entity recognition is performed on the preprocessed text to extract place and person entities, and the recognized place and person entities are associated with the original data.

[0075] In step S4, the entity recognition algorithm is calculated using the following formula:

[0076]

[0077] Among them, y is the entity label sequence, x is the input text, and y i is the label of the i-th word, n is the total number of word labels in the entity label sequence y; P(y i |x,y i-1 ) means that given the input x and the previous element y i-1 Under the condition that the current element y i probability.

[0078] In step S5, a machine learning classifier is used to implement article classification, and the machine learning classifier uses a support vector machine (SVM) or a neural network;

[0079] Calculate the similarity of article content through cosine similarity;

[0080] Keywords are identified by calculating the centrality of nodes in the graph. The iterative formula of the centrality measurement PageRank is as follows:

[0081]

[0082] Where: PR(v i ) is the node v iPageRank value, d is the damping coefficient, usually set to 0.85; In(v i ) is pointing to node v i The node set, L(v i ) is the node v i The number of outgoing links.

[0083] Combining graph-centricity and text similarity, we define a summary extraction formula as follows:

[0084] score(s i )=α·centrality(s i )+(1-α)·similarity(s i ,document)

[0085] Where: s i is a sentence in the document, centrality(s i ) is the sentence s i The centrality score, similarity(s i , document) is the sentence s i The similarity with the whole document, α is the weight coefficient used to balance the influence of centrality and similarity.

[0086] In step S6, the processed data is imported into the ElasticSearch search engine, and the aggregation analysis function of the ElasticSearch search engine is used to group, count and sort the data, and a public opinion analysis report is generated according to the sentiment tags, location and person entity information;

[0087] The public opinion analysis report includes hot topics, sentiment distribution and regional distribution;

[0088] In addition, users can customize monitoring thresholds according to their needs, monitor public opinion trends in real time, and issue timely warnings when abnormal situations are discovered.

[0089] The public opinion analysis system based on data collection includes:

[0090] The data collection module is responsible for collecting various public opinion information on the Internet in real time through batch data collection technology. The collection sources include but are not limited to websites, forums and social media;

[0091] The data preprocessing module is responsible for processing the collected unstructured data using optical character recognition (OCR) technology and converting it into structured text data for subsequent analysis and utilization. At the same time, it removes irrelevant characters, advertisements and duplicate content through data cleaning and data deduplication to improve data quality and reduce the error rate in the data processing process. It uses natural language processing technology to segment the text and perform part-of-speech tagging on the segmented text to prepare for subsequent sentiment analysis and entity recognition.

[0092] The sentiment analysis module is responsible for part-of-speech tagging of text after word segmentation and uses natural language processing technology to perform sentiment analysis. The sentiment analysis module adopts a simplified BERT model and uses a bidirectional attention mechanism to reform the input layer. It adds encoding of positive and negative sentiment words based on position vectors, word vectors, and sentence vectors to effectively solve the timing problem, make the semantic dependence on the context more accurate, and judge whether public opinion information is positive, negative, or neutral, providing basic data for subsequent public opinion analysis.

[0093] The entity recognition module is responsible for searching and collecting encyclopedia information about people and global locations, and extracting and classifying location and person features of entities using the named entity recognition (NER) algorithm, the BERT model, or a combination of the bidirectional long short-term memory network (BiLSTM) and the conditional random field (CRF);

[0094] The event aggregation module is responsible for aggregating similar events based on article classification and article content similarity, applying graph-centric thinking to keyword extraction, and using similarity and graph-centric thinking to improve the effect of abstract extraction;

[0095] In the sentence proximity algorithm, the sentence proximity and word contribution are comprehensively considered to extract the summary, and the comprehensive evaluation is improved by about 6% compared with the best open source products on the market;

[0096] The public opinion report generation module is responsible for conducting in-depth analysis of public opinion information that has undergone sentiment analysis based on the ElasticSearch aggregation analysis function, exploring the value of the information behind it, including the development trend of events, the people involved and their opinions, and forming a public opinion analysis report; the public opinion analysis results are displayed in the form of charts, lists and / or maps to facilitate users to intuitively understand and grasp the public opinion dynamics; at the same time, various public opinion analysis reports are generated according to user needs to provide support for decision-making.

[0097] The public opinion analysis device based on data collection includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above-mentioned method steps when executing the computer program.

[0098] The readable storage medium stores a computer program, and when the computer program is executed by a processor, the above method steps are implemented.

[0099] Compared with the existing technology, the public opinion analysis method and system based on data collection has the following characteristics:

[0100] (1) Improved the efficiency of public opinion monitoring and early warning: By collecting various public opinion information on the Internet in real time, intelligent analysis and effective use of massive public opinion data are achieved, thereby improving the efficiency of public opinion monitoring and early warning.

[0101] (2) Ability to provide real-time public opinion information services: The public opinion analysis system design method based on data collection can conduct in-depth analysis of public opinion information that has undergone sentiment analysis, explore the information value behind it, and form a public opinion analysis report. This patented technology can generate various public opinion analysis reports and provide users with real-time public opinion information services.

[0102] (3) Improved decision-making accuracy: Through in-depth analysis of public opinion information, it can provide support for decision-making, enabling decision makers to better understand public opinion trends and make more accurate and effective decisions.

[0103] The above is a detailed introduction to a method and system for public opinion analysis based on data collection in an example of the present invention. This section uses specific examples to illustrate the principles and implementation methods of the invention. The above examples are only used to help understand the core ideas of the present invention. Without departing from the principles of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

Claims

1. A public opinion analysis method based on data collection, characterized by: The following steps are involved: Step S1: Data collection Collect all kinds of public opinion information on the Internet in real time through batch data collection technology, including but not limited to websites, forums and social media; Step S2: Data preprocessing Optical character recognition (OCR) technology is used to process the collected unstructured data and convert it into structured text data for subsequent analysis and utilization. At the same time, data cleaning and data deduplication methods are used to remove irrelevant characters, advertisements, and duplicate content, thereby improving data quality and reducing the error rate during data processing. Use natural language processing technology to segment the text and perform part-of-speech tagging on the segmented text to prepare for subsequent sentiment analysis and entity recognition; Step S3: Sentiment Analysis Based on the part-of-speech tagging of the text after word segmentation, sentiment analysis is performed using natural language processing technology; Sentiment analysis uses a simplified BERT model and a bidirectional attention mechanism to reform the input layer. It also adds encoding for positive and negative sentiment words based on position vectors, word vectors, and sentence vectors. This addresses timing issues, enhances semantic reliance on context, and determines whether public opinion information is positive, negative, or neutral, providing basic data for subsequent public opinion analysis. Step S4: Entity Recognition Search and collect encyclopedia information on people and global locations, and use the named entity recognition (NER) algorithm, the BERT model, or a combination of a bidirectional long short-term memory (BiLSTM) network and a conditional random field (CRF) to extract and classify location and person features. Step S5: Event aggregation Based on article classification and article content similarity, similar events are aggregated, graph-centric thinking is applied to keyword extraction, and the effect of abstract extraction is improved by using similarity and graph-centric thinking; Step S6: Public Opinion Report Based on the ElasticSearch aggregation analysis function, we conduct in-depth analysis of public opinion information that has undergone sentiment analysis, explore the information value behind it, including the development trend of events, the people involved and their opinions, and form a public opinion analysis report; display the public opinion analysis results in the form of charts, lists and / or maps, so that users can intuitively understand and grasp the public opinion dynamics; at the same time, generate various public opinion analysis reports according to user needs to provide support for decision-making.

2. The method for public opinion analysis based on data collection according to claim 1, characterized in that: In step S1, the web crawler Scrapy technology framework is used to write a crawler script and set crawler rules to ensure compliance with data collection and avoid infringing user privacy; the collected data is stored in a database.

3. The method for public opinion analysis based on data collection according to claim 1, characterized in that: In step S3, a sentiment analysis model is first trained using a machine learning or deep learning algorithm BERT, and then the preprocessed text is input into the sentiment analysis model; the sentiment analysis model calculates based on the part-of-speech tags of the text and outputs sentiment labels, including positive, negative, and neutral; finally, the sentiment labels are associated with the original data for subsequent analysis.

4. The method for public opinion analysis based on data collection according to claim 1, characterized in that: In step S4, first, a named entity recognition (NER) algorithm, a BERT model, or a combination of a bidirectional long short-term memory (BiLSTM) network and a conditional random field (CRF) is used to train a location and person entity recognition model; then, entity recognition is performed on the preprocessed text to extract location and person entities, and the recognized location and person entities are associated with the original data.

5. The method for public opinion analysis based on data collection according to claim 4 is characterized in that: In step S4, the entity recognition algorithm is calculated using the following formula: Among them, y is the entity label sequence, x is the input text, and y i is the label of the i-th word, n is the total number of word labels in the entity label sequence y; P(y i |x,y i-1 ) means that given input x and previous element y i-1 Under the condition that the current element y i probability.

6. The method for public opinion analysis based on data collection according to claim 1, characterized in that: In step S5, a machine learning classifier is used to classify articles, and the machine learning classifier uses a support vector machine (SVM) or a neural network; Calculate the content similarity of articles through cosine similarity; Keywords are identified by calculating the centrality of nodes in the graph. The iterative formula for centrality measurement PageRank is as follows: Among them: PR (v i ) is the node v i PageRank value, d is the damping coefficient, usually set to 0.85; In(v i ) points to node v i The node set, L(v i ) is the node v i The number of outgoing links; Combining graph-centricity and text similarity, we define a summary extraction formula as follows: score(s i )=α·centrality(s i )+(1-α)·similarity(s i ,document) Where: s i is a sentence in the document, centrality(s i ) is the sentence s i The centrality score, similarity(s i ,document) is the sentence s i The similarity with the entire document, α is the weight coefficient used to balance the influence of centrality and similarity.

7. The method for public opinion analysis based on data collection according to claim 1, characterized in that: In step S6, the processed data is imported into the ElasticSearch search engine, and the aggregation analysis function of the ElasticSearch search engine is used to group, count and sort the data, and generate a public opinion analysis report based on the emotional tags, location and person entity information; The public opinion analysis report includes hot topics, sentiment distribution and regional distribution; In addition, users can customize monitoring thresholds according to their needs, monitor public opinion trends in real time, and issue timely warnings when abnormal situations are discovered.

8. A system for implementing the public opinion analysis method based on data collection as described in any one of claims 1 to 7, Its characteristics are: include: The data collection module is responsible for collecting various public opinion information on the Internet in real time through batch data collection technology. The collection sources include but are not limited to websites, forums and social media; The data preprocessing module is responsible for processing the collected unstructured data using optical character recognition (OCR) technology, converting it into structured text data for subsequent analysis and utilization. At the same time, through data cleaning and data deduplication, irrelevant characters, advertisements, and duplicate content are removed to improve data quality and reduce the error rate during data processing. Natural language processing technology is used to segment the text and perform part-of-speech tagging on the segmented text to prepare for subsequent sentiment analysis and entity recognition. The sentiment analysis module is responsible for part-of-speech tagging based on the text after word segmentation and uses natural language processing technology to perform sentiment analysis. The sentiment analysis module adopts a simplified BERT model and uses a bidirectional attention mechanism to reform the input layer. Based on position vectors, word vectors, and sentence vectors, it adds encoding for positive and negative sentiment words, effectively solving time sequence issues and more accurately relying on contextual semantics. It determines whether public opinion information is positive, negative, or neutral, providing basic data for subsequent public opinion analysis. The entity recognition module is responsible for searching and collecting encyclopedia information on people and global locations. It uses the named entity recognition (NER) algorithm, the BERT model, or a combination of a bidirectional long short-term memory (BiLSTM) network and a conditional random field (CRF) to extract and classify location and person features. The event aggregation module is responsible for aggregating similar events based on article classification and content similarity, applying graph-centric thinking to keyword extraction, and using proximity and graph-centric thinking to improve summary extraction results. It also extracts summaries by comprehensively considering sentence proximity and word contribution to calculate sentence proximity. The public opinion report generation module is responsible for conducting in-depth analysis of public opinion information that has undergone sentiment analysis based on the ElasticSearch aggregation analysis function, exploring the information value behind it, including the development trend of events, the people involved and their opinions, and forming a public opinion analysis report; displaying the public opinion analysis results in the form of charts, lists and / or maps to facilitate users to intuitively understand and grasp the public opinion dynamics; at the same time, generating various public opinion analysis reports based on user needs to provide support for decision-making.

9. A public opinion analysis device based on data collection, characterized by: The method comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method according to any one of claims 1 to 7 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images