Public opinion data analysis method, system, equipment and product
Through the sentiment analysis method combining multi-source data acquisition and multi-language pre-training language model, the accuracy and real-time problems of sentiment analysis in public opinion data analysis are solved, and efficient and accurate public opinion monitoring is achieved.
Patent Information
- Application Number
- CN202510056593.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-14
- Publication Date
- 2025-05-16
AI Technical Summary
In the face of the explosive growth of social media data volume and multilingual environments, the accuracy and real-time nature of sentiment analysis are insufficient, making it difficult to meet the needs of real-time monitoring.
Multi-source data acquisition, distributed computing and real-time stream processing technology are used to pre-process data, combined with word vector transformation and multi-language pre-trained language models (such as BERT or GPT) for sentiment analysis, and through the comprehensive scores of sentiment dictionary and public opinion analysis models, the accuracy and real-time nature of sentiment analysis are improved.
It improves the accuracy and real-time processing capabilities of public opinion data analysis, can better understand complex contexts, and meet the needs of efficient and accurate public opinion monitoring.
Smart Images
Figure CN120011654A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of data processing technology, and specifically relates to a public opinion data analysis method, system, equipment and product. Background Art
[0002] With the rapid development of the Internet and social media, public opinion data analysis has become an important tool for monitoring public sentiment, predicting market trends and assisting decision-making. Existing public opinion data analysis methods mainly rely on natural language processing (NLP), machine learning and data mining technologies to collect and analyze text data on online platforms such as social media, news websites and forums to identify the public's attitudes and emotions towards specific events, brands or issues.
[0003] Traditional public opinion analysis methods usually include the following steps: first, collect raw text data through web crawlers and API (Application Programming Interface) interfaces; then, pre-process the raw text data by cleaning, word segmentation, and removing stop words; then, use a pre-built sentiment dictionary or machine learning model to perform sentiment analysis to determine the sentiment tendency in the text data, such as positive, negative, or neutral; finally, extract the hidden topic information in the text data based on a topic model (such as the LDA topic model).
[0004] However, in the process of using the prior art, the inventors found that the prior art has at least the following problems: With the explosive growth of social media data and the diversification of online language expressions, existing technologies face many challenges in terms of the accuracy and real-time performance of sentiment analysis. Specifically, existing sentiment analysis methods are unstable when dealing with multilingual environments or complex contexts (such as sarcasm, metaphor, etc.), which affects the accuracy of sentiment classification. In addition, traditional sentiment analysis models often have computational delays when facing large-scale real-time data, making it difficult to meet the needs of real-time monitoring, affecting the timeliness and effectiveness of decision-making. Summary of the invention
[0005] The present invention aims to solve the above technical problems at least to a certain extent, and provides a public opinion data analysis method, system, device and product.
[0006] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a method for analyzing public opinion data, comprising: Collect initial public opinion data from different target platforms, and pre-process the initial public opinion data from different target platforms to obtain target public opinion data; Performing word vector conversion on the target public opinion data to obtain a word vector data set corresponding to the target public opinion data; Performing matching processing on the word vector data set based on a preset sentiment dictionary to obtain a sentiment dictionary score of the target public opinion data; Inputting the word vector data set into a preset public opinion analysis model to obtain a sentiment classification score of the target public opinion data; wherein the public opinion analysis model is constructed based on a multilingual pre-trained language model; According to the sentiment dictionary score and the sentiment classification score, a comprehensive sentiment score of the target public opinion data is obtained.
[0007] In a possible design, initial public opinion data from different target platforms are collected and preprocessed to obtain target public opinion data, including: Use multi-source data collection technology to collect initial public opinion data from different target platforms; Perform data fusion processing on the initial public opinion data of different target platforms to obtain the fused initial public opinion data; Performing data cleaning on the fused initial public opinion data to obtain cleaned public opinion data; The cleaned public opinion data is standardized to obtain target public opinion data.
[0008] In a possible design, after collecting the initial public opinion data from different target platforms, the initial public opinion data of different target platforms are preprocessed using distributed computing and real-time stream processing technology.
[0009] In a possible design, the Word2Vec model is used to convert the target public opinion data into word vectors.
[0010] In a possible design, the sentiment dictionary score of the target public opinion data is: ; In the formula, is the word vector dataset i The sentiment scores of word vector data, is the total number of word vector data in the word vector dataset.
[0011] In one possible design, the multilingual pre-trained language model adopts a BERT model or a GPT model.
[0012] In one possible design, the composite sentiment score is: S = α · S 1+ β ·S 2; In the formula, S 1 is the sentiment dictionary score of the target public opinion data, α is the weight parameter of the preset sentiment dictionary score, S 2 is the sentiment classification score of the target public opinion data, β It is the weight parameter of the preset sentiment classification score.
[0013] In a second aspect, the present invention provides a public opinion data analysis system for implementing any one of the public opinion data analysis methods described above; the public opinion data analysis system comprises: The data collection module is used to collect initial public opinion data from different target platforms and pre-process the initial public opinion data of different target platforms to obtain target public opinion data; A word vector conversion module, connected to the communication, for performing word vector conversion on the target public opinion data to obtain a word vector data set corresponding to the target public opinion data; A first scoring module is connected to the word vector conversion module for matching the word vector data set based on a preset sentiment dictionary to obtain a sentiment dictionary score of the target public opinion data; A second scoring module is connected to the word vector conversion module for inputting the word vector data set into a preset public opinion analysis model to obtain a sentiment classification score of the target public opinion data; wherein the public opinion analysis model is constructed based on a multilingual pre-trained language model; The score aggregation module is respectively connected to the first scoring module and the second scoring module for obtaining a comprehensive sentiment score of the target public opinion data according to the sentiment dictionary score and the sentiment classification score.
[0014] In a third aspect, the present invention provides an electronic device, comprising: a memory for storing computer program instructions; and, A processor is used to execute the computer program instructions to complete the operation of the public opinion data analysis method as described in any one of the above.
[0015] In a fourth aspect, the present invention provides a computer program product, including a computer program or instructions, which, when executed by a computer, implements a public opinion data analysis method as described in any one of the above.
[0016] The beneficial effects of the present invention are: The present invention discloses a method, system, device and product for analyzing public opinion data, which can improve the accuracy and real-time processing capability of public opinion data analysis. Specifically, in the implementation process of the present invention, first, after collecting initial public opinion data from different target platforms, the present invention pre-processes the initial public opinion data of different target platforms to obtain target public opinion data, and then performs word vector conversion on the target public opinion data to obtain a word vector data set corresponding to the target public opinion data, thereby facilitating subsequent sentiment score analysis. In terms of sentiment analysis, the present invention combines the public opinion analysis model with the traditional sentiment dictionary to realize sentiment score analysis of the word vector data set, and finally obtains the comprehensive sentiment score of the target public opinion data, which can enhance the present invention's ability to understand and analyze complex contexts, and is conducive to improving the accuracy of public opinion data analysis, while meeting the user's needs for efficient and accurate public opinion monitoring.
[0017] Other beneficial effects of the present invention will be further described in the specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 is a flow chart of the public opinion data analysis method in Example 1; Figure 2 It is a module block diagram of the public opinion data analysis system in Example 2; Figure 3 This is a module block diagram of the electronic device in Example 3. DETAILED DESCRIPTION
[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.
[0020] Embodiment 1: This embodiment discloses a method for analyzing public opinion data, which can be executed by, but is not limited to, a computer device or a virtual machine with certain computing resources, such as a personal computer, a smart phone, a personal digital assistant, or a wearable device, or by a virtual machine.
[0021] like Figure 1 As shown, a method for analyzing public opinion data may include but is not limited to the following steps: S1. Collect initial public opinion data from different target platforms, and pre-process the initial public opinion data of different target platforms to obtain target public opinion data; In step S1, initial public opinion data from different target platforms are collected and preprocessed to obtain target public opinion data, including: S101. Use multi-source data collection technology to collect initial public opinion data from different target platforms; specifically, in this embodiment, the initial public opinion data can be collected by using, but not limited to, API interfaces and web crawlers, which are not limited here. The target platform is a platform set according to the public opinion analysis goal, including social media and news websites, which are not limited here.
[0022] S102. Perform data fusion processing on the initial public opinion data of different target platforms to obtain fused initial public opinion data; S103. Perform data cleaning on the fused initial public opinion data to obtain cleaned public opinion data; in this embodiment, the data cleaning process includes steps such as denoising, text segmentation, stop word removal and stem extraction to ensure the quality and consistency of the data.
[0023] S104. Standardize the cleaned public opinion data to obtain target public opinion data. Specifically, in this embodiment, standardize the fused initial public opinion data to achieve the unification of data format and timestamp, eliminate the heterogeneity of data in the fused initial public opinion data, and thus help improve the subsequent data processing efficiency and accuracy.
[0024] Based on the above steps S101-S104, the collection, fusion, cleaning and standardization of data from different platforms can be achieved for subsequent analysis.
[0025] In step S1, after collecting the initial public opinion data from different target platforms, the initial public opinion data of different target platforms are preprocessed using distributed computing and real-time stream processing technology, thereby achieving real-time analysis of large-scale data and reducing latency. It should be noted that this embodiment uses a distributed computing framework and edge computing technology to improve the real-time processing capability of large-scale data.
[0026] Specifically, in this embodiment, a real-time stream processing framework such as Apache Flink or Apache Spark Streaming is used to process a large amount of real-time initial public opinion data streams. In addition, when the initial public opinion data is pre-processed, it is performed at the network edge node to reduce the load of the central server.
[0027] It should also be noted that during the implementation of this embodiment, differential privacy technology is used to protect user data in the initial public opinion data to ensure that user privacy is not violated during the analysis process, and the entire process of data collection, storage and use is compliant and complies with relevant laws and regulations, thereby avoiding infringement of user privacy rights.
[0028] S2. Convert the target public opinion data into word vectors to obtain a word vector data set corresponding to the target public opinion data; it should be noted that, in this embodiment, the target public opinion data is converted into word vectors to obtain multiple word vectors, which retain the semantic relationship between words, so that similar words are close in the vector space. Specifically, each word vector represents the semantic features of a word in the text, and multiple word vectors constitute a word vector data set.
[0029] In step S2 of this embodiment, the Word2Vec model (a model for generating word vectors) is used to convert the target public opinion data into word vectors. Specifically, the Word2Vec model can take context into consideration, has better effects, and has fewer model dimensions, faster training speed, strong versatility, and is suitable for various NLP tasks.
[0030] S3. Match the word vector data set based on a preset sentiment dictionary to obtain the sentiment dictionary score of the target public opinion data; it should be noted that, in this embodiment, the preset sentiment dictionary can be constructed based on a preset sentiment polarity classifier and a domain corpus of the field in which the public opinion analysis target is located, and the sentiment dictionary contains a large number of words marked with sentiment tendencies, and each word corresponds to a sentiment value (such as positive, negative or neutral).
[0031] In this embodiment, in the process of matching the word vector data set based on a preset sentiment dictionary, a preliminary sentiment score is given to each word in combination with the sentiment dictionary. Specifically, for each word vector data in the word vector data set, it is matched with the vocabulary in the sentiment dictionary, and the sentiment value corresponding to the matching vocabulary in the sentiment dictionary is queried. If the word vector data exists in the sentiment dictionary, it is assigned a corresponding sentiment score. If the word vector data is not in the sentiment dictionary, it can be ignored or assigned a neutral score. After obtaining the sentiment scores of all word vector data in the word vector data set, the sentiment scores are weighted averaged to obtain the sentiment dictionary score of the target public opinion data.
[0032] Specifically, in step S3, the sentiment dictionary score of the target public opinion data is: ; In the formula, is the word vector dataset iThe sentiment score of the word vector data is obtained by matching the sentiment dictionary. It is the total number of word vector data in the word vector dataset, which can also be called the total number of valid words.
[0033] S4. Input the word vector data set into the preset public opinion analysis model to obtain the sentiment classification score of the target public opinion data; wherein, the public opinion analysis model is constructed based on a multilingual pre-trained language model; it should be noted that the public opinion analysis model can capture the dependencies between words and retain contextual information in the entire text sequence. In addition, in this embodiment, the public opinion analysis model is constructed based on a multilingual pre-trained language model, which can enable the enhanced public opinion analysis model to process text data in different languages, thereby enhancing the understanding ability of this embodiment for texts in different languages.
[0034] In step S4, the multilingual pre-trained language model adopts a BERT (Bidirectional Encoder Representations from Transformers, a deep bidirectional model based on Transformer) model or a GPT (Generative Pre-trained Transformer, a generative pre-trained model) model.
[0035] Specifically, in this embodiment, the correlation between each word vector data and other word vector data in the word vector data set can be calculated based on the self-attention mechanism, and then the global semantic feature vector of the target public opinion data is obtained, and then the global semantic feature vector is subjected to sentiment classification through a fully connected layer and an activation function (such as Softmax), and the probability distribution P(H)=Softmax(W·H+b) of each sentiment category (positive, negative and neutral) is output, where Softmax() represents the activation function, H is the global semantic feature vector, and W and b are preset weights and bias parameters; and finally the sentiment classification score of the target public opinion data is obtained, that is, the probability value of the corresponding category in the predicted probability distribution.
[0036] In addition, in order to enhance the accuracy of sentiment analysis, this embodiment may further introduce a special sarcasm and metaphor detection module to further improve the performance of the public opinion analysis model in complex situations.
[0037] S5. Obtain a comprehensive sentiment score of the target public opinion data according to the sentiment dictionary score and the sentiment classification score.
[0038] In step S5, the comprehensive sentiment score is: S = α · S 1+β · S 2; In the formula, S 1 is the sentiment dictionary score of the target public opinion data, α is the weight parameter of the preset sentiment dictionary score, S 2 is the sentiment classification score of the target public opinion data, β It should be understood that the parameters α and β are used to balance the influence of the sentiment dictionary score and the sentiment classification score, and are usually obtained through model training optimization, which is not limited here.
[0039] In addition, in this embodiment, the method further includes: S6. Monitor the changing trend of public opinion through time series analysis method and generate a public opinion analysis visualization report.
[0040] Specifically, in this embodiment, an ARIMA (Autoregressive Integrated Moving Average model) model or an LSTM (long and short-term memory) network is used to perform time series analysis on sentiment scores to predict future sentiment trends. In addition, this embodiment also uses visualization tools such as D3.js (Data-Driven Documents, a JavaScript library for creating dynamic, interactive data visualization) or Grafana (an open source data visualization and monitoring platform) to generate real-time sentiment trend charts and topic distribution maps to provide decision makers with intuitive analysis results.
[0041] This embodiment can improve the accuracy and real-time processing capability of public opinion data analysis. Specifically, during the implementation of this embodiment, first, after collecting the initial public opinion data from different target platforms, this embodiment pre-processes the initial public opinion data of different target platforms to obtain the target public opinion data, and then performs word vector conversion on the target public opinion data to obtain a word vector data set corresponding to the target public opinion data, thereby facilitating subsequent sentiment score analysis. In terms of sentiment analysis, this embodiment combines the public opinion analysis model with the traditional sentiment dictionary to realize sentiment score analysis of the word vector data set, and finally obtains the comprehensive sentiment score of the target public opinion data, which can enhance the understanding and analysis capabilities of this embodiment for complex contexts, help improve the accuracy of public opinion data analysis, and at the same time meet the needs of users for efficient and accurate public opinion monitoring.
[0042] Embodiment 2: This embodiment discloses a public opinion data analysis system for implementing the public opinion data analysis method in Embodiment 1; Figure 2 As shown, the public opinion data analysis system includes: The data collection module is used to collect initial public opinion data from different target platforms and pre-process the initial public opinion data of different target platforms to obtain target public opinion data; A word vector conversion module, connected to the communication, for performing word vector conversion on the target public opinion data to obtain a word vector data set corresponding to the target public opinion data; A first scoring module is connected to the word vector conversion module for matching the word vector data set based on a preset sentiment dictionary to obtain a sentiment dictionary score of the target public opinion data; A second scoring module is connected to the word vector conversion module for inputting the word vector data set into a preset public opinion analysis model to obtain a sentiment classification score of the target public opinion data; wherein the public opinion analysis model is constructed based on a multilingual pre-trained language model; The score aggregation module is respectively connected to the first scoring module and the second scoring module for obtaining a comprehensive sentiment score of the target public opinion data according to the sentiment dictionary score and the sentiment classification score.
[0043] It should be noted that the working process, working details and technical effects of the public opinion data analysis system provided in this embodiment 2 can be found in embodiment 1 and will not be repeated here.
[0044] Embodiment 3: Based on Embodiment 1 or 2, this embodiment discloses an electronic device, which may be a smart phone, a tablet computer, a laptop computer, or a desktop computer. The electronic device may be referred to as a user terminal, a portable terminal, a desktop terminal, etc. Figure 3 As shown, the electronic equipment includes: a memory for storing computer program instructions; and, A processor is used to execute the computer program instructions to complete the operation of the public opinion data analysis method as described in any one of Example 1.
[0045] Specifically, the processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen.
[0046] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 302 is used to store at least one instruction, which is used to be executed by the processor 301 to implement the public opinion data analysis method provided in Example 1 of the present application.
[0047] In some embodiments, the terminal may further optionally include: a communication interface 303 and at least one peripheral device. The processor 301, the memory 302 and the communication interface 303 may be connected via a bus or a signal line. Each peripheral device may be connected to the communication interface 303 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 304, a display screen 305 and a power supply 306.
[0048] The communication interface 303 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 301 and the memory 302. In some embodiments, the processor 301, the memory 302, and the communication interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 301, the memory 302, and the communication interface 303 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.
[0049] The radio frequency circuit 304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 304 communicates with a communication network and other communication devices through electromagnetic signals.
[0050] The display screen 305 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof.
[0051] The power supply 306 is used to supply power to various components in the electronic device.
[0052] Embodiment 4: Based on any one of Examples 1 to 3, this embodiment discloses a computer program product, including a computer program or instructions, which, when executed by a computer, implements the public opinion data analysis method as described in any one of Example 1.
[0053] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for analyzing public opinion data, characterized in that: include: Collect initial public opinion data from different target platforms, and pre-process the initial public opinion data from different target platforms to obtain target public opinion data; Performing word vector conversion on the target public opinion data to obtain a word vector data set corresponding to the target public opinion data; Performing matching processing on the word vector data set based on a preset sentiment dictionary to obtain a sentiment dictionary score of the target public opinion data; Inputting the word vector data set into a preset public opinion analysis model to obtain a sentiment classification score of the target public opinion data; wherein the public opinion analysis model is constructed based on a multilingual pre-trained language model; According to the sentiment dictionary score and the sentiment classification score, a comprehensive sentiment score of the target public opinion data is obtained.
2. A method for analyzing public opinion data according to claim 1, characterized in that: Collect initial public opinion data from different target platforms, and pre-process the initial public opinion data from different target platforms to obtain target public opinion data, including: Use multi-source data collection technology to collect initial public opinion data from different target platforms; Perform data fusion processing on the initial public opinion data of different target platforms to obtain the fused initial public opinion data; Performing data cleaning on the fused initial public opinion data to obtain cleaned public opinion data; The cleaned public opinion data is standardized to obtain target public opinion data.
3. A method for analyzing public opinion data according to claim 1, characterized in that: After collecting the initial public opinion data from different target platforms, the initial public opinion data of different target platforms are preprocessed using distributed computing and real-time stream processing technology.
4. A method for analyzing public opinion data according to claim 1, characterized in that: The Word2Vec model is used to convert the target public opinion data into word vectors.
5. A method for analyzing public opinion data according to claim 1, characterized in that: The sentiment dictionary score of the target public opinion data is: ； In the formula, is the word vector dataset i The sentiment scores of word vector data, is the total number of word vector data in the word vector dataset.
6. A method for analyzing public opinion data according to claim 1, characterized in that: The multilingual pre-trained language model adopts a BERT model or a GPT model.
7. A method for analyzing public opinion data according to claim 1, characterized in that: The comprehensive sentiment score is: S = α · S 1+ β · S 2； In the formula, S 1 is the sentiment dictionary score of the target public opinion data, α is the weight parameter of the preset sentiment dictionary score, S 2 is the sentiment classification score of the target public opinion data, β It is the weight parameter of the preset sentiment classification score.
8. A public opinion data analysis system, characterized in that: Used to implement the public opinion data analysis method according to any one of claims 1 to 7; the public opinion data analysis system comprises: The data collection module is used to collect initial public opinion data from different target platforms and pre-process the initial public opinion data of different target platforms to obtain target public opinion data; A word vector conversion module, connected to the communication, for performing word vector conversion on the target public opinion data to obtain a word vector data set corresponding to the target public opinion data; A first scoring module is connected to the word vector conversion module for matching the word vector data set based on a preset sentiment dictionary to obtain a sentiment dictionary score of the target public opinion data; A second scoring module is connected to the word vector conversion module for inputting the word vector data set into a preset public opinion analysis model to obtain a sentiment classification score of the target public opinion data; wherein the public opinion analysis model is constructed based on a multilingual pre-trained language model; The score aggregation module is respectively connected to the first scoring module and the second scoring module for obtaining a comprehensive sentiment score of the target public opinion data according to the sentiment dictionary score and the sentiment classification score.
9. An electronic device, characterized in that: include: a memory for storing computer program instructions; as well as, A processor is used to execute the computer program instructions to complete the operation of the public opinion data analysis method as described in any one of claims 1 to 7.
10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or the instruction is executed by a computer, the method for analyzing public opinion data as described in any one of claims 1 to 7 is implemented.