An Opinion Information Recognition Method and System for Big Data Processing

Abstract

The present invention discloses a method and system for identifying public opinion information for big data processing, which relates to the technical field of public opinion data, including collecting public opinion information data and performing preprocessing, constructing a multi-level knowledge graph; using a graph neural network to learn the multi-level knowledge graph to construct a public opinion information recognition model; using the public opinion information recognition model to identify and classify public opinion information data; formulating a feedback mechanism to update the multi-level knowledge graph and the public opinion information recognition model. By using a multi-level knowledge graph and a graph convolutional network, the public opinion information recognition model of the present invention can perform multi-level abstraction from the underlying entity relationship to the high-level theme layer, improving the accuracy of public opinion classification; by introducing a graph attention mechanism, the public opinion information recognition model can more accurately capture the connections between nodes, which helps to improve the node update effect of the multi-level knowledge graph.

Description

Technical Field

[0001] The present invention relates to the technical field of public opinion data, and in particular to a method and system for identifying public opinion information for big data processing. Background Art

[0002] With the advent of the big data era, the rapid growth and complexity of public opinion information have become the focus of social attention. In this context, the method for identifying public opinion information for big data processing has gradually become crucial. In the past technological development, the processing of public opinion information data mainly focused on the fields of text mining and natural language processing. However, due to the diversity, real-time nature, and huge scale of public opinion information, traditional processing methods have become insufficiently flexible and efficient. The existing technologies mainly focus on mining and analyzing text data to understand the trend of public opinion. Traditional text mining methods include keyword extraction and sentiment analysis, but these methods have certain limitations in dealing with large-scale data and complex relationships. Currently, the graph neural network model, as a new machine learning method, is widely used in the learning and analysis of complex data. The graph neural network model performs excellently in processing graph-structured data and can quickly and accurately capture the non-linear relationships in the data. Therefore, it has become a new option for processing public opinion information.

[0003] However, although the existing technologies have made some progress in public opinion information processing, the current common solutions have many drawbacks, including: First, traditional text mining methods have limited understanding of the context of information and are difficult to accurately capture the deep relationships in public opinion information. Second, the existing methods have limited construction and utilization of multi-level knowledge graphs, resulting in insufficient analysis ability for complex relationships. In addition, the real-time nature of public opinion information also poses certain pressure on the processing speed of the existing technologies. Therefore, the application of graph neural networks and the construction of multi-level knowledge graphs provide a new way for more accurate and efficient identification and classification of large-scale public opinion information. Summary of the Invention

[0004] In view of the problems in the prior art, such as the difficulty in accurately capturing the deep relationships in public opinion information and the insufficient analysis ability for complex relationships when identifying the types of public opinion data, the present invention is proposed.

[0005] Therefore, the problem to be solved by the present invention is how to provide a method for deeply analyzing the relevance between public opinion information, more comprehensively understanding and analyzing public opinion information, and improving the speed of public opinion recognition.

[0006] To solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides a public opinion information recognition method for big data processing, which includes collecting public opinion information data and performing preprocessing, constructing a multi-level knowledge graph; using a graph neural network to learn the multi-level knowledge graph to construct a public opinion information recognition model; using the public opinion information recognition model to identify and classify public opinion information data; formulating a feedback mechanism to update the multi-level knowledge graph and the public opinion information recognition model.

[0008] As a preferred solution of the public opinion information recognition method for big data processing according to the present invention, wherein: the public opinion information data includes social media information and news reports; the construction of the multi-level knowledge graph includes the following steps: using the named entity recognition NER technology to extract information from the preprocessed public opinion information data; integrating the extracted information to construct a multi-level knowledge graph; using a graph database to store the constructed multi-level knowledge graph; the entity recognition includes people, places and organizations; the multi-level knowledge graph includes a basic information layer, an entity relationship layer and a theme layer.

[0009] As a preferred solution of the public opinion information recognition method for big data processing according to the present invention, wherein: the construction of the public opinion information recognition model includes the following steps: converting the constructed multi-level knowledge graph into graph structure data; using a graph convolutional network GCN model to learn the graph structure data to capture the semantic relationship between nodes and construct a public opinion information recognition model; the public opinion information recognition model captures the connection between nodes by introducing a graph attention mechanism to update the nodes of the multi-level knowledge graph.

[0010] As a preferred solution of the public opinion information recognition method for big data processing according to the present invention, wherein: the specific formula of the graph attention mechanism is as follows:

[0011]

[0012] Wherein, is the attention weight of node v to neighbor node u; is the attention score between node v and node u; N(v) is the set of neighbor nodes of node v; is the attention score between node k and node u; the specific formula for updating the nodes of the multi-level knowledge graph is as follows:

[0013]

[0014] Wherein, is the representation of node v at the l+1 layer; w uv is the edge weight between node v and node u; W (l) is the weight matrix; is the representation of node u in the l-th layer.

[0015] As a preferred solution of the public opinion information recognition method for big data processing according to the present invention, wherein: the recognition and classification include sentiment classification and topic classification; the recognition and classification of public opinion information data include the following steps: collecting the public opinion information data to be recognized and performing preprocessing; using the public opinion information recognition model to recognize and classify the preprocessed public opinion information data, and outputting the probability distribution of each node; calculating the loss function of the public opinion information recognition model; visualizing the results of the recognition and classification; the specific formula of the probability distribution of each node is as follows:

[0016]

[0017]

[0018] where P(y i = c) is the probability that node i belongs to category c; z ic is the score that node i belongs to category c; C is the number of categories; z ij is the score that node i belongs to category j; D is the dimension of the node representation; is the weight connecting node i and node j; is the representation of node j in the l-th layer; is the bias term of category c.

[0019] As a preferred solution of the public opinion information recognition method for big data processing according to the present invention, wherein: the specific situation of recognizing and classifying the preprocessed public opinion information data is as follows: when the node belongs to the sentiment category, if the probability distribution of each node is greater than or equal to the first threshold, it is determined that the node belongs to the corresponding sentiment category; when the node belongs to the sentiment category, if the probability distribution of each node is less than the first threshold, it is determined that the node does not belong to the corresponding sentiment category; when the node data is the theme category, if the probability distribution of each node is greater than or equal to the second threshold, it is determined that the node belongs to the corresponding theme category; when the node data is the theme category, if the probability distribution of each node is less than the second threshold, it is determined that the node does not belong to the corresponding theme category; if the probability distribution of each node is greater than the first threshold or the second threshold continuously for N times, the corresponding threshold is increased to reduce the sensitivity to the category; if the probability distribution of each node is less than the first threshold or the second threshold continuously for N times, the corresponding threshold is decreased to increase the sensitivity to the category; if the probability distribution of each node is equal to the first threshold or the second threshold continuously for N times, the corresponding threshold remains unchanged.

[0020] As a preferred solution of the public opinion information recognition method for big data processing according to the present invention, wherein: the specific formula of the loss function is as follows:

[0021]

[0022] Among them, Loss is the loss function of the public opinion information recognition model; N is the number of nodes in the dataset; y ic is the label indicating whether node i belongs to category c.

[0023] In a second aspect, to further solve the security problems existing in the recognition of public opinion data, the embodiments of the present invention provide a public opinion information recognition system for big data processing, which includes: a knowledge graph module for collecting public opinion information data and using named entity recognition (NER) technology for information extraction to construct a multi-level knowledge graph; a model construction module for using a graph convolutional network (GCN) model to learn the multi-level knowledge graph and construct a public opinion information recognition model; a category judgment module for using the public opinion information recognition model to identify and classify the public opinion information data to be recognized and judge the category of the public opinion information data; and a model optimization module for optimizing the model using real-time data and user feedback.

[0024] In a third aspect, the embodiments of the present invention provide a computer device, including a memory and a processor, where the memory stores a computer program, and: when the computer program is executed by the processor, any step of a public opinion information recognition method for big data processing as described in the first aspect of the present invention is implemented.

[0025] In a fourth aspect, the embodiments of the present invention provide a computer-readable storage medium, on which a computer program is stored, and: when the computer program is executed by the processor, any step of a public opinion information recognition method for big data processing as described in the first aspect of the present invention is implemented.

[0026] The beneficial effects of the present invention are as follows: By proposing a public opinion information recognition method for big data processing and using a multi-level knowledge graph and a graph convolutional network, the public opinion information recognition model can perform multi-level abstraction from the underlying entity relationships to the high-level theme layer, which helps to more comprehensively understand and analyze public opinion information and improve the accuracy of public opinion classification; by introducing a graph attention mechanism, the public opinion information recognition model can more accurately capture the connections between nodes, which helps to improve the node update effect of the multi-level knowledge graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. Among them:

[0028] Figure 1It is the flowchart for identifying public opinion information of the present invention in Embodiment 1. Detailed implementation manners

[0029] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following will describe the detailed implementation manners of the present invention with reference to the accompanying drawings of the specification.

[0030] In the following description, many specific details are set forth to facilitate a thorough understanding of the present invention. However, the present invention may be implemented in other ways different from those described herein. Those skilled in the art can make similar generalizations without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0031] Secondly, the so-called "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation manner of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it an individual or alternative embodiment that is mutually exclusive with other embodiments.

[0032] Embodiment 1

[0033] Referring to Figure 1 , which is the first embodiment of the present invention. This embodiment provides a method for identifying public opinion information for big data processing, including the following steps:

[0034] S1: Collect public opinion information data and perform preprocessing to construct a multi-level knowledge graph.

[0035] Preferably, as Figure 1 shown in the flowchart for identifying public opinion data of the present invention, by collecting public opinion data and performing preprocessing, a multi-level knowledge graph is constructed. Then, a graph neural network is used to learn the multi-level knowledge graph and construct a public opinion information recognition model. After that, the public opinion data is recognized and classified by the public opinion information recognition model, and the category of the public opinion data and whether to adjust the threshold are judged according to the comparison between the probability distribution of each node and the threshold. Finally, a feedback mechanism is established to update the multi-level knowledge graph and the public opinion information recognition model according to real-time data and user feedback.

[0036] Preferably, the public opinion information data includes social media information and news reports.

[0037] Specifically, constructing the multi-level knowledge graph includes the following steps: using the named entity recognition (NER) technology to extract information from the preprocessed public opinion information data, including entity recognition and relationship extraction.

[0038] Integrate the extracted information to construct a multi-level knowledge graph.

[0039] A multi - level knowledge graph constructed by using a graph database for storage.

[0040] Furthermore, entity recognition includes people, locations, and organizations.

[0041] Furthermore, the multi - level knowledge graph includes a basic information layer, an entity relationship layer, and a theme layer.

[0042] S2: Use a graph neural network to learn the multi - level knowledge graph and construct a public opinion information recognition model.

[0043] Preferably, constructing the public opinion information recognition model includes the following steps: Convert the constructed multi - level knowledge graph into graph - structured data, where nodes represent entities and edges represent the relationships between entities.

[0044] Use the graph convolutional network GCN model to learn the graph - structured data, capture the semantic relationships between nodes, and construct a public opinion information recognition model.

[0045] Furthermore, the public opinion information recognition model captures the connections between nodes by introducing a graph attention mechanism and updates the nodes of the multi - level knowledge graph.

[0046] Specifically, the specific formula of the graph attention mechanism is as follows:

[0047]

[0048] Among them, is the attention weight of node v for neighbor node u; is the attention score between node v and node u; N(v) is the set of neighbor nodes of node v; is the attention score between node k and node u.

[0049] Specifically, the specific formula for updating the nodes of the multi - level knowledge graph is as follows:

[0050]

[0051] Among them, is the representation of node v at the l + 1 layer; w uv is the edge weight between node v and node u; W (l) is the weight matrix; is the representation of node u at the l layer.

[0052] S3: Use the public opinion information recognition model to identify and classify public opinion information data.

[0053] Preferably, the identification and classification include sentiment classification and topic classification.

[0054] Further, the identification and classification of public opinion information data include the following steps: Collect the public opinion information data to be identified and perform preprocessing.

[0055] Use the public opinion information recognition model to identify and classify the preprocessed public opinion information data, and output the probability distribution of each node, indicating the possibility that the node belongs to different categories.

[0056] Calculate the loss function of the public opinion information recognition model.

[0057] Visualize the results of the identification and classification.

[0058] Specifically, the specific formula for the probability distribution of each node is as follows:

[0059]

[0060]

[0061] Among them, P(y i = c) is the probability that node i belongs to category c; z ic is the score that node i belongs to category c; C is the number of categories; z ij is the score that node i belongs to category j; D is the dimension represented by the node; is the weight connecting node i and node j; is the representation of node j in the l-th layer; is the bias term of category c.

[0062] Further, the specific situation of identifying and classifying the preprocessed public opinion information data is as follows: When the node belongs to the sentiment category, if the probability distribution of each node is greater than or equal to the first threshold, it is determined that the node belongs to the corresponding sentiment category.

[0063] When the node belongs to the sentiment category, if the probability distribution of each node is less than the first threshold, it is determined that the node does not belong to the corresponding sentiment category.

[0064] When the node is of the data topic category, if the probability distribution of each node is greater than or equal to the second threshold, it is determined that the node belongs to the corresponding topic category.

[0065] When the node is of the data topic category, if the probability distribution of each node is less than the second threshold, it is determined that the node does not belong to the corresponding topic category.

[0066] If the probability distribution of each node is greater than the first threshold or the second threshold for N consecutive times, the corresponding threshold is increased to reduce the sensitivity to the category.

[0067] If the probability distribution of each node is less than the first threshold or the second threshold for N consecutive times, the corresponding threshold is decreased to increase the sensitivity to the category.

[0068] If the probability distribution of each node is equal to the first threshold or the second threshold continuously for N times, the corresponding threshold remains unchanged.

[0069] Specifically, the specific formula of the loss function is as follows:

[0070]

[0071] Among them, Loss is the loss function of the public opinion information recognition model; N is the number of nodes in the dataset; y ic is a label indicating whether node i belongs to category c.

[0072] S4: Establish a feedback mechanism to update the multi-level knowledge graph and the public opinion information recognition model.

[0073] Specifically, the feedback mechanism includes the following steps: regularly collect the latest public opinion information data and update the constructed multi-level knowledge graph.

[0074] Use the updated multi-level knowledge graph to retrain the public opinion information recognition model to improve the accuracy and generalization ability of the public opinion information recognition model.

[0075] Collect user evaluations and suggestions to evaluate and improve the public opinion information recognition model, and adjust the parameters of the public opinion information recognition model.

[0076] This embodiment also provides a public opinion information recognition system for big data processing, including: a knowledge graph module for collecting public opinion information data and using named entity recognition (NER) technology for information extraction to construct a multi-level knowledge graph; a model construction module for using a graph convolutional network (GCN) model to learn the multi-level knowledge graph and construct a public opinion information recognition model; a category judgment module for using the public opinion information recognition model to identify and classify the public opinion information data to be recognized and judge the category of the public opinion information data; a model optimization module for optimizing the model using real-time data and user feedback.

[0077] This embodiment also provides a computer device applicable to a public opinion information recognition method for big data processing, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement a public opinion information recognition method for big data processing as proposed in the above embodiment.

[0078] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a carrier network, NFC (Near Field Communication), or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or may be a button, a trackball, or a touchpad provided on the casing of the computer device, or may also be an external keyboard, a touchpad, or a mouse, etc.

[0079] This embodiment also provides a storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the method for identifying public opinion information for big data processing proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM for short), Electrically Erasable Programmable Read-Only Memory (EEPROM for short), Erasable Programmable Read-Only Memory (EPROM for short), Programmable Read-Only Memory (PROM for short), Read-Only Memory (ROM for short), magnetic memory, flash memory, a magnetic disk, or an optical disc.

[0080] In summary, by proposing a method for identifying public opinion information for big data processing, the present invention utilizes a multi-level knowledge graph and a graph convolutional network, enabling the public opinion information recognition model to perform multi-level abstraction from the underlying entity relationships to the high-level theme layer, which helps to more comprehensively understand and analyze public opinion information and improve the accuracy of public opinion classification; by introducing a graph attention mechanism, the public opinion information recognition model can more accurately capture the connections between nodes, which helps to improve the node update effect of the multi-level knowledge graph.

[0081] Embodiment 2

[0082] Referring to Table 1 and Table 2, this is the second embodiment of the present invention. The difference between this embodiment and the first embodiment is that, in order to verify its beneficial effects, the operation data and related descriptions of the present invention in the actual environment are provided.

[0083] In this example, public opinion information data is collected from social media information and news reports. The multi-level knowledge graph and graph convolutional network are used to analyze the public opinion information, extract the relevance between public opinion information, and then use the constructed public opinion recognition model to recognize the public opinion information and conduct public opinion classification. Among them, as shown in Table 1 and Table 2, the present invention provides data tables for the sentiment classification and topic classification of public opinion information.

[0084] Table 1 Probability Table for Sentiment Classification of Public Opinion Information

[0085] Knowledge graph node Sentiment classification probability Node 1 0.85 Node 2 0.62 Node 3 0.91 ... ... Node n 0.35

[0086] Table 2 Probability Table for Topic Classification of Public Opinion Information

[0087] Knowledge graph node Subject classification probability Node 1 0.78 Node 2 0.55 Node 3 0.82 ... ... Node n 0.41

[0088] As can be seen from the above two tables, the present invention classifies the sentiment and topic of public opinion information by using a multi-level knowledge graph and a graph convolutional network. Through the analysis of the probability distribution, the topic category to which the node belongs can be judged more accurately. At the same time, by dynamically adjusting the threshold, the robustness of the public opinion recognition model is improved, which helps to enhance the adaptability of the public opinion recognition model.

[0089] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not restrictive. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

Claims

1. A method for identifying public opinion information for big data processing, characterized in that: include: Collect public opinion information data and pre-process it to build a multi-level knowledge graph; Using a graph neural network to learn the multi-level knowledge graph and construct a public opinion information recognition model; Using the public opinion information recognition model to identify and classify public opinion information data; Formulate a feedback mechanism to update the multi-level knowledge graph and the public opinion information identification model; The public opinion information data includes social media information and news reports; The construction of a multi-level knowledge graph includes the following steps: Use named entity recognition (NER) technology to extract information from pre-processed public opinion information data; Integrate the extracted information to build a multi-level knowledge graph; Multi-level knowledge graphs built using graph database storage; The entity identification includes persons, places and organizations; The multi-level knowledge graph includes a basic information layer, an entity relationship layer and a subject layer; The identification and classification include sentiment classification and topic classification; The identification and classification of public opinion information data includes the following steps: Collect public opinion information data to be identified and perform pre-processing; Use the public opinion information recognition model to identify and classify the preprocessed public opinion information data, and output the probability distribution of each node; Calculate the loss function of the public opinion information recognition model; Visualize the results of recognition and classification; The specific formula for the probability distribution of each node is as follows: Among them, P(y i =c) is the probability that node i belongs to category c; ic is the score of node i belonging to category c; C is the number of categories; z ij is the score of node i belonging to category j; D is the dimension represented by the node; is the weight connecting node i and node j; is the representation of node j at layer l; is the bias term of category c; The specific situation of identifying and classifying the pre-processed public opinion information data is as follows: When the node belongs to an emotion category, if the probability distribution of each node is greater than or equal to the first threshold, it is determined that the node belongs to the corresponding emotion category; When a node belongs to an emotion category, if the probability distribution of each node is less than the first threshold, it is determined that the node does not belong to the corresponding emotion category; When the node data is a topic category, if the probability distribution of each node is greater than or equal to the second threshold, it is determined that the node belongs to the corresponding topic category; When the node data is a topic category, if the probability distribution of each node is less than the second threshold, it is determined that the node does not belong to the corresponding topic category; If the probability distribution of each node is greater than the first threshold or the second threshold for N consecutive times, the corresponding threshold is increased to reduce the sensitivity to the category; If the probability distribution of each node is less than the first threshold or the second threshold for N consecutive times, the corresponding threshold is lowered to increase the sensitivity to the category; If the probability distribution of each node is equal to the first threshold or the second threshold N times in a row, the corresponding threshold is kept unchanged.

2. The method for identifying public opinion information for big data processing according to claim 1, characterized in that: The construction of the public opinion information identification model includes the following steps: Convert the constructed multi-level knowledge graph into graph structure data; Use the graph convolutional network (GCN) model to learn graph structure data, capture the semantic relationship between nodes, and build a public opinion information recognition model; The public opinion information recognition model captures the connections between nodes by introducing a graph attention mechanism and updates the nodes of the multi-level knowledge graph.

3. The method for identifying public opinion information for big data processing according to claim 2, characterized in that: The specific formula of the graph attention mechanism is as follows: in, is the attention weight of node v to its neighbor node u; is the attention score of node v and node u; N(v) is the set of neighbor nodes of node v; is the attention score of node k and node u; The specific formula for updating the nodes of the multi-level knowledge graph is as follows: in, is the representation of node v at the l+1th layer; W (l) is the weight matrix; is the representation of node u at layer l.

4. The method for identifying public opinion information for big data processing according to claim 3, characterized in that: The specific formula of the loss function is as follows: Among them, Loss is the loss function of the public opinion information recognition model; N is the number of nodes in the data set; y ic is a label indicating whether node i belongs to category c.

5. A public opinion information identification system for big data processing, based on a public opinion information identification method for big data processing according to any one of claims 1 to 4, characterized in that: include, The knowledge graph module is used to collect public opinion information data and use named entity recognition (NER) technology to extract information and build a multi-level knowledge graph; The model building module is used to use the graph convolutional network (GCN) model to learn multi-level knowledge graphs and build a public opinion information recognition model; A category judgment module is used to identify and classify the public opinion information data to be identified by using the public opinion information recognition model, and to determine the category of the public opinion information data; The model optimization module is used to optimize the model using real-time data and user feedback.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of a public opinion information identification method for big data processing as described in any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a public opinion information identification method for big data processing as described in any one of claims 1 to 4 are implemented.

Images