Method, device and storage medium for extracting and standardizing subject information in network public opinion

Abstract

The application discloses a network public opinion subject information extraction and standardization method and device and a storage medium, relates to the technical field of network public opinion analysis, and comprises the following steps: acquiring network public opinion data, and performing multi-modal data fusion and preprocessing; a risk subject identification model based on multi-level semantic fusion is used to extract preliminary subject information from the preprocessed data; the preliminary subject information is subjected to standardization processing, including entity disambiguation, form normalization and relationship verification, to generate standardized subject information; based on an evolution analysis model, behavior trajectory tracking and trend analysis are performed on the standardized subject information; and the standardized subject information and analysis results are output. The application adopts multiple technologies such as DOM tree structure analysis, information entropy evaluation, multi-modal cross verification and context semantic understanding based on a Transformer, can accurately locate and identify core subject information in a webpage, and effectively understands the context.

Description

Technical Field

[0001] This invention relates to the field of online public opinion analysis technology, and in particular to methods, equipment and storage media for extracting and standardizing key information in online public opinion. Background Technology

[0002] With the widespread adoption of the internet and social media, cyberspace has become a crucial venue for the public to express opinions and disseminate information. Online public opinion monitoring and analysis are of paramount importance for government social governance, corporate brand management, and public crisis response. One of the core tasks of online public opinion analysis is to accurately identify, classify, and deeply analyze the entities involved in public opinion. However, public opinion data comes from a wide range of sources, including news websites, social media, forums, and short video platforms, and its formats encompass various modalities such as plain text, posts with accompanying images, and video content. This multi-source, heterogeneous data has inconsistent formats and varying quality, making it difficult for traditional methods to effectively integrate and process it uniformly.

[0003] In the online context, the same entity often has multiple ways of being expressed, and there are numerous abbreviations, aliases, and even online nicknames. At the same time, name ambiguity is a prominent issue. Traditional methods based on keyword matching or simple rules are insufficient for accurately disambiguating and standardizing entities, easily leading to information confusion and misjudgment.

[0004] Furthermore, existing technologies primarily focus on the static identification and statistical analysis of entities, failing to effectively track the evolution of their behavioral trajectories during public opinion events. An entity's influence, emotional inclinations, and relationships change dynamically over time. Without analysis of the entity's temporal behavioral patterns, the evolution of its relationship networks, and the path of influence propagation, it is impossible to truly understand the inherent laws and driving factors of public opinion development, and thus, to achieve effective risk warnings. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method, device, and storage medium for extracting and standardizing key information in online public opinion.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] Methods for extracting and standardizing key information in online public opinion include the following steps:

[0008] Step S1: Acquire online public opinion data and perform multimodal data fusion and preprocessing;

[0009] Step S2: Based on the risk subject identification model of multi-level semantic fusion, extract preliminary subject information from the preprocessed data;

[0010] Step S3: Standardize the preliminary subject information, including entity disambiguation, form normalization and relation verification, to generate standardized subject information;

[0011] Step S4: Based on the evolutionary analysis model, perform behavioral trajectory tracking and trend analysis on standardized subject information;

[0012] Step S5: Output the standardized main information and analysis results.

[0013] A device for extracting and standardizing key information in online public opinion, the device being equipped with a standardization system, the standardization system comprising:

[0014] Data Acquisition and Preprocessing Module: Responsible for acquiring public opinion data from online platforms and performing preprocessing operations such as data cleaning, deduplication, and format standardization;

[0015] Subject Information Extraction Module: Employs semantic fusion technology to identify and extract various types of subject information from online public opinion;

[0016] Subject information standardization module: Through multi-strategy fusion, the extracted subject information is disambiguated, normalized, and verified;

[0017] Subject Behavior Analysis Module: Based on time series analysis and network relationship diagrams, it tracks the subject's behavioral trajectory and analyzes the evolution of subject relationships;

[0018] Output module: Outputs the processing results in a structured format.

[0019] Preferably, the data acquisition and preprocessing module includes:

[0020] The multi-source heterogeneous data acquisition unit adopts a pluggable collector architecture. For web page data, it uses the Python-based Scrapy or Requests framework for targeted crawling, and complies with website specifications through Robots.txt detection and dynamic latency control. For API data, it uses the OAuth2.0 protocol for authorization and a token polling mechanism to avoid call frequency limits. Direct database acquisition is achieved by configuring a JDBC / ODBC connection pool and using SQL queries or listening to database logs to achieve incremental collection.

[0021] The multimodal data fusion unit assigns globally unique UUIDs and timestamps to all incoming data streams. For non-text data, it uses the Tesseract OCR engine to recognize text in images and performs layout analysis using a CNN convolutional neural network. It converts audio into text based on the Whisper speech recognition model and separates the vocal channels of different speakers. It uses PP-OCRv3 to extract frames and recognize subtitles and the main subject of the image. Finally, it associates multimodal data generated by the same event together using UUIDs to form a unified data record.

[0022] Preferably, the data acquisition and preprocessing module further includes a distributed caching unit, which uses Kafka or Pulsar message queues as buffers to decouple data acquisition from subsequent processing. After the data flows into the queue, it is processed in real time by Flink or Spark Streaming engines. The cleaned and fused data is stored in a distributed file system or columnar database, and a Redis cache is established for frequently queried hot data to improve data retrieval efficiency.

[0023] Preferably, the main information extraction module includes:

[0024] The webpage structure analysis unit parses the webpage source code into a DOM tree and then uses a visual block detection algorithm to calculate the layout density and semantic weight of each DOM node by parsing CSS styles and element coordinates. By setting a threshold, the main text block can be accurately located and extracted. The layout density is the ratio of the effective text length within the node to the pixel area it occupies, and the semantic weight is the cosine similarity between the node content and the title.

[0025] The node evaluation unit, by calculating information entropy, helps extract the content of nodes containing core subject information in the DOM tree;

[0026] The multimodal subject recognition unit uses the NER model for Chinese text and the en_core_web_lg model from the SpaCy library for English text. For text extracted from audio and video, the acoustic features and context are combined, and all recognition results are cross-validated and assigned a confidence score.

[0027] The context semantic understanding unit uses a Transformer-based deep learning model for context semantic encoding; it analyzes the context surrounding the target word to generate a high-dimensional semantic vector; and it determines the most likely result by calculating the cosine similarity between this vector and the candidate meaning vector.

[0028] Preferably, the calculation formula for the node evaluation unit is as follows:

[0029] S=-Σ(p(x i )*log(p(x i )))

[0030] Where p(x) i ) represents the probability distribution of a specific feature; the node evaluation unit sorts the nodes and prioritizes extracting content with high entropy values.

[0031] Preferably, the subject information standardization module includes:

[0032] The entity linking unit links each extracted subject name to a standard entity in the knowledge base;

[0033] The relational graph construction unit uses a graph database to store and represent entity relationships; where nodes are standardized entities and edges are relationships between entities; the attributes of edges include relation strength, start and end time, and data source.

[0034] Preferably, the entity linking unit calculates entity similarity using the following formula:

[0035] S(e1,e2)=α*S text +β*S attr +γ*S rel

[0036] Among them, S text It is text similarity based on edit distance or the BM25 algorithm; S attr It is attribute similarity; S rel α represents the relationship similarity; β and γ are the weights; the entity linking unit selects the candidate entity with the highest comprehensive similarity for linking. If the highest score is lower than the threshold, it is determined to be a new entity.

[0037] Preferably, the subject behavior analysis module includes:

[0038] The behavioral profiling unit constructs a dynamic profile vector for each subject. The vector dimensions include activity index, emotional tone, and topic tags. Among them, the activity index is the number of times the subject speaks per unit time after logarithmic smoothing; the emotional tone is obtained by calculating the emotional value of each text segment using an emotional dictionary and then calculating the weighted average over the period; and the topic tags are the key topic distributions extracted from the subject's speech using an LDA topic model.

[0039] The time series pattern mining unit transforms the subject's behavioral indicators into time series data; first, it uses a sliding window to smooth the data and remove short-term fluctuations; then, it uses an ARIMA model or LSTM to predict the sequence; by analyzing the residuals between the predicted and actual values ​​and combining them with CUSUM, it discovers abnormal fluctuations in the subject's behavior.

[0040] The influence propagation analysis unit models the interaction between subjects as a directed weighted graph; subjects are nodes, the direction of the edges represents information flow, and the weights represent the intensity of the interaction; independent cascade model (ICM) or linear threshold model (LTM) is used for simulation; through multiple simulations, the propagation range and propagation efficiency of each node are calculated; thus, key subjects on the propagation path are identified, where the propagation range is the number of nodes finally activated, and the propagation efficiency is the activation speed.

[0041] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0042] The beneficial effects of this invention are as follows:

[0043] 1. This invention employs multiple technologies, including DOM tree structure analysis, information entropy evaluation, multimodal cross-validation, and Transformer-based contextual semantic understanding, to accurately locate and identify the core information in a webpage and effectively understand its context, thereby greatly improving the accuracy of information extraction.

[0044] 2. This invention associates the extracted fuzzy subjects with standard entities in the knowledge base, achieving entity disambiguation and form normalization, and constructs a dynamic relationship graph, so that scattered information is transformed into a structured and traceable knowledge system.

[0045] 3. This invention not only performs static analysis, but also tracks the behavior trajectory of the subject, uncovers potential patterns, and identifies key nodes through time-series analysis, behavioral profiling, and influence propagation simulation. Based on preset rules and anomaly detection algorithms, it achieves dynamic and intelligent public opinion early warning. Attached Figure Description

[0046] Figure 1 This is a flowchart of the method for extracting and standardizing main information in online public opinion proposed in this invention. Detailed Implementation

[0047] The technical solution of the present invention will be further described in detail below with reference to specific embodiments.

[0048] Example 1:

[0049] Methods for extracting and standardizing key information in online public opinion include the following steps:

[0050] Step S1: Acquire online public opinion data, including data in various forms such as text, images, and videos, and perform multimodal data fusion and preprocessing;

[0051] Step S2: Based on the risk subject identification model of multi-level semantic fusion, extract preliminary subject information from the preprocessed data;

[0052] Step S3: Standardize the preliminary subject information, including entity disambiguation, form normalization and relation verification, to generate standardized subject information;

[0053] Step S4: Based on the evolutionary analysis model, perform behavioral trajectory tracking and trend analysis on standardized subject information;

[0054] Step S5: Output the standardized main information and analysis results.

[0055] Example 2:

[0056] Equipment for extracting and standardizing key information in online public opinion, equipped with a standardization system, the standardization system including:

[0057] Data Acquisition and Preprocessing Module: Responsible for acquiring public opinion data from online platforms and performing preprocessing operations such as data cleaning, deduplication, and format standardization;

[0058] Subject Information Extraction Module: Employs semantic fusion technology to identify and extract various types of subject information from online public opinion;

[0059] Subject information standardization module: Through multi-strategy fusion, the extracted subject information is disambiguated, normalized, and verified;

[0060] Subject Behavior Analysis Module: Based on time series analysis and network relationship diagrams, it tracks the subject's behavioral trajectory and analyzes the evolution of subject relationships;

[0061] Output module: Outputs the processing results in a structured format.

[0062] The data acquisition and preprocessing module includes:

[0063] The multi-source heterogeneous data acquisition unit adopts a pluggable collector architecture. For web page data, it uses the Python-based Scrapy or Requests framework for targeted crawling, and complies with website specifications through Robots.txt detection and dynamic latency control. For API data, it uses the OAuth2.0 protocol for authorization and a token polling mechanism to avoid call frequency limits. Direct database acquisition is achieved by configuring a JDBC / ODBC connection pool and using SQL queries or listening to database logs to achieve incremental collection.

[0064] The multimodal data fusion unit assigns globally unique UUIDs and timestamps to all incoming data streams; for non-text data, it uses the TesseractOCR engine to recognize text in images and performs layout analysis using a CNN convolutional neural network; it converts audio into text based on the Whisper speech recognition model and separates the vocal channels of different speakers; it uses PP-OCRv3 to extract frames and recognize subtitles and the main subject of the image; finally, it associates multimodal data generated by the same event together using UUIDs to form a unified data record.

[0065] The distributed caching unit uses Kafka or Pulsar message queues as buffers to decouple data collection from subsequent processing. After the data flows into the queue, it is processed in real time by Flink or Spark Streaming engines. For example, it can count the frequency of mentions of a certain topic within a time window and detect public opinion hotspots in real time. The cleaned and integrated data will be stored in a distributed file system or columnar database, and a Redis cache will be established for frequently queried hot data to improve data retrieval efficiency.

[0066] The subject information extraction module includes:

[0067] The webpage structure analysis unit parses the webpage source code into a DOM tree and then uses a visual block detection algorithm to calculate the layout density and semantic weight of each DOM node by parsing CSS styles and element coordinates. By setting a threshold, the main text block can be accurately located and extracted. The layout density is the ratio of the effective text length within the node to the pixel area it occupies, and the semantic weight is the cosine similarity between the node content and the title.

[0068] The node evaluation unit, by calculating information entropy, helps extract the content of nodes containing core information from the DOM tree; the formula is as follows:

[0069] S=-Σ(p(x i )*log(p(x i )))

[0070] Where p(x) i ) represents the probability distribution of a specific feature (such as tag type, keyword frequency);

[0071] For a DOM node, if its text length is moderate, its punctuation is standard, and its link density is low, its information entropy will be high, indicating that the node may carry a larger amount of main text information; based on this characteristic, the node evaluation unit sorts the nodes and prioritizes extracting content with high entropy values.

[0072] The multimodal subject recognition unit uses the NER model for Chinese text and the en_core_web_lg model from the SpaCy library for English text. For text extracted from audio and video, the acoustic features and context are combined, and all recognition results are cross-validated and assigned a confidence score.

[0073] The context semantic understanding unit uses a Transformer-based deep learning model (ERNIE) for context semantic encoding; it analyzes the context surrounding the target word to generate a high-dimensional semantic vector; and it determines the most likely result by calculating the cosine similarity between this vector and the candidate meaning vector.

[0074] The subject information standardization module includes:

[0075] The entity linking unit links each extracted subject name to a standard entity in the knowledge base; entity similarity is calculated using the following formula:

[0076] S(e1,e2)=α*S text +β*S attr +γ*S rel

[0077] Among them, S text It is text similarity based on edit distance or the BM25 algorithm; S attr It is attribute similarity; S rel α represents the relationship similarity; β and γ are the weights; the entity linking unit selects the candidate entity with the highest comprehensive similarity for linking. If the highest score is lower than the threshold, it is determined to be a new entity.

[0078] The relational graph construction unit uses a graph database to store and represent entity relationships; where nodes are standardized entities and edges are relationships between entities; the attributes of edges include relation strength, start and end time, and data source.

[0079] The subject behavior analysis module includes:

[0080] The behavioral profiling unit constructs a dynamic profile vector for each subject. The vector dimensions include activity index, emotional tone, and topic tags. Among them, the activity index is the number of times the subject speaks per unit time after logarithmic smoothing; the emotional tone is obtained by calculating the emotional value of each text segment using an emotional dictionary and then calculating the weighted average over the period; and the topic tags are the key topic distributions extracted from the subject's speech using an LDA topic model.

[0081] The time series pattern mining unit transforms the subject's behavioral indicators into time series data. First, a sliding window is used for smoothing to remove short-term fluctuations. Then, the sequence is predicted using an ARIMA model or LSTM. By analyzing the residuals between the predicted and actual values ​​and combining them with CUSUM, abnormal fluctuations in the subject's behavior are discovered.

[0082] The influence propagation analysis unit models the interaction between subjects as a directed weighted graph; subjects are nodes, the direction of the edges represents information flow, and the weights represent the intensity of the interaction; independent cascade model (ICM) or linear threshold model (LTM) is used for simulation; through multiple simulations, the propagation range and propagation efficiency of each node are calculated; thus, key subjects on the propagation path are identified, where the propagation range is the number of nodes finally activated, and the propagation efficiency is the activation speed.

[0083] The dynamic early warning unit sets early warning rules, as follows:

[0084] A subject's negative sentiment value exceeds its historical mean by M standard deviations for N consecutive hours;

[0085] The isolated forest algorithm was used to detect that the comprehensive behavioral vector of the subject deviated from its normal cluster;

[0086] The density of the associated subject cluster increases sharply by more than P percent over time K.

[0087] When any of the above rules is triggered, the system automatically generates warnings of different levels, where N, M, K, and P are set values.

[0088] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for extracting and standardizing key information in online public opinion, characterized in that, Includes the following steps: Step S1: Acquire online public opinion data and perform multimodal data fusion and preprocessing; Step S2: Based on the risk subject identification model of multi-level semantic fusion, extract preliminary subject information from the preprocessed data; Step S3: Standardize the preliminary subject information, including entity disambiguation, form normalization and relation verification, to generate standardized subject information; Step S4: Based on the evolutionary analysis model, perform behavioral trajectory tracking and trend analysis on standardized subject information; Step S5: Output the standardized main information and analysis results.

2. Equipment for extracting and standardizing key information in online public opinion, characterized in that, For implementing the method of claim 1, the device is equipped with a standardization system, the standardization system comprising: Data Acquisition and Preprocessing Module: Responsible for acquiring public opinion data from online platforms and performing preprocessing operations such as data cleaning, deduplication, and format standardization; Subject Information Extraction Module: Employs semantic fusion technology to identify and extract various types of subject information from online public opinion; Subject information standardization module: Through multi-strategy fusion, the extracted subject information is disambiguated, normalized, and verified; Subject Behavior Analysis Module: Based on time series analysis and network relationship diagrams, it tracks the subject's behavioral trajectory and analyzes the evolution of subject relationships; Output module: Outputs the processing results in a structured format.

3. The device for extracting and standardizing main information in online public opinion according to claim 2, characterized in that, The data acquisition and preprocessing module includes: The multi-source heterogeneous data acquisition unit adopts a pluggable collector architecture. For web page data, it uses the Python-based Scrapy or Requests framework for targeted crawling, and complies with website specifications through Robots.txt detection and dynamic latency control. For API data, it uses the OAuth2.0 protocol for authorization and a token polling mechanism to avoid call frequency limits. Direct database acquisition is achieved by configuring a JDBC / ODBC connection pool and using SQL queries or listening to database logs to achieve incremental collection. The multimodal data fusion unit assigns globally unique UUIDs and timestamps to all incoming data streams. For non-text data, it uses the Tesseract OCR engine to recognize text in images and performs layout analysis using a CNN convolutional neural network. It converts audio into text based on the Whisper speech recognition model and separates the vocal channels of different speakers. It uses PP-OCRv3 to extract frames and recognize subtitles and the main subject of the image. Finally, it associates multimodal data generated by the same event together using UUIDs to form a unified data record.

4. The device for extracting and standardizing subject information in online public opinion according to claim 3, characterized in that, The data acquisition and preprocessing module also includes a distributed caching unit, which uses Kafka or Pulsar message queues as buffers to decouple data acquisition from subsequent processing. After the data flows into the queue, it is processed in real time by Flink or Spark Streaming engines. The cleaned and fused data is stored in a distributed file system or columnar database, and a Redis cache is established for frequently queried hot data to improve data retrieval efficiency.

5. The device for extracting and standardizing subject information in online public opinion according to claim 2, characterized in that, The main information extraction module includes: The webpage structure analysis unit parses the webpage source code into a DOM tree and then uses a visual block detection algorithm to calculate the layout density and semantic weight of each DOM node by parsing CSS styles and element coordinates. By setting a threshold, the main text block can be accurately located and extracted. The layout density is the ratio of the effective text length within the node to the pixel area it occupies, and the semantic weight is the cosine similarity between the node content and the title. The node evaluation unit, by calculating information entropy, helps extract the content of nodes containing core subject information in the DOM tree; The multimodal subject recognition unit uses the NER model for Chinese text and the en_core_web_lg model from the SpaCy library for English text. For text extracted from audio and video, the acoustic features and context are combined, and all recognition results are cross-validated and assigned a confidence score. The context semantic understanding unit uses a Transformer-based deep learning model for context semantic encoding; it analyzes the context surrounding the target word to generate a high-dimensional semantic vector; and it determines the most likely result by calculating the cosine similarity between this vector and the candidate meaning vector.

6. The device for extracting and standardizing subject information in online public opinion according to claim 5, characterized in that, The calculation formula for the node evaluation unit is as follows: S=-Σ(p(x i )*log(p(x i ))) Where p(x) i ) represents the probability distribution of a specific feature; the node evaluation unit sorts the nodes and prioritizes extracting content with high entropy values.

7. The device for extracting and standardizing subject information in online public opinion according to claim 2, characterized in that, The subject information standardization module includes: The entity linking unit links each extracted subject name to a standard entity in the knowledge base; The relational graph construction unit uses a graph database to store and represent entity relationships; where nodes are standardized entities and edges are relationships between entities; the attributes of edges include relation strength, start and end time, and data source.

8. The device for extracting and standardizing subject information in online public opinion according to claim 7, characterized in that, The entity linking unit calculates entity similarity using the following formula: S(e1,e2)=α*S text +β*S attr +γ*S rel Among them, S text It is text similarity based on edit distance or the BM25 algorithm; S attr It is attribute similarity; S rel α represents the relationship similarity; β and γ are the weights; the entity linking unit selects the candidate entity with the highest comprehensive similarity for linking. If the highest score is lower than the threshold, it is determined to be a new entity.

9. The device for extracting and standardizing subject information in online public opinion according to claim 8, characterized in that, The subject behavior analysis module includes: The behavioral profiling unit constructs a dynamic profile vector for each subject. The vector dimensions include activity index, emotional tone, and topic tags. Among them, the activity index is the number of times the subject speaks per unit time after logarithmic smoothing; the emotional tone is obtained by calculating the emotional value of each text segment using an emotional dictionary and then calculating the weighted average over the period; and the topic tags are the key topic distributions extracted from the subject's speech using an LDA topic model. The time series pattern mining unit transforms the subject's behavioral indicators into time series data; first, it uses a sliding window to smooth the data and remove short-term fluctuations; then, it uses an ARIMA model or LSTM to predict the sequence; by analyzing the residuals between the predicted and actual values ​​and combining them with CUSUM, it discovers abnormal fluctuations in the subject's behavior. The influence propagation analysis unit models the interaction between subjects as a directed weighted graph; subjects are nodes, the direction of the edges represents information flow, and the weights represent the intensity of the interaction; independent cascade model (ICM) or linear threshold model (LTM) is used for simulation; through multiple simulations, the propagation range and propagation efficiency of each node are calculated; thus, key subjects on the propagation path are identified, where the propagation range is the number of nodes finally activated, and the propagation efficiency is the activation speed.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in claim 1.

Images