A network public opinion data collection and management method

Abstract

This application belongs to the field of public opinion analysis technology, and relates to a method for data collection and management of online public opinion. It employs Selenium to simulate browser behavior, periodically accessing trending search pages to parse and store keywords, achieving automated and dynamic tracking of trending search data. This effectively replaces manual configuration or fixed rules, solving the problem of real-time capture and omission of key topics. Secondly, it utilizes simulated browser technology to penetrate dynamically rendered content, and collects relevant articles by constructing search URLs and recursively collects sub-comments to establish a comment tree structure. This overcomes the limitations of APIs on historical data and comment details, as well as the barrier of static crawlers' inability to handle dynamic content, obtaining complete article key fields and hierarchical comment data, improving data integrity and collection stability. Finally, it optimizes data storage through partitioned storage, batch writing, and hot / cold data separation strategies, and improves query performance by combining primary key search, parent-child comment indexes, and full-text indexes, ensuring efficient data management and subsequent analysis.

Description

Technical Field

[0001] This application belongs to the field of public opinion analysis technology, and more specifically, relates to a method for data collection and management of online public opinion. Background Technology

[0002] With the rapid development of mobile internet and social media, online platforms (such as microblogs, forums, and short video comment sections) have become the core arena for information dissemination and public opinion formation. User-generated content (UGC), especially online comment data, contains the public's true attitudes and emotional inclinations towards social events, products and services, and policy measures, providing crucial data support for fields such as public opinion analysis, sentiment computing, and social behavior research. For example, by analyzing comment data on hot topics, it is possible to track public opinion dynamics in real time and warn of potential risks; by mining the emotional tendencies in user comments, it is possible to provide decision-making basis for enterprise product improvement and government policy optimization. Therefore, efficiently and accurately acquiring and managing online comment data has become a fundamental prerequisite for the implementation of public opinion research and applications.

[0003] Traditional online data collection methods primarily rely on two types of technologies: first, API calls based on platform access. However, most social media platforms impose strict restrictions on API access frequency and data scope (such as historical data and comment details) to protect data security or limit traffic, resulting in incomplete or unreal-time data collection. Second, web crawling technology based on static webpage parsing. However, modern online platforms generally employ dynamic rendering (such as asynchronous JavaScript loading and infinite scrolling) and anti-crawling mechanisms (such as CAPTCHAs and IP blocking), making it difficult for static crawlers to penetrate dynamic content barriers and susceptible to anti-crawling strategies, resulting in poor stability. Furthermore, trending keywords serve as a "weathervane" for online public opinion, and their dynamic changes (such as real-time ranking updates and the emergence of sudden topics) require data collection systems to have timed and automated keyword tracking capabilities. However, existing tools mostly rely on manual configuration or fixed rules, making it difficult to achieve real-time capture and dynamic response of trending search data, leading to delayed data sources or omissions of key topics. Summary of the Invention

[0004] This invention provides a method for collecting and managing online public opinion data, aiming to solve the technical problems of insufficient automation and comprehensiveness in data collection.

[0005] A method for collecting and managing online public opinion data includes the following steps: S1. Obtain trending search lists or user-defined keywords from the internet, use Selenium to simulate browser behavior, periodically access trending search pages, parse and store trending search keywords; S2. Based on trending keywords, search for the latest search records in the existing data; S3. Based on trending keywords and the latest search history, construct a search URL, collect relevant articles, and extract key fields; S4. Collect comments level by level, using a recursive approach to collect sub-comments and build a comment tree structure; S5. Preprocess the collected data and write the preprocessed data into the database and search engine. Adopt partitioned storage, batch writing and hot and cold data separation strategies, and optimize query performance through primary key search, parent and child comment index and full text index. S6. Count the number of blog posts and comments by time, draw trend charts, calculate blog post popularity scores, and analyze user activity.

[0006] This invention employs Selenium to simulate browser behavior, periodically accessing trending search pages to parse and store keywords. This automates and dynamically tracks trending search data, effectively replacing manual configuration or fixed rules and solving the problem of real-time capture and omission of key topics. Secondly, it utilizes simulated browser technology to penetrate dynamically rendered content and constructs search URLs to collect relevant articles and recursively collects sub-comments to build a comment tree structure. This overcomes the limitations of APIs on historical data and comment details, as well as the inability of static crawlers to handle dynamic content, obtaining complete article key fields and hierarchical comment data, thus improving data integrity and collection stability. Finally, it optimizes data storage through partitioned storage, batch writing, and hot / cold data separation strategies, and improves query performance by combining primary key search, parent-child comment indexes, and full-text indexes, ensuring efficient data management and subsequent analysis.

[0007] Preferably, the timed hot search page for model articles automatically adjusts the current access interval based on the ranking changes of keywords in the historical list. If the ranking fluctuates drastically recently, the waiting time is shortened, and vice versa.

[0008] Preferably, step S2 includes the following steps: Each trending keyword is read, and its semantics are expanded using a pre-built dynamic semantic lexicon. The original words and expanded words are combined into a logical query statement, which is then executed in the full-text index database to match semantically related records, resulting in a candidate set containing semantically matched records. Based on the type of trending search terms and the current system configuration, a dynamic time window is determined, and records whose publication time is within the most recent time window are filtered from the candidate set to obtain a candidate record set after time filtering. Based on the candidate record set, a comprehensive score is calculated for each record. The comprehensive score consists of the relevance score returned by the search engine, the time decay factor, and the popularity factor, resulting in a list of candidate records sorted from high to low by the comprehensive score. Based on the preset number of records to be returned, a specified number of records are selected sequentially from the sorted list to obtain the final record set.

[0009] Preferably, constructing the search URL includes the following steps: For each trending keyword, its semantic similarity to the user's historical search terms is calculated. At the same time, a time decay weight is assigned based on the search time. The trending search terms and historical search terms are then dynamically merged to generate combined query terms. The encoded query terms are then concatenated to the target platform's basic search URL to obtain a set of platform search URLs with semantic weights.

[0010] Preferably, the recursive method for collecting sub-comments includes the following steps: For each collected comment, first determine if the current recursion depth exceeds the maximum limit. If it does not exceed the depth limit, construct the URL of the corresponding comment's sub-comment page, load the page using Selenium, and parse the sub-comment data. After parsing, calculate the priority score based on the comment's likes and replies. When the score exceeds a predetermined threshold, continue recursively collecting the next level of comments for that sub-comment. Stop recursion when the recursion depth has reached the dynamically calculated maximum depth, the page parsing shows that the comment has no sub-comments, or the sub-comment page fails to load or parse.

[0011] Preferably, establishing the comment tree structure includes the following steps: A tree-like data structure is constructed in memory, with the article as the root node, the top-level comment as the first-level child node, and the child comments recursively nested as the lower-level nodes. Each node contains the extracted core fields and a child comment list attribute to store the next level of comments. After the data collection is completed, the tree structure is converted into a storable format; the top-level comment is a comment without a parent comment.

[0012] Preferably, the preprocessing includes the following steps: Data with missing core fields is deleted directly, while data with missing non-core fields is filled with NULL. For duplicate collection of the same blog post or comment due to pagination and multithreading during data collection, the user ID or article ID is used for judgment. If they belong to the same user ID or the same article ID, the duplicate data is directly deleted. For duplicate content published by different users, the SimHash algorithm is used to determine whether they belong to similar text. If they do, they are deleted. Based on the method of blacklisting keywords, delete data containing blacklisted keywords; Regular expressions are used to remove HTML tags and URL-related content from the processed data.

[0013] Preferably, the partitioned storage strategy is as follows: The data is split into independent tables according to business type. The table partitioning adopts composite partitioning. First, the parent comment ID range is partitioned, and then each partition is sub-partitioned according to the posting time. The hot and cold data are determined according to a predefined time threshold. Data less than or equal to the time threshold is considered hot data, and data greater than the time threshold is considered cold data. The time threshold is dynamically adjusted based on historical access frequency and storage cost.

[0014] Preferably, the blog post popularity score is obtained based on the following steps: For each blog post, the number of likes, comments, reposts, and favorites are first logarithmically transformed. Then, the transformed results are multiplied by preset weights, and finally, the weighted values ​​are summed to obtain the blog post popularity score.

[0015] Preferably, the user activity level is obtained based on the following steps: For each user, calculate the total number of interactions and then divide it by the number of active days to get the average daily number of interactions. If a user has zero active days, the activity level is recorded as zero; if the number of active days is 1, the total number of interactions is taken as the activity level.

[0016] The beneficial effects of this invention include: This invention employs Selenium to simulate browser behavior, periodically accessing trending search pages to parse and store keywords. This automates and dynamically tracks trending search data, effectively replacing manual configuration or fixed rules and solving the problem of real-time capture and omission of key topics. Secondly, it utilizes simulated browser technology to penetrate dynamically rendered content and constructs search URLs to collect relevant articles and recursively collects sub-comments to build a comment tree structure. This overcomes the limitations of APIs on historical data and comment details, as well as the inability of static crawlers to handle dynamic content, obtaining complete article key fields and hierarchical comment data, thus improving data integrity and collection stability. Finally, it optimizes data storage through partitioned storage, batch writing, and hot / cold data separation strategies, and improves query performance by combining primary key search, parent-child comment indexes, and full-text indexes, ensuring efficient data management and subsequent analysis. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the overall process provided for an embodiment of the present invention. Detailed Implementation

[0019] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0020] See Figure 1 As shown, a method for collecting and managing online public opinion data includes the following steps: S1. Obtain trending search lists or user-defined keywords from the internet, use Selenium to simulate browser behavior, periodically access trending search pages, parse and store trending search keywords; The timed trending search page automatically adjusts the access interval based on the ranking changes of keywords in the historical list. If the ranking fluctuates drastically recently, the waiting time is shortened; otherwise, the interval is lengthened. The specific expression is as follows: ; In the formula: Indicates the time interval between this visit; Indicates the basic access interval; Represents the sensitivity coefficient to the rate of change; Indicates the rate of change of keywords; ; In the formula: This indicates the total number of keywords in the ranking list; This indicates the ranking of the i-th keyword during the previous data collection. This indicates the ranking of the i-th keyword at the current time of data collection; Indicates the highest ranking number on the list; when A high number indicates significant changes in the rankings. As T approaches 0, it decreases; conversely, as T increases, it approaches zero. .

[0021] In this embodiment, a browser is used to simulate human operation to access the target URL and load the complete page (including JavaScript-rendered content). If a network timeout or page error occurs, the proxy IP is automatically switched and the page is retried (up to 3 times), with the waiting time doubling with each retry (1 second → 2 seconds → 4 seconds).

[0022] S2. Based on trending keywords, search for the latest search records in the existing data; Each trending keyword is read, and its semantics are expanded using a pre-built dynamic semantic lexicon. The original words and expanded words are combined into a logical query statement, which is then executed in the full-text index database to match semantically related records, resulting in a candidate set containing semantically matched records. Determine a dynamic time window based on the type of trending search terms and the current system configuration. The process involves filtering the candidate set to identify records whose publication time falls within the most recent time window, resulting in a time-filtered candidate record set; for example, trending topics related to breaking news. Hours; Continuous trending topics Hourly; Regular trending topics Hour.

[0023] Based on the candidate record set, a comprehensive score is calculated for each record. The comprehensive score consists of the relevance score returned by the search engine, the time decay factor, and the popularity factor. This results in a list of candidate records sorted from highest to lowest comprehensive score. The specific expression is as follows: ; In the formula: This indicates that r records the trending searches. The final overall score; This indicates the recording of r content and trending search keywords. The original relevance score; This represents the calorific value of record r itself; This represents the time-freshness decay value of record r; and Indicates the weighting coefficient; ; In the formula: , , These represent the number of likes, comments, and shares of record r, respectively. and Indicates the interaction weight coefficient; ; In the formula: This represents the decay rate parameter; This indicates the time difference between when the record was published and when it was released.

[0024] Based on the preset number of records to be returned (e.g., 20 records), select the specified number (20 records) of records from the sorted list in order to obtain the final record set.

[0025] S3. Based on trending keywords and the latest search history, construct a search URL, collect relevant articles, and extract key fields, including article ID, user ID, number of followers, authentication information, article content, and publication time. In this embodiment, constructing the search URL includes the following steps: For each trending keyword, its semantic similarity to the user's historical search terms is calculated. A time decay weight is assigned based on the search time. Trending and historical search terms are then dynamically merged to generate combined query terms. These encoded query terms are then appended to the target platform's base search URL to obtain a set of semantically weighted platform search URLs. The URL generation formula is as follows: ; In the formula: Indicates the base path of the search interface of the target platform; This indicates the currently trending search keywords; Indicates the dominant factor in trending searches; This represents a function that indicates the relevance of search records. This indicates selecting from S and... The top-K most similar words are concatenated; This represents the k-th generated URL; Represents a collection of search records; ; In the formula: express Search terms semantic similarity; Represents the time decay function. Indicates the attenuation coefficient. This represents the difference between the current time and the search time. This represents the minimum value to prevent division by zero errors; This represents the i-th search record.

[0026] In this embodiment, each URL is opened using a headless Chrome browser, with a 20-second page load timeout and a 5-second element search wait. For dynamically loaded content, an intelligent scrolling strategy is executed: the page is continuously scrolled until the height of the newly added content is less than 80% of the current page or the maximum number of scrolls (5 times) is reached. A random delay of 1-2.5 seconds is inserted between scrolls to simulate manual operation, and finally the fully rendered page DOM tree is captured.

[0027] S4. Collect comments level by level, using a recursive approach to collect sub-comments and build a comment tree structure; In this embodiment, step S4 includes the following steps: Comments are collected using a depth-first search recursive algorithm. The core function is `collect_comments()`, and the specific process is as follows: For each article, access the article_url, load the page using Selenium, parse the HTML / CSS selectors to extract the top-level comments (i.e., parent_comment_id = NULL), and parse fields including: comment_id, user_id, content, publish_time, like_count (number of likes for the comment), reply_count (number of replies), etc.

[0028] To calculate the number of pages for paginated display of the comment list, the formula is as follows: ; In the formula: P represents the total number of pages; This indicates the total number of comments on the article; This indicates the number of comments displayed per page; Loop through the page numbers (from 1 to (P)), construct the pagination URL (e.g., article_url?page=i), and Selenium loads and parses it.

[0029] Define a recursive function `collect_comments(parent_id, current_depth, article_id)`: `parent_id` represents the ID of the current parent comment (NULL for top-level comments, which are comments without a parent, i.e., not child comments); `current_depth` represents the current recursion depth (initialized to 0 for top-level comments); `article_id` represents the ID of the current article, used to construct the URL. In this embodiment, to prevent stack overflow and optimize data collection (e.g., deep comments have low value), the maximum depth is... Dynamically adjust based on article popularity, the specific expression is as follows: ; In the formula: This indicates the default maximum depth constant (with a value of 5). This indicates the total number of comments on the article; Represents the base of the logarithm (with a value of 2); Check depth limit: If If the condition is met, the recursion terminates and a log is recorded. Constructing sub-comment URLs: based on parent_id and platform API (such as article_url / comment / parent_id / replies), accessed via Selenium; Parse sub-comments: extract fields (same as top-level comments, such as article_id (article ID), article_url (article URL), like_count (number of likes), comment_count (total number of comments), etc.), and check the has_replies flag (a boolean value indicating whether there are deeper replies); In this embodiment, to optimize efficiency, sub-comments are recursively collected only for highly interactive comments, and priority scores are calculated first: ; ; In the formula: Indicates priority score; Represents the normalization function; and Indicates the weighting parameter; Indicates the number of replies; This indicates the number of likes; (0.3) Recursive call ,in Indicates the sub-comment ID; During the recursive process, the current branch will terminate its data collection if any of the following situations occur: 1) The current recursion depth has reached the maximum depth that can be dynamically calculated; 2) The page parsing shows that the comment has no sub-comments (e.g., the platform marks "No reply yet" or the "View more replies" button is not clickable); 3) The sub-comment page fails to load or parse (e.g., network timeout, abnormal HTML structure). In these cases, the error will be captured and the recursion will stop through the exception handling mechanism.

[0030] A tree-like data structure is constructed in memory, with the article as the root node, the top-level comment as the first-level child node, and the child comments recursively nested as the lower-level nodes. Each node contains the extracted core fields and a child comment list attribute to store the next level of comments. After the data collection is completed, the tree structure is converted into a storable format; the top-level comment is a comment without a parent comment.

[0031] S5. Preprocess the collected data and write the preprocessed data into the database and search engine. Adopt partitioned storage, batch writing and hot and cold data separation strategies, and optimize query performance through primary key search, parent and child comment index and full text index. The preprocessing includes the following steps: Missing values ​​in the original data are handled by directly deleting core fields such as article ID, user ID, and missing content; and filling non-core fields such as missing identity information with "NULL". To handle duplicate data, for the same blog post or comment that was repeatedly collected due to pagination and multithreading during data collection, it was directly deleted based on user ID or post ID; for duplicate content posted by different users, the SimHash algorithm was used to identify and delete similar text. ; in For the i-th word to be processed, The weight of the word is h(·), where h(·) represents the weight of the word. The function hashes the data into a k-bit 0 / 1 vector. The `sign(·)` method checks the sign of each bit; if the sign is greater than 0, it outputs 1; otherwise, it outputs 0, ultimately yielding a k-bit binary fingerprint. The similarity between two texts is calculated simply by determining the Hamming distance between their fingerprints. ; In the formula: fingerprints The j-th binary bit; fingerprints The j-th binary bit; To process noisy data, a blacklist keyword approach is used to set and remove keywords such as "promotion," "advertisement," and "add WeChat." The processed data is normalized by using regular expressions to remove HTML tags and URL-related content.

[0032] The partitioned storage strategy is as follows: Based on data access frequency and business query patterns, a composite partitioning strategy combining "time + business dimension" is designed: Table-level partitioning: Split data into independent tables based on business type to avoid single tables becoming too large. The posts table stores core information about blog posts (post ID, user ID, publication time, number of likes / comments / shares, etc.). The comments table stores comment data (comment ID, parent comment ID, user ID, content, posting time, etc.). The users table stores static user information (user ID, number of followers, authentication information, etc., reducing redundancy).

[0033] Row-level partitioning: Partitioning by time + business key: The posts table can be partitioned by publication time (publish_time) or by month (e.g., posts_202401, posts_202402), with each partition storing blog post data for a single month; time partitioning allows direct location of data within a target time range, reducing the number of rows scanned.

[0034] Partition key calculation: partition_key = to_char(publish_time, 'YYYYMM') (converts the time to a "year and month" string as the partition identifier).

[0035] The comments table is partitioned using a composite partitioning approach—first, it is partitioned by the parent comment ID (parent_comment_id) range (the parent_comment_id of the top-level comment is the article ID), and then each partition is further sub-partitioned by publication time. When querying the comment tree of a specific article, the partition can be directly located using parent_comment_id, and then recent comments can be filtered by time sub-partitions, improving the efficiency of recursive queries.

[0036] Partition key calculation: range_partition_key = floor(parent_comment_id / 10000) (divided into a range partition for every 10000 article IDs), subpartition_key = to_char(publish_time,'YYYYMM').

[0037] The hot and cold data are determined according to a predefined time threshold; data less than or equal to the time threshold is considered hot data, and data greater than the time threshold is considered cold data. The time threshold is... The system dynamically adjusts based on historical access frequency and storage costs, as shown in the following expression: ; In the formula: Indicates the average access interval for historical data; This represents the storage cost coefficient; and This represents the weighting parameter.

[0038] Primary key index: The posts table: Primary key post_id (B+ tree index, uniquely identifies a blog post); The comments table has a composite primary key (comment_id, parent_comment_id) (ensuring that comment IDs are unique within the same parent comment to avoid duplicate comments). The users table has a primary key, user_id, which uniquely identifies the user.

[0039] Business Query Index: Parent-child comment index: A composite index (parent_comment_id, publish_time) is built on the comments table, which supports quick querying of all child comments by parent comment ID (such as article ID) and sorting by publication time; User activity index: Create a joint index of (user_id, publish_time) for the posts table and the comments table respectively; Time trend index: Create a joint index of (publish_time, hot_score) for the posts table (hot_score is the popularity score), to support fast query of high-popularity posts within a certain time period (e.g., filtering Top N high-popularity posts when drawing a trend chart).

[0040] Full-text search index: Create a full-text index for the posts.content and comments.content fields in Elasticsearch, use the IK tokenizer (supporting Chinese word segmentation), and configure the following analyzers: Tokenizer setting: "analyzer": "ik_max_word" (fine-grained word segmentation to improve keyword matching accuracy); Stop word filtering: Filter meaningless words such as "de" and "shi" to reduce the index volume; Synonym expansion: Configure industry-specific synonyms (e.g., "hot search" = "hot list") to improve search recall rate.

[0041] S6. Count the number of posts and comments by time, draw a trend chart, calculate the post popularity score, and analyze user activity.

[0042] The post popularity score is obtained based on the following steps: For each post, first perform logarithmic transformation on the number of likes, comments, reposts, and collections respectively, then multiply the transformation results by preset weights respectively, and finally accumulate the weighted values to obtain the post popularity score , the specific expression is as follows: ; In the formula: represents the time decay factor, , is the decay coefficient, represents the time difference between the current time and the post time; represents the number of times the post is liked; represents the number of times the post is commented; represents the number of times the post is reposted; represents the number of times the post is collected; , , , represent the interaction indicator weights.

[0043] The user activity is obtained based on the following steps: For each user, calculate the total number of interactions, then divide by the number of active days to obtain the average daily interaction volume. If a user has zero active days, their activity level is recorded as zero; if they have one active day, the total number of interactions is used as their activity level. The specific expression is as follows: ; ; In the formula: This represents the effective contribution value of user u; This indicates the number of active days user u had within the statistical period; This indicates the number of followers of user u; Represents the normalization constant; This indicates the number of blog posts published by user u within the statistical period; This indicates the number of highly popular blog posts published by user u, i.e., the blog post popularity score is greater than the predetermined popularity threshold. This indicates the number of highly interactive comments posted by user u, meaning comments with more than a predetermined threshold of likes are considered highly interactive.

[0044] This invention employs Selenium to simulate browser behavior, periodically accessing trending search pages to parse and store keywords. This automates and dynamically tracks trending search data, effectively replacing manual configuration or fixed rules and solving the problem of real-time capture and omission of key topics. Secondly, it utilizes simulated browser technology to penetrate dynamically rendered content and constructs search URLs to collect relevant articles and recursively collects sub-comments to build a comment tree structure. This overcomes the limitations of APIs on historical data and comment details, as well as the inability of static crawlers to handle dynamic content, obtaining complete article key fields and hierarchical comment data, thus improving data integrity and collection stability. Finally, it optimizes data storage through partitioned storage, batch writing, and hot / cold data separation strategies, and improves query performance by combining primary key search, parent-child comment indexes, and full-text indexes, ensuring efficient data management and subsequent analysis.

[0045] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for collecting and managing online public opinion data, characterized in that, Includes the following steps: S1. Obtain trending search lists or user-defined keywords from the internet, use Selenium to simulate browser behavior, periodically access trending search pages, parse and store trending search keywords; S2. Based on trending keywords, search for the latest search records in the existing data; S3. Based on trending keywords and the latest search history, construct a search URL, collect relevant articles, and extract key fields; S4. Collect comments level by level, using a recursive approach to collect sub-comments and build a comment tree structure; S5. Preprocess the collected data and write the preprocessed data into the database and search engine. Adopt partitioned storage, batch writing and hot and cold data separation strategies, and optimize query performance through primary key search, parent and child comment index and full text index. S6. Count the number of blog posts and comments by time, draw trend charts, calculate blog post popularity scores, and analyze user activity.

2. The method for data collection and management of online public opinion according to claim 1, characterized in that, The timed trending search page automatically adjusts the access interval based on the ranking changes of keywords in the historical list. If the ranking fluctuates drastically recently, the waiting time will be shortened, and vice versa.

3. The method for data collection and management of online public opinion according to claim 1, characterized in that, Step S2 includes the following steps: Each trending keyword is read, and its semantics are expanded using a pre-built dynamic semantic lexicon. The original words and expanded words are combined into a logical query statement, which is then executed in the full-text index database to match semantically related records, resulting in a candidate set containing semantically matched records. Based on the type of trending search terms and the current system configuration, a dynamic time window is determined, and records whose publication time is within the most recent time window are filtered from the candidate set to obtain a candidate record set after time filtering. Based on the candidate record set, a comprehensive score is calculated for each record. The comprehensive score consists of the relevance score returned by the search engine, the time decay factor, and the popularity factor, resulting in a list of candidate records sorted from high to low by the comprehensive score. Based on the preset number of records to be returned, a specified number of records are selected sequentially from the sorted list to obtain the final record set.

4. The method for data collection and management of online public opinion according to claim 1, characterized in that, Constructing the search URL includes the following steps: For each trending keyword, its semantic similarity to the user's historical search terms is calculated. At the same time, a time decay weight is assigned based on the search time. The trending search terms and historical search terms are then dynamically merged to generate combined query terms. The encoded query terms are then concatenated to the target platform's basic search URL to obtain a set of platform search URLs with semantic weights.

5. The method for data collection and management of online public opinion according to claim 1, characterized in that, The recursive method for collecting sub-comments includes the following steps: For each collected comment, first determine if the current recursion depth exceeds the maximum limit. If it does not exceed the depth limit, construct the URL of the corresponding comment's sub-comment page, load the page using Selenium, and parse the sub-comment data. After parsing, calculate the priority score based on the comment's likes and replies. When the score exceeds a predetermined threshold, continue recursively collecting the next level of comments for that sub-comment. Stop recursion when the recursion depth has reached the dynamically calculated maximum depth, the page parsing shows that the comment has no sub-comments, or the sub-comment page fails to load or parse.

6. The method for data collection and management of online public opinion according to claim 1, characterized in that, The process of establishing the comment tree structure includes the following steps: A tree-like data structure is constructed in memory, with the article as the root node, the top-level comment as the first-level child node, and the child comments recursively nested as the lower-level nodes. Each node contains the extracted core fields and a child comment list attribute to store the next level of comments. After the data collection is completed, the tree structure is converted into a storable format; the top-level comment is a comment without a parent comment.

7. The method for data collection and management of online public opinion according to claim 1, characterized in that, The preprocessing includes the following steps: Data with missing core fields is deleted directly, while data with missing non-core fields is filled with NULL. For duplicate collection of the same blog post or comment due to pagination and multithreading during data collection, the user ID or article ID is used for judgment. If they belong to the same user ID or the same article ID, the duplicate data is directly deleted. For duplicate content published by different users, the SimHash algorithm is used to determine whether they belong to similar text. If they do, they are deleted. Based on the method of blacklisting keywords, delete data containing blacklisted keywords; Regular expressions are used to remove HTML tags and URL-related content from the processed data.

8. The method for data collection and management of online public opinion according to claim 1, characterized in that, The partitioned storage strategy is as follows: The data is split into independent tables according to business type. The table partitioning adopts composite partitioning. First, the parent comment ID range is partitioned, and then each partition is sub-partitioned according to the posting time. The hot and cold data are determined according to a predefined time threshold. Data less than or equal to the time threshold is considered hot data, and data greater than the time threshold is considered cold data. The time threshold is dynamically adjusted based on historical access frequency and storage cost.

9. The method for data collection and management of online public opinion according to claim 1, characterized in that, The blog post popularity score is obtained based on the following steps: For each blog post, the number of likes, comments, reposts, and favorites are first logarithmically transformed. Then, the transformed results are multiplied by preset weights, and finally, the weighted values ​​are summed to obtain the blog post popularity score.

10. The method for data collection and management of online public opinion according to claim 1, characterized in that, The user activity level is obtained based on the following steps: For each user, calculate the total number of interactions and then divide it by the number of active days to get the average daily number of interactions. If a user has zero active days, the activity level is recorded as zero. If the activity period is 1 day, then the total number of interactions is taken as the activity level.

Images