An opinion information processing method

Abstract

The application discloses an opinion information processing method, and relates to the technical field of information processing.The application monitors the user retrieval data storage frequency of target literature through a sliding window mechanism, triggers preset analysis only when the literature is browsed frequently for a short time, ensures the analysis pertinence, constructs a mapping set, a strong correlation set and an incremental crawling set after triggering the analysis, mines potential incremental retrieval words of the target literature from the user retrieval data, constructs a retrieval formula based on the incremental retrieval words, accurately crawls resources meeting the demand in the whole network and stores the resources into a literature storage library, solves the problems of blind resource expansion and disconnection with user demand in traditional crawling, makes the target website literature resources continuously rich around the core demand of users, improves the content reserve, and attracts more users with similar demands to visit, effectively improves the platform traffic and competitiveness.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, specifically to a method for processing public opinion information. Background Technology

[0002] In today's world where the dissemination of public opinion information is becoming increasingly convenient, various information service websites have become the core channel for the public to obtain relevant documents. These websites collect diverse documents such as statements, event analysis reports, and commentary articles, and set exclusive keywords for each document to optimize the search experience, attract users to browse and view, and thus increase website traffic.

[0003] However, in actual operation, existing websites essentially terminate their service chain after uploading documents and configuring keywords, lacking in-depth analysis and response mechanisms for user search behavior. In particular, some documents experience explosive growth in high-frequency searches and browsing in a short period of time due to factors such as the fermentation of social hotspots and sudden public events. This phenomenon should have been a key opportunity for websites to seize traffic dividends and enhance user stickiness, but existing websites have neither conducted special analysis on the search user data of such high-frequency search documents nor been able to expand the crawling of related resources based on user search information. As a result, they can only provide users with single-target documents, making it difficult to meet their extended information needs. This not only causes websites to miss the opportunity to attract a large amount of traffic, but also reduces the user experience due to the single nature of information supply, gradually losing its advantage in the competition with similar information service platforms.

[0004] To address the above problems, this invention proposes a solution. Summary of the Invention

[0005] The purpose of this invention is to provide a method for processing public opinion information in order to solve the problems mentioned in the background art.

[0006] This invention provides a method for processing public opinion information, comprising the following steps:

[0007] Step 1: The data acquisition module collects the search data of any user who clicks to browse the target document on the target website and transmits it to the data analysis module for storage;

[0008] Step 2: The data analysis module monitors the storage frequency of the search data of users who click to browse the target document through a sliding window mechanism to determine whether to execute the preset analysis steps of the target document. If it is determined to execute the analysis steps, the analysis steps are executed immediately, and the incremental crawl set of the target document is determined according to the analysis steps. The incremental crawl set is then transmitted to the incremental crawling module.

[0009] Step 3: After receiving the transmitted incremental crawl set, the incremental crawling module constructs several search terms for the target documents and performs a search across the entire network based on each constructed search term.

[0010] Furthermore, in step one, for any user who clicks to browse the target document, the data collection process is as follows: taking the moment when the user clicks on the target document as the reference point, backtracking to a preset first collection period, and collecting all the search information entered by the user within the first collection period; simultaneously, taking the moment when the user stops browsing the target document as the starting point, backtracking to a preset second collection period, and collecting all the search information entered by the user within the second collection period, wherein the search information includes several keywords.

[0011] Furthermore, in step two, the process of determining whether to execute the preset analysis steps for the target document is as follows:

[0012] The system uses a preset statistical duration as the fixed window duration to count the total number of times the search data of users who clicked to browse the target document is stored in the current window. When the total number of times the data is stored in the window reaches the preset fixed amount, the system immediately executes the analysis steps for the target document. After the execution, all search data in the current window is discarded, and a new window is opened after a preset monitoring duration. The window duration is still the preset statistical duration, and the total number of times the search data is stored in the new window is counted from zero.

[0013] If the total number of times the data in the window is stored does not reach the preset fixed amount, the sliding window will move over time, always maintaining the window duration at the preset statistical duration, and continuously monitoring whether the number reaches the preset fixed amount. The storage frequency refers to the total number of times the target document's search data is stored within a unit statistical duration, that is, the ratio of the total number of times the data in the window is stored to the statistical duration.

[0014] Furthermore, in step two, the analysis steps for determining the incremental crawl set of target documents according to the aforementioned analysis steps are as follows:

[0015] S11: Obtain all search data of all users who clicked to browse the target document stored in the window when determining to immediately execute the analysis step of the target document, and label them as A1, A2, ..., Aa, where a≥1;

[0016] S12: Obtain the mapping whitelist of the target document, and construct the mapping set B1={C1,C2,...,Cc} of the target document based on all the mapping keywords C1, C2,...,Cc contained in the mapping whitelist, where c is the total number of mapping keywords in the mapping whitelist;

[0017] S13: Mark all search information contained in search data A1 sequentially as D1, D2, ..., Dd, where d ≥ 1, according to the order of the entered time from the furthest to the nearest time.

[0018] S14: Create a hit variable E1 for retrieval information D1. The initial value of the hit variable E1 is 0. Determine whether retrieval information D1 is a relevant retrieval of the target document according to the preset judgment rules.

[0019] S15: Determine whether the search information D2, D3, ..., Dd is a relevant search for the target document in sequence according to S14, and re-label all the search information that is determined to be a relevant search for the target document as I1, I2, ..., Ii, 1≤i≤d respectively;

[0020] S16: Extract all keywords from the search information I1, I2, ..., Ii that indicate no match for the mapping set B1, and construct the filter set B2 based on them;

[0021] S17: Extract all keywords that are strongly related to the target set B2 according to the preset extraction rules, and construct a strongly related set of search data A1.

[0022] S18: Construct a strongly correlated set of the search data A2, A3, ..., Aa in sequence from S13 to S17;

[0023] S19: Select several keywords from the strongly related sets of search data A1, A2, ..., Aa according to the preset selection rules, and use them as incremental search terms for the target documents to construct an incremental crawl set of the target documents.

[0024] Furthermore, in S14, the rules for determining whether retrieval information D1 is a relevant retrieval of the target document are as follows:

[0025] S141: Label all keywords contained in the search information D1 as F1, F2, ..., Ff, respectively, where f≥1, and f represents the total number of keywords in the search information D1;

[0026] S142: Calculate the similarity between keyword F1 and the mapped keywords C1, C2, ..., Cc respectively, and compare the largest similarity with P1. If the similarity is greater than or equal to P1, it is determined that keyword F1 hits the mapping set B1, and the value of the hit variable E1 is incremented by 1. Otherwise, it is determined that keyword F1 does not hit the mapping set B1, and no processing is performed. Here, P1 is a preset similarity hit determination threshold.

[0027] S143: Perform a hit determination on the keyword F2, F3, ..., Ff according to S141 to S142, and obtain the value G1 of the hit variable E1 after all determinations are completed;

[0028] S144: Calculate the hit rate ratio H1 between the search information D1 and the mapped whitelist using the formula H1=G1 / f. Compare H1 with P2. If H1≥P2, then the search information D1 is determined to be a relevant search for the target document; otherwise, the search information D1 is determined not to be a relevant search for the target document. P2 is a preset relevance determination threshold.

[0029] Furthermore, in S17, extract all keywords that are strongly related to the target set B2, and construct the extraction rules for the strongly related set of search data A1 as follows:

[0030] S171: Extract all keywords in the set B2 to be filtered, and relabel them as J1, J2, ..., Jj, j≥1, where j is the total number of keywords that were determined not to have hit the mapping set B1 extracted from the search information I1, I2, ..., Ii;

[0031] S172: Calculate the similarity between keyword J1 and each keyword in the filter set B2 except for keyword J1. After the calculation, count the total number of keywords whose similarity value obtained after similarity calculation with keyword J1 is greater than or equal to P3. Compare the total number with P4. If the total number is greater than or equal to P4, it is determined that keyword J1 is strongly related in the filter set B2. Otherwise, it is determined that keyword J1 is not strongly related in the filter set B2. P3 and P4 are the preset strong correlation similarity threshold and strong correlation quantity threshold, respectively.

[0032] S173: According to S172, determine whether keywords J2, J3, ..., Jj are strongly related to the filter set B2. After determining whether keywords J1, J2, J3, ..., Jj are strongly related to the filter set B2, construct a strongly related set of search data A1 based on all keywords that are determined to be strongly related to the filter set B2.

[0033] Furthermore, in S19, several keywords are selected from the strongly related sets of the search data A1, A2, ..., Aa as incremental search terms for the target documents. The selection rules for constructing the incremental crawl set of the target documents are as follows:

[0034] S191: Extract each keyword from the strongly related set of search data A1. Following the same determination method as step S172, determine whether the keyword is strongly related to the strongly related sets of search data A2, A3, ..., Aa. If the keyword is determined to be strongly related to a strongly related set greater than or equal to P5, then the keyword is marked as an incremental search term for the target document. P5 is a preset cross-set strong correlation determination threshold.

[0035] S192: Select several keywords from the strongly related sets of search data A2, A3, ..., Aa as incremental search terms for the target documents, according to S191.

[0036] S193: Obtain the incremental search terms for all selected target documents and construct an incremental crawl set of target documents based on them.

[0037] Compared with existing technologies, it has the following advantages:

[0038] This invention monitors the frequency of user-retrieved data storage for target documents using a sliding window mechanism. Pre-set analysis is triggered only when a document is viewed frequently in a short period of time, ensuring the analysis is targeted. Data collection is performed by tracing back the first collection time from the moment the user clicks on the document and extending the second collection time backward from the moment the browsing stops, making the data coverage more comprehensive. Combined with the Word2Vec model, keyword semantic similarity is calculated. By statistically analyzing hit variables and calculating the hit rate ratio, relevant content is filtered from massive amounts of information, ensuring that the analysis results are highly consistent with the user's actual needs. This solves the problem that websites do not know the core concerns of users.

[0039] In this invention, the dynamic window mechanism uses a preset statistical duration as a fixed window, and counts the total number of times the retrieved data is stored in real time. The preset analysis steps are executed only when the data reaches a preset fixed amount. If the threshold is not reached, the dynamic sliding window continues to monitor, realizing on-demand triggering of analysis only when high attention is paid. After the analysis is executed, the system discards the data in the current window and opens a new window after a preset monitoring interval. This avoids interference from continuous window judgment, reserves time for the previous analysis to end, prevents repeated triggering in a short period of time, reduces ineffective computing power consumption, and improves system operating efficiency.

[0040] After triggering analysis, this invention constructs a mapping set, a strongly correlated set, and an incremental crawling set to mine potential incremental search terms for target documents from user search data. Based on these incremental search terms, the incremental crawling module constructs a search formula to accurately crawl resources that meet the requirements across the entire network and store them in the document repository. This solves the problems of blind resource expansion and disconnection from user needs in traditional crawling, enabling the document resources of the target website to be continuously enriched around the core needs of users. This not only improves the content reserves but also attracts more users with similar needs, effectively enhancing the platform's traffic and competitiveness. Attached Figure Description

[0041] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Please see Figure 1 This application provides a method for processing public opinion information. The above method is executed through a public opinion information processing system, which includes a data acquisition module, a data analysis module, an incremental crawling module, and a document repository.

[0044] The data acquisition module is used to collect search data from any user who clicks to browse a target document. The data collection process for any user who clicks to browse a target document is as follows: Using the moment the user clicks on the target document as a reference point, a preset first collection duration is traced backward, and all search information entered by the user within this first collection duration is collected. Simultaneously, using the moment the user stops browsing the target document as a starting point, a preset second collection duration is traced backward, and all search information entered by the user within this second collection duration is collected. Each search entry contains several keywords entered by the user, used to retrieve document data within the target website.

[0045] The criteria for determining whether a user stops browsing a target document are: the user enters new search information or the user closes the target document.

[0046] It should be noted that the first collection duration and the second collection duration are preset by the administrator. The setting is based on the following: numerically, the first collection duration is longer than the second collection duration, that is, the scope of the search information collected before the user clicks is greater than the scope of the search information collected after the user clicks, so as to more fully capture the user's active search intent before clicking on the target document, while also taking into account the record of related search behavior after the user clicks.

[0047] The data collection module collects the search data of each user who clicks to browse the target document and then transmits it to the data analysis module for storage.

[0048] The data analysis module is used to monitor the storage frequency of search data from users who click to view the target document through a sliding window mechanism to determine whether to immediately execute the preset analysis steps of the target document, as follows:

[0049] The system uses a preset statistical duration as the fixed window duration to count the total number of times the search data of users who clicked to browse the target document is stored in the current window. When the total number of times the data is stored in the window reaches the preset fixed amount, the system immediately executes the analysis steps for the target document. After the execution, all search data in the current window is discarded, and a new window is opened after a preset monitoring duration. The window duration is still the preset statistical duration, and the total number of times the search data is stored in the new window is counted from zero.

[0050] If the total number of times the data stored in the window does not reach the preset fixed amount, the dynamic sliding window will continue to maintain the window duration as the preset statistical duration over time, and continuously monitor whether the number reaches the preset fixed amount. Through the above sliding window mechanism, it can be determined whether the target document is frequently searched in a short period of time, thereby determining whether it is necessary to expand the search efforts for the target document to increase the traffic attractiveness of the target website.

[0051] It should be noted that the statistical duration, monitoring duration, and fixed quantity are preset by the management personnel. The purpose of setting the statistical duration and fixed quantity is to quantify the judgment criteria for short-term high-frequency retrieval by using the quantity threshold within a fixed time window. The basis for setting the monitoring duration is, on the one hand, to avoid the judgment interference of continuous windows and reserve buffer time for the execution of the previous analysis step, and on the other hand, to avoid triggering too many repeated analyses in a short period of time and reduce system resource consumption.

[0052] Storage frequency refers to the total number of times the target document's search data is stored within a unit of statistical time, that is, the ratio of the total number of times the data in the window is stored to the statistical time.

[0053] If it is determined that the analysis step will be performed, then the analysis step will be performed immediately. The analysis step is as follows:

[0054] S11: Obtain all search data of all users who clicked to browse the target document stored in the window when determining to immediately execute the analysis step of the target document, and label them as A1, A2, ..., Aa, where a≥1;

[0055] S12: Obtain the mapping whitelist of target documents from the incremental crawling module, and construct the mapping set B1={C1,C2,...,Cc} of target documents based on all mapping keywords C1, C2,...,Cc in the mapping whitelist, where c is the total number of mapping keywords in the mapping whitelist;

[0056] S13: According to the order of the time entered from the farthest to the nearest time, mark all the search information contained in the search data A1 as D1, D2, ..., Dd, where d≥1 and d is the total number of search information in the search data A1;

[0057] S14: Create a hit variable E1 for retrieval information D1. The initial value of the hit variable E1 is 0. Determine whether retrieval information D1 is a relevant search to the target document according to the preset judgment rules. The judgment steps are as follows:

[0058] S141: Label all keywords contained in the search information D1 as F1, F2, ..., Ff, respectively, where f≥1, and f represents the total number of keywords in the search information D1;

[0059] S142: Determine whether keyword F1 matches mapping set B1, as follows:

[0060] The similarity between keyword F1 and the mapped keywords C1, C2, ..., Cc is calculated, and the largest similarity is compared with P1. If the similarity is greater than or equal to P1, keyword F1 is determined to hit the mapping set B1, and the value of the hit variable E1 is incremented by 1. Otherwise, keyword F1 is determined not to hit the mapping set B1, and no processing is performed. Here, P1 is a preset similarity hit determination threshold. In this application, the value of P1 is set to 0.7, which can effectively distinguish between them.

[0061] It should be noted that when calculating the similarity between keyword F1 and each mapped keyword in mapping set B1, semantic similarity is used. In this application, the Word2Vec model is used to calculate the similarity.

[0062] The specific calculation process is illustrated by calculating the similarity between keyword F1 and the mapped keyword C1, as follows:

[0063] The Word2Vec model, based on the Skip-gram pattern, was used for training on a pre-defined corpus of documents within the technical field of the target literature. The word vector dimension was set to 300 dimensions, generating semantic vectors V for each keyword F1. F1 And the semantic vector V of the mapping keyword C1 C1 ;

[0064] Similarity based on semantic similarity is calculated using the vector cosine value: Sim(F1,C1) = ;

[0065] S143: Perform a hit determination on the keyword F2, F3, ..., Ff according to S141 to S142, and obtain the value G1 of the hit variable E1 after the hit determination on the keyword Ff is completed;

[0066] S144: Calculate the hit rate ratio H1 between the search information D1 and the mapped whitelist using the formula H1=G1 / f. It should be noted that the hit rate ratio is a human definition used to quantify the similarity matching degree between the keywords in the search information D1 and the search keywords in the mapped whitelist.

[0067] Compare H1 and P2. If H1 ≥ P2, then the search information D1 is determined to be a relevant search for the target document. Otherwise, the search information D1 is determined not to be a relevant search for the target document. P2 is the preset relevance determination threshold.

[0068] S15: Determine whether the search information D2, D3, ..., Dd is a relevant search for the target document in sequence according to S14, and re-label all the search information that is determined to be a relevant search for the target document as I1, I2, ..., Ii, 1≤i≤d, where i is the total number of search information that is determined to be a relevant search for the target document in search information D1, D2, ..., Dd;

[0069] S16: Extract all keywords from the search information I1, I2, ..., Ii that are determined to be out of target mapping set B1, and construct a filter set B2 based on them. Here, extracting all keywords from the search information I1 that are determined to be out of target mapping set B1 refers to extracting all keywords from the search information I1 that are determined to be out of target mapping set B1 when relevance search is used to determine whether the search information I1 is a target document. The search information I2, I3, ..., Ii are the same as the search information I1.

[0070] S17: Extract all keywords that are strongly related to the target set B2 according to the preset extraction rules, and construct a strongly related set of search data A1. The extraction rules are as follows:

[0071] S171: Extract all keywords in the set B2 to be filtered, and relabel them as J1, J2, ..., Jj, j≥1, where j is the total number of keywords that were determined not to have hit the mapping set B1 extracted from the search information I1, I2, ..., Ii;

[0072] S172: Determine whether keyword J1 is strongly correlated with the filter set B2. The determination criteria are as follows:

[0073] Following the similarity calculation steps in S142, the similarity between keyword J1 and each keyword in the filter set B2 (excluding keyword J1) is calculated. After the calculation, the total number of keywords whose similarity scores with keyword J1 are greater than or equal to P3 is counted. The total number is compared with P4. If the total number is greater than or equal to P4, it indicates that there are enough keywords in the filter set B2 that are highly similar to keyword J1, and keyword J1 is determined to be strongly correlated in the filter set B2. Otherwise, it indicates that there are not enough keywords in the filter set B2 that are highly similar to keyword J1, and keyword J1 is determined to be not strongly correlated in the filter set B2. P3 and P4 are the preset first and second thresholds, respectively.

[0074] S173: According to S172, determine whether keywords J2, J3, ..., Jj are strongly related to the filter set B2 respectively. After determining whether keywords J1, J2, J3, ..., Jj are strongly related to the filter set B2, construct a strongly related set of search data A1 based on all keywords that are determined to be strongly related to the filter set B2.

[0075] S18: Construct a strongly correlated set of the search data A2, A3, ..., Aa in sequence from S13 to S17;

[0076] S19: According to the preset selection rules, select several keywords from the strongly related sets of search data A1, A2, ..., Aa as incremental search terms for the target documents, and construct the incremental crawl set of the target documents. The selection rules are as follows:

[0077] S191: Extract each keyword from the strongly related set of search data A1. Following the same determination method as step S172, determine whether the keyword is strongly related to the strongly related sets of search data A2, A3, ..., Aa. If the keyword is determined to be strongly related to a strongly related set greater than or equal to P5, then the keyword is marked as an incremental search term for the target document. P5 is a preset cross-set strong correlation determination threshold to ensure the cross-user commonality of incremental search terms.

[0078] S192: Select several keywords from the strongly related sets of search data A2, A3, ..., Aa as incremental search terms for the target documents, according to S191.

[0079] S193: Obtain the incremental search terms for all selected target documents and construct an incremental crawl set of target documents based on them;

[0080] The data analysis module transmits the incremental crawl set of the target documents to the incremental crawling module;

[0081] The incremental crawling module is used to crawl data across the entire network. The incremental crawling module pre-stores a mapping whitelist of target documents. The mapping whitelist contains several search keywords. In this application, the search keywords are extracted from the target documents using a keyword extraction algorithm, which can be any one of TF-IDF or TextRank.

[0082] After receiving the transmitted incremental crawl set of the target document, the incremental crawling module constructs several search terms for the target document based on the incremental crawl set.

[0083] In one embodiment of the present invention, one way to construct several search terms for target documents is to display the incremental crawled set of target documents to the administrator for viewing, and the administrator manually constructs several search terms for the target documents.

[0084] In one embodiment of the present invention, another way to construct several search terms for target documents is to have the administrator pre-specify the maximum and minimum number of incremental search terms contained in a single search term, and group all incremental search terms contained in the incremental crawl set according to semantic relevance. Incremental search terms within each group are connected by "or", and groups are connected by "AND" to form a structured search term. The grouping can be based on any one of K-means clustering or Word2Vec vector similarity clustering.

[0085] For example, keywords can be grouped according to semantic relevance, such as "public opinion monitoring" and "public opinion analysis" as one group, and "big data collection" and "data crawling" as another group. Keywords within each group are connected by "OR", and between groups are connected by "AND", forming a structured search query: Public opinion monitoring OR public opinion analysis AND big data collection OR data crawling;

[0086] For each constructed search query, the entire network is searched and crawled using several preset search engines to obtain crawled data of the target documents. The crawled data includes, but is not limited to, news and academic documents that conform to the search query logic. The crawled data is stored in a document repository to populate the data of the target website.

[0087] A literature repository is used for data storage; the literature repository stores the target literature.

[0088] Some of the data in the above formulas are numerical calculations with dimensions removed, and the contents not described in detail in this specification are all prior art known to those skilled in the art.

[0089] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A method for processing public opinion information, characterized in that, Includes the following steps: Step 1: The data acquisition module collects the search data of any user who clicks on and browses a target document on the target website and transmits it to the data analysis module for storage. The acquisition process is as follows: taking the moment when the user clicks on the target document as the baseline, it traces back to a preset first acquisition time and collects all the search information entered by the user within the first acquisition time; at the same time, taking the moment when the user stops browsing the target document as the starting point, it traces back to a preset second acquisition time and collects all the search information entered by the user within the second acquisition time. The search information includes several keywords. Step Two: The data analysis module monitors the storage frequency of search data from users who clicked on and viewed the target document using a sliding window mechanism to determine whether to execute the preset analysis steps for the target document. If it is determined to execute the analysis steps, the analysis steps are executed immediately, and the incremental crawl set of the target document is determined according to the analysis steps. The incremental crawl set is then transmitted to the incremental crawling module. The content of determining whether to execute the preset analysis steps for the target document is as follows: The system uses a preset statistical duration as the fixed window duration to count the total number of times the search data of users who clicked to browse the target document is stored in the current window. When the total number of times the data is stored in the window reaches the preset fixed amount, the system immediately executes the analysis steps for the target document. After the execution, all search data in the current window is discarded, and a new window is opened after a preset monitoring duration. The window duration is still the preset statistical duration, and the total number of times the search data is stored in the new window is counted from zero. If the total number of times the data in the window is stored does not reach the preset fixed amount, the sliding window will move over time, always maintaining the window duration at the preset statistical duration, and continuously monitoring whether the number reaches the preset fixed amount. The storage frequency refers to the total number of times the target document clicks and browses the user's search data is stored within a unit statistical duration, that is, the ratio of the total number of times the data in the window is stored to the statistical duration. The analysis steps for determining the incremental crawl set of target documents according to the aforementioned analysis steps are as follows: S11: Obtain all search data of all users who clicked to browse the target document stored in the window when determining to immediately execute the analysis step of the target document, and label them as A1, A2, ..., Aa, where a≥1; S12: Obtain the mapping whitelist of the target document, and construct the mapping set B1={C1,C2,...,Cc} of the target document based on all the mapping keywords C1, C2,...,Cc contained in the mapping whitelist, where c is the total number of mapping keywords in the mapping whitelist; S13: Mark all search information contained in search data A1 sequentially as D1, D2, ..., Dd, where d ≥ 1, according to the order of the entered time from the furthest to the nearest time. S14: Create a hit variable E1 for retrieval information D1. The initial value of the hit variable E1 is 0. Determine whether retrieval information D1 is a relevant retrieval of the target document according to the preset judgment rules. S15: Determine whether the search information D2, D3, ..., Dd is a relevant search for the target document in sequence according to S14, and re-label all the search information that is determined to be a relevant search for the target document as I1, I2, ..., Ii, 1≤i≤d respectively; S16: Extract all keywords from the search information I1, I2, ..., Ii that indicate no match for the mapping set B1, and construct the filter set B2 based on them; S17: Extract all keywords that are strongly related to the target set B2 according to the preset extraction rules, and construct a strongly related set of search data A1. S18: Construct a strongly correlated set of the search data A2, A3, ..., Aa in sequence from S13 to S17; S19: Select several keywords from the strongly related sets of search data A1, A2, ..., Aa according to the preset selection rules, and use them as incremental search terms for the target documents to construct an incremental crawling set of the target documents; Step 3: After receiving the transmitted incremental crawl set, the incremental crawling module constructs several search terms for the target documents and performs a search across the entire network based on each constructed search term.

2. The method for processing public opinion information according to claim 1, characterized in that, In S14, the rules for determining whether retrieval information D1 is a relevant retrieval of the target document are as follows: S141: Label all keywords contained in the search information D1 as F1, F2, ..., Ff, respectively, where f≥1, and f represents the total number of keywords in the search information D1; S142: Calculate the similarity between keyword F1 and the mapped keywords C1, C2, ..., Cc respectively, and compare the largest similarity with P1. If the similarity is greater than or equal to P1, it is determined that keyword F1 hits the mapping set B1, and the value of the hit variable E1 is incremented by 1. Otherwise, it is determined that keyword F1 does not hit the mapping set B1, and no processing is performed. Here, P1 is a preset similarity hit determination threshold. S143: Perform a hit determination on the keyword F2, F3, ..., Ff according to S141 to S142, and obtain the value G1 of the hit variable E1 after all determinations are completed; S144: Calculate the hit rate ratio H1 between the search information D1 and the mapped whitelist using the formula H1=G1 / f. Compare H1 with P2. If H1≥P2, then the search information D1 is determined to be a relevant search for the target document; otherwise, the search information D1 is determined not to be a relevant search for the target document. P2 is a preset relevance determination threshold.

3. The method for processing public opinion information according to claim 1, characterized in that, S17, extract all keywords that are strongly related to the target set B2 from the target set B2, and construct the extraction rules for the strongly related set of search data A1 as follows: S171: Extract all keywords in the set B2 to be filtered, and relabel them as J1, J2, ..., Jj, j≥1, where j is the total number of keywords that were determined not to have hit the mapping set B1 extracted from the search information I1, I2, ..., Ii; S172: Calculate the similarity between keyword J1 and each keyword in the filter set B2 except for keyword J1. After the calculation, count the total number of keywords whose similarity value obtained after similarity calculation with keyword J1 is greater than or equal to P3. Compare the total number with P4. If the total number is greater than or equal to P4, it is determined that keyword J1 is strongly related in the filter set B2. Otherwise, it is determined that keyword J1 is not strongly related in the filter set B2. P3 and P4 are the preset strong correlation similarity threshold and strong correlation quantity threshold, respectively. S173: According to S172, determine whether keywords J2, J3, ..., Jj are strongly related to the filter set B2. After determining whether keywords J1, J2, J3, ..., Jj are strongly related to the filter set B2, construct a strongly related set of search data A1 based on all keywords that are determined to be strongly related to the filter set B2.

4. The method for processing public opinion information according to claim 3, characterized in that, S19, select several keywords from the strongly related sets of search data A1, A2, ..., Aa as incremental search terms for the target documents. The selection rules for constructing the incremental crawl set of the target documents are as follows: S191: Extract each keyword from the strongly related set of search data A1. Following the same determination method as step S172, determine whether the keyword is strongly related to the strongly related sets of search data A2, A3, ..., Aa. If the keyword is determined to be strongly related to a strongly related set greater than or equal to P5, then the keyword is marked as an incremental search term for the target document. P5 is a preset cross-set strong correlation determination threshold. S192: Select several keywords from the strongly related sets of search data A2, A3, ..., Aa as incremental search terms for the target documents, according to S191. S193: Obtain the incremental search terms for all selected target documents and construct an incremental crawl set of target documents based on them.

Images