Public opinion information processing method

Abstract

The invention discloses a public opinion information processing method, and relates to the technical field of information processing, the user retrieval data storage frequency of a target literature is monitored through a sliding window mechanism, preset analysis is triggered only when the literature is browsed at high frequency for a short time, the analysis pertinence is ensured, and after the analysis is triggered, the user retrieval data storage frequency of the target literature is monitored. Potential incremental retrieval words of a target document are mined from user retrieval data by constructing a mapping set, a strong correlation set and an incremental crawling set, an incremental crawling module constructs a retrieval formula based on the incremental retrieval words, and resources meeting requirements are accurately crawled in the whole network and stored in a document storage library. According to the method, the problems of blind resource expansion and disjoint with user requirements in traditional crawling are solved, the target website literature resources are continuously enriched around the core requirements of the users, content reserve is perfected, more users with the same requirements can be attracted to access, and the flow and competitiveness of the platform are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, specifically to a method for processing public opinion information. Background Technology

[0002] In today's world where the dissemination of public opinion information is becoming increasingly convenient, various information service websites have become the core channel for the public to obtain relevant documents. These websites collect diverse documents such as statements, event analysis reports, and commentary articles, and set exclusive keywords for each document to optimize the search experience, attract users to browse and view, and thus increase website traffic.

[0003] However, in actual operation, existing websites essentially terminate their service chain after uploading documents and configuring keywords, lacking in-depth analysis and response mechanisms for user search behavior. In particular, some documents experience explosive growth in high-frequency searches and browsing in a short period of time due to factors such as the fermentation of social hotspots and sudden public events. This phenomenon should have been a key opportunity for websites to seize traffic dividends and enhance user stickiness, but existing websites have neither conducted special analysis on the search user data of such high-frequency search documents nor been able to expand the crawling of related resources based on user search information. As a result, they can only provide users with single-target documents, making it difficult to meet their extended information needs. This not only causes websites to miss the opportunity to attract a large amount of traffic, but also reduces the user experience due to the single nature of information supply, gradually losing its advantage in the competition with similar information service platforms.

[0004] To address the above problems, this invention proposes a solution. Summary of the Invention

[0005] The purpose of this invention is to provide a method for processing public opinion information in order to solve the problems mentioned in the background art.

[0006] This invention provides a method for processing public opinion information, comprising the following steps:

[0007] Step 1: The data acquisition module collects the search data of any user who clicks to browse the target document on the target website and transmits it to the data analysis module for storage;

[0008] Step 2: The data analysis module monitors the storage frequency of the search data of users who click to browse the target document through a sliding window mechanism to determine whether to execute the preset analysis steps of the target document. If it is determined to execute the analysis steps, the analysis steps are executed immediately, and the incremental crawl set of the target document is determined according to the analysis steps. The incremental crawl set is then transmitted to the incremental crawling module.

[0009] Step 3: After receiving the transmitted incremental crawl set, the incremental crawling module constructs several search terms for the target documents and performs a search across the entire network based on each constructed search term.

[0010] Furthermore, in step one, for any user who clicks to browse the target document, the data collection process is as follows: taking the moment when the user clicks on the target document as the reference point, backtracking to a preset first collection period, all search information entered by the user during this period is collected; simultaneously, taking the moment when the user stops browsing the target document as the starting point, backtracking to a preset second collection period, all search information entered by the user during this period is collected, and the search information includes several keywords.

[0011] Furthermore, in step two, the process of determining whether to execute the preset analysis steps for the target document is as follows:

[0012] The system uses a preset statistical duration as the fixed window duration to count the total amount of search data stored in the current window for users who clicked and viewed the target document. When the total amount of data in the window reaches the preset fixed amount, the system immediately executes the analysis steps for the target document. After the execution, all search data in the current window is discarded, and a new window is opened after a preset monitoring duration. The window duration is still the preset statistical duration, and the total amount of search data in the new window is counted from zero.

[0013] If the total amount of data in the window does not reach the preset fixed amount, the dynamic sliding window will move over time, always maintaining the window duration at the preset statistical duration, and continuously monitoring whether the quantity reaches the preset fixed amount. The storage frequency refers to the total amount of search data of the target document clicked and viewed by the user stored within a unit statistical duration, that is, the ratio of the total amount of data in the window to the statistical duration.

[0014] Furthermore, in step two, the analysis steps for determining the incremental crawl set of target documents according to the aforementioned analysis steps are as follows:

[0015] S11: Obtain all retrieval data of all browsing users who clicked to browse the target document stored in the window when determining to immediately execute the analysis step of the target document, and label them as A1, A2, ..., Aa, where a≥1;

[0016] S12: Obtain the mapping whitelist of the target document, and construct the mapping set B1={C1,C2,...,Cc} of the target document based on all the mapping keywords C1, C2,...,Cc contained in the mapping whitelist, where c is the total number of mapping keywords in the mapping whitelist;

[0017] S13: Mark all search information contained in search data A1 sequentially as D1, D2, ..., Dd, where d ≥ 1, according to the order of the entered time from the furthest to the nearest time.

[0018] S14: Create a hit variable E1 for retrieval information D1. The initial value of the hit variable E1 is 0. Determine whether retrieval information D1 is a relevant retrieval of the target document according to the preset judgment rules.

[0019] S15: Determine whether the search information D2, D3, ..., Dd is a relevant search for the target document in sequence according to S14, and re-label all the search information that is determined to be a relevant search for the target document as I1, I2, ..., Ii, 1≤i≤d respectively;

[0020] S16: Extract all keywords that were determined not to have hit the mapping set B1 from the search information I1, I2, ..., Ii respectively, and construct the filter set B2 based on them;

[0021] S17: Extract all keywords that are strongly related to the target set B2 according to the preset extraction rules, and construct a strongly related set of search data A1.

[0022] S18: Construct a strongly correlated set of the search data A2, A3, ..., Aa in sequence according to S13 to S17;

[0023] S19: Select several keywords from the strongly related sets of search data A1, A2, ..., Aa according to the preset selection rules, and use them as incremental search terms for the target documents to construct an incremental crawl set of the target documents.

[0024] Furthermore, in S14, the rules for determining whether retrieval information D1 is a relevant retrieval of the target document are as follows:

[0025] S141: Label all keywords contained in the search information D1 as F1, F2, ..., Ff, respectively, where f≥1, and f represents the total number of keywords in the search information D1;

[0026] S142: Calculate the similarity between keyword F1 and the mapped keywords C1, C2, ..., Cc respectively, and compare the largest similarity with P1. If the similarity is greater than or equal to P1, it is determined that keyword F1 hits the mapping set B1, and the value of the hit variable E1 is incremented by 1. Otherwise, it is determined that keyword F1 does not hit the mapping set B1, and no processing is performed. Here, P1 is a preset similarity hit determination threshold.

[0027] S143: Perform a hit determination on the keyword F2, F3, ..., Ff according to S141 to S142, and obtain the value G1 of the hit variable E1 after all determinations are completed;

[0028] S144: Calculate the hit rate ratio H1 between the search information D1 and the mapped whitelist using the formula H1=G1 / f. Compare H1 with P2. If H1≥P2, then the search information D1 is determined to be a relevant search for the target document; otherwise, the search information D1 is determined not to be a relevant search for the target document. P2 is a preset relevance determination threshold.

[0029] Furthermore, in S17, extract all keywords that are strongly related to the target set B2, and construct the extraction rules for the strongly related set of search data A1 as follows:

[0030] S171: Extract all keywords in the set B2 to be filtered, and relabel them as J1, J2, ..., Jj, j≥1, where j is the total number of keywords that were determined not to have hit the mapping set B1 extracted from the search information I1, I2, ..., Ii;

[0031] S172: Calculate the similarity between keyword J1 and each keyword in the filter set B2 except for keyword J1. After the calculation, count the total number of keywords whose similarity value obtained after similarity calculation with keyword J1 is greater than or equal to P3. Compare the total number with P4. If the total number is greater than or equal to P4, it is determined that keyword J1 is strongly related in the filter set B2. Otherwise, it is determined that keyword J1 is not strongly related in the filter set B2. P3 and P4 are the preset strong correlation similarity threshold and strong correlation quantity threshold, respectively.

[0032] S173: According to S172, determine whether keywords J2, J3, ..., Jj are strongly related to the filter set B2. After determining whether keywords J1, J2, J3, ..., Jj are strongly related to the filter set B2, construct a strongly related set of search data A1 based on all keywords that are determined to be strongly related to the filter set B2.

[0033] Furthermore, in S19, several keywords are selected from the strongly related sets of the search data A1, A2, ..., Aa as incremental search terms for the target documents. The selection rules for constructing the incremental crawl set of the target documents are as follows:

[0034] S191: Extract each keyword from the strongly related set of search data A1. Following the same determination method as step S172, determine whether the keyword is strongly related to the strongly related sets of search data A2, A3, ..., Aa. If the keyword is determined to be strongly related to a strongly related set greater than or equal to P5, then the keyword is marked as an incremental search term for the target document. P5 is a preset cross-set strong correlation determination threshold.

[0035] S192: Select several keywords from the strongly related sets of search data A2, A3, ..., Aa as incremental search terms for the target documents, according to S191.

[0036] S193: Obtain the incremental search terms for all selected target documents and construct an incremental crawl set of target documents based on them.

[0037] Compared with existing technologies, it has the following advantages:

[0038] This invention monitors the frequency of user-retrieved data storage for target documents using a sliding window mechanism. Pre-set analysis is triggered only when a document is viewed frequently in a short period of time, ensuring the analysis is targeted. Data collection is performed by tracing back the first collection time from the moment the user clicks on the document and extending the second collection time backward from the moment the browsing stops, making the data coverage more comprehensive. Combined with the Word2Vec model, keyword semantic similarity is calculated. By statistically analyzing hit variables and calculating the hit rate ratio, relevant content is filtered from massive amounts of information, ensuring that the analysis results are highly consistent with the user's actual needs. This solves the problem that websites do not know the core concerns of users.

[0039] In this invention, the dynamic window mechanism uses a preset statistical duration as a fixed window to count the total amount of data retrieved within the window in real time. The preset analysis steps are executed only when the data reaches the preset fixed amount. If the threshold is not reached, the dynamic sliding window continues to monitor, realizing on-demand triggering of analysis only when high attention is paid. After the analysis is executed, the system discards the data in the current window and opens a new window after a preset monitoring interval. This avoids interference from continuous window judgments, reserves time for the previous analysis to finish, prevents repeated triggering in a short period of time, reduces ineffective computing power consumption, and improves system operating efficiency.

[0040] After triggering analysis, this invention constructs a mapping set, a strongly correlated set, and an incremental crawling set to mine potential incremental search terms for target documents from user search data. Based on these incremental search terms, the incremental crawling module constructs a search formula to accurately crawl resources that meet the requirements across the entire network and store them in the document repository. This solves the problems of blind resource expansion and disconnection from user needs in traditional crawling, enabling the document resources of the target website to be continuously enriched around the core needs of users. This not only improves the content reserves but also attracts more users with similar needs, effectively enhancing the platform's traffic and competitiveness. Attached Figure Description

[0041] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Please see Figure 1 This application provides a method for processing public opinion information. The above method is executed through a public opinion information processing system, which includes a data acquisition module, a data analysis module, an incremental crawling module, and a document repository.

[0044] The data acquisition module is used to collect search data from any user who clicks to browse a target document. The data collection process for any user who clicks to browse a target document is as follows: taking the moment the user clicks on the target document as a reference point, the module traces back a preset first collection period and collects all search information entered by the user during that period; simultaneously, taking the moment the user stops browsing the target document as a starting point, the module traces back a preset second collection period and collects all search information entered by the user during that period; each search entry contains several keywords entered by the user to retrieve document data within the target website.

[0045] The criteria for determining whether a user stops browsing a target document are: the user enters new search information or the user closes the target document.

[0046] It should be noted that the first collection duration and the second collection duration are preset by the administrator. The setting is based on the following: numerically, the first collection duration is longer than the second collection duration, that is, the scope of the search information collected before the user clicks is greater than the scope of the search information collected after the user clicks, so as to more fully capture the user's active search intent before clicking on the target document, while also taking into account the record of related search behavior after the user clicks.

[0047] The data collection module collects the search data of each user who clicks to browse the target document and then transmits it to the data analysis module for storage.

[0048] The data analysis module is used to monitor the storage frequency of search data from users who click to view the target document through a sliding window mechanism to determine whether to immediately execute the preset analysis steps of the target document, as follows:

[0049] The system uses a preset statistical duration as the fixed window duration to count the total amount of search data stored in the current window for users who clicked and viewed the target document. When the total amount of data in the window reaches the preset fixed amount, the system immediately executes the analysis steps for the target document. After the execution, all search data in the current window is discarded, and a new window is opened after a preset monitoring duration. The window duration is still the preset statistical duration, and the total amount of search data in the new window is counted from zero.

[0050] If the total amount of data in the window does not reach the preset fixed amount, the dynamic sliding window will move over time, always maintaining the window duration at the preset statistical duration, and continuously monitoring whether the number reaches the preset fixed amount. Through the above sliding window mechanism, it can be determined whether the target document is frequently searched in a short period of time, thereby determining whether it is necessary to expand the search efforts for the target document to increase the traffic attractiveness of the target website.

[0051] It should be noted that the statistical duration, monitoring duration, and fixed quantity are preset by the management personnel. The purpose of setting the statistical duration and fixed quantity is to quantify the judgment criteria for short-term high-frequency retrieval by using the quantity threshold within a fixed time window. The basis for setting the monitoring duration is, on the one hand, to avoid the judgment interference of continuous windows and reserve buffer time for the execution of the previous analysis step, and on the other hand, to avoid triggering too many repeated analyses in a short period of time and reduce system resource consumption.

[0052] Storage frequency refers to the total amount of search data of users who clicked to browse the target document stored within a unit of statistical time, that is, the ratio of the total amount of data in the window to the statistical time.

[0053] If it is determined that the analysis step will be performed, then the analysis step will be performed immediately. The analysis step is as follows:

[0054] S11: Obtain all retrieval data of all browsing users who clicked to browse the target document stored in the window when determining to immediately execute the analysis step of the target document, and label them as A1, A2, ..., Aa, where a≥1;

[0055] S12: Obtain the mapping whitelist of target documents from the incremental crawling module, and construct the mapping set B1={C1,C2,...,Cc} of target documents based on all mapping keywords C1, C2,...,Cc in the mapping whitelist, where c is the total number of mapping keywords in the mapping whitelist;

[0056] S13: According to the order of the time entered from the farthest to the nearest time, mark all the search information contained in the search data A1 as D1, D2, ..., Dd, where d≥1 and d is the total number of search information in the search data A1;

[0057] S14: Create a hit variable E1 for retrieval information D1. The initial value of the hit variable E1 is 0. Determine whether retrieval information D1 is a relevant search to the target document according to the preset judgment rules. The judgment steps are as follows:

[0058] S141: Label all keywords contained in the search information D1 as F1, F2, ..., Ff, respectively, where f≥1, and f represents the total number of keywords in the search information D1;

[0059] S142: Determine whether keyword F1 matches mapping set B1, as follows:

[0060] The similarity between keyword F1 and the mapped keywords C1, C2, ..., Cc is calculated, and the largest similarity is compared with P1. If the similarity is greater than or equal to P1, keyword F1 is determined to hit the mapping set B1, and the value of the hit variable E1 is incremented by 1. Otherwise, keyword F1 is determined not to hit the mapping set B1, and no processing is performed. Here, P1 is a preset similarity hit determination threshold. In this application, the value of P1 is set to 0.7, which can effectively distinguish between them.

[0061] It should be noted that when calculating the similarity between keyword F1 and each mapped keyword in mapping set B1, semantic similarity is used. In this application, the Word2Vec model is used to calculate the similarity.

[0062] The specific calculation process is illustrated by calculating the similarity between keyword F1 and the mapped keyword C1, as follows:

[0063] The Word2Vec model, based on the Skip-gram pattern, was used for training on a pre-defined corpus of documents within the technical field of the target literature. The word vector dimension was set to 300 dimensions, generating semantic vectors V for each keyword F1. F1 And the semantic vector V of the mapping keyword C1 C1 ;

[0064] Similarity based on semantic similarity is calculated using the vector cosine value: Sim(F1,C1) = ;

[0065] S143: Perform a hit determination on the keyword F2, F3, ..., Ff according to S141 to S142, and obtain the value G1 of the hit variable E1 after the hit determination on the keyword Ff is completed;

[0066] S144: Calculate the hit rate ratio H1 between the search information D1 and the mapped whitelist using the formula H1=G1 / f. It should be noted that the hit rate ratio is a human definition used to quantify the similarity matching degree between the keywords in the search information D1 and the search keywords in the mapped whitelist.

[0067] Compare H1 and P2. If H1 ≥ P2, then the search information D1 is determined to be a relevant search for the target document. Otherwise, the search information D1 is determined not to be a relevant search for the target document. P2 is the preset relevance determination threshold.

[0068] S15: Determine whether the search information D2, D3, ..., Dd is a relevant search for the target document in sequence according to S14, and re-label all the search information that is determined to be a relevant search for the target document as I1, I2, ..., Ii, 1≤i≤d, where i is the total number of search information that is determined to be a relevant search for the target document in search information D1, D2, ..., Dd;

[0069] S16: Extract all keywords from the search information I1, I2, ..., Ii that are determined to be out of target mapping set B1, and construct a filter set B2 based on them. Here, extracting all keywords from the search information I1 that are determined to be out of target mapping set B1 refers to extracting all keywords from the search information I1 that are determined to be out of target mapping set B1 when relevance search is used to determine whether the search information I1 is a target document. The search information I2, I3, ..., Ii are the same as the search information I1.

[0070] S17: Extract all keywords that are strongly related to the target set B2 according to the preset extraction rules, and construct a strongly related set of search data A1. The extraction rules are as follows:

[0071] S171: Extract all keywords in the set B2 to be filtered, and relabel them as J1, J2, ..., Jj, j≥1, where j is the total number of keywords that were determined not to have hit the mapping set B1 extracted from the search information I1, I2, ..., Ii;

[0072] S172: Determine whether keyword J1 is strongly correlated with the filter set B2. The determination criteria are as follows:

[0073] Following the similarity calculation steps in S142, the similarity between keyword J1 and each keyword in the filter set B2 (excluding keyword J1) is calculated. After the calculation, the total number of keywords whose similarity scores with keyword J1 are greater than or equal to P3 is counted. The total number is compared with P4. If the total number is greater than or equal to P4, it indicates that there are enough keywords in the filter set B2 that are highly similar to keyword J1, and keyword J1 is determined to be strongly correlated in the filter set B2. Otherwise, it indicates that there are not enough keywords in the filter set B2 that are highly similar to keyword J1, and keyword J1 is determined to be not strongly correlated in the filter set B2. P3 and P4 are the preset first and second thresholds, respectively.

[0074] S173: According to S172, determine whether keywords J2, J3, ..., Jj are strongly related to the filter set B2 respectively. After determining whether keywords J1, J2, J3, ..., Jj are strongly related to the filter set B2, construct a strongly related set of search data A1 based on all keywords that are determined to be strongly related to the filter set B2.

[0075] S18: Construct a strongly correlated set of the search data A2, A3, ..., Aa in sequence according to S13 to S17;

[0076] S19: According to the preset selection rules, select several keywords from the strongly related sets of search data A1, A2, ..., Aa as incremental search terms for the target documents, and construct the incremental crawl set of the target documents. The selection rules are as follows:

[0077] S191: Extract each keyword from the strongly related set of search data A1. Following the same determination method as step S172, determine whether the keyword is strongly related to the strongly related sets of search data A2, A3, ..., Aa. If the keyword is determined to be strongly related to a strongly related set greater than or equal to P5, then the keyword is marked as an incremental search term for the target document. P5 is a preset cross-set strong correlation determination threshold to ensure the cross-user commonality of incremental search terms.

[0078] S192: Select several keywords from the strongly related sets of search data A2, A3, ..., Aa as incremental search terms for the target documents, according to S191.

[0079] S193: Obtain the incremental search terms for all selected target documents and construct an incremental crawl set of target documents based on them;

[0080] The data analysis module transmits the incremental crawl set of the target documents to the incremental crawling module;

[0081] The incremental crawling module is used to crawl data across the entire network. The incremental crawling module pre-stores a mapping whitelist of target documents. The mapping whitelist contains several search keywords. In this application, the search keywords are extracted from the target documents using a keyword extraction algorithm, which can be any one of TF-IDF or TextRank.

[0082] After receiving the transmitted incremental crawl set of the target document, the incremental crawling module constructs several search terms for the target document based on the incremental crawl set.

[0083] In one embodiment of the present invention, one way to construct several search terms for target documents is to display the incremental crawled set of target documents to the administrator for viewing, and the administrator manually constructs several search terms for target documents.

[0084] In one embodiment of the present invention, another way to construct several search terms for target documents is to have the administrator pre-specify the maximum and minimum number of incremental search terms contained in a single search term, and group all incremental search terms contained in the incremental crawl set according to semantic relevance. Incremental search terms within each group are connected by "or", and groups are connected by "AND" to form a structured search term. The grouping can be based on any one of K-means clustering or Word2Vec vector similarity clustering.

[0085] For example, keywords can be grouped according to semantic relevance, such as "public opinion monitoring" and "public opinion analysis" as one group, and "big data collection" and "data crawling" as another group. Keywords within each group are connected by "OR", and between groups are connected by "AND", forming a structured search query: Public opinion monitoring OR public opinion analysis AND big data collection OR data crawling;

[0086] For each constructed search query, the entire network is searched and crawled using several preset search engines to obtain crawled data of the target documents. The crawled data includes, but is not limited to, news and academic documents that conform to the search query logic. The crawled data is stored in a document repository to populate the data of the target website.

[0087] A literature repository is used for data storage; the literature repository stores the target literature.

[0088] Some of the data in the above formulas are numerical calculations with dimensions removed, and the contents not described in detail in this specification are all prior art known to those skilled in the art.

[0089] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. An opinion information processing method characterized by comprising: The method comprises the following steps: Step one: the data collection module collects the search data of the browsing user who clicks and browses the target literature in the target website and transmits the search data to the data analysis module for storage; Step two: the data analysis module monitors the storage frequency of the search data of the browsing user who clicks and browses the target literature through the sliding window mechanism to determine whether to execute the preset analysis step of the target literature, if it is determined to execute the analysis step, the analysis step is immediately executed, and the incremental crawling set of the target literature is determined according to the analysis step, and the incremental crawling set is transmitted to the incremental crawling module; Step three: the incremental crawling module receives the transmitted incremental crawling set and constructs a plurality of search formulas of the target literature, and searches the whole network based on each search formula.

2. The public opinion information processing method according to claim 1, characterized in that, In step one, for any browsing user who clicks and browses the target literature, the collection process of the search data of the browsing user is as follows: taking the time when the browsing user clicks the target literature as the reference point, the first preset collection time is traced back, all search information input by the browsing user in the time period is collected; at the same time, taking the time when the browsing user stops browsing the target literature as the starting point, the second preset collection time is traced back, all search information input by the browsing user in the time period is collected, and the search information contains a plurality of keywords.

3. The public opinion information processing method of claim 1, wherein In step two, the content of determining whether to execute the preset analysis step of the target literature is as follows: Taking the preset statistical time length as the fixed window length, the total amount of search data of the browsing user who clicks and browses the target literature stored in the current window is counted in real time; when the total amount of data in the window reaches the preset fixed amount, it is determined to immediately execute the analysis step of the target literature, and after the determination is executed, all search data in the current window is discarded, a new window is restarted after a preset monitoring time interval, and the window length is still the preset statistical time length, and the total amount of search data in the new window is counted from zero; If the total amount of data in the window does not reach the preset fixed amount, the sliding window is dynamically pushed with time, the window length is always maintained as the preset statistical time length, and whether the amount reaches the preset fixed amount is continuously monitored, and the storage frequency refers to the total amount of search data of the browsing user who clicks and browses the target literature stored in the unit statistical time length, that is, the ratio of the total amount of data in the window to the statistical time length.

4. The public opinion information processing method according to claim 3, characterized in that, In step two, the analysis step of determining the incremental crawling set of the target literature according to the analysis step is as follows: S11: obtaining all search data of all browsing users who click and browse the target literature stored in the window when it is determined to immediately execute the analysis step of the target literature, and marking the search data as A1, A2,..., Aa, a≥1 respectively; S12: obtaining the mapping white list of the target literature, constructing the mapping set B1={C1,C2,...,Cc} of the target literature according to all mapping keywords C1, C2,..., Cc contained in the mapping white list, and c is the total number of mapping keywords in the mapping white list; S13: all search information contained in the search data A1 is sequentially marked as D1, D2,..., Dd in the order from far to near according to the time interval between the input time and the current time, d≥1; S14: Create a hit variable E1 of the search information D1, the initial value of the hit variable E1 is 0, and determine whether the search information D1 is the coherent search of the target document according to the preset determination rule; S15: Determine whether the search information D2, D3,..., Dd is the coherent search of the target document according to S14 in turn, and re-label all the search information determined as the coherent search of the target document as I1, I2,..., Ii, 1≤i≤d; S16: Extract all the keywords determined not to hit the mapping set B1 from the search information I1, I2,..., Ii respectively, and construct a to-be-screened set B2 according to the same; S17: Extract all the keywords that are strongly related in the to-be-screened set B2 from the to-be-screened set B2 according to the preset extraction rule, and construct a strong correlation set of the search data A1; S18: Construct the strong correlation set of the search data A2, A3,..., Aa according to S13 to S17 in turn; S19: Select a number of keywords from the strong correlation set of the search data A1, A2,..., Aa as the incremental search words of the target document according to the preset selection rule, and construct an incremental crawling set of the target document.

5. The public opinion information processing method according to claim 4, characterized in that, In S14, the determination rule for determining whether the search information D1 is the coherent search of the target document is as follows: S141: Label all the keywords contained in the search information D1 as F1, F2,..., Ff, f≥1, f represents the total number of keywords in the search information D1; S142: Calculate the similarity of the keyword F1 with the mapping keywords C1, C2,..., Cc respectively, and compare the maximum similarity with P1, if the similarity is greater than or equal to P1, it is determined that the keyword F1 hits the mapping set B1, at this time the value of the hit variable E1 is increased by 1, otherwise it is determined that the keyword F1 does not hit the mapping set B1, and no processing is performed, wherein P1 is a preset similarity hit determination threshold; S143: Determine whether the keywords F2, F3,..., Ff hit the mapping set B1 according to S141 to S142 in turn, and obtain the value G1 of the hit variable E1 at this time after all the determinations are completed; S144: Calculate the hit rate of the search information D1 with the mapping white list H1 by using the formula H1=G1 / f, and compare H1 with P2, if H1≥P2, it is determined that the search information D1 is the coherent search of the target document, otherwise it is determined that the search information D1 is not the coherent search of the target document, P2 is a preset coherent determination threshold.

6. The public opinion information processing method according to claim 4, characterized in that, In S17, the extraction rule for extracting all the keywords that are strongly related in the to-be-screened set B2 from the to-be-screened set B2 to construct the strong correlation set of the search data A1 is as follows: S171: Extract all the keywords in the to-be-screened set B2, and re-label them as J1, J2,..., Jj, j≥1, j is the total number of keywords extracted from the search information I1, I2,..., Ii and determined not to hit the mapping set B1; S172: Calculate the similarity between keyword J1 and each keyword in the set B2 except keyword J1, and count the number of keywords whose similarity with keyword J1 is greater than or equal to P3. Compare the number with P4. If the number is greater than or equal to P4, determine that keyword J1 is strongly related in the set B2. Otherwise, determine that keyword J1 is not strongly related in the set B2. P3 and P4 are respectively the preset strong correlation similarity threshold and the strong correlation quantity threshold. S173: Determine whether keywords J2, J3,..., Jj are strongly related in the set B2 according to S172. After determining whether keywords J1, J2, J3,..., Jj are strongly related in the set B2, construct the strong correlation set of the search data A1 according to all keywords determined to be strongly related in the set B2.

7. The public opinion information processing method according to claim 6, characterized in that, S19: Select several keywords from the strong correlation sets of the search data A1, A2,..., Aa as the incremental retrieval words of the target literature. The selection rules for the incremental crawling set of the target literature are as follows: S191: Extract each keyword from the strong correlation set of the search data A1. Determine whether the keyword is strongly related in the strong correlation sets of the search data A2, A3,..., Aa according to the same determination method in step S172. If the keyword is strongly related in more than or equal to P5 strong correlation sets, mark the keyword as an incremental retrieval word of the target literature. P5 is the preset cross-set strong correlation determination threshold. S192: Select several keywords from the strong correlation sets of the search data A2, A3,..., Aa as the incremental retrieval words of the target literature according to S191. S193: Obtain all selected incremental retrieval words of the target literature and construct the incremental crawling set of the target literature according to them.

Images