A bandit algorithm-based topic crawler method

Abstract

The application discloses a topic crawler method based on a Bandit algorithm, finds out key words of correlation based on correlation dictionary API and extension of correlation of keywords, optimizes a target model based on training of the Bandit algorithm, calculates the income of each keyword, that is, the degree of interest, distinguishes the keyword range corresponding to each topic crawler, and performs content grabbing of online topic crawlers. The method can extend the similarity of keywords, obtain more valuable information, highly find effective pages, reflect the real interest points of users, timely adjust and correct crawler behaviors, and avoid waste of time and space resources.

Description

Technical Field

[0001] This invention belongs to the field of network information acquisition technology in the computer information management industry, and specifically relates to a topic crawling method based on the Bandit algorithm. Background Technology

[0002] With the development of digital technology and the iteration of mobile media, network information is becoming increasingly abundant. Network information management is a complex system engineering project that is multi-faceted, multi-element, and involves the entire process. Its function and value lie in its ability to identify the evolutionary process and utility mechanism of the initiation, occurrence, and development of events. By collecting and analyzing network information, it is possible to predict secondary and derivative events, prevent and respond to potential risks from a systematic and process-oriented perspective, and reduce operations.

[0003] Web information management typically relies on the value of content crawled by web crawlers. Traditional topic crawlers are programs or scripts that automatically crawl web pages according to certain rules, such as specifying an initial URL or a specific site. Their main purpose is to obtain a large amount of data containing topic-related information within a certain time. However, the use of traditional web crawlers is limited by some technical and professional aspects, such as limitations in collection speed, value density, and professional domain knowledge. The returned web pages often contain a lot of valueless information and cannot meet the diverse needs of customers.

[0004] Existing web crawling methods for managing web information typically follow fixed rules—such as URL keyword matching or text content keyword matching—to extract web page information. While these methods can obtain a certain amount of information, there is a significant difference between accuracy and the amount of information retrieved. Existing technologies have two main shortcomings:

[0005] The reliance on a single keyword, without expanding on keyword similarity, prevents the acquisition of more valuable information and results in a low success rate in finding effective pages.

[0006] The lack of user participation in the information acquisition process makes it impossible to reflect the user's true interests (e.g., topic content) and to correct the crawler's behavior (e.g., keyword range) in a timely manner, resulting in a waste of time and space resources.

[0007] The web information obtained by existing web crawlers suffers from high homogeneity and one-sidedness. Summary of the Invention

[0008] To address the aforementioned problems in existing technologies, the present invention aims to provide a topic crawling method based on the Bandit algorithm, which can expand the similarity of keywords, obtain more valuable information, and highly search for effective pages; it can reflect the user's true interests and can promptly correct crawling behavior, saving time and space resources.

[0009] The technical solution adopted in this invention is as follows:

[0010] A topic crawler method based on the Bandit algorithm includes the following steps:

[0011] S01: Create an associated thesaurus;

[0012] S02: Create a topic-based crawler, pre-select keywords and sites related to the topic, and perform word segmentation preprocessing on the keywords to distinguish between user keywords and relevant keywords;

[0013] S03: Create the Bandit algorithm model. The initial input parameters of the Bandit algorithm model are the user-set preset maximum number of attempts threshold n and preset rate of return threshold p, as well as the dynamic parameter information of the keywords in S02.

[0014] The dynamic parameter information of the keywords includes the interest coefficient α and the disinterest coefficient β. The initial values ​​of the interest coefficient α and the disinterest coefficient β for each keyword are preset to 1.

[0015] S04: The web crawler retrieves a new webpage;

[0016] Use a Bloom filter to quickly identify whether it is a new webpage;

[0017] If it is a new webpage, proceed to step S05;

[0018] If it is not a new webpage, discard it directly;

[0019] S05: Format the key information of the new webpage, extract the current information, and organize it into a data packet;

[0020] S06: Match the keyword attribute data of the data packet with user keywords and relevant keywords;

[0021] If the keyword attribute data of the data packet matches the user's keyword, then proceed directly to step S11 to download the webpage;

[0022] If the keyword attribute data of the data packet does not match the user keyword, proceed to step S07;

[0023] S07: Continue matching with relevant keywords;

[0024] If the keyword attribute data of a data packet does not match any relevant keywords, the data packet is discarded.

[0025] If the keyword attribute data of the data packet matches the relevant keyword, then proceed to step S08;

[0026] S08: Manually label the data and provide feedback to the user, asking whether the data package should be further crawled.

[0027] S09: The user filters and judges the data packet, performs a "mark" operation on the page, and then provides feedback;

[0028] S10: After receiving user feedback, the Beta parameter value in the relevant keyword changes accordingly;

[0029] If the user accepts, meaning the user agrees to further crawl the contents of the data packet, then α changes to α+1; otherwise, β changes to β+1, and the process jumps to step S11.

[0030] S11: The crawler download queue crawls content from known URLs;

[0031] S12: Based on the input keywords and the corresponding Beta parameters, calculate the yield p for each keyword using the Bandit algorithm model;

[0032] S13: Sort keywords based on their interest level according to the return on each keyword p.

[0033] Furthermore, step S01 includes the following operations:

[0034] By crawling online dictionary websites, raw keywords and corresponding synonyms are obtained. Then, words are manually filtered and classified to form a custom related word library, establishing a network of relationships between keywords and related words. Based on this related word library, an API service is encapsulated to provide the function of obtaining keywords related to words.

[0035] Furthermore, step S02 includes the following operations:

[0036] Using the related keyword library API, a series of relevant keywords are obtained based on the keywords set by the user, and the top 5 relevant keywords are selected.

[0037] Further, in step S03, the dynamic parameter information of each keyword is used to calculate the rate of return of each keyword according to the expected value formula α / (α+β) of the Beta distribution, that is:

[0038] p = α / (α+β).

[0039] Furthermore, in step S05, the key information of the new webpage includes the following:

[0040] S0501, the URL information obtained;

[0041] S0502, the text content of the hyperlink;

[0042] S0503, the content of the nearby text;

[0043] The text content of hyperlinks and the text content near them will be segmented into words to extract keywords.

[0044] Furthermore, step S08 also includes the following operations:

[0045] If the keyword attribute data of a data packet matches a relevant keyword, but the Bandit algorithm model identifies that the keyword has been marked as "keywords of no interest" multiple times in the history, then the data packet is discarded directly.

[0046] Furthermore, in step S13, the keyword identifier is determined based on the comparison results of the number of times the formatted data packet of the keyword was not crawled by the user with the preset upper limit threshold, and the comparison results of the yield p of the keyword with the preset yield threshold.

[0047] Furthermore, step S13 also includes the following operations:

[0048] If the value of a keyword is greater than or equal to 1000, and 0.8 ≤ rate of return p ≤ 1, then the keyword is marked as a "keyword of interest".

[0049] Furthermore, step S13 also includes the following operations:

[0050] If a keyword has an α+β value greater than or equal to 1000, a return p less than or equal to 0.1, and the number of times the keyword is not used by users is greater than or equal to a preset number of times, and the keyword's return p is less than a preset return threshold, then the keyword is marked as a "keyword that is definitely not of interest".

[0051] Finally, step S13 also includes the following operations:

[0052] If the α+β value of a keyword is less than or equal to 10, then the keyword is marked as a "keyword that may be of interest".

[0053] The beneficial effects of this invention are as follows:

[0054] A topic crawling method based on the Bandit algorithm is proposed. This method leverages a related thesaurus API to expand keyword relevance, identify relevant keywords, and trains and optimizes the target model using the Bandit algorithm. It calculates the benefit (i.e., interest level) of each keyword in real-time, distinguishes the keyword range corresponding to each topic crawler, and performs online topic crawling. It can expand keyword similarity to obtain more valuable information and effectively find relevant pages; it can reflect the user's true interests, adjust and correct crawler behavior in a timely manner, and avoid wasting time and space resources.

[0055] Applied to internet information gathering, this approach, while focusing on key information, uncovers more online information without the risk of cocooning or silence effects caused by subjective keyword and site selection. It avoids getting bogged down in a large amount of homogeneous content during information gathering and analysis, thus enabling comprehensive management of online information and obtaining holistic analytical results. It also solves the problem of highly homogeneous and one-sided online information content obtained by web crawlers.

[0056] This invention's topic crawling method based on the Bandit algorithm is mainly applied to web clothing information crawling, but it is not limited to this type. This invention's topic crawling method based on the Bandit algorithm is also applicable to information crawling in fields such as entertainment, sports, and games. This invention's topic crawling method based on the Bandit algorithm is a technological innovation based on the publicly available algorithms in the prior art, thereby promoting its application in the field of web information crawling. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating the topic crawling method based on the Bandit algorithm of this invention.

[0058] Figure 2 This is the yield p and Beta curve for each keyword in the topic crawling method based on the Bandit algorithm of this invention. Detailed Implementation

[0059] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0060] like Figures 1-2 As shown, this invention provides a topic crawler method based on the Bandit algorithm, and the overall planning scheme is as follows:

[0061] Internet information collection can be achieved through the following steps:

[0062] S01: Create an associated thesaurus;

[0063] By crawling online dictionary websites, raw keywords and corresponding synonyms are obtained. Then, words are manually filtered and classified to form a custom related thesaurus, establishing a network of relationships between keywords and related words. Based on this related thesaurus, an API service is encapsulated to provide the function of obtaining keywords related to words.

[0064] S02: Create a topic-based crawler, pre-select keywords and sites related to the topic, and perform word segmentation preprocessing on the keywords to distinguish between user keywords and relevant keywords;

[0065] Using the related keyword library API, a series of relevant keywords are obtained based on the keywords set by the user, and the top 5 relevant keywords are selected.

[0066] S03: Create the Bandit algorithm model. The initial input parameters of the Bandit algorithm model are the user-set preset maximum number of attempts threshold n and preset rate of return threshold p, as well as the dynamic parameter information of the keywords in S02.

[0067] The dynamic parameter information of the keywords includes the interest coefficient α and the non-interest coefficient β. The initial values ​​of the interest coefficient α and the non-interest coefficient β for each keyword are preset to 1.

[0068] The dynamic parameter information for each keyword is used to calculate the return rate of each keyword according to the Beta distribution expectation formula α / (α+β), that is:

[0069] p = α / (α+β);

[0070] S04: The web crawler retrieves a new webpage;

[0071] Use a Bloom filter to quickly identify whether it is a new webpage;

[0072] If it is a new webpage, then execute S05;

[0073] If it is not a new webpage, discard it directly;

[0074] S05: Format the key information of the new webpage, extract the current information, and organize it into a data packet;

[0075] The key information on the new webpage includes the following:

[0076] S0501, the URL information obtained;

[0077] S0502, the text content of the hyperlink;

[0078] S0503, the content of the nearby text;

[0079] S06: Match the keyword attribute data of the data packet with user keywords and relevant keywords;

[0080] If the keyword attribute data of the data packet matches the user's keyword, then proceed directly to step S13 to download the webpage;

[0081] If the keyword attribute data of the data packet does not match the user keyword, proceed to step S07;

[0082] S07: Continue matching with relevant keywords based on the keyword attribute data of the data packet;

[0083] If the keyword attribute data of a data packet does not match any relevant keywords, the data packet is discarded.

[0084] If the keyword attribute data of the data packet matches a relevant keyword, then execute S08;

[0085] S08: Manually label the data and provide feedback to the user, asking whether the data package should be further crawled.

[0086] S09: The user filters and judges the data packet, performs a "mark" operation on the page, and then provides feedback;

[0087] S10: After receiving user feedback, the Beta parameter value in the relevant keyword changes accordingly; if the user accepts, that is, the user agrees to further crawl the content of the data packet, then α changes to α+1; otherwise, β changes to β+1, and jumps to step S11.

[0088] S11: The crawler download queue crawls content from known URLs;

[0089] S12: Based on the input keywords and the corresponding Beta parameters, calculate the yield p for each keyword using the Bandit algorithm model;

[0090] S13: Sort keywords by their level of interest based on their return on investment (p).

[0091] The keyword identifier is determined by comparing the number of times the formatted data packet for the keyword was not crawled by the user with the preset upper limit threshold, and the yield p of the keyword with the preset yield threshold.

[0092] If the value of a keyword is greater than or equal to 1000, the value of α / (α+β) is greater than or equal to 0.8, and the rate of return p≈1, that is, 0.8≤rate of return p≤1, then the keyword is marked as a "keyword of interest";

[0093] If the α+β value of a keyword is greater than or equal to 1000, while the α / (α+β) value (i.e. the yield p value) is less than or equal to 0.1, and the number of times the keyword is not used by users is greater than or equal to a preset number of times, and the yield p of the keyword is less than a preset yield threshold, then the keyword is marked as a "keyword that is definitely not of interest".

[0094] If the α+β value of a keyword is less than or equal to 10, then the keyword is marked as a "keyword that may be of interest".

[0095] Bandit algorithm is not a single algorithm, but a class of algorithms. The most representative implementation is the Thompson sampling algorithm, whose core is the Beta distribution. Bandit algorithm addresses the question of whether to continue with the existing method to obtain stable returns, or to explore a new method to try for higher returns, finding a balance between accuracy and diversity. In the process of making decisions about accuracy and diversity, the algorithm continuously updates its understanding of the decision and optimizes the corresponding target model. Its core idea is: try multiple times; if an option is determined to be good, use it more often; if an option is determined to be bad, use it less often. If the good or bad of an option is not yet determined, give it more opportunities to be selected until it is determined whether it is good or bad. Simply put, it categorizes options into "things that are definitely interesting" and "things that might be interesting," and eliminates "things that are not interesting."

[0096] This invention presents a topic-based web crawling method based on the Bandit algorithm. Leveraging a related thesaurus API, it expands keyword relevance, identifies relevant keywords, and trains and optimizes the target model using the Bandit algorithm. It calculates the benefit (i.e., interest level) of each keyword in real time, distinguishes the keyword range corresponding to each topic crawler, and performs online topic-based content scraping. It can expand keyword similarity to obtain more valuable information and effectively find relevant pages; it can reflect the user's true interests, adjust and correct crawler behavior in a timely manner, and avoid wasting time and space resources.

[0097] When applied to internet information gathering, this method can uncover more relevant online information while identifying key areas of interest, avoiding the cocoon effect and silence effect caused by subjective keyword and site selection. It also prevents content acquisition and analysis from becoming bogged down in a large amount of homogeneous content, failing to comprehensively represent public opinion, and resulting in biased analysis. This solves the problem of highly homogeneous and biased information obtained by web crawlers.

[0098] This invention's topic crawling method based on the Bandit algorithm is mainly applied to crawling clothing network information, but it is not limited to this type. This invention's topic crawling method based on the Bandit algorithm is also applicable to crawling information in fields such as entertainment, sports, and games. This invention's topic crawling method based on the Bandit algorithm is a technological innovation based on the algorithms already disclosed in the prior art, thereby promoting its application in the field of network information crawling.

[0099] Specific implementation methods:

[0100] Bandit algorithm is not a single algorithm, but a class of algorithms. The most representative implementation is the Thompson sampling algorithm, whose core is the Beta distribution. Bandit algorithm addresses the question of whether to continue with the existing method to obtain stable returns, or to explore a new method to try for higher returns, finding a balance between accuracy and diversity. In the process of making decisions about accuracy and diversity, the algorithm continuously updates its understanding of the decision and optimizes the corresponding target model. Its core idea is: through multiple tests, if an option is determined to be good, use it more often; if an option is determined to be bad, use it less often. If the good or bad of an option is not yet determined, give it more opportunities to be selected until it is determined whether it is good or bad. Simply put, it categorizes options into "things that are definitely interesting" and "things that might be interesting," and eliminates "things that are not interesting."

[0101] S01: Use conventional web crawlers to crawl online dictionary websites—such as HaiCi (http: / / dict.cn / ) and Kingsoft PowerWord (http: / / www.iciba.com / )—to obtain raw keywords and corresponding synonyms. Then, manually filter and classify the words to form a custom related thesaurus, establishing a network of relationships between keywords and related words. Based on this thesaurus, encapsulate an API service to provide the function of obtaining keywords and related words.

[0102] S02: Create a topic-based crawler, specify relevant topic keywords and a subset of websites, and perform word segmentation preprocessing on the keywords. Use a related keyword database API to obtain a series of relevant keywords based on the user-defined keywords (e.g., inputting "apple" will yield keywords such as "Dell," "laptop," "computer," and "tablet"). Only the first 5 keywords need to be retrieved. It is necessary to distinguish between user-defined keywords and relevant keywords.

[0103] S03: When crawling a new webpage, the crawler uses a Bloom filter to quickly identify whether it is a new webpage.

[0104] A Bloom filter is essentially a long binary vector and a series of random mapping functions, saving space for storing URLs and significantly reducing the probability of hash collisions. It can be used to check if an element is in a set. Its advantages are that its space efficiency and query time are much better than general algorithms; its disadvantages are a certain false positive rate and difficulty in deletion. Theoretically, 256MB of space can store the feature information of 2 billion URLs.

[0105] The specific principle behind using Bloom filters to quickly identify whether a webpage is new is as follows:

[0106] Using a bit array of length m and initializing all elements to 0, perform k hash operations on the elements using k hash functions, take the remainder of len(m) to obtain k positions, and set the corresponding positions in m to 1. Generally, k are three different hash functions.

[0107] The insertion process, for example, maps "baidu" to a Bloom filter, generating three hash values ​​(1, 3, and 6) using three different hash functions. Then, the corresponding bit values ​​are set to 1.

[0108] The query process involves using "baidu" to check if a value exists in the Bloom filter. Three different hash functions are used to generate three hash values: 1, 3, and 6. The Bloom filter is then checked to see if the corresponding bit values ​​for these three values ​​are all 1. We find that 1, 3, and 6 all return 1, proving that the value exists. If any bit returns 0, it proves that the value does not exist.

[0109] In practice, a query is performed first, and if the result is not found, the insertion is then performed.

[0110] S04: Format the data, including obtaining URL information, hyperlink text content, and nearby text content. Perform text segmentation, keyword extraction, and generate a formatted data package.

[0111] S05: The data packet has matched the user's keywords. Jump directly to S10 to download the webpage.

[0112] S06: If no relevant keyword is matched, the data packet is discarded directly. Alternatively, if a relevant keyword is matched but the Bandit algorithm model identifies the keyword as "a keyword that is definitely not of interest", the data packet is also discarded directly.

[0113] S07: Relevant keywords have been hit. Manual annotation and feedback are required to determine whether further content should be crawled from this data package.

[0114] S08: If after the user's feedback information, the Beta parameter value in the relevant keyword will change, receive α + 1, otherwise β + 1, and execute S10.

[0115] The Beta parameter is a name for the Beta beta distribution function; the probability distribution logic behind the keyword is the Beta distribution, which is calculated through the derivation formula of the Beta beta distribution expectation:

[0116]

[0117] S09: The crawler download queue crawls the content of the already determined URL addresses.

[0118] S10: Calculate the return rate p of each keyword according to the input keyword and the corresponding Beta parameter.

[0119] The process of calculating the return rate is as follows: The probability distribution of the return rate p conforms to the Beta(α, β) distribution. The Beta distribution is a continuous probability distribution curve defined on the interval [0, 1]. The function formula of the Beta distribution is as follows:

[0120]

[0121] It has two parameters: α, β. It is especially suitable for the probability of something happening or succeeding. Its characteristics include: when the value of α + β is larger, the distribution curve is narrower; when the value of α / (α + β) is larger, the center of the distribution is closer to 1, otherwise the center of the distribution is closer to 0. We can regard it as a distribution of probability density. When we don't know what the specific probability of something is, it gives the possibility of all probabilities occurring, which can be understood as a probability density distribution. The return rate p of each keyword can be calculated through the Beta distribution expectation formula α / (α + β), that is, p = α / (α + β).

[0122] The return rate p of each keyword is actually the degree of interest of the corresponding user in this keyword. The Bandit algorithm model updates the return rate p, thereby sorting the degree of interest in the keywords. According to the characteristics of this algorithm model, when the number of our selections and feedback is more, it is easier to obtain the global optimal solution, and the sorting of the keywords can be stable. For keywords that have been feedback by users not to be crawled more than n times (n is a positive integer, and this value is a preset number threshold summarized according to past operation experience and can be adjusted), and calculate that the return rate p of this keyword is less than m (0 < m < 1, and this value is a preset return rate threshold summarized according to past operation experience and can be adjusted), then this keyword will be marked as "keyword determined not to be interested" in the algorithm model.

[0123] The derivation formula of the Beta beta distribution expectation of the algorithm model is:

[0124]

[0125] If a keyword not only has a large α+β, but also a large α / (α+β) value (i.e., the value of the return p), and the calculated return p is close to 1, that is, 0.8≤return p≤1, then it is determined that this is a keyword of interest, with a good average return and a great advantage in each selection, and can be marked as a "keyword of interest".

[0126] If a keyword has a large α+β and a small α / (α+β) (i.e., the value of the return p), and users report that they do not use the keyword more than 10 times (this value is a threshold and can be adjusted), and the calculated return p of the keyword is less than 0.3 (this value is a threshold and can be adjusted), then the keyword is marked as a "keyword that is determined to be of no interest" in the algorithm model.

[0127] If a keyword has a small α+β, meaning it hasn't been selected or evaluated many times, it indicates that the keyword's effectiveness is uncertain. Mark it as a "keyword you might be interested in".

[0128] The key points of this application are: based on the related thesaurus API, it expands the relevance of keywords, identifies relevant keywords, trains and optimizes the target model based on the Bandit algorithm, calculates the benefit (i.e., interest level) of each keyword in a timely manner, distinguishes the keyword range corresponding to each topic crawler, and performs content crawling for online topic crawlers. It also allows for timely adjustments to crawler behavior to avoid wasting time and space resources.

[0129] This application addresses the problem of highly homogenized and one-sided information obtained by existing web crawlers. It proposes a topic-based web crawling method for internet information collection based on the Bandit algorithm. This method can uncover more relevant online information while still focusing on key monitored content, avoiding the cocoon effect and silence effect caused by subjective keyword and site selection. It also prevents content acquisition and analysis from becoming bogged down in a large amount of homogenized content, failing to comprehensively represent public opinion, and obtaining one-sided analytical results, ultimately affecting the effectiveness of online information collection.

[0130] This invention is not limited to the above-described optional embodiments. Anyone can derive other various forms of products under the guidance of this invention. However, regardless of any changes made in their shape or structure, any technical solution that falls within the scope of the claims of this invention shall be protected by this invention.

Claims

1. A topic crawler method based on the Bandit algorithm, characterized in that: It includes the following steps: S01: Create an associated thesaurus; S02: Create a topic-based crawler, pre-select keywords and sites related to the topic, and perform word segmentation preprocessing on the keywords to distinguish between user keywords and relevant keywords; S03: Create the Bandit algorithm model. The initial input parameters of the Bandit algorithm model are the user-set preset maximum number of attempts threshold n and preset rate of return threshold p, as well as the dynamic parameter information of the keywords in S02. The dynamic parameter information of the keywords includes the interest coefficient α and the non-interest coefficient β. The initial values ​​of the interest coefficient α and the non-interest coefficient β for each keyword are preset to 1. S04: The web crawler retrieves a new webpage; Use a Bloom filter to quickly identify whether it is a new webpage; If it is a new webpage, then execute S05; If it is not a new webpage, discard it directly; S05: Format the key information of the new webpage, extract the current information, and organize it into a data packet; S06: Match the keyword attribute data of the data packet with user keywords and relevant keywords; If the keyword attribute data of the data packet matches the user's keyword, then proceed directly to step S13 to download the webpage; If the keyword attribute data of the data packet does not match the user keyword, proceed to step S07; S07: Continue matching with relevant keywords based on the keyword attribute data of the data packet; If the keyword attribute data of a data packet does not match any relevant keywords, the data packet is discarded. If the keyword attribute data of the data packet matches a relevant keyword, then execute S08; S08: Manually label the data and provide feedback to the user, asking whether the data package should be further crawled. S09: The user filters and judges the data packet, performs a "marking" operation on the page, and then provides feedback; S10: After receiving user feedback, the Beta parameter value in the relevant keyword changes accordingly; if the user accepts, that is, the user agrees to further crawl the content of the data packet, then α changes to α+1; otherwise, β changes to β+1, and jumps to step S11. S11: The crawler download queue crawls content from known URLs; S12: Based on the input keywords and the corresponding Beta parameters, calculate the yield p for each keyword using the Bandit algorithm model; S13: Sort keywords by their level of interest based on the return on each keyword, p; In step S03, the dynamic parameter information of each keyword is used to calculate the rate of return of each keyword according to the expected formula of Beta distribution α / (α+β), that is: p=α / (α+β); Step S08 also includes the following operations: If the keyword attribute data of a data packet matches a relevant keyword, but the Bandit algorithm model identifies that the keyword has been marked as "keywords that are definitely not of interest" multiple times in the history, then the data packet will be discarded directly. In step S13, the keyword identifier is determined by comparing the number of times the formatted data packet of the keyword is not crawled by the user with the preset upper limit threshold, and the return rate p of the keyword with the preset return rate threshold. Step S13 also includes the following operations: If a keyword has an α+β value greater than or equal to 1000, a return p value less than or equal to 0.1, and the number of times the keyword is not used by users is greater than or equal to a preset number of times, and the return p of the keyword is less than a preset return threshold, then the keyword is marked as a "keyword that is definitely not of interest".

2. The topic crawler method based on the Bandit algorithm according to claim 1, characterized in that: Step S01 includes the following operations: By crawling online dictionary websites, we obtain raw keywords and corresponding synonyms. Then, we manually filter and classify the words to form a custom related thesaurus and establish a network of relationships between keywords and related words. Furthermore, based on this related thesaurus, an API service is encapsulated to provide the function of obtaining keyword-related words.

3. The topic crawler method based on the Bandit algorithm according to claim 2, characterized in that: Step S02 includes the following operations: Using the related keyword library API, a series of relevant keywords are obtained based on the keywords set by the user, and the top 5 relevant keywords are selected.

4. The topic crawler method based on the Bandit algorithm according to claim 1, characterized in that: In step S05, the key information of the new webpage includes the following: S0501, the URL information obtained; S0502, the text content of the hyperlink; S0503, Content of nearby text; The text content of hyperlinks and the text content near them will be segmented into words to extract keywords.

5. The topic crawler method based on the Bandit algorithm according to claim 1, characterized in that: Step S13 also includes the following operations: If the α+β value of a keyword is greater than or equal to 1000, and 0.8 ≤ return p ≤ 1, then the keyword is marked as a "keyword of interest".

6. The topic crawler method based on the Bandit algorithm according to claim 1, characterized in that: Step S13 also includes the following operations: If the α+β value of a keyword is less than or equal to 10, then the keyword is marked as a "keyword that may be of interest".

Images