A topic crawler method, system, and storage medium based on the PTB model
By combining the PTB model with improved PageRank and Bayesian classification algorithms, the problem of insufficient topic relevance in topic crawling is solved, achieving more efficient and accurate web page crawling and avoiding topic drift and lack of authority.
Patent Information
- Application Number
- CN202410330465.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-22
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-03-22
AI Technical Summary
Existing topic-based web crawling methods suffer from insufficient accuracy in determining topic relevance, resulting in low accuracy in the topics crawled and issues such as topic drift and lack of authority.
We employ a PTB-based approach, combined with an improved PageRank algorithm and a Bayesian classification algorithm, to calculate topic relevance from both page link and content perspectives. We also utilize the Scrapy framework to implement asynchronous multi-threaded crawling, thereby improving the accuracy of topic relevance determination.
It improves the accuracy of topic-based crawling methods and systems in determining topic degree, avoids topic drift and lack of authority, and achieves more efficient and accurate web page crawling.
Smart Images

Figure CN118170962B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a topic crawler method, system, and storage medium based on the PTB model, belonging to the field of computer network information acquisition technology. Background Technology
[0002] With the rapid development of technology and the generation of massive amounts of data, the internet has become a vast database and a crucial means for people to access information. This vast amount of online data contains a wealth of valuable information that can be mined and utilized by users. The most important question is how to effectively extract this valuable information from this enormous database. Currently, web crawling technology has become a vital technique for acquiring web data, automatically scraping information from the World Wide Web. However, the broad scope of the crawled content results in a large amount of advertising, redundant, and irrelevant information. Thematic crawling, on the other hand, only crawls web pages related to a specific topic, meeting the requirements for specialized data acquisition. Therefore, acquiring data in a specific field requires the use of thematic crawling technology.
[0003] Topic crawling has become a research hotspot in web mining. Key aspects of topic crawling technology research include determining topic relevance and developing search strategies, primarily divided into content-based and link-based research. Link-based topic crawling strategies assume that web links are interconnected, possess an organized and directional structure, and represent the authority and importance of web pages. Content-based topic crawling strategies, on the other hand, focus more on whether the specific page content is relevant to the topic, thus avoiding "topic drift" to some extent.
[0004] Currently, many topic-based web crawling methods use a single strategy or overly simplistic topic relevance algorithms, resulting in insufficient accuracy in determining the topic relevance of crawled web pages. Therefore, improving the accuracy of topic relevance determination is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] The purpose of this invention is to provide a topic crawling method, system, and storage medium based on the PTB model, which can improve the accuracy of topic degree discrimination in the topic crawling method and system.
[0006] To achieve the above objectives, the present invention provides the following technical solution:
[0007] In a first aspect, the present invention provides a topic crawler method based on the PTB model, comprising:
[0008] Based on the user-defined topic keywords and related seed URL set, a request is sent to the web server and a crawling response is obtained. The seed URL set is crawled sequentially, and all URL links within the seed URL set are obtained and placed into the URL queue in sequence.
[0009] The URLs in the URL queue are filtered, and the filtered URLs are parsed sequentially to obtain the corresponding page links and page content.
[0010] Based on the improved PageRank algorithm, the link topic score is calculated for the page links to obtain the link topic score;
[0011] Based on the improved Bayesian classification algorithm, the content topic degree of the page content is calculated to obtain the content topic degree, and irrelevant web pages in the content topic degree calculation are discarded.
[0012] Based on the PTB model, the topic relevance is calculated according to the connection topic degree and content topic degree. Web pages with topic relevance less than a preset threshold are discarded, and web pages with topic relevance greater than or equal to the preset threshold are stored in the database.
[0013] In conjunction with the first aspect, further, after sequentially crawling the seed URL set, it also includes: obtaining a page with an HTML document structure or an XML document structure from the HTTP response and passing it to the parser component, which uses an HTML parser or an XML parser to convert the HTML document structure or XML document structure into the data structure inside the webpage.
[0014] In conjunction with the first aspect, furthermore, when sequentially crawling the seed URL set, the Scrapy web crawling framework is used for asynchronous requests and multi-threaded crawling, including:
[0015] Create a Scrapy project using the Scrapy command line, and determine the data to scrape and the page from which to start scraping.
[0016] Create a Spider, and define the starting URL and tracking links in the Spider to obtain the necessary data;
[0017] Define the data structure Item that needs to be retrieved, and ensure that each data field is correctly mapped to Item;
[0018] Define a Pipeline, a process controller for processing and storing the extracted data. Use the Pipeline to send the extracted data to an external system or database and use the Twiested framework for asynchronous operations.
[0019] Enable asynchronous requests by adding the asyncio framework to the Spider to support asynchronous network request operations;
[0020] Configure multithreading to perform asynchronous processing using different modules and settings in Scrapy;
[0021] To start the crawler, run Scrapy-Spider to launch the entire crawler framework.
[0022] In conjunction with the first aspect, further, based on the improved PageRank algorithm, the link topicality of the page links is calculated, and the link topicality includes:
[0023] Based on the improved PageRank algorithm, the PR value of the parent node's webpage is obtained;
[0024] Based on the PR value of the parent node webpage, according to the reference relationship structure between webpages, obtain all nodes that point to the parent node webpage, and calculate the PR value of all nodes that point to the parent node webpage.
[0025] Based on the PageRank (PR) values of all nodes pointing to the parent page, the number of outbound links from all nodes pointing to the parent page is calculated, which is the number of external links contained in the parent page, and thus the link topicality is obtained.
[0026] Combining the first aspect, the further improved PageRank algorithm's calculation formula is as follows:
[0027]
[0028] Among them, PR(g) τ+1 ) represents the parent node g in the (τ+1)th iteration. τ+1 PR value, PR(g) τ ) represents the parent node g in the τth iteration. τ PR value, PR(T) τ ) represents node T at the τth iteration. τ PR value, L(T) τ ) represents node T at the τth iteration. τ The number of outgoing chains, where g τ+1 It is g τ The parent node, T τ It is except g τ Outside of pointing to g τ+1 For the other nodes, k represents the first adjustment factor, and T represents the number of iteration rounds.
[0029] In conjunction with the first aspect, further improvements to the Bayesian classification algorithm include:
[0030] Linear normalization is used to normalize the various indicators and their corresponding solutions in the decision problem of Bayesian classification algorithm, and the indicator weights and information redundancy of each category in the training set are calculated.
[0031] Based on the index weights and information redundancy of each category in the training set, the entropy method is used to perform a weighted operation on the Bayesian classification algorithm to obtain an improved Bayesian classification algorithm.
[0032] Combining the first aspect, the further improved Bayesian classification algorithm's calculation formula is as follows:
[0033]
[0034] Where, x i Let C represent the i-th attribute value object in the training set, where n represents the total number of attribute value objects in the training set. j Let w represent the j-th category in the training set. There are two categories in the training set: category C1 represents topic-related categories, and category C2 represents topic-irrelevant categories. j This represents the weight of the index for the j-th category in the training set. Where, d j P(C) represents the information redundancy of the j-th category in the training set. j Let P(x) be the prior probability. i |C j ) represents the conditional probability, indicating the category C. j The attribute value object x i The probability of occurrence, P ij The weight of each indicator.
[0035] In conjunction with the first aspect, the PTB model is further defined as follows:
[0036]
[0037] Where e represents the second adjustment factor, x i Let C represent the i-th attribute value object in the training set, where n represents the total number of attribute value objects in the training set. j Let w represent the j-th category in the training set. There are two categories in the training set: category C1 represents topic-related categories, and category C2 represents topic-irrelevant categories. j This represents the weight of the index for the j-th category in the training set. Where, d j P(C) represents the information redundancy of the j-th category in the training set. j Let P(x) be the prior probability. i |C j ) represents the conditional probability, indicating the class C. j The attribute value object x iThe probability of occurrence, P ij The weight of each indicator, PR(g) τ+1 ) represents the parent node g in the (τ+1)th iteration. τ+1 PR value.
[0038] Secondly, the present invention provides a topic crawler system based on the PTB model, including a processor and a storage medium;
[0039] The storage medium is used to store instructions;
[0040] The processor is configured to operate according to the instructions to perform the steps of the method according to any one of the first aspects.
[0041] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the first aspects.
[0042] Compared with the prior art, the beneficial effects of the present invention are:
[0043] The topic crawling method based on the PTB model provided by this invention starts from two perspectives: page links and page content. It combines the topicality of links and the topicality of content to make a comprehensive topicality judgment. This ensures that the importance of the crawled web pages is guaranteed, while also preserving the topic relevance of the web pages. It avoids the phenomenon of "topic drift" and the situation where the web pages lack authority, thereby improving the accuracy of topicality judgment in the topic crawling method and system.
[0044] This invention employs an improved PageRank algorithm, making the calculation of link topicality more accurate; it also uses an improved Bayesian classification algorithm to extract key sentences from the processed text content, and then extracts keywords based on the key sentences, which can greatly improve the text's comprehensiveness.
[0045] This invention also employs the Scrapy framework to implement a multi-threaded crawler for asynchronous requests, improving the efficiency of topic crawling methods and systems. Compared to other topic crawling methods and systems, the topic crawling method and system based on the PTB model provided by this invention are more efficient and accurate. Attached Figure Description
[0046] Figure 1 This is a flowchart of a topic crawler method based on the PTB model provided in an embodiment of the present invention. Detailed Implementation
[0047] The technical solution of this application will be further described in detail below with reference to specific embodiments.
[0048] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. Unless otherwise specified, the embodiments of this application and the technical features within them can be combined with each other.
[0049] Example 1:
[0050] Figure 1 This is a flowchart illustrating a topic crawler method based on the PTB model provided in this embodiment. This flowchart only shows the logical order of the method in this embodiment; however, it can be implemented in different ways without conflict. Figure 1 Complete the steps shown or described in the order indicated.
[0051] See Figure 1 The topic crawling method based on the PTB model in this embodiment specifically includes the following steps:
[0052] Step 1: Based on the user's determination of the topic keywords and related seed URLs to be collected;
[0053] In this embodiment, taking the topic of "intelligent vehicle" as an example, the topic keyword "intelligent vehicle" is selected, and the relevant information portal websites with a certain degree of authority are found by searching for keywords from commonly used search engines as the seed URL set.
[0054] Step 2: Send a request to the web server and obtain the crawling response;
[0055] In this embodiment, the crawler program sends an HTTP request to the web server by simulating a browser. After obtaining a permission response, the subsequent crawling work can be carried out.
[0056] Step 3: Crawl the seed URL set sequentially, obtain all URL links within the seed URL set, and put them into the URL queue in sequence;
[0057] In this embodiment, after sequentially crawling the seed URL set, the process further includes: obtaining a page with an HTML or XML document structure from the HTTP response and passing it to the parser component. The parser component then uses an HTML parser or an XML parser to convert the HTML or XML document structure into the webpage's internal data structure. Afterward, users can use webpage information extraction tools such as XPath expressions to locate and extract useful information.
[0058] In this embodiment, when sequentially crawling the seed URL set, the Scrapy web crawler framework is used to perform asynchronous requests and multi-threaded crawling, specifically including the following steps:
[0059] S1: Create Scrapy. Use the Scrapy command line to create a new project and determine the data to be scraped and the page from which to start scraping.
[0060] S2: Create a Spider, define the starting URL and tracking links in the Spider, and obtain the necessary data;
[0061] S3: Define the data structure Item that needs to be retrieved, and ensure that each data field is correctly mapped to Item;
[0062] S4: Define a Pipeline, a process controller for processing and storing extracted data. Use the Pipeline to send the extracted data to external systems or databases to ensure the accuracy and integrity of the data, and use the Twiested framework for asynchronous operations.
[0063] S5: Enable asynchronous requests. Add the asyncio framework to the Spider to support asynchronous network request operations. This can improve efficiency and greatly reduce the load on sites with high access frequency.
[0064] S6: Configure multithreading to use different modules and settings in Scrapy for asynchronous processing via multithreading;
[0065] S7: Start the crawler, run Scrapy-Spider, and start the entire crawler framework.
[0066] In addition, there is the distribution of URL queues. All web page links inside the seed URL are obtained through page parsing and information extraction technologies. These links are first placed in the URL queue to be crawled. After steps four to seven, they are then placed in the already crawled URL queue to avoid duplicate crawling and waste of resources.
[0067] Step 4: Filter the URLs in the URL queue, and parse the filtered URLs one by one to obtain the corresponding page links and page content;
[0068] In this embodiment, URL links in the URL queue are filtered to prevent duplicate access and parsing. The Duplicate Filter middleware is used in the Scrapy framework to control the deduplication behavior, specifically including the following steps:
[0069] Step a: Use the Duplicate Filter component in Scrapy. This component is treated as middleware. Therefore, the deduplication middleware needs to be added to Scrapy's middleware list;
[0070] Step b: The Duplicate Filter middleware treats the URL as the only unique object for deduplication by default. That is, if Scrapy uses this middleware for deduplication, two URLs will only be considered the same URL if they are completely identical.
[0071] Step c: Control the specific deduplication method by customizing certain parameters of the Duplicate Filter middleware. For example, use the scrapy.utils.request.request_fingerprint function to perform a hash operation on the request object, thereby including other request characteristics besides the URL in the deduplication consideration, further enhancing the deduplication effect;
[0072] Step d: In some special cases, it may be necessary to implement deduplication logic manually. In such cases, you can implement custom deduplication by overriding the `equals` method of the `Request` class in Scrapy or the `URL.Canonicalizer.canonicalize` method. It's important to note that this technique is relatively advanced and is typically only used when specific deduplication logic is required.
[0073] In this embodiment, when parsing the filtered URLs sequentially, the page crawling requests and page parsing operations in steps two and three are repeated.
[0074] Step 5: Based on the improved PageRank algorithm, calculate the link topicality of page links to obtain the link topicality.
[0075] In this embodiment, based on the improved PageRank algorithm, the link topicality of page links is calculated, and the specific steps to obtain the link topicality include the following:
[0076] Step 1: Obtain the PR value of the parent node's webpage based on the improved PageRank algorithm;
[0077] In this embodiment, the calculation formula for the improved PageRank algorithm is as follows:
[0078]
[0079] Among them, PR(g) τ+1 ) represents the parent node g in the (τ+1)th iteration.τ+1 PR value, PR(g) τ ) represents the parent node g in the τth iteration. τ PR value, PR(T) τ ) represents node T at the τth iteration. τ PR value, L(T) τ ) represents node T at the τth iteration. τ The number of outgoing chains, where g τ+1 It is g τ The parent node, T τ It is except g τ Outside of pointing to g τ+1 For the other nodes, k represents the first adjustment factor, and T represents the number of iteration rounds.
[0080] In this embodiment, after multiple iterative simulations, the first adjustment factor k is set to 0.6.
[0081] Step 2: Based on the PR value of the parent node webpage, according to the reference relationship structure between webpages, obtain all nodes pointing to the parent node webpage, and calculate the PR value of all nodes pointing to the parent node webpage;
[0082] Step 3: Based on the PR values of all nodes pointing to the parent node webpage, calculate the number of outbound links from all nodes pointing to the parent node webpage, that is, the number of external links contained in the parent node webpage, and then obtain the link topic degree.
[0083] Step 6: Based on the improved Bayesian classification algorithm, calculate the content topicality of the page content, obtain the content topicality, and discard irrelevant web pages in the content topicality calculation;
[0084] In this embodiment, the improvement of the Bayesian classification algorithm specifically includes the following steps:
[0085] Step ①: Use linear normalization to normalize the various indicators and their corresponding solutions in the decision problem of the Bayesian classification algorithm, and calculate the indicator weights and information redundancy of each category in the training set;
[0086] Step 2: Based on the index weights and information redundancy of each category in the training set, the Bayesian classification algorithm is weighted using the entropy method to obtain an improved Bayesian classification algorithm.
[0087] In this embodiment, the calculation formula of the improved Bayesian classification algorithm is as follows:
[0088]
[0089] Where, x i Let C represent the i-th attribute value object in the training set, where n represents the total number of attribute value objects in the training set. jLet w represent the j-th category in the training set. There are two categories in the training set: category C1 represents topic-related categories, and category C2 represents topic-irrelevant categories. j This represents the weight of the index for the j-th category in the training set. Where, d j P(C) represents the information redundancy of the j-th category in the training set. j Let P(x) be the prior probability. i |C j ) represents the conditional probability, indicating the class C. j The attribute value object x i The probability of occurrence, P ij The weight of each indicator.
[0090] In addition, before calculating the content topicality, the page content needs to be segmented and keywords extracted. This is necessary for similarity matching with the topic keywords provided by the user in step one. This invention uses the jieba segmenter for segmentation and the Text Rank algorithm for keyword extraction, specifically including the following steps:
[0091] Step A: First, the extracted page content is segmented using the jieba word segmenter, and each sentence is divided into many words;
[0092] Step B: Based on the co-occurrence relationships between words, construct a graph model, where each node represents a word and the edges represent the co-occurrence relationships between words;
[0093] Step C: Convert each word into a corresponding word vector using the Word2Vec word vector model, calculate the similarity between each word using cosine similarity, the higher the similarity value, the greater the weight of the corresponding edge, and use the similarity value as the weight to weight the graph;
[0094] Step D: Use the PageRank algorithm to calculate the score of each node (i.e., each word), sort the nodes according to their calculated scores, and output the top few words with the highest scores as the keywords of the text.
[0095] In this embodiment, the calculation formula for the Text Rank algorithm is as follows:
[0096]
[0097] Among them, WS(V α ) represents sentence V at node α. α Weights; W βα Sentence V at node α α Sentence V at node β βThe similarity, ln V α W represents other nodes pointing to node α. βγ Sentence V at node β β The word V in γ h represents the damping coefficient value, h = 0.8, and the right side of the formula is the contribution of the adjacent sentences to the current sentence, finally obtaining the training samples to be predicted.
[0098] Step 7: Based on the PTB model, calculate the topic relevance according to the connection topic degree and content topic degree, discard web pages with topic relevance less than a preset threshold, and store web pages with topic relevance greater than or equal to the preset threshold in the database.
[0099] In this embodiment, the PTB model is:
[0100]
[0101] Where e represents the second adjustment factor, x i Let C represent the i-th attribute value object in the training set, where n represents the total number of attribute value objects in the training set. j Let w represent the j-th category in the training set. There are two categories in the training set: category C1 represents topic-related categories, and category C2 represents topic-irrelevant categories. j This represents the weight of the index for the j-th category in the training set. Where, d j P(C) represents the information redundancy of the j-th category in the training set. j Let P(x) be the prior probability. i |C j ) represents the conditional probability, indicating the class C. j The attribute value object x i The probability of occurrence, P ij The weight of each indicator, PR(g) τ+1 ) represents the parent node g in the (τ+1)th iteration. τ+1 PR value.
[0102] Given Overall numerical ratio PR(g) τ+1 The value is much smaller. In order to make the PTB model more accurate in judging the topic relevance of web pages, a second adjustment factor e needs to be added. The second adjustment factor e = 0.85 is set.
[0103] In this embodiment, the user can adjust the threshold range of topic relevance to meet the accuracy requirements of the topic-related web pages to be collected.
[0104] The topic crawling method based on the PTB model provided in this embodiment starts from two perspectives: page links and page content. It combines link topicality and content topicality for comprehensive topicity judgment, ensuring both the importance of crawled pages and the topic relevance of the pages, avoiding the phenomenon of "topic drift" and the lack of authority of the pages, and improving the accuracy of topicity judgment in the topic crawling method and system. An improved PageRank algorithm is used, making the calculation of link topicality more accurate; an improved Bayesian classification algorithm is used to extract key sentences from the processed text content, and then keywords are extracted based on the key sentences, which can greatly improve the text's comprehensiveness. Furthermore, the Scrapy framework is used to implement asynchronous request multi-threaded crawling, improving the efficiency of the topic crawling method and system. Compared with other topic crawling methods and systems, the topic crawling method and system based on the PTB model provided in this embodiment are more efficient and accurate.
[0105] Example 2:
[0106] This embodiment provides a topic crawler system based on the PTB model, including a processor and a storage medium;
[0107] Storage media are used to store instructions;
[0108] The processor is used to perform operations according to instructions to execute the steps of the method in Embodiment 1.
[0109] Example 3:
[0110] This embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in Embodiment 1.
[0111] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0112] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0113] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0114] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0115] The above are merely preferred embodiments of this application. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A topic crawler method based on the PTB model, characterized in that, include: Based on the user-defined topic keywords and related seed URL set, a request is sent to the web server and a crawling response is obtained. The seed URL set is crawled sequentially, and all URL links within the seed URL set are obtained and placed into the URL queue in sequence. The URLs in the URL queue are filtered, and the filtered URLs are parsed sequentially to obtain the corresponding page links and page content. Based on the improved PageRank algorithm, the link topicality of page links is calculated to obtain the link topicality. Based on the improved Bayesian classification algorithm, the content topicality of the page content is calculated, and irrelevant web pages are discarded in the content topicality calculation. Based on the PTB model, the topic relevance is calculated according to the connection topic degree and content topic degree. Web pages with topic relevance less than a preset threshold are discarded, and web pages with topic relevance greater than or equal to the preset threshold are stored in the database. Improvements to the Bayesian classification algorithm include: Linear normalization is used to normalize the various indicators and their corresponding solutions in the decision problem of Bayesian classification algorithm, and the indicator weights and information redundancy of each category in the training set are calculated. Based on the index weights and information redundancy of each category in the training set, the entropy method is used to perform a weighted operation on the Bayesian classification algorithm to obtain an improved Bayesian classification algorithm. The improved Bayesian classification algorithm is calculated using the following formula: ; in, Indicates the first training set An object of attribute values, This represents the total number of objects with attribute values in the training set. Indicates the first training set There are 2 categories in the training set, the first category is... Indicates topic relevance, second category This indicates that the topic is unrelated. Indicates the first training set The weights of the indicators for each category, ,in, Indicates the first training set Information redundancy of each category For prior probability, Let be the conditional probability, representing the category. The attribute value object The probability of occurrence The weight of each indicator; The PTB model is: ; in, Indicates the second regulating factor. Indicates the first training set An object of attribute values, This represents the total number of objects with attribute values in the training set. Indicates the first training set There are 2 categories in the training set, the first category is... Indicates topic relevance, second category This indicates that the topic is unrelated. Indicates the first training set The weights of the indicators for each category, ,in, Indicates the first training set Information redundancy of each category For prior probability, Let be the conditional probability, representing the category. The attribute value object The probability of occurrence The weight of each indicator, Indicates the first Parent node during round iteration PR value.
2. The topic crawler method based on the PTB model according to claim 1, characterized in that, After crawling the seed URL set in sequence, the process also includes: obtaining a page with an HTML document structure or an XML document structure from the HTTP response and passing it to the parser component, which then uses an HTML parser or an XML parser to convert the HTML document structure or XML document structure into the data structure inside the webpage.
3. The topic crawler method based on the PTB model according to claim 1, characterized in that, When sequentially crawling the seed URL set, the Scrapy web crawling framework is used for asynchronous requests and multi-threaded crawling, including: Create a Scrapy project using the Scrapy command line, and determine the data to scrape and the page from which to start scraping. Create a Spider, and define the starting URL and tracking links in the Spider to obtain the necessary data; Define the data structure Item that needs to be retrieved, and ensure that each data field is correctly mapped to Item; Define a Pipeline, a process controller for processing and storing the extracted data. Use the Pipeline to send the extracted data to an external system or database and use the Twiested framework for asynchronous operations. Enable asynchronous requests by adding the asyncio framework to the Spider to support asynchronous network request operations; Configure multithreading to perform asynchronous processing using different modules and settings in Scrapy; To start the crawler, run Scrapy-Spider to launch the entire crawler framework.
4. The topic crawler method based on the PTB model according to claim 1, characterized in that, Based on the improved PageRank algorithm, the link topicality of the page links is calculated, and the link topicality includes: Based on the improved PageRank algorithm, the PR value of the parent node's webpage is obtained; Based on the PR value of the parent node webpage, according to the reference relationship structure between webpages, obtain all nodes that point to the parent node webpage, and calculate the PR value of all nodes that point to the parent node webpage. Based on the PageRank (PR) values of all nodes pointing to the parent page, the number of outbound links from all nodes pointing to the parent page is calculated, which is the number of external links contained in the parent page, and thus the link topicality is obtained.
5. The topic crawler method based on the PTB model according to claim 4, characterized in that, The improved PageRank algorithm is calculated as follows: ; in, Indicates the first Parent node during round iteration PR value, Indicates the first Parent node during round iteration PR value, Indicates the first Nodes during round iteration PR value, Indicates the first Nodes during round iteration The number of outgoing chains, of which, yes The parent node, Except Pointing outwards Other nodes, Indicates the first regulating factor. Indicates the number of iteration rounds.
6. A topic crawler system based on the PTB model, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 5.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1 to 5.
