A data capture method and system

Abstract

The present invention belongs to the technical field of network data processing, and discloses a data crawling method and system, which includes: analyzing the links between web pages of a website, constructing a transition probability matrix between web pages, wherein each element in the transition probability matrix represents the link quality from one web page to another; calculating the TF-IDF value of the keywords to be queried in the website, and constructing a TF-IDF feature vector based on the TF-IDF value; generating a comprehensive score for each web page according to the transition probability matrix and the TF-IDF feature vector; and determining the priority order of the corresponding web page in the website according to the comprehensive score of each web page; crawling the web page data of the website based on the determined priority order of the web page in the website. The present invention significantly improves the accuracy of web page data crawling and greatly enhances the quality of information obtained by users.

Description

Technical Field

[0001] The present invention relates to the technical field of network data processing, and in particular to a data crawling method and system. Background Art

[0002] A web crawler is an efficient automated tool designed to systematically access and retrieve web pages on the Internet. When performing web page collection tasks, a graph theory model is generally used to view the Internet as a directed graph consisting of links. Using the graph theory model, crawlers can apply breadth-first search (BFS) or depth-first search (DFS) algorithms to traverse web pages and download information. Although these basic search strategies can quickly collect data, they usually do not conduct in-depth value assessment and screening of the visited web pages. This results in a fast data crawling speed, but the accuracy and relevance of search results are often unsatisfactory. Therefore, in order to improve the quality of search results, the key is to optimize the web crawler so that it can identify and prioritize the crawling and processing of high-value web resources.

[0003] The crawler strategy based on value return focuses on the value evaluation of web pages during the crawling process, and gives priority to high-value web page resources. Currently, the following two crawler strategies based on value return are commonly used: Bandwidth First Strategy: This strategy determines the priority of crawling based on the size and loading time of the page, and gives priority to crawling pages with smaller page size and shorter loading time to improve crawling efficiency and speed. Backlink Count Strategy: This strategy evaluates the value of a web page based on the number of backlinks, that is, the number of times a page is referenced by other pages. Generally speaking, pages with more backlinks may be more valuable.

[0004] The advantage of the bandwidth priority strategy is its efficient data collection capability. It can quickly crawl a large number of pages from the Internet, which is especially suitable for crawling tasks that require extensive information collection in a short period of time. However, a potential disadvantage of this approach is that it may not fully consider the correlation between the loading time of a page and the value of its content. Some pages may contain extremely valuable information even though they take a long time to load. If these pages do not receive enough attention, important data may be missed.

[0005] The backlink count strategy evaluates the importance and authority of a page by analyzing the number of backlinks. This method is effective in identifying pages that are widely recognized and frequently cited on the web. These pages usually contain key or authoritative content. However, this method also has its limitations, especially when facing link farms or spam links, which may produce misleading evaluation results due to these improper link practices.

[0006] Therefore, how to provide a data crawling method and system that can improve the accuracy of web page data crawling is a problem that needs to be solved urgently. Summary of the invention

[0007] The embodiments of the present invention provide a data capture method and system to solve the above technical problems in the prior art.

[0008] In order to have a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not a general review, nor is it intended to identify key / important components or to describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0009] According to a first aspect of an embodiment of the present invention, a data crawling method is provided.

[0010] In one embodiment, the data capture method includes:

[0011] Analyze the links between web pages of the website and construct a transition probability matrix between web pages, wherein each element in the transition probability matrix represents the quality of the link from one web page to another web page;

[0012] Calculate the TF-IDF value of the keyword to be searched in the website, and construct a TF-IDF feature vector based on the TF-IDF value;

[0013] Generate a comprehensive score for each web page based on the transition probability matrix and the TF-IDF feature vector; and determine the priority order of the corresponding web page in the website based on the comprehensive score of each web page;

[0014] Based on the determined priority order of web pages in the website, the web page data of the website is crawled.

[0015] In one embodiment, analyzing the links between web pages of a website and constructing a transition probability matrix between the web pages includes:

[0016] Analyze the historical click frequency of each web page in the website, and determine the authority weight of the web page based on the domain name type of the web page and the pre-set correspondence between the domain name type and the authority weight;

[0017] According to the historical click frequency and the authority weight, the link scores between the web pages are calculated; and the link scores between the web pages are integrated to construct a transition probability matrix between the web pages.

[0018] In one embodiment, the calculation formula of the link score is:

[0019] ;

[0020] In the formula, Link Score is the link score; λ is the authority weight of the web page; Click Rate is the historical click frequency of the web page.

[0021] In one embodiment, analyzing the historical click frequency of each web page in the website includes:

[0022] Obtaining the website log of the website, and analyzing the website log to obtain the click data of the user when browsing the web page, the click data including the click jump link between the web pages and the click frequency;

[0023] Based on the click-through links between web pages and the click frequency, determine the historical click frequency of each web page in the website.

[0024] In one embodiment, calculating the TF-IDF value of a keyword to be searched in a website and constructing a TF-IDF feature vector based on the TF-IDF value includes:

[0025] For each web page, count the number of occurrences of the keyword to be searched to determine the frequency of the keyword in the web page; and calculate the IDF value of the web page based on the total number of web pages in the website and the number of web pages containing the keyword;

[0026] Multiply the frequency of the keyword in the web page by the corresponding IDF value to calculate the TF-IDF value of the keyword;

[0027] The TF-IDF values ​​of all keywords are integrated to obtain a TF-IDF feature vector, where each element in the TF-IDF feature vector represents the relevance between the web page and the keyword.

[0028] In one embodiment, the calculation formula of the IDF value is:

[0029] ;

[0030] In the formula, is the IDF value of keyword t; N is the total number of web pages in the website; Docs containing t is the number of web pages containing the keyword.

[0031] In one embodiment, generating a comprehensive score for each web page according to the transition probability matrix and the TF-IDF feature vector; and determining the priority order of the corresponding web page in the website according to the comprehensive score of each web page includes:

[0032] The transition probability matrix and the TF-IDF feature vector are multiplied to obtain a comprehensive score of each web page; wherein the comprehensive score of each web page is proportional to the priority of each web page in the website;

[0033] According to the positive relationship between the comprehensive score of each web page and the priority of each web page in the website and the obtained comprehensive score of each web page, the priority order of the corresponding web page in the website is determined.

[0034] In one embodiment, crawling the web page data of the website based on the determined priority order of the web pages in the website includes: crawling the web page data of the website using a web crawler tool based on the determined priority order of the web pages in the website.

[0035] According to a second aspect of an embodiment of the present invention, a data crawling system is provided.

[0036] In one embodiment, the data capture system includes:

[0037] A transition probability matrix construction module is used to analyze the links between web pages of the website and construct a transition probability matrix between the web pages, wherein each element in the transition probability matrix represents the quality of the link from one web page to another web page;

[0038] A TF-IDF feature vector construction module, used to calculate the TF-IDF value of the keyword to be queried in the website, and to construct a TF-IDF feature vector based on the TF-IDF value;

[0039] A priority order determination module is used to generate a comprehensive score for each web page according to the transition probability matrix and the TF-IDF feature vector; and to determine the priority order of the corresponding web page in the website according to the comprehensive score of each web page;

[0040] The data crawling module is used to crawl the web page data of the website based on the determined priority order of the web pages in the website.

[0041] In one embodiment, when the transition probability matrix construction module analyzes the links between web pages of a website and constructs the transition probability matrix between web pages, it analyzes the historical click frequency of each web page in the website, and determines the authority weight of the web page based on the domain name type of the web page and the preset correspondence between the domain name type and the authority weight; calculates the link score between the web pages based on the historical click frequency and the authority weight; and integrates the link scores between the web pages to construct the transition probability matrix between the web pages.

[0042] In one embodiment, the calculation formula of the link score is:

[0043] ;

[0044] In the formula, Link Score is the link score; λ is the authority weight of the web page; Click Rate is the historical click frequency of the web page.

[0045] In one embodiment, when analyzing the historical click frequency of each web page in the website, the transition probability matrix construction module obtains the website log of the website and analyzes the website log to obtain the click data of users when browsing the web pages, wherein the click data includes the click jump links between the web pages and the click frequency; based on the click jump links between the web pages and the click frequency, the historical click frequency of each web page in the website is determined.

[0046] In one embodiment, when the TF-IDF feature vector construction module calculates the TF-IDF value of the keyword to be queried in the website and constructs the TF-IDF feature vector based on the TF-IDF value, for each web page, the number of occurrences of the keyword to be queried is counted to determine the frequency of the keyword in the web page; and the IDF value of the web page is calculated based on the total number of web pages in the website and the number of web pages containing the keyword; the frequency of the keyword in the web page is multiplied by the corresponding IDF value to calculate the TF-IDF value of the keyword; and the TF-IDF values ​​of all keywords are integrated to obtain the TF-IDF feature vector, each element of which represents the correlation between the web page and the keyword.

[0047] In one embodiment, the calculation formula of the IDF value is:

[0048] ;

[0049] In the formula, is the IDF value of keyword t; N is the total number of web pages in the website; Docs containing t is the number of web pages containing the keyword.

[0050] In one embodiment, the priority order determination module generates a comprehensive score for each web page according to the transition probability matrix and the TF-IDF feature vector; and when determining the priority order of the corresponding web page in the website according to the comprehensive score of each web page, multiplies the transition probability matrix and the TF-IDF feature vector to obtain the comprehensive score of each web page; wherein the comprehensive score of each web page is proportional to the priority of each web page in the website; and according to the proportional relationship between the comprehensive score of each web page and the priority of each web page in the website and the obtained comprehensive score of each web page, the priority order of the corresponding web page in the website is determined.

[0051] In one embodiment, when the data capture module captures the web page data of the website based on the determined priority order of the web pages in the website, it uses a web crawler tool to capture the web page data of the website based on the determined priority order of the web pages in the website.

[0052] According to a third aspect of an embodiment of the present invention, a computer device is provided.

[0053] In some embodiments, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0054] According to a fourth aspect of embodiments of the present invention, a computer-readable storage medium is provided.

[0055] In one embodiment, the computer-readable storage medium stores a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0056] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0057] The present invention utilizes the authority of web page sources (a measure of the reliability of web pages) and user click rates to construct a transition probability matrix to evaluate the link quality between web pages, and uses TF-IDF technology to deeply analyze the relevance between web page content and query topics; through matrix-vector multiplication, the two key factors of link quality and content relevance are deeply integrated to ensure that high-reliability web pages with highly relevant content and active user interaction are preferentially identified and processed in search results; this not only significantly improves the accuracy of web page data crawling, but also greatly enhances the quality of information users obtain.

[0058] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0060] Figure 1 is a flowchart of a data crawling method according to an exemplary embodiment;

[0061] Figure 2 is a structural block diagram of a data capture system according to an exemplary embodiment;

[0062] Figure 3 The figure is a schematic diagram showing the structure of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0063] The following description and accompanying drawings fully illustrate the specific embodiments of this article so that those skilled in the art can practice them. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of the embodiments of this article includes the entire scope of the claims, as well as all available equivalents of the claims. Herein, the terms "first", "second", etc. are only used to distinguish one element from another, without requiring or implying any actual relationship or order between these elements. In fact, the first element can also be called the second element, and vice versa. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the structure, device or equipment including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also include elements inherent to such structure, device or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the structure, device or equipment including the elements. Each embodiment is described in a progressive manner herein, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other.

[0064] The terms "longitudinal", "lateral", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. in this document indicate the orientation or position relationship based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing this document and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In the description of this document, unless otherwise specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a mechanical connection or an electrical connection, it can also be the internal communication of two elements, it can be a direct connection, or it can be an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0065] As used herein, the term "plurality" means two or more than two, unless otherwise specified.

[0066] In this document, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B.

[0067] In this article, the term "and / or" is a description of the association relationship between objects, indicating that three relationships may exist. For example, A and / or B means: A or B, or, A and B.

[0068] It should be understood that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0069] Each module in the device or system of the present application can be implemented in whole or in part by software, hardware, or a combination thereof. The above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to the above modules.

[0070] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0071] Figure 1 An embodiment of a data crawling method of the present invention is shown.

[0072] In this optional embodiment, the data capture method includes:

[0073] Step S101, analyzing the links between web pages of the website, and constructing a transition probability matrix between web pages, wherein each element in the transition probability matrix represents the quality of a link from one web page to another web page;

[0074] Step S103, calculating the TF-IDF value of the keyword to be searched in the website, and constructing a TF-IDF feature vector based on the TF-IDF value;

[0075] Step S105, generating a comprehensive score for each web page according to the transition probability matrix and the TF-IDF feature vector; and determining the priority order of the corresponding web page in the website according to the comprehensive score of each web page;

[0076] Step S107: crawling web page data of the website based on the determined priority order of the web pages in the website.

[0077] Figure 2 An embodiment of a data crawling system of the present invention is shown.

[0078] In this optional embodiment, the data capture system includes:

[0079] A transition probability matrix construction module 201 is used to analyze the links between web pages of the website and construct a transition probability matrix between the web pages, wherein each element in the transition probability matrix represents the quality of a link from one web page to another web page;

[0080] A TF-IDF feature vector construction module 203 is used to calculate the TF-IDF value of the keyword to be searched in the website, and to construct a TF-IDF feature vector based on the TF-IDF value;

[0081] The priority order determination module 205 is used to generate a comprehensive score for each web page according to the transition probability matrix and the TF-IDF feature vector; and determine the priority order of the corresponding web page in the website according to the comprehensive score of each web page;

[0082] The data capture module 207 is used to capture the web page data of the website based on the determined priority order of the web pages in the website.

[0083] In the above embodiment, when analyzing the links between web pages of the website and constructing the transition probability matrix between web pages, the historical click frequency of each web page in the website can be analyzed, and the authority weight of the web page can be determined based on the domain name type of the web page and the pre-set corresponding relationship between the domain name type and the authority weight; the link score between the web pages is calculated based on the historical click frequency and the authority weight; and the link scores between the web pages are integrated to construct the transition probability matrix between the web pages. The calculation formula of the link score is:

[0084] ;

[0085] In the formula, Link Score is the link score; λ is the authority weight of the web page; Click Rate is the historical click frequency of the web page.

[0086] In the above embodiment, when analyzing the historical click frequency of each web page in the website, the website log of the website can be obtained and analyzed to obtain the click data of users when browsing the web pages, and the click data includes the click jump links between the web pages and the click frequency; according to the click jump links between the web pages and the click frequency, the historical click frequency of each web page in the website is determined.

[0087] In the above embodiment, when calculating the TF-IDF value of the keyword to be queried in the website and constructing the TF-IDF feature vector based on the TF-IDF value, for each web page, the number of occurrences of the keyword to be queried is counted to determine the frequency of the keyword in the web page; and the IDF value of the web page is calculated based on the total number of web pages in the website and the number of web pages containing the keyword; the frequency of the keyword in the web page is multiplied by the corresponding IDF value to calculate the TF-IDF value of the keyword; the TF-IDF values ​​of all keywords are integrated to obtain the TF-IDF feature vector, and each element in the TF-IDF feature vector represents the correlation between the web page and the keyword. Among them, the calculation formula of the IDF value is:

[0088] ;

[0089] In the formula, is the IDF value of keyword t; N is the total number of web pages in the website; Docs containing t is the number of web pages containing the keyword.

[0090] In the above embodiment, when generating a comprehensive score for each web page according to the transition probability matrix and the TF-IDF feature vector, and determining the priority order of the corresponding web page in the website according to the comprehensive score of each web page, the transition probability matrix and the TF-IDF feature vector can be multiplied to obtain the comprehensive score of each web page; wherein the comprehensive score of each web page is proportional to the priority of each web page in the website; and according to the proportional relationship between the comprehensive score of each web page and the priority of each web page in the website and the obtained comprehensive score of each web page, the priority order of the corresponding web page in the website is determined.

[0091] In the above embodiment, when crawling the web page data of the website based on the determined priority order of the web pages in the website, the web crawler tool can be used to crawl the web page data of the website based on the determined priority order of the web pages in the website.

[0092] In order to facilitate understanding of the above technical solutions of the present invention, the above technical solutions of the present invention are specifically described below in the form of specific cases.

[0093] In specific use, before data crawling, there are four steps including data collection, construction of transition probability matrix, construction of TF-IDF feature vector and comprehensive evaluation of web page importance, as follows:

[0094] 1. Collect data and traverse the web pages from the known technology news website A as the starting point of the crawler. Assume that the following four web pages A, B, C and D are collected, and the link relationship between them is as follows: 1) A links to B, C and D; 2) B links to C; 3) C links to A and D; 4) D has no outgoing links (Dead End).

[0095] By analyzing the website logs, we know the following data: A is an authoritative news website, B is a well-known blog, C is an ordinary information website, and D is a social media platform. The historical click frequency data includes: the click rate of the link from A to B is 30%, A to C is 20%, A to D is 15%, B to C is 15%, C to A is 15%, and C to D is 10%.

[0096] 2. Construct the transition probability matrix

[0097] 1) Link A to B: Source authority: A is an authoritative news website, given the highest weight: λ=1.0; Historical click frequency: The click rate of the link A to B is 30%; Calculate the score of the link A to B: .

[0098] 2) Link from B to C: Source authority: B is a well-known blog, with a weight of λ=0.8; Historical click frequency: The click rate of the link from B to C is 15%; Calculate the score of the link from B to C: .

[0099] And so on, calculate the scores of A to C, A to D, C to A, and C to D. Construct the transition probability matrix P:

[0100] ;

[0101] Note: The matrix elements represent the scores of different web page links. For example, CA=0.08 means that the link score from web page C to web page A is 0.08.

[0102] 3. Constructing TF-IDF feature vector of web page

[0103] 3.1. Define the set of keywords related to science and technology news

[0104] For example, t = {“artificial intelligence”, “blockchain”, “5G”}

[0105] 3.2. Keyword matching

[0106] Take the keyword set in web page A as an example:

[0107] Calculate the frequency of keyword set t appearing in web page A:

[0108] ;

[0109] Calculate the inverse of the frequency of keyword set t appearing in the web page set:

[0110] ;

[0111] Multiply TF and IDF to calculate the TF-IDF value of web page A:

[0112] .

[0113] Similarly, the TF-IDF values ​​of web pages B, C, and D are calculated as follows: TF-IDF(t,B)=0.080

[0114] TF-IDF(t,C)=0.121, TF-IDF(t,D)=0.059,

[0115] 3.3. Web Content Feature Vectorization

[0116] Construct a web page feature vector based on TF-IDF:

[0117] ;

[0118] This vector represents the relevance of the web page set to the keyword.

[0119] 4. Comprehensive evaluation of web page importance

[0120] Multiply equation 1 and equation 2: ,According to the ranking, the order of crawling and processing of web pages is A, C, B, D.

[0121] Figure 3 An embodiment of a computer device of the present invention is shown. The computer device may be a server, and the computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store static information and dynamic information data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the steps in the above method embodiment are implemented.

[0122] Those skilled in the art will understand that Figure 3The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0123] In addition, the above-mentioned computer device provided by the present invention includes a memory and a processor, the memory stores a computer program, and the processor implements the steps in the above-mentioned method embodiment when executing the computer program.

[0124] In addition, the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0125] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0126] The present invention is not limited to the structures which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A data capture method, characterized in that: include: Analyze the links between web pages of the website and construct a transition probability matrix between web pages, wherein each element in the transition probability matrix represents the quality of the link from one web page to another web page; Calculate the TF-IDF value of the keyword to be searched in the website, and construct a TF-IDF feature vector based on the TF-IDF value; Generate a comprehensive score for each web page based on the transition probability matrix and the TF-IDF feature vector; and determine the priority order of the corresponding web page in the website based on the comprehensive score of each web page; Based on the determined priority order of web pages in the website, crawl the web page data of the website; Among them, analyzing the links between web pages of the website and constructing the transition probability matrix between web pages includes: Analyze the historical click frequency of each web page in the website, and determine the authority weight of the web page based on the domain name type of the web page and the pre-set correspondence between the domain name type and the authority weight; Calculating link scores between web pages based on the historical click frequency and the authority weight; and integrating the link scores between the web pages to construct a transition probability matrix between the web pages; The calculation formula of the link score is: ； In the formula, Link Score is the link score; λ is the authority weight of the web page; Click Rate is the historical click frequency of the web page.

2. The data capture method according to claim 1, characterized in that: Analyze the historical click frequency of each web page in the website including: Obtaining the website log of the website, and analyzing the website log to obtain the click data of the user when browsing the web page, the click data including the click jump link between the web pages and the click frequency; Based on the click-through links between web pages and the click frequency, determine the historical click frequency of each web page in the website.

3. The data capture method according to claim 1, characterized in that: Calculating the TF-IDF value of the keyword to be searched in the website and constructing the TF-IDF feature vector based on the TF-IDF value includes: For each web page, count the number of occurrences of the keyword to be searched to determine the frequency of the keyword in the web page; and calculate the IDF value of the web page based on the total number of web pages in the website and the number of web pages containing the keyword; Multiply the frequency of the keyword in the web page by the corresponding IDF value to calculate the TF-IDF value of the keyword; The TF-IDF values ​​of all keywords are integrated to obtain a TF-IDF feature vector, where each element in the TF-IDF feature vector represents the relevance between the web page and the keyword.

4. The data capture method according to claim 1, characterized in that: Generate a comprehensive score for each web page according to the transition probability matrix and the TF-IDF feature vector; And according to the comprehensive score of each web page, the priority order of the corresponding web page in the website is determined, including: The transition probability matrix and the TF-IDF feature vector are multiplied to obtain a comprehensive score of each web page; wherein the comprehensive score of each web page is proportional to the priority of each web page in the website; According to the positive relationship between the comprehensive score of each web page and the priority of each web page in the website and the obtained comprehensive score of each web page, the priority order of the corresponding web page in the website is determined.

5. A data capture system, characterized in that: include: A transition probability matrix construction module is used to analyze the links between web pages of the website and construct a transition probability matrix between the web pages, wherein each element in the transition probability matrix represents the quality of the link from one web page to another web page; A TF-IDF feature vector construction module, used to calculate the TF-IDF value of the keyword to be queried in the website, and to construct a TF-IDF feature vector based on the TF-IDF value; A priority order determination module is used to generate a comprehensive score for each web page according to the transition probability matrix and the TF-IDF feature vector; and to determine the priority order of the corresponding web page in the website according to the comprehensive score of each web page; A data crawling module, used for crawling web page data of the website based on the determined priority order of the web pages in the website; The transition probability matrix construction module analyzes the links between web pages of the website and constructs the transition probability matrix between web pages by analyzing the historical click frequency of each web page in the website, and determines the authority weight of the web page based on the domain name type of the web page and the preset corresponding relationship between the domain name type and the authority weight; calculates the link scores between the web pages based on the historical click frequency and the authority weight; and integrates the link scores between the web pages to construct the transition probability matrix between the web pages; The calculation formula of the link score is: ； In the formula, Link Score is the link score; λ is the authority weight of the web page; Click Rate is the historical click frequency of the web page.

6. The data capture system according to claim 5, characterized in that: When analyzing the historical click frequency of each web page in the website, the transition probability matrix construction module obtains the website log of the website and analyzes the website log to obtain the click data of the user when browsing the web pages, wherein the click data includes the click jump links between the web pages and the click frequency; and determines the historical click frequency of each web page in the website according to the click jump links between the web pages and the click frequency.

7. The data capture system according to claim 5, characterized in that: When the TF-IDF feature vector construction module calculates the TF-IDF value of the keyword to be queried in the website and constructs the TF-IDF feature vector based on the TF-IDF value, for each web page, the number of occurrences of the keyword to be queried is counted to determine the frequency of the keyword in the web page; and the IDF value of the web page is calculated according to the total number of web pages in the website and the number of web pages containing the keyword; the frequency of the keyword in the web page is multiplied by the corresponding IDF value to calculate the TF-IDF value of the keyword; and the TF-IDF values ​​of all keywords are integrated to obtain the TF-IDF feature vector, each element of which represents the correlation between the web page and the keyword.

8. The data capture system according to claim 5, characterized in that: The priority order determination module generates a comprehensive score for each web page according to the transition probability matrix and the TF-IDF feature vector; and when determining the priority order of the corresponding web page in the website according to the comprehensive score of each web page, the transition probability matrix and the TF-IDF feature vector are multiplied to obtain the comprehensive score of each web page; wherein the comprehensive score of each web page is proportional to the priority of each web page in the website; and the priority order of the corresponding web page in the website is determined according to the proportional relationship between the comprehensive score of each web page and the priority of each web page in the website and the obtained comprehensive score of each web page.

Images