Web crawler filtering method and device

Abstract

A web crawler filtering method and apparatus thereof are provided. A web crawler filtering method and apparatus thereof are provided. The performance of a search engine depends to a certain extent on the ability of a web crawler (web content acquisition). This application proposes (1) a web page vector representation method; and proposes a crawler-filtering algorithm based on web page vector similarity for crawling and filtering web pages to be crawled. Experiments show that the crawler-filtering algorithm can bypass low-quality or irrelevant content while covering reasonable content during the web crawling process. This optimizes the quality of web crawling.

Description

Technical Field

[0001] The present application relates to web crawler technology, specifically, to a crawler-filtering method constructed by using machine learning technology to analyze the similarity between web pages and web crawler targets, and an information processing device for implementing the method. Background Art

[0002] With the rapid development of the Internet and mobile Internet, a huge amount of data has been generated. In order to enable users to quickly and conveniently query and obtain information from such a huge amount of data, network information acquisition and retrieval technologies represented by search engines have greatly promoted the process of information exchange and sharing. For example, commercial search engines such as Google and Baidu use specific information acquisition programs, namely web crawlers, based on certain strategies (Olston C and Najork M. Web Crawling, Foundations and In Information Retrieval: Vol. 4: No. 3, pp 175-246. http: / / dx.doi.org / 10.1561 / 1500000017), vertical search engines collect information from the entire internet, build web databases and directories (indexes), provide retrieval services for users, and deliver relevant information to users. However, at present, traditional commercial search engines are unable to provide accurate search capabilities in a specific field, which has led to the emergence of vertical search engines. Vertical search engines do not provide users with the ability to search for content from millions or even tens of millions of web pages, but rather provide a narrow, highly targeted service for finding information in a specific field.

[0003] Whether it is a commercial search engine or a vertical search engine, web crawler technology is an important part of search engine technology. It can automatically find and store web pages in the ocean of network content, providing a data basis for "needle in a haystack" type of information acquisition.

[0004] Figure 1 A schematic diagram of a web crawler in the prior art is shown.

[0005] A web crawler is, for example, a program or automated script that is a way to browse and collect information on the Internet. A list of one or a group of URLs (Uniform Resource Locators, Unified Resource Locators) to be accessed (also called seed URLs) is provided for the crawler to process. The web crawler obtains URLs from the list of URLs to be accessed, downloads web pages based on the URLs, analyzes the content and network resources (e.g., pictures, videos, etc.) in the web pages, and creates an index, and records the index in a local storage or network storage system in association with the downloaded web pages and / or their network resources. This provides search engines with the ability to retrieve / query indexes. The downloaded web pages usually include further URLs, and the crawler also picks up the URLs included in the downloaded web pages and adds them to the list of URLs to be accessed for further crawling and processing. The goal of a web crawler is to collect the most web page information in the shortest time.

[0006] There are several types of web crawlers:

[0007] (1) General web crawler

[0008] General purpose web crawlers are not limited to web pages of a specific subject or domain. They continuously follow hyperlinks and fetch all web pages they come across.

[0009] (2) User interest crawler

[0010] User interest crawlers prioritize the interests of users and prioritize URLs. These web crawlers do not crawl all the links they encounter, but instead guide the priority crawler to selectively search for web pages related to a predefined set of topics based on a condition or topic of interest submitted by the user.

[0011] (3) Hidden content crawler

[0012] Information on a webpage cannot be accessed or retrieved through hyperlinks. It is hidden behind a search or query interface, making it inaccessible directly. This part of the webpage is called a hidden webpage or deep webpage. Using crawler technology to mine hidden information is called a hidden content crawler (Gravano L, Ipeirotis PG, and Sahami M. 2003. Qprober: a system for automatic classification of hidden content).

[0013] (4) Continuous content crawling

[0014] Web content is dynamic, and the data on web pages is constantly changing. Continuous content crawling is used to maintain up-to-date search engines and index databases. However, there's a trade-off between data freshness and resource consumption.

[0015] The behavior of web crawlers may affect normal network services. Therefore, web crawlers need to follow several strategies:

[0016] (1) Politeness strategy: Web crawlers should not interfere with the normal operation of the website and crawl only pages that are allowed to be accessed.

[0017] (2) Parallel strategy: The web crawler program starts in multi-threaded mode to minimize the download time, and the new URLs assigned will be distributed to new threads and run in parallel.

[0018] (3) Revisit strategy: Web crawlers need to revisit pages that have been visited to keep the crawled data fresh.

[0019] (4) Robust Strategy: Web crawlers need to avoid being "stuck" on a specific page and constantly retrieving duplicate web pages. A robust crawler strategy needs to circumvent this situation.

[0020] Neural network technology based on deep learning has made great progress. Deep learning (DL) mainly uses multi-layer neural networks to enable machines to learn, thereby obtaining the inherent laws and representation levels of data. Figure 2 This diagram shows the structure of a deep neural network in the prior art. A deep neural network consists of an input layer, hidden layers, and an output layer. Each layer of a deep neural network consists of neurons with basic computing capabilities, connected to each other through inputs and outputs. The input layer receives input from the neural network, while the hidden layer includes multiple layers of neurons, such as a fully connected structure, to identify inherent patterns in the data. The output layer outputs the processing results of the deep neural network.

[0021] By using deep neural networks, it is possible to represent speech, pictures, videos, language, text and other content as vectors suitable for computer processing.

[0022] The existing text vectorization technology is used to convert words and sentences into vectors that are easy to be processed by computers (Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv: 1301.3781; Pennington Jeffrey, Socher Richard, and Manning Christopher. 2014. GloVe: Global vectors for word representation. pages 1532–1543.). In addition, the vectors generated by the text vectorization technology can express semantic similarity. Words / sentences with similar semantics, after being converted into vectors, have a closer distance in the vector space, so that the computer can judge the degree of semantic similarity of the words / sentences used to generate the corresponding vectors by comparing the distance between the vectors. For example, between "cat," "dog," "cow," and "sheep," "cat" and "dog" are similar, while "cow" and "sheep" are similar. Consequently, after these words are vectorized, the distance between the vector generated by "cat" and the vector generated by "dog" in vector space is smaller than the distance between the vector generated by "cat" and the vector generated by "cow." Similarly, the distance between the vector generated by "cow" and the vector generated by "sheep" is smaller than the distance between the vector generated by "dog" and the vector generated by "cow." As another example, the wording of "This team is ready to win this game" and "The balance of victory in this game has tilted towards this team" is almost completely different. However, using text vectorization technology, the two sentences are close to each other in vector space. From another perspective, this vector body reflects the meaning of the sentence.

[0023] The BERT model (a neural network model) based on a pre-trained model (Jacob Devlin, Wei Chang, and Kenton Lee. 2018. Bert: pre-training of deep bidirectional transformers for language understanding. CoRR, abs / 1810.04805, pages 770–778.) has already performed exceptionally well across various natural language processing tasks. The BERT model processes an input sentence and outputs a vectorized representation of the sentence (called a sentence vector). The generated sentence vectors are able to express phonetic similarity, so sentences with similar semantics have a close distance between their generated sentence vectors in vector space.

[0024] Existing image vectorization technology is used to convert images into vectors that are easy for computers to process. In addition, the vectors generated by image vectorization technology can express the similarity of image content. After being converted into vectors, images with similar content have a closer distance in the vector space. The idea of ​​image vectorization representation is to use large image datasets (such as ImageNet (Deng J, Dong W, Socher R, Li LJ, Li K, and Fei-Fei L. 2009. ImageNet: A large-scale hierarchical image database. pages 248–255.)) to train a deep neural network to obtain an image classification model. Then, using the layer that does not contain the final classifier part, the feature vector corresponding to the input image is extracted. The obtained feature vector can effectively represent the image in search or similarity matching tasks.

[0025] Figure 3A and Figure 3B A schematic diagram showing image vectorization technology.

[0026] The trained deep neural network processes the input image, and its output layer neurons output, for example, the classification result of the image. The vector obtained by combining or splicing the outputs of several neurons in the hidden layer is used as the image vector obtained by vectorizing the input image. This image vector reflects the content of the image. Figure 3A In the example, the outputs of multiple neurons in the hidden layer connected to the output layer are selected to form the image vector. Optionally, the outputs of one or more other neurons in the hidden layer are selected to form the image vector representing the image input to the deep neural network.

[0027] exist Figure 3B In this example, the classic image classification network VGGNet (Karen Simonyan and Andrew Zisserman (2014): Very Deep Convolutional Networks for Large-Scale Image Recognition.arXiv:1409.1556[cs] (September 2014)) is used to obtain an image vector. The VGGNet network consists of multiple convolutional layers, pooling layers, and fully connected layers connected in sequence. When the VGGNet network processes an input image, the output of the pooling layer immediately preceding the fully connected layer is selected as the image vector for that image. This image vector has a dimension of, for example, 7*7*512. Summary of the Invention

[0028] However, existing web crawler technologies face the following challenges:

[0029] (1) The data structure of web page content is complex. Web page content contains a large amount of semi-structured and unstructured data (text, pictures, videos), which makes it difficult to collect, analyze, and crawl data.

[0030] (2) The scale of web crawlers is difficult to control

[0031] The scale of web content acquisition cannot be estimated in advance. Due to resource limitations, a trade-off needs to be made between content crawling coverage and database freshness. Web content crawlers need to bypass irrelevant and low-quality content.

[0032] (3) Insufficient ability to understand web content

[0033] Web pages are linked to each other through URLs. There is a lack of analysis and judgment on whether the network resources that need to be accessed next are related to the content topic, and there is insufficient understanding of the web page content.

[0034] In order to solve one or more of the above technical problems and obtain one or more of the above technical effects, various embodiments of the present application are provided.

[0035] According to the first aspect of the present application, a method for crawling a network using a first web crawler according to the first aspect of the present application is provided, comprising: obtaining a first-category URL from a URL cache, crawling web pages corresponding to the first-category URL and extracting one or more second-category URLs and adding them to a preload cache; obtaining a second-category URL from the preload cache, preloading web pages corresponding to the second-category URL and generating a web page vector based on the preloaded web pages corresponding to the second-category URL; if the distance between the generated web page vector and a target vector generated according to the crawling target is less than a specified threshold, adding the second-category URL as a first-category URL to the URL cache.

[0036] According to the first web crawler method for crawling a network in the first aspect of the present application, a second web crawler method for crawling a network in the first aspect of the present application is provided, further comprising: if the distance between the generated web page vector and the target vector generated according to the crawling target is not less than a specified threshold, the second-category URL will be preloaded instead of being added to the URL cache as a first-category URL.

[0037] According to the first or second web crawler method for crawling a network according to the first aspect of the present application, a third web crawler method for crawling a network according to the first aspect of the present application is provided, wherein in response to the presence of a first category URL in the URL cache, the web crawler method for crawling a network is repeatedly executed; in response to the presence of a second category URL in the preload cache, the steps of obtaining the second category URL from the preload cache, preloading the web page corresponding to the second category URL and generating a web page vector based on the web page corresponding to the preloaded second category URL are repeatedly executed; and if the distance between the generated web page vector and the target vector generated according to the crawling target is less than a specified threshold, the second category URL is added to the URL cache as a first category URL.

[0038] According to the first or second web crawler method for crawling a network according to the first aspect of the present application, a fourth web crawler method for crawling a network according to the first aspect of the present application is provided, which also includes: obtaining a URL seed and adding it to the URL cache; and obtaining a crawl target and generating a target vector, wherein the text of one or more words and / or one or more sentences of the crawl target is vectorized to generate a target vector, and / or one or more images of the crawl target are input into a deep neural network, and the output is obtained from the intermediate layer neurons that are neither the input layer nor the output layer of the deep neural network to generate a picture vector as the target vector.

[0039] According to the fourth method of crawling a network with a web crawler in the first aspect of the present application, a fifth method of crawling a network with a web crawler in the first aspect of the present application is provided, wherein a web page corresponding to a second-category URL is preloaded and a web page vector is generated based on the web page corresponding to the preloaded second-category URL, including: extracting one or more text blocks from the web page corresponding to the preloaded second-category URL; calculating a vector of the text block based on each of the one or more text blocks; calculating a statistical value of the vector of each of the one or more text blocks as a web page text content vector of the web page corresponding to the preloaded second-category URL; extracting one or more images from the web page corresponding to the preloaded second-category URL; calculating an image vector based on each of the one or more images; calculating a statistical value of the image vector of each of the one or more images as a web page image content vector of the web page corresponding to the preloaded second-category URL; and obtaining a web page vector of the web page corresponding to the preloaded second-category URL based on the web page text content vector and the web page image content vector.

[0040] According to the fifth method for crawling a network with a web crawler according to the first aspect of the present application, a sixth method for crawling a network with a web crawler according to the first aspect of the present application is provided, wherein a text block includes one or more sentences; and calculating a vector of a text block based on each of the one or more text blocks includes: for each text block based on the one or more text blocks, obtaining one or more sentences constituting the current text block; inputting each of the one or more sentences into a BERT model instance to obtain a sentence vector corresponding to the input sentence; and calculating a statistical value of the sentence vector of each of the one or more sentences as the vector of the current text block.

[0041] According to the sixth web crawler method for crawling a network according to the first aspect of the present application, a seventh web crawler method for crawling a network according to the first aspect of the present application is provided, wherein the vector corresponding to the i-th text block Bi according to the one or more text blocks is recorded as V Bi , Bi=(Si1,Si2,…,Sij,…,Sini) means that the text block Bi includes multiple sentences, sentence Sij is the jth sentence of the text block Bi, and the sentence vector of sentence Sij is recorded as V Sij , ni is the number of sentences included in the text block Bi, and ni, i and j are positive integers 1<=j<=ni.

[0042] According to the seventh web crawler method for crawling a network according to the first aspect of the present application, an eighth web crawler method for crawling a network according to the first aspect of the present application is provided, wherein the web page text content vector of the web page corresponding to the preloaded second type URL is recorded as P text , Wherein N is the number of text blocks in the web page corresponding to the preloaded second-category URL, and N is a positive integer, 1<=i<=N.

[0043] According to the eighth web crawler method for crawling a network according to the first aspect of the present application, a ninth web crawler method for crawling a network according to the first aspect of the present application is provided, wherein the web page image content vector of the web page corresponding to the preloaded second type URL is recorded as P image , Where M is the number of images in the web page corresponding to the preloaded second-category URL, M is a positive integer, VI p is the image vector of the pth image in the web page corresponding to the preloaded second-category URL, where p is a positive integer, 1<=p<=M; and the web page vector V of the web page corresponding to the preloaded second-category URL page =(P text , P image ); or the webpage vector V of the webpage corresponding to the preloaded second type URL page =Ptext +P image .

[0044] According to the second aspect of the present application, a first information processing device according to the second aspect of the present application is provided, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, one of the first to ninth web crawler methods for crawling a network according to the first aspect of the present application is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The present application, together with the preferred mode of use and further objects and advantages thereof, will be best understood by reference to the following detailed description of illustrative embodiments when read in conjunction with the accompanying drawings, in which:

[0046] Figure 1 A schematic diagram of a web crawler in the prior art is shown.

[0047] Figure 2 A schematic diagram showing the deep neural network architecture of the prior art.

[0048] Figure 3A and Figure 3B A schematic diagram showing image vectorization technology.

[0049] Figure 4 A flowchart of calculating sentence similarity according to an embodiment of the present application is shown.

[0050] Figure 5A Shows the flowchart for calculating the text content vector of a web page. Figure 5B Shows the flowchart for calculating the content vector of web page images.

[0051] Figure 6 This is a flowchart of a web crawler crawling network data according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0053] Figure 4 The flowchart of calculating sentence similarity according to an embodiment of the present application is shown. Sentence similarity is the basis for the web crawler to understand and filter URLs to be crawled according to an embodiment of the present application.

[0054] See Figure 4, by processing sentences using a pre-trained deep neural network model such as the BERT model to convert the sentences into sentence vectors. The sentences are, for example, sentences obtained by a crawler from the text of a web page. In order to compare two sentences (denoted as sentence A and sentence B), two pre-trained BERT model instances are provided (these two BERT model instances can be obtained through the same or different pre-training processes and have the same or different model parameters), and sentence A and sentence B are respectively input into the corresponding BERT model instances, and the sentence vector U corresponding to sentence A and the sentence vector V corresponding to sentence B are obtained from the output of the BERT model instance. The distance between the sentence vector U and the sentence vector V is calculated as the similarity between sentence A and sentence B. It can be understood that in Figure 4 In the example, the smaller the distance between sentence vector U and sentence vector V, the greater the similarity between sentence A and sentence B.

[0055] Those skilled in the art will understand that other existing technologies or deep neural network technologies or other natural language processing technologies to be developed in the future can be used to obtain the similarity between sentences without being limited to the use of the BERT model.

[0056] According to an embodiment of the present application, a method for generating a webpage vector based on a webpage (also known as the Page2Vec algorithm) is also provided. A webpage typically includes text and images. According to an embodiment of the present application, a webpage vector is obtained by combining a webpage text content vector and a webpage image content vector. Figure 5A Shows the flowchart for calculating the text content vector of a web page. Figure 5B Shows the flowchart for calculating the content vector of web page images.

[0057] The text content of a web page usually includes multiple parts, for example, a header marked by the " " tag of HTML (Hypertext Markup Language), a body part marked by the " " tag, and similarly, " <iframe>Tags such as ”、"” also usually mark regions with text content. Generally, the text content of a web page is regarded as consisting of multiple text blocks. The text blocks are identified by tags such as HTML. Still as an example, each paragraph of the text in the web page is regarded as a text block. The i-th text block in the web page is denoted as Bi, where i is a positive integer.

[0058] A text block consists of one or more sentences. Generally, the text block Bi = (S1, S2, …, Sn), indicating that the text block Bi includes n sentences, where n is a positive integer, and Sj represents the j-th sentence in this text block, 1 <= j <= n. The vector corresponding to the text block Bi is denoted as VBi, and the sentence vector of the sentence Sj is denoted as VSj. According to the embodiments of the present application, that is, the vector of the text block is the average value of the sentence vectors of all the sentences included in this text block.

[0059] The web page content vector of the web page Page is denoted as Ptext. The web page Page includes N text blocks, denoted as Page = (B1, B2, …, BN), where Bi represents the i-th text block in the web page Page, 1 <= i <= N. According to the embodiments of the present application, that is, the web page content vector is the average value of the vectors of all the text blocks included in this web page.

[0060] Referring to Figure 5A, to generate the web page text content vector (Ptext) of the web page Page, one or more text blocks (Bi) are extracted from the web page Page (510). To calculate the vector VBi of the text block Bi, the sentence Sj is extracted from the text block Bk (520), and the sentence vector VSj of the extracted sentence Sj is calculated (530).

[0061] The text block Bi includes one or more sentences. The vector VBi of this text block is generated according to the sentence vectors of one or more sentences included in this text block Bi (540).For example, the vector V Bi of the text block Bi is obtained by taking the average of the sentence vectors of all sentences in the text block Bi as the vector of the text block and calculating the sum, weighted average or other statistical values ​​of the vector V Sj of each sentence Sj . Based on the vectors of one or more text blocks included in the web page Page, a web page text content vector (P text ) of the web page is generated (550). For example, the web page text content vector (P text ) is obtained by calculating the average value of the vectors V Bi of each text block Bi. Alternatively, the web page text content vector (P text ) is obtained by calculating the sum, weighted average, or other statistical value of the vectors V Bi of each text block Bi. In an optional embodiment, text blocks located at different positions on a web page have respective weights when calculating the web page text content vector. For example, a text block located in the center of a web page has a relatively high weight, while a text block located in the header / title bar of the web page also has a relatively high weight, while a text block located in the footer or side of the web page has a relatively low weight.

[0064] Web pages also typically include pictures. For example, through HTML " tag to identify images on a web page. Those skilled in the art are also familiar with other methods for obtaining images from a web page. According to an embodiment of the present application, the image vector (denoted as V1 ) of the image included in the web page is used to generate the web page image content vector (P image ). As an example, the average value of the image vectors (V1) of all the images included in the web page is used as the web page image content vector (P image ) of the web page. Where M is a positive integer representing the number of images in a web page, VI p represents the image vector of the p-th image in the web page, 1<=p<=M, and p is a positive integer.

[0066] Optionally, the web page image content vector (P image ) of the web page is obtained by calculating the sum, weighted average or other statistical values ​​of the image vectors (VI) of one, multiple or all images of the web page.

[0067] In still optional embodiments, pictures located at different positions on a web page have respective weights when calculating the web page picture content vector. For example, a picture located in the center of the web page has a relatively high weight, a picture located in the header / title bar of the web page also has a relatively high weight, while a picture located in the footer or side of the web page has a relatively low weight.

[0068] Referring to Figure 5B, to generate the web page picture content vector (Pimage) of web page Page, one or more pictures (560) are extracted from web page Page. The picture vector (VI) of each extracted picture is calculated (570). The picture vector (VI) is generated based on the picture by a method shown in, for example, <a href='javascript:;' class="see-img-anchor" img-id="HDA0003364181930000As an example, the web page vector (Vpage=(Ptext, Pimage)) is obtained by connecting the web page text content vector (Ptext) and the web page image content vector (Pimage). As another example, the web page vector (Vpage=Ptext+Pimage) is obtained by adding the web page text content vector (Ptext) and the web page image content vector (Pimage).

[0070] Furthermore, according to an embodiment of the present application, the similarity of two corresponding web pages is evaluated by calculating the distance between two web page vectors. The smaller the distance between two web page vectors, the higher the similarity of the web pages represented by these two web page vectors.

[0071] Figure 6 is a flowchart of a web crawler for crawling network data according to an embodiment of the present application.

[0072] In order to improve the harvest ratio (HR, Harvest Ratio) of the web crawler (Behl A Banati H Thukral A, Mendiratta V. 2011. A social semantic focused crawler. pages 273–283.), and reduce the interference of a large number of web pages and diverse web page content on the crawling results of the web crawler, according to the web crawler of an embodiment of the present application, during the crawling of web pages, the URLs to be crawled extracted from the web pages are filtered, and only the web pages that the web crawler is interested in are selected for crawling, and the web pages irrelevant to the crawling target are not crawled. The web pages that the web crawler is interested in are the web pages with a relatively high similarity to the crawling target.

[0073] As an example, the harvest ratio is considered to be the proportion of web page crawls that meet the requirements. The specific definition is: HR = r / p, HR ∈ (0,1), r is the number of web pages related to the crawling target that are crawled, and p is the total number of web pages crawled. Thus, HR represents the proportion of web pages related to the crawling target among all the crawled web pages, and its size reflects the crawling effect of the web crawler.

[0074] Furthermore, since the content and the URL in the web page are mixed together, the process of crawling the web page involves identifying the web page content, and recording information such as recommended indexes, web addresses, and / or web page snapshots, introducing overheads of computing and storage resources. Therefore, it is desired to identify the similarity between the web page and the crawling target before the crawler crawls the web page. For this purpose, a preloading mechanism for further web pages corresponding to the URL obtained from the web page is also introduced. During the preloading process of the web page, only the similarity between the preloaded web page and the crawling target is identified, and operations such as analyzing the web page and building an index are not performed. Thus, the overheads of computing, storage, power consumption, time, etc. in the web page preloading process are less than those in the process of crawling the web page.

[0075] Referring to Figure 6, a web crawler according to an embodiment of the present application obtains a crawling target (610), and starts a crawling process according to the given crawling target and the seed URL. The crawling target is, for example, one or more topic words (e.g., football, shopping), one or more sentences, and / or one or more web pages (e.g., URL). A target vector (denoted as VD) (620) is generated according to the obtained crawling target. The process of generating the target vector VD is similar to the content of generating the web page vector. As an example, if the crawling target is one or more words / sentences, text vectorization techniques such as the BERT model are used to generate word vectors or sentence vectors as the target vector. It can be understood that for a crawling target including multiple words / sentences, the word vectors / sentence vectors are also summed, averaged, weighted averaged, or other statistical calculations are performed to obtain the target vector. If the crawling target is a picture, the picture vector (VI) corresponding to the picture or the statistical result of multiple picture vectors is calculated as the target vector. If the crawling target is a web page including, for example, both text and pictures, the web page vector of the web page is generated as the crawling target VD.

[0076] The seed URL is the crawling starting point of the web crawler, such as the URL of a portal website or a URL specified by the user. There can be one or more seed URLs. The seed URL is added to the URL cache accessible by the web crawler. For example, the URL cache is implemented in the form of a queue, so that the web crawler processes the URLs in the cache in a first-in-first-out manner.

[0077] In response to the start of the crawling process, the web crawler obtains a URL from the URL cache and crawls the obtained URL (630). The crawling process includes analyzing the web page content corresponding to the URL and building an index, etc.During the crawling process, one or more URLs included in the web page are also extracted and recorded in the preloading cache (640). For example, the preloading cache is implemented in the form of a queue.

[0078] Understandably, using the cache helps the various stages of the web crawler to be processed concurrently. For example, multiple tasks or operation steps of the web crawler are implemented by multiple threads respectively, and each thread exchanges data through the cache.

[0079] The web crawler extracts URLs from the preloading cache and preloads the web page Page corresponding to the URL (650). For the preloaded web page, the web crawler may not analyze the web page content and / or build an index to reduce the computing / storage overhead of the web crawler. For the preloaded web page Page, the web crawler calculates its web page vector Vpage (660). The web page vector Vpage is calculated by, for example, combining the methods described in Figure 5A and Figure 5B. The web crawler also determines whether the preloaded web page Page is similar to the crawling target (670). As an example, the distance between the web page vector Vpage of the web page Page and the target vector VD is provided to determine whether the preloaded web page Page is similar to the crawling target. For example, if the distance between the web page vector Vpage and the target vector VD is less than a specified threshold, it is considered that the preloaded web page Page is similar to the crawling target. Optionally or further, the web crawler also adjusts the target vector VD during the crawling process to obtain a clearer or richer voice for retrieving the target according to the crawling result. For example, the target vector VD is updated by calculating VD = f(VD, Vpage), where Vpage is the web page vector that has been identified as close to the existing target vector VD, and the function f is, for example, averaging, weighted averaging, etc. Still optionally, the target vector VD is updated during the crawling process through a target tracking algorithm such as the Kalman filter.In step 670 , if it is determined that the web page vector Vpage of the web page preloaded according to a certain URL is similar to the target vector VD, the URL is added to the URL cache for subsequent crawling of the web page indicated by the URL (630). If the web page vector Vpage of the web page preloaded according to the URL is less similar to the target vector VD, the URL is discarded and not added to the URL cache, and the web page indicated by it is not subsequently crawled. Then, the process returns to step 650, and the web crawler obtains other URLs from the preload cache and continues processing.

[0081] It is understandable that the URL cache and preload cache may include multiple URLs to be processed. The web crawler continuously extracts URLs from the URL cache and preload cache and processes them accordingly until the URL cache is empty or other stopping conditions are met, and the crawling process stops.

[0082] In order to evaluate the crawling effect of the web crawler according to the embodiment of the present application, the web crawler according to the embodiment of the present application, the Focus crawler algorithm based on user interest (Batsakis S, Petrakis EGM, and Milios E. 2009. Improving the performance of focused web) and the URL filtering crawler algorithm (Wang Hui-chang, Ruan Shu-hua, and Tang Qi-jie. 2009. The implementation of a web crawler URL filter algorithm based on caching. In 2009 Second International Workshop on Computer Science) were used respectively. Under different initial seed URLs, web pages were crawled according to different target crawling goals. 1000 web pages were crawled for each goal, and the harvest rate (HR) was calculated. ( 1 ) Initial URL seed

[0084] Tencent (www.qq.com) and Sina (www.sina.com.cn) were selected as URL seeds respectively. ( 2 ) Climbing target

[0086] Select two crawling targets, "shopping" and "football", respectively.Table 1 below shows the crawling results of selecting Tencent (www.qq.com) as the initial URL seed page and crawling the two crawling targets "shopping" and "football". In the table, the HR indicator represents the harvest rate.

[0088] Table 1 Table 2 below shows the results of crawling the two crawling targets "shopping" and "football" by selecting Sina News (www.sina.com.cn) as the initial URL seed page.

[0091] Table 2

[0093] By comparing the harvest rate results of the web crawler, Focus algorithm and URL filtering algorithm of the embodiment of the present application under different URL seeds and crawling targets, it is obvious that the HR result of the web crawler of the embodiment of the present application is higher than that of other algorithms.

[0094] Although the present application is described with reference to examples, this is for illustrative purposes only and is not intended to limit the present application, and changes, additions and / or deletions to the embodiments may be made without departing from the scope of the present application. Those skilled in the art who benefit from the teachings presented in the above description and the associated drawings to which these embodiments relate will recognize that many modifications and other embodiments of the present application described herein. Therefore , it should be understood that the present application is not limited to the disclosed specific embodiments and is intended to include modifications and other embodiments within the scope of the appended claims. Although specific terms are used herein, they are used only in a general sense and in a descriptive sense and are not used for the purpose of limitation.< / iframe>

Claims

1. A method for crawling a network using a web crawler, comprising: Get URL seeds and add them to the URL cache; as well as Acquire a crawl target and generate a target vector, wherein one or more words and / or one or more sentences of the crawl target are generated using a BERT model, and one or more images of the crawl target are input into a deep neural network, and outputs are obtained from neurons in an intermediate layer that is neither an input layer nor an output layer of the deep neural network to generate an image vector as the target vector; Get the first category URL from the URL cache, crawl the web page corresponding to the first category URL and extract one or more second category URLs to add to the preload cache; Obtaining a second-category URL from a preload cache, preloading a webpage corresponding to the second-category URL, and generating a webpage vector based on the preloaded webpage corresponding to the second-category URL; If the distance between the generated web page vector and the target vector generated according to the crawling target is less than a specified threshold, the target vector is updated by using the generated web page vector and the target vector, and the second-category URL is added to the URL cache as a first-category URL for subsequent crawling; if the distance between the generated web page vector and the target vector generated according to the crawling target is not less than a specified threshold, the preloaded second-category URL is discarded and is not added to the URL cache as a first-category URL; The preloading of web pages corresponding to the second category URLs and generating web page vectors according to the preloaded web pages corresponding to the second category URLs includes: Extracting one or more text blocks from a preloaded web page corresponding to the second type of URL, wherein the text blocks in the web page are identified by HTML tags; calculating a text block vector based on each of the one or more text blocks; Calculating a statistical value of a vector of each of the one or more text blocks as a webpage text content vector of a preloaded webpage corresponding to the second-category URL; Extracting one or more images from a preloaded web page corresponding to the second type of URL, wherein the images in the web page are identified by HTML tags; calculating a picture vector based on each of the one or more pictures; Calculating a statistical value of each image vector of the one or more images as a webpage image content vector of a preloaded webpage corresponding to the second type of URL; Obtaining a webpage vector of the webpage corresponding to the preloaded second-category URL based on the webpage text content vector and the webpage image content vector; and The step of calculating a vector of a text block according to each of the one or more text blocks comprises: For each text block according to the one or more text blocks, obtaining one or more sentences constituting the current text block; Input each of the one or more sentences into the BERT model instance to obtain a sentence vector corresponding to the input sentence; Calculate the statistical value of the sentence vector of each of the one or more sentences as the vector of the current text block; The vector corresponding to the i-th text block Bi according to the one or more text blocks is recorded as V Bi , Bi=(Si1,Si2,…,Sij,…,Sin i ) indicates that the text block Bi includes multiple sentences, sentence Sij is the jth sentence of the text block Bi, and the sentence vector of sentence Sij is recorded as V Sij , n i is the number of sentences included in the text block Bi, n i 、i and j are positive integers 1<=j<=n i ; The webpage text content vector of the webpage corresponding to the preloaded second type of URL is recorded as P text , Wherein N is the number of text blocks in the web page corresponding to the preloaded second-category URL, and N is a positive integer, 1<=i<=N; The web page image content vector of the web page corresponding to the preloaded second type URL is recorded as P image , Where M is the number of images in the web page corresponding to the preloaded second-category URL, M is a positive integer, VI p is the image vector of the p-th image in the webpage corresponding to the preloaded second-category URL, where p is a positive integer, 1<=p<=M; and The webpage vector V of the webpage corresponding to the preloaded second type of URL page =(P text , P image ); or the webpage vector of the webpage corresponding to the preloaded second type URL V page =P text +P image 。 2. The method according to claim 1, wherein In response to the existence of the first category of URLs in the URL cache, repeatedly executing the method of crawling the network with the web crawler; In response to the presence of the second-category URL in the preload cache, the steps of obtaining the second-category URL from the preload cache, preloading the web page corresponding to the second-category URL and generating a web page vector based on the preloaded web page corresponding to the second-category URL are repeated, and if the distance between the generated web page vector and the target vector generated according to the crawling target is less than a specified threshold, the second-category URL is added to the URL cache as a first-category URL.

3. An information processing device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to claim 1 or 2 is implemented.

Images