Regular expression based URL filtering method

Abstract

The invention discloses a regular expression based URL filtering method. The method comprises: step 1, obtaining a first URL required to be crawled, and crawling a page corresponding to the first URL; step 2, displaying text content of the page corresponding to the first URL and a plurality of second URLs, and prompting a user to input a URL filtering rule and a text filtering rule; step 3, in response to the URL filtering rule submitted by the user, filtering the plurality of second URLs by applying the URL filtering rule to obtain one or more third URLs; and step 4, adding the one or more third URLs into a crawling queue.

Description

Technical field: [0001] The invention relates to the technical field of network information processing, in particular to a method for obtaining a network crawling scheme by using a crawler program to capture a user-defined network crawler. Background technique: [0002] With the development of Internet technology, search engines have become an important way for people to obtain information. Existing search engines are all implemented based on a technology commonly known as a web crawler (Crawler). When a web crawler crawls, it is difficult to delete the valuable information you want. There are chrome-plated URL links in the crawled webpage. It is difficult to judge whether the link is a catalog page or a detailed page through the source code of the webpage. The working principle of the crawler is that the search engine regularly executes the web crawler program, starts from the specified initial URL list as the root of the search tree to access the webpage resources located...

AI Technical Summary

Problems solved by technology

[0003] At present, most web crawler programs are based on the page structure to obtain data. By obtaining the web page document, the web page document is parsed into a DOM tree form. According to the rules of the DOM tree, all elements in the HTML document are represented by nodes, and the extraction is constructed according to the DOM tree. Rules for data extraction. During the data extraction process, due to the heterogeneity of web page information sources, in order not to lose the extraction accuracy, it is necessary to construct corresponding extraction rules for each website, so that the coverage rate of the crawler program is very low and extremely large. The possibility of obtaining network resources is limited; the network page acquisition technology based on the DOM tree can improve the efficiency of data acquisition and the utilization rate of system resources, but in the process of data extraction, it will depend on the specific label nodes of the page. Once the corresponding page structure Changes or new label node naming rules will not only prevent the crawler from obtaining correct data, but will further cause the crawler to stop data acquisition

Images