Webpage data capturing and filtering method

Abstract

The invention discloses a webpage data capturing and filtering method, which comprises the following steps: a) obtaining webpage Html codes; b) converting the webpage Html codes into an XML (extensive makeup language) sequence table; and c) performing fuzzy matching on the XML sequence table by using information key words to obtain webpage data. According to the webpage data capturing and filtering method provided by the invention, the webpage Html codes are comprehensively subjected to XML serialization at first, and the webpage data in XML files are obtained in a fuzzy filtering way, so that massive amounts of webpage data can be captured and filtered quickly and more accurately.

Description

technical field [0001] The invention belongs to the technical field of computer databases, and in particular relates to a method for capturing and filtering webpage data. Background technique [0002] Web crawling and data extraction technology has a long history of development, through various technical means to achieve the purpose of collecting web content, and, in each stage of computing technology development, people are trying to use more advanced technology and programming language to achieve more A powerful website content acquisition tool, the existing common web page data capture methods are as follows: [0003] 1. The method of web crawling and data extraction using web crawler technology [0004] Web crawler is Web Spider, which is a very vivid name. Comparing the Internet to a spider web, then a spider is a spider crawling around on the web. Web spiders search for web pages through the link addresses of the web pages. Starting from a certain page (usually the ...

AI Technical Summary

Problems solved by technology

Obtain the required information by filtering the information on the webpage (for example, BBS needs to capture the article title, author, posting time, hits, replies, etc.), such as making regular expressions for filtering according to the rules of Html language tags in the webpage , so that the above information can be accurately filtered out, but when the area to be captured is very wide, such as covering nearly 80,000 forums, more than 200 large news websites, and many well-known search engines, blogs, post bars, etc. website, so the amount of daily data updates is very large, it is impossible to independently create a set of tailor-made filtering programs for each BBS and each section of each website, even if it is produced at a cost, with a large number of websites. The revision of the crawling program must also be accurately modified accordingly. This kind of maintenance workload and this update speed are obviously unrealistic

[0023] It can be seen that, usually, the crawling program made by the existing general method can only capture data for individual websites or a small number of websites in order to achieve accurate filtering of data capture. Considering the production of too many data matching programs and the Maintenance, it is impossible to crawl unknown massive websites, so it is necessary to provide a new webpage data crawling and filtering method

Images