A Web Data Record Extraction Method Based on Incomplete Subtree Matching

Abstract

The invention discloses a Web data record extraction method based on incomplete subtree matching. The Web data record extraction method comprises the following steps: downloading hypertext markup language (HTML) source codes according to hyper text transport protocol (HTTP), and encoding the downloaded characters according to UNICODE; filtering noise, and marking information; analyzing the HTML source codes by using assembles of NEKO or HTMLParser and the like, and constructing Document trees of webpage; extracting candidate subtree sets; matching incomplete subtrees; and determining data record set. The method is based on the subtree matching and does not depend on a template structure of the webpage so as to have high generality. Through label filtering and determination of the candidate subtrees, performance of a data extraction process can be improved effectively. Based on intercepted incomplete subtree matching, the method judges similarity between subtree structures, can effectively eliminate structural difference caused by the fact that a temperate is filled by data, and improves accuracy of data record extraction.

Description

technical field [0001] The invention relates to a method for extracting Web data records based on incomplete subtree matching. Background technique [0002] With the rapid development of the Internet and the continuous improvement of Web technology, more and more organizations and individuals distribute information to the Internet. Every day, thousands of web pages are generated on the Internet, and the Internet has become a huge "library library" for information sharing. How to find and extract effective data information from massive Web information has become an important topic. [0003] HTML webpage is one of the most important data formats on the Internet. It is a label language, which is displayed by the browser after combining scripts and styles. The essence of HTML is a semi-structured language, which is suitable for browsing by humans after being rendered, but it is not conducive to identifying and extracting data by computer programs. In the definition of HTML ta...

AI Technical Summary

Problems solved by technology

[0013] The statistics-based method for this type of page is no longer applicable, because the statistical method generally needs to use the statistical information of a long text, and the webpage of the data record type does not meet this characteristic

The rule-based method requires a large training data set. Manually labeling web pages is a rather labor-intensive process, and the rules generally apply to one website. It is unrealistic to obtain a general rule with high accuracy for data extraction from multiple websites. of

Currently, the most widely used method is the method of manual programming. This method has high accuracy, but its outstanding disadvantages are that it consumes a lot of manpower and is difficult to maintain.

For each website, the corresponding extraction code must be written. In the case of a revision of the target website, the program failure is not easy to detect, and the code still needs to be changed after detection

Images