Automatic extraction method oriented to data of deep web pages

Abstract

The invention discloses an automatic extraction method oriented to data of deep web pages, and belongs to the field of computer data mining. The automatic extraction method includes obtaining two deep web pages of the same website at first, and respectively marking the two deep web pages as a first page and a second page; converting HTML (hypertext markup language) documents of the first page and the second page into XHTML (extensible hypertext markup language) documents; then removing noise of the first page and the second page; eliminating repeated modes of the first page and the second page to generate a webpage data extraction wrapper; removing noise of the page with the data to be extracted at first when the page is extracted; marking the page by the webpage data extraction wrapped after the noise of the webpage is removed, and finally extracting the marked page. By the aid of the automatic extraction method, efficiency of a repeated mode elimination algorithm and efficiency of a matching algorithm are improved, extraction complexity is reduced, the matching algorithm and an extraction algorithm, which are designed according to characteristics of the repeated mode elimination algorithm, in the method are simple and speedy in process, and data extraction accuracy is improved.

Description

technical field [0001] The invention belongs to the field of computer data mining, in particular to a method for automatically extracting page data facing deep webs. Background technique [0002] At present, in the field of deep web page data extraction research, many automated or semi-automated data extraction tools have been developed. Among them, there is a method of obtaining web page data extraction wrappers by learning and using manually marked sample pages. Such methods require a lot of manual participation. , leading to a low degree of automation, difficult generation and maintenance of wrappers, and a data extraction method based on vision. This method overcomes the dependence of existing methods on the original HTML page to a certain extent, but due to the It is also very difficult to obtain accurate visual information due to semi-structured or unstructured features. At the same time, there is an automatic data extraction method, which can automatically summarize t...

AI Technical Summary

Problems solved by technology

[0002] At present, in the field of deep web page data extraction research, many automated or semi-automated data extraction tools have been developed. Among them, there is a method of obtaining web page data extraction wrappers by learning and using manually marked sample pages. Such methods require a lot of manual participation. , leading to a low degree of automation, difficult generation and maintenance of wrappers, and a data extraction method based on vision. This method overcomes the dependence of existing methods on the original HTML page to a certain extent, but due to the It is also very difficult to obtain accurate visual information due to semi-structured or unstructured features. At the same time, there is an automatic data extraction method, which can automatically summarize the extraction rules and then extract data. It is especially suitable for the extraction of data-intensive pages, but , the backtracking method is used to deal with the repeated pattern problem when obtaining the web page data extraction wrapper from the web page, resulting in exponential time complexity, and because the existing automated data extraction tools do not preprocess the page, resulting in inefficiency high

[0003] Therefore those skilled in the art are committed to developing a method for eliminating repetitive patterns for automatic extraction of deep web page data, so as to improve the low efficiency and low accuracy in the existing automated data extraction technology

Images