Self-adaptive data extraction method of structure change webpage

Abstract

The invention provides a self-adaptive data extraction method for a structural change webpage, which comprises the following steps of: acquiring webpage extraction data, and judging whether the webpage extraction data is abnormal or not; when the webpage extraction data is abnormal, selecting a plurality of extracted webpages before the abnormality, and obtaining the extracted contents of the extracted webpages; obtaining core semantics of the extracted content according to the extracted content; carrying out similarity comparison on the core semantics and the information of each node of the current webpage after abnormality, and obtaining a most matched node; repeating the above steps until the similarity judgment of the contents of all the selected extracted webpages and the corresponding current webpages is completed; according to a similarity judgment result, determining all content positions in the current webpage corresponding to the extracted content, further acquiring an acquisition path, and finishing self-adaptive data extraction. According to the method, the anti-risk capability of the automatic webpage extraction method is improved, higher stability is achieved, and themethod can cope with complex and changeable network environments.

Description

technical field [0001] The invention relates to the field of computer applications, in particular to an adaptive data extraction method for web pages with structural changes. Background technique [0002] The data structure in a webpage is generally hidden in each webpage, and there are a large amount of one-class data in a webpage with the same structure. There are already many webpage data extraction methods in the prior art, among which, the simplest method is for each type of website Writing a specific template, but the workload of this method is huge. In order to reduce the workload, in 2010, three people including Nie Tiezheng from Northeastern University published a web page data extraction method based on Extensible Markup Language query. This method The workload of web page data extraction is reduced and the automation is improved. In 2014, Huang Yihua of Nanjing University and others published a full-scale web information extraction integration method. This method ...

AI Technical Summary

Problems solved by technology

[0003] At present, although the automatic web page data extraction method is mature, there is no solution specifically for the risk resistance and stability of the automated web page extraction method. Once the data structure of the same type of web page changes during the data extraction process, the previously automatically generated The data extraction method will become meaningless, so a new technical means is needed to improve the risk resistance and stability of the automated web page extraction method

Images