Adaptive information extraction method for webpage characteristics

Abstract

The invention discloses a method for extracting information from an academic home page. The method comprises the following steps of: (1) finding an academic home page from Internet; (2) crawling and analyzing the academic home page, wherein the crawling of an irrelevant page is reduced by using a heuristic strategy so as to accelerate analysis speed; (3) analyzing the page into a form of documentobject module (DOM), and dividing according to attributes and contents of elements so as to acquire a cohesive text unit list; (4) identifying the text unit by using an information recognizer, wherein each information recognizer only identifies one information type, and performing subfield extraction on the text information; (5) performing association analysis on the extraction result, eliminating different meanings by using the association of the information, and complementing the missing field; and (6) matching the extraction result and a database, and eliminating the redundant data, wherein the extraction result is stored in a semantic database in a form of semantic data. In the method, by combination of heuristic rules, a machine learning method and a conditional probability model, academic information can be extracted efficiently and accurately from the academic home page.

Description

technical field [0001] The invention belongs to the field of information extraction systems, and in particular relates to an information extraction method adaptive to web page features. The method is especially suitable for extracting information such as author names, email addresses, institution information, and published articles from academic homepages. Background technique [0002] With the advent of the information age, the Internet has gradually become the main way for people to share and obtain information. Various information is published on the Internet in the form of web pages for people to read. However, with the explosive growth of Internet information, people find it more and more difficult to find the information they need on the Internet. On the one hand, the amount of information is huge, and on the other hand, the way information is presented is very flexible and free, which increases people's ability to identify targets. The cost of information. Therefore,...

AI Technical Summary

Problems solved by technology

(3) Redundancy, the same information may appear repeatedly on multiple sites

This method has a certain automatic extraction ability, but because the bottom layer still relies on the regularization method, it cannot achieve high accuracy for complex extraction tasks

Images