Web page structural data extraction method and system

Abstract

The invention relates to a web page structural data extraction method which is characterized by comprising the following steps: choosing a training web page content set and extracting target structural data; training the training web page content set to obtain a regular expression matched with the target structural data; writing the regular expression in a configuration module; utilizing the configuration module to collect a web page; and extracting the structural data from the collected web page. The web page structural data extraction method and system can realize the structural data extraction of static web pages and dynamic web pages and are suitable for acquiring various types of website information and extracting the structural data.

Description

technical field [0001] The invention relates to a network information collection technology, in particular to a system and method for extracting structured data of web pages, and belongs to the technical field of network information. Background technique [0002] With the development of network information technology, the information of websites, forums, blogs and other web pages is getting larger and larger. Search engines, content analysis, public opinion analysis and other technologies all analyze and process this information. They all use the web crawler. a technology. A web crawler, also known as a web spider, is a data collection method that automatically analyzes web page connections and automatically obtains information and stores it locally. At present, not only search engines, but also many applications use web crawlers as the main source of data collection, such as intelligent analysis of network content; not only traditional keyword retrieval, but also structure...

AI Technical Summary

Problems solved by technology

However, the production of these configuration files often requires personnel who are very familiar with web design, which requires the user's computer software knowledge and limits the scope of users.

[0007] 4. Different websites have different website structures such as channels, sections, depth, advertisements, and user attention information. In view of this situation, there is currently no web crawler development that allows users to choose independently, and then automatically collects relevant pages;

[0008] 5. A large number of javascript scripting languages ​​are used on web pages, and current web crawlers still lack effective methods for how to extract structured data controlled by javascript

[0009] It can be seen that with the continuous expansion of the application range of web crawlers, higher requirements are put forward for the structured data extraction of web pages, and the existing web crawler technologies and products cannot fully meet the needs of more intelligent data collection. Require

Images