Web text data crawler method and system

Abstract

The invention discloses a web text data crawler method and system. The method comprises the steps that a target keyword is input into a keyword retrieval bar of a crawler system, and the crawler system automatically calls a search engine to obtain a search result page; analyzing the entry websites in the search result page, and deleting the entry websites belonging to the set website type according to an analysis result to obtain a target object list; according to the association degree between the target keyword and the description text in the target object list, filtering the deleted entry websites in the target object list; analyzing the webpage corresponding to the filtered entry website by adopting a webpage analysis library to obtain a text content list of each webpage, and filtering the text content in the text content list according to the text probability distribution and the text length; and screening paragraphs and sentences of the text content in the filtered text content list according to the target keyword to obtain web text data of the target keyword. According to the invention, the crawling precision and the crawling efficiency can be improved.

Description

technical field [0001] The invention relates to the technical field of web crawlers, in particular to a web text data crawler method and system. Background technique [0002] A web crawler, also known as a web crawler, is a program or script that automatically grabs information on the Internet according to certain rules. Nowadays, there is a huge amount of information stored in the World Wide Web, how to effectively extract this information and how to use it has become a major challenge. Web crawlers access network resources by imitating browsers, and then use regular expressions, XPath and other data extraction libraries, as well as specific rules to automatically obtain specific target information in web pages. Web crawlers are currently widely used in search engines, public opinion monitoring, and data analysis. [0003] At present, the network text data crawler method has following several: 1, be used for the method, device and computer program of collecting data from ...

AI Technical Summary

Problems solved by technology

[0004] The above network text data crawler technology has the following disadvantages: 1. The data source is single, and the content information of the pages is often crawled with a certain website as the root directory, which greatly reduces the The amount of network text data obtained, the crawling accuracy is not high

At the same time, some useless web content such as pictures, videos, and maps will be processed, which reduces the efficiency of data crawling

Images