Topic web crawler method and system based on text classification

Abstract

The invention provides a topic web crawler method based on text classification. The method comprises the following steps: S1, receiving a topic, and initializing a URL task queue; S2, taking out a crawler task from the URL task queue, and obtaining network document content; S3, obtaining the classification of the network document content; S4, judging whether the classification is the same as the theme or not; if yes, extracting the URL in the network document, and adding the URL into a URL task queue; if the URL task queue is different from the URL task queue and there is a task in the URL task queue, executing the step S2; and S5, circularly executing the steps S2-S4 until no task exists in the URL task queue. According to the invention, the classification model can learn the features of the web text, and the classification accuracy of the classification task can be effectively improved.

Description

technical field [0001] The invention relates to the fields of artificial intelligence and deep learning, in particular to a method and system for a subject web crawler based on text classification. Background technique [0002] The rapid growth of the scale of the network, while bringing extremely rich information to people, has also brought great challenges to the retrieval of this information. Even the very large-scale Baidu and even Google search engines are also very difficult Information coverage is less than 50%. And the search server resources are far behind the ever-increasing speed of the network scale. If the traditional information crawling method is still adopted, it will only make the coverage of information retrieval smaller and smaller. [0003] General search engines use crawler programs to retrieve websites, such as Google, Baidu and other large search engines for all users, use seed pages as the starting point of search, and try to traverse the entire net...

AI Technical Summary

Problems solved by technology

[0002] The rapid growth of the scale of the network, while bringing extremely rich information to people, has also brought great challenges to the retrieval of this information. Even the very large-scale Baidu and even Google search engines are also very difficult Information coverage is less than 50%

However, for a specific topic, general search engines have problems such as large information redundancy, high memory usage, consumption of system resources, low precision and weak personalized requirements.

Images