Distributed vertical crawler method and terminal equipment

Abstract

The invention discloses a distributed vertical crawler method and terminal equipment. The distributed vertical crawler method comprises the following steps: A, crawling webpage resource data by a webcrawler in a data crawling unit; B, preprocessing the captured webpage resource data; C, classifying the preprocessed webpage resource data to obtain classified data; D, transmitting the classified data to a data analysis unit for data analysis; E, transmitting the analyzed data to a storage unit for encrypted storage; and F, finally transmitting the encrypted and stored data to a background monitoring terminal. The webpage resource data can be quickly captured, preprocessed, classified and encrypted, data processing efficiency is improved, safety is high, and data leakage is avoided.

Description

technical field [0001] The invention relates to the technical field of information retrieval, in particular to a distributed vertical crawler method and terminal equipment. Background technique [0002] With the rapid development of the network, the World Wide Web has become the carrier of a large amount of information, how to effectively extract and use this information has become a huge challenge. As a tool to assist people in retrieving information, search engines become the entrance and guide for users to access the World Wide Web. However, existing general search engines also have certain limitations, such as: users in different fields and backgrounds often have different retrieval purposes and needs, and the results returned by general search engines include a large number of web pages that users do not care about; The goal of a general search engine is to cover the network as much as possible, and the contradiction between limited search engine server resources and u...

AI Technical Summary

Problems solved by technology

However, existing general search engines also have certain limitations, such as: users in different fields and backgrounds often have different retrieval purposes and needs, and the results returned by general search engines include a large number of web pages that users do not care about; The goal of a general search engine is to cover the network as much as possible, and the contradiction between limited search engine server resources and unlimited network data resources will be further deepened; the abundance of data forms on the World Wide Web and the continuous development of network technology, pictures, databases, A large number of different data such as audio, video and multimedia appear, and general search engines are often powerless to these data with dense information content and certain structure, and cannot be well discovered and acquired; most general search engines provide keyword-based retrieval, which is difficult to support semantic Inquiries made for information, etc.

At present, when the amount of data that web crawlers need to crawl is huge, the crawling efficiency of the existing distributed crawler architecture is low, so it is necessary to improve

Images