Keyword-based oriented webpage collection method

Abstract

The invention relates to a keyword-based oriented webpage collection method. The method comprises the steps of introducing a text weighting algorithm to set a weight for a keyword; calculating a webpage theme relevancy in combination with a spatial vector model algorithm; judging the importance of a webpage by use of a webpage link structure and the theme relevancy; gathering related subject webpage files according to a text clustering algorithm; calculating a probability of a to-be-grabbed webpage which belongs to the subject theme by use of a naive bayesian algorithm; setting a fitness function and screening a webpage related to the theme; and dynamically adjusting a system model according to the real-time crawling situation of the webpage. Based on a distributed platform, the oriented crawling of the theme webpage is realized by use of an adaptive theme algorithm in combination with an open source network collection architecture; parallel webpage crawling is realized by use of a distributed technology, each node calculation resource is fully utilized and the crawling rate of the webpage is improved.

Description

Technical field [0001] The invention relates to a keyword-based directional webpage collection method, which belongs to the cross-technical field of subject web crawlers and distributed computing. Background technique [0002] With the rapid development of information technologies such as electronic computers, storage devices, and mobile communication networks, the rapid popularization of mobile Internet, social networking, and the Internet of Things has led to the rapid growth of data on Internet platforms, and the era of big data has come. According to statistics, as of mid-March 2016, the total number of known web pages (excluding hidden web pages) on the Internet alone has exceeded 4.6 billion globally, and how to efficiently collect network data becomes particularly important. [0003] Data collection is the prerequisite for subsequent data mining, analysis and decision-making. The capture efficiency of network data collection directly determines the effect of data processing....

AI Technical Summary

Problems solved by technology

[0005] In the traditional face, the current network data directional collection technology mainly has the following problems: (1) The topic crawler needs to judge the relevance of the page before saving the web page, and only saves the web page related to the topic

At present, most of the topic identification methods are based on classifiers. The accuracy and efficiency of classifiers are very low, and it is difficult to accurately capture the topic web pages.

(2) The main problem of the topic collection algorithm based on the link structure is that the calculated link value has little correlation with the topic, which is easy to cause "topic drift", and collects web pages that have nothing to do with the topic, and the topic search strategy based on the link content evaluation low efficiency

(3) In the current distributed directional data acquisition system, frequent communication between nodes is required, and the scalability of the system is not high

Images