Webpage spider theme type search system based on improved cloud platform

Abstract

According to the webpage spider theme type search system based on the improved cloud platform, improvement is carried out on a webpage analysis algorithm HITS algorithm based on a link structure and awebpage analysis algorithm based on theme similarity calculation of a VSM vector space model, and an improved webpage spider model algorithm is provided; according to the method, an overall frameworkmodel based on a Hadoop cloud platform webpage spider is provided, a storage structure of the cloud platform webpage spider is designed and realized on a file system HDFS, and MapReduce algorithm realization is carried out on each functional module based on module division; an improved task allocation algorithm is provided, uniform allocation and the load condition of each crawling child node canbe considered, and the crawling efficiency and accuracy of the cloud platform webpage spider system are improved; results show that the cloud platform webpage spider system based on Hadoop provided and realized by the invention is feasible and effective, the accuracy and efficiency of theme type search can be greatly improved, and theme associated information can be comprehensively, quickly and accurately retrieved.

Description

technical field [0001] The invention relates to a web spider theme search system, in particular to a web spider theme search system based on an improved cloud platform, belonging to the theme search system technical field. Background technique [0002] With the increasing popularity and rapid development of Internet technology, in the face of such a large and diverse information data, general-purpose search engines, as the main means of obtaining information, are far from meeting people's retrieval needs for specific fields or topic-related information. . In view of this, thematic search engines and web spider technologies for specific fields or topics have been widely used. However, with the exponential growth of network information, traditional web spiders that only rely on a single computer for crawling cannot adapt to big data. Thematic search in the environment requires performance and scalability, and the computing framework of the Hadoop cloud platform can solve this...

AI Technical Summary

Problems solved by technology

In view of this, thematic search engines and web spider technologies for specific fields or topics have been widely used. However, with the exponential growth of network information, traditional web spiders that only rely on a single computer for crawling cannot adapt to big data. Thematic search in the environment requires performance and scalability, and the Hadoop cloud platform computing framework can solve this problem very well

Images