Construction of hotspot mining system under Hadoop framework

Abstract

The invention relates to construction of a hotspot mining system under a Hadoop framework, which comprises the following specific steps of acquiring data information A from a network by using a cloudcomputer Hadoop cluster module and preprocessing the data information A to obtain preprocessed data information B and sending the preprocessed data information B to the mining system; performing wordsegmentation on the preprocessed data information B to obtain a keyword set C; screening each keyword D in the keyword set C through a previous hotspot information word bank; sorting the keywords D from high to low, screening out hot words E, and constructing a hot word set F; constructing a word co-occurrence network according to the correlation among the hot words D in the hot word set F; and dividing the hot word set F by adopting a clustering algorithm according to the word co-occurrence network to obtain a hot topic set. According to the method, the hot topic set can be quickly obtained,and the accuracy of obtaining the hot topics from the network can be improved.

Description

technical field [0001] The invention relates to the technical field of language data processing, in particular to the construction of a hotspot mining system under the framework of Hadoop. Background technique [0002] With the rapid development of the Internet, more and more users participate in the Internet. Users can publish information on the Internet or obtain the information they need. The information on the Internet is increasing every day; And paying attention to key hotspot information is, brings trouble; Now more and more users pay close attention to hotspot, but people often cannot effectively obtain the hotspot information that oneself needs from the network; For this reason, propose a kind of Hadoop framework in this application The following hot topic mining system is constructed to help users quickly obtain hot topic collections from the Internet, and to improve the accuracy of obtaining hot topics from the Internet. Contents of the invention [0003] (1) P...

AI Technical Summary

Problems solved by technology

[0002] With the rapid development of the Internet, more and more users participate in the Internet. Users can publish information on the Internet or obtain the information they need. The information on the Internet is increasing every day; And paying attention to key hotspot information is, brings trouble; Now more and more users pay close attention to hotspot, but people of...

Images