A distributed topic discovery method and system for big data

Abstract

The invention relates to a distributed subject finding system and method for big data. The system and the method comprise three parallel processing processes. Each process is composed of one or more of a mapping functional module, a combination functional module and a simplification functional module. Normalization is carried out on the feature vector of each input document, and the number of times, occurring in the documents, of each word forming the documents is counted; at the end of each mapping functional module, each document is used as an original cluster, frequency of the documents is counted, at the end of each combination functional module, local clustering is carried out on the original clusters generated by ends of the mapping functional modules so that local clusters can be generated, and at the end of each simplification functional module, clustering is carried out on the local clusters generated by combination of a plurality of far end physical nodes so that overall clusters can be generated; grading and sorting are carried out on internal keywords of the overall clusters generated in the second mapping / simplification process, and K high-mark keyword expression subjects needed by users are output. Accordingly, the TB level data are processed, the calculation capacity is improved in a linear mode, the distributed calculation can be truly achieved, and the performance and the efficiency are improved.

Description

technical field [0001] The present invention relates to the Internet industry, the news media industry, and the Web big data analysis industry, and in particular to a distributed topic discovery method and system for big data. Background technique [0002] The main task of topic discovery is to aggregate a large number of news reports discussing the same event or related topics under the same cluster to reduce duplication and redundancy. For governments and telecom operators, massive news and comment topic discovery technology can help them understand social conditions and public opinions faster and in real time. According to the processing process, topic discovery can be divided into the following steps: event-related web page crawling, web page text parsing, text word segmentation, dictionary generation, text modeling, and text single-path incremental clustering. Event-related web page crawling is to use crawler tools to capture original Internet information related to cu...

AI Technical Summary

Problems solved by technology

[0008] In order to solve the above problems, the purpose of the present invention is to solve the problems that the traditional single-path incremental clustering method cannot handle massive data, relies heavily on the input order and the imperfect keyword scoring mechanism, and proposes a distributed theme for big data Discover the core method process, improve it based on the traditional single-path method process, use the big data processing framework Hadoop and Map / Reduce mechanism, and perform local clustering on smaller data blocks after segmentation on multiple Mapper ends, reducing the pressure on the memory and calculation of a single machine , re-aggregate the generated classes on the Mapper side at the Reducer side to form a global clustering; use randomly extracted documents as the original clustering seeds to solve the problem of relying heavily on the input order; use an improved scoring mechanism to take into account both keyword frequency and document frequency , to reduce the interference of very few "abnormal documents"

Images