Parallel topic mining method and device

Abstract

An embodiment of the invention provides a parallel topic mining method and device. The method comprises steps as follows: a first node of the parallel topic mining device receives a second word-topic submatrix sent by a second node and a second remainder submatrix, wherein the second remainder submatrix comprises a row whose row accumulated value is the largest in a remainder matrix as well as a column whose column accumulated value is the largest; the second word-topic submatrix comprises a row, corresponding to the row number of the row with the largest row accumulated value in the remainder matrix, in a word-topic matrix as well as a column, corresponding to the column number of the column with the largest column accumulated value in the remainder matrix, in the word-topic matrix; a first word-topic submatrix is updated according to the second word-topic submatrix, a first remainder submatrix is updated according to the second remainder submatrix, and the updated first word-topic submatrix and the updated first remainder submatrix are sent to the second node. Accordingly, the communication capacity in the topic mining process is reduced, and the topic mining speed is increased.

Description

technical field [0001] The invention relates to computer technology, in particular to a parallel topic mining method and device. Background technique [0002] Mining semantically related word clusters from massive document or image collections is called topic mining. Usually, the bag-of-words matrix is ​​used to represent the document collection and input into the latent Dirichlet Allocation system (Latent Dirichlet Allocation, referred to as: LDA), and the document-topic matrix and word-topic matrix are estimated and output through automatic reasoning methods. [0003] Existing parallel topic mining methods use large-scale multi-processor clusters, which include a parent computing node and multiple child computing nodes, and the child computing nodes and parent computing nodes are usually connected and communicated through a network. When performing topic mining, LDA first divides the input bag of words matrix evenly according to the document index, and inputs them to each...

AI Technical Summary

Problems solved by technology

[0004] When doing large-scale topic mining, the communication module in LDA needs to upload and download all the elements of the word-topic matrix in each cycle. The communication volume is large, resulting in a large time delay, and the efficiency of parallel topic mining is low.

Images