Parallel data processing method based on latent dirichlet allocation model

Abstract

The invention discloses a parallel data processing method based on the hidden Dirichlet distribution model, which belongs to the data mining field. The method includes three solutions, including the multi-process parallel processing, the multi-thread parallel processing and the composite multi-process multi-thread processing; the data DM for being processed is divided into data segments in equal or unequal length in the three solutions; each data segment is provided with an index; each computer process / thread processes the corresponding data segment through applying the index, so as to obtainthe subject information of each data item and generate the local sufficient statistic; when the whole DM is processed, the global sufficient statistic is obtained through the merge of the local sufficient statistics so that the current Mi model is obtained through the estimation until the model becomes convergence. The parallel data processing method can utilize the multi-kernel parallel frame ofa single computer and the cluster large-scale parallel capability of multi-computer to realize the high-speed processing of the large-scale text sets and effectively reduce the memory usage during theparallel processing process.

Description

technical field [0001] The invention relates to a text data mining method, in particular to a high-efficiency data processing method based on hidden subject text representation, and belongs to the field of computer data mining. Background technique [0002] computer data mining [0003] Computer data mining refers to the intelligent information processing process that uses computers to obtain effective, useful, and understandable information or knowledge from large amounts of data. Early computer data mining mainly focused on mining regular numerical data in database systems. With the continuous expansion of the Internet scale and the great variety of applications, computer data mining has gradually turned to Internet information processing. The data carried by the Internet is very different from the data in the database system: First, the data on the Internet is mainly text written in natural language, while the data in the database system is mainly numerical; It mainly s...

AI Technical Summary

Problems solved by technology

[0005] Common text data representation methods can be divided into two main types, one type is only the text data representation method of basic word information, and its representative is the vector space method (Vector Space Model) (reference: Salton, G. The SMART Retrieval System. Englewood Cliffs, Prentice-Hall, 1971.), but this kind of method has the problems of large information loss and high data sparsity, which are not good for text information processing

Images