Method and device of excavation of subject of text big data based on characteristic space decomposition

Abstract

The invention relates to a method and a device of excavation of a subject of text big data based on characteristic space decomposition. The method comprises two associated parts: one part is a space decomposition method based on the subject characteristic, the second part is an acceleration method based on model solution of multiple sub-spaces. The key of the space decomposition method is to utilize model characteristics to decouple the data samples and the subject assemblies, and therefore segmentation and decomposition of the data space and the subject space are achieved simultaneously, a plurality of sub-model spaces smaller than a full model space are obtained, and complexity of a storage space of a calculation solution algorithm is effectively reduced. At the same time relevant independence among the sub spaces can be utilized simultaneously to reflect the sub spaces to all kinds of parallel entities, and therefore time complexity of the calculation algorithm is effectively reduced. The method of the excavation of the subject of the text big data based on the characteristic space decomposition is capable of sufficiently utilizing parallel processing capability of a calculation device, and achieving parallel expansion processing of large-scaled subject modeling spaces and large-scaled data assemblies.

Description

technical field [0001] The invention belongs to the technical field of computer data mining, and in particular relates to a high-efficiency data processing method and device for topic mining text big data based on a topic analysis model, which is used for typical big data such as Internet web pages and large digital library documents. Perform topic analysis and mining efficiently. Background technique [0002] Computer data mining technology is mainly an intelligent information processing process that uses computers to mine human-understandable information or knowledge from large amounts of data and utilizes them. The rise of the knowledge economy and the vigorous development of the Internet have created an urgent need for computer data mining technology. The main carrier of knowledge is the text of natural language, and the text without further processing is just raw data, which must be analyzed semantically so that the computer can obtain useful knowledge from it. The ke...

AI Technical Summary

Problems solved by technology

[0006] Due to the above characteristics, the probabilistic topic model analysis technology is limited to small-scale feature sets and small-scale topic sets, and it is difficult to efficiently perform topic analysis on typical big data such as Internet web page information and large digital libraries.

Images