Method and system for processing efficient relating subject model data

Abstract

The invention discloses an effective correlated theme model data process method and a system, wherein the method comprises: in task initiation phase, firstly offering an initiation model M0 through a master control node and synchronizing the model to all computation nodes and then dividing task set and distributing the set onto a plurality of computation nodes for computation; in task execution phase, processing a plurality of data, wherein in each turn, firstly processing local parallel computation on working thread of each computation node to obtain theme distribution and model statistics of the node file sub set, and then sending the theme distribution and model statistics to a master control node for collection and judging whether the data process results converge or not. The system of the invention comprises a master control node and a plurality of computation nodes, which form a cluster computer system for computation. The invention can greatly improve computation speed and expand computation aims.

Description

technical field [0001] The invention relates to a text representation method and system thereof, in particular to a high-efficiency data processing method and system based on hidden subject text representation, and belongs to the field of computer information retrieval. Background technique [0002] Computer information retrieval is one of the important infrastructures of the information society, and the services provided run through from basic network information search to information filtering and classification to various advanced data mining. In computer information retrieval, the representation method of text is a fundamentally important issue: first, the processing object of computer information retrieval is mainly text information, and other types of information generally must depend on text information or additional text information to exist; Furthermore, the text representation method is a prerequisite for computer information retrieval services, because the basic m...

AI Technical Summary

Problems solved by technology

[0004]Although the associated topic model provides an ideal means of high-level text representation in terms of function, it is still mainly limited to a small amount of data, and it is difficult to implement it in the real environment. The fundamental reason for the use of large-scale data is that there is a serious bottleneck in its solution method: first, its classic implementation is based on the conventional serial computing method, that is, each step of the computing task must be performed sequentially, and the previous step processing The result is the start of the subsequent processing

In this way, at any point in time, all computing tasks can only be performed on one hardware computing unit, so even if it is placed on a high-performance computer with multiple hardware computing units (such as multi-core, multi-processor), it cannot Speed ​​up task solving

Furthermore, since the calculation process itself cannot be split in the serial mode, the processed data must be gathered together for the calculation process to access at any time, which increases the storage load of the system, such as hard disk, memory, especially memory The impact of this aspect is very obvious. Excessive memory usage will lead to a sharp drop in computing speed and even cause the system to refuse the execution of computing tasks.

Images