Parallel autoencoder based feature learning method and system

Abstract

The invention provides a parallel autoencoder based feature learning method. The method comprises that 1) a managing machine executes Map operation, plans tasks for different working machines, and issues the tasks to each working machine, the tasks of the different working machines are consistent, the task includes training a weight matrix of the autoencoder on the basis of input data, and the weight matrix comprises all weights of the autoencoder; 2) each working machine reads part of data set corresponding to the working machine; 3) the working machines execute the tasks issued by the managing machine in parallel, train the weight matrixes of the autoencoder, and back feed the trained weight matrixes to the managing machine; and 4) the managing machine executes Reduce operation, and the weight matrixes back fed by the working machines are averaged arithmetically. According to the invention, feature learning can be realized more efficiently, and the time complexity of data processing of a parallel autoencoder is reduced from secondary complexity to linear complexity.

Description

technical field [0001] The present invention relates to the technical field of data mining. Specifically, the present invention relates to a feature learning method and system based on a parallel automatic encoding machine, which can be applied to multiple fields such as finance, communication, image processing, Web data analysis, and big data analysis. industry. Background technique [0002] With the informatization and networking of social development, information technology has increasingly affected all aspects of social life, and the development of computers has greatly improved social production efficiency. More and more people rely on computers and the Internet to improve living standards and work efficiency, continuously transform society and improve the quality of life. However, with the gradual application of computers in various fields, a large amount of information and data will inevitably be generated. And these information and data are not just data, the infor...

AI Technical Summary

Problems solved by technology

However, when the data scale increases, the computing power required by this parallel autoencoder scheme will increase non-linearly. Therefore, when this scheme faces massive data computing tasks, it is often difficult to meet the requirements in terms of data processing efficiency and time.

Images