Spark-based extreme learning machine parallelization calculation method

Abstract

The invention discloses a big data processing platform Spark-based extreme learning machine parallelization calculation method which is applicable to being used in the field of machine learning of a big data processing platform Spark. The method comprises the following steps: firstly, saving a sample dataset into a distributed file system row by row according to each sample, and preprocessing the sample set to obtain a characteristic matrix and a classification label vector T; secondly, randomly initializing hidden node parameters, namely a weight matrix omega and a hidden layer deviation vector b, according to the number of sample characteristics and the number of hidden nodes; thirdly, solving a matrix multiplication parallelization solution to obtain a hidden layer output matrix H; finally, performing calculation by using the matrix multiplication parallelization solution and matrix single inverse operation to obtain a unique optimal solution of a weight vector beta. The steps are simple, and the calculation amount is small; the calculation step can be carried out in parallel on multiple computers, so that the calculation efficiency is effectively improved; meanwhile, the fault tolerance is high, and the efficiency for model training of an extreme learning machine algorithm under a big data background is greatly improved.

Description

technical field [0001] The invention relates to a parallel computing method, in particular to a Spark-based extreme learning machine parallel computing method used in the field of big data processing platform Spark machine learning computing. Background technique [0002] Machine learning is one of the most popular research fields at present. In recent years, with the continuous growth of data volume, the efficiency of machine learning has attracted much attention, and the learning efficiency of neural networks needs to be solved urgently. For the extreme learning machine algorithm, because it randomly initializes the hidden node parameters and directly obtains the hidden node output weight through matrix operations, there is no large number of iterative operations in the traditional learning algorithm, which greatly improves the operation speed at the algorithm level. [0003] Since the data to be processed in the extreme learning machine needs to be loaded into the memory ...

AI Technical Summary

Problems solved by technology

[0005] Purpose of the invention: In order to overcome the problem that the efficiency of the current extreme learning machine parallelization scheme is still not high, the present invention provides a Spark-based extreme learning machine parallelization scheme, which can make the extreme learning machine run at the same efficiency when processing large data. Compared with the existing hadoop-based parallelization scheme, it is greatly improved

Images