Task configuration method for heterogeneous distributed machine learning cluster

Abstract

The invention discloses a task configuration method for a heterogeneous distributed machine learning cluster. The method comprises the following steps: constructing the heterogeneous distributed machine learning cluster; generating a training set and a prediction set; pre-training the convolutional neural network; generating a random forest training sample subset of the parameter server; constructing a random forest model; generating an inferred training time for each node; configuring a task for each node; updating the pre-trained convolutional neural network; training a convolutional neural network; and when the number of training times of the convolutional neural network corresponding to each node reaches the maximum number of times, reconfiguring node tasks for node resource feature parameter changes existing in the distributed machine learning cluster. According to the method, the matching degree of the task configured for each node in the heterogeneous distributed machine learning cluster by the parameter server and the resource of the node is improved, and the task can be dynamically reconfigured for each node according to the resource change of each node.

Description

Technical field [0001] The present invention belongs to the field of physical technology, and more is a further involve a task-oriented configuration method for a shared machine learning cluster in the field of distributed machine learning. The present invention can be used in a large-scale heterogeneous distributed machine learning cluster through reasonable task configuration to make full use of the cluster node to calculate resources. Background technique [0002] With the arrival of the big data era, the data set for training machine learning models presents the trend of explosion growth. Fast training, dynamic flexible distributed machine learning clusters have become an inevitable trend of large-scale machine learning development. Distributed Machine Learning Through the parameter server to put the model training task to a large number of computing performance cluster nodes, shorten the time of the entire model training. However, most existing node resources are inconsisten...

AI Technical Summary

Problems solved by technology

[0005] The purpose of the present invention is to address the above-mentioned deficiencies in the prior art, and propose a task configuration method based on heterogeneous distributed machine learning clusters, which is used to solve the problem of low resource utilization of heterogeneous distributed machine learning cluster nodes and heterogeneous parameter servers. The problem of large time overhead for each node configuration task in a distributed machine learning cluster

The present invention dynamically configures tasks for each node by replacing the characteristic parameters of the node resources before the change with the characteristic parameters of the changed node resources. Compared with the traditional method, it does not need to count the number of tasks completed by each node training, and solves the parameter server problem. The problem of high time overhead for configuring tasks for each node in a heterogeneous distributed machine learning cluster

Images