A kind of machine learning tool middleware and machine learning training method

Abstract

A machine learning tool middleware and a machine learning training method. The machine learning tool comprises at least one training unit, wherein each training unit is provided with a middleware combined with a machine learning tool. Each middleware comprises a low-level communication module and at least one of a data distribution module, a model parameter updating module, a training parameter adjusting module and a training cessation determining module. The training unit distributes required data from a data storage device to a storage unit accessible to each training unit through the middleware, so that each training unit reads the data from the storage unit to implement the training, completes the update of model parameters of the training unit and the adjustment of training parameters of each training unit through communication between corresponding modules of the middleware, and determines whether to cease the training according to the training information of all training units. The middleware is responsible for required processing for large-scale parallel training, and is convenient for expanding various machine learning tools.

Description

technical field [0001] The invention belongs to the technical field of machine learning, and in particular relates to a machine learning tool middleware and a machine learning training method. Background technique [0002] Machine learning is a branch of artificial intelligence, and in many cases, it has almost become synonymous with artificial intelligence. Simply put, machine learning is to use machine learning algorithm models to enable machines to learn laws from a large amount of historical data, so as to intelligently identify new samples or make predictions for the future. The general process of machine learning is to calculate the machine learning algorithm model parameters from the input data (input data), form the machine algorithm model according to the calculated model parameters, and intelligently identify new samples or make predictions for the future. In many practical applications, the input data is very large and must be processed by multiple computing devi...

AI Technical Summary

Problems solved by technology

However, in doing so, a large number of experimental comparisons and verifications are required, and existing products need to be modified to be compatible with the data and model formats of the platform.

In addition, there is no guarantee that the existing implementation of the platform can meet the business requirements

At the same time, it is necessary to have an in-depth understanding of the implementation of the platform, and it takes a lot of time to realize the data format, model format and training method, which has high requirements for users

Images