A kind of machine learning tool middleware and machine learning training method

Abstract

The invention discloses a machine learning tool middleware and a machine learning training method. The machine learning tool includes at least one training unit, each training unit is provided with middleware combined with the machine learning tool, the middleware includes a bottom communication module, andAt least one of the data distribution module, the model parameter update module, the training parameter adjustment module, and the training stop judgment module.The training unit distributes the required data from the data storage device to the storage unit accessible to each training unit through the middleware, so that each training unit reads the data from the storage unit for training; and through the corresponding module between the middlewareThe communication completes the update of the model parameters of the training unit, the adjustment of the training parameters of each training unit; and determines whether to stop training based on the training information of all the training units.The middleware of the present invention is responsible for the processing required for large-scale parallel training, and facilitates the expansion of various machine learning tools.

Description

technical field [0001] The invention belongs to the technical field of machine learning, and in particular relates to a machine learning tool middleware and a machine learning training method. Background technique [0002] Machine learning is a branch of artificial intelligence, and in many cases, it has almost become synonymous with artificial intelligence. Simply put, machine learning is to use machine learning algorithm models to enable machines to learn laws from a large amount of historical data, so as to intelligently identify new samples or make predictions for the future. The general process of machine learning is to calculate the machine learning algorithm model parameters from the input data (input data), form the machine algorithm model according to the calculated model parameters, and intelligently identify new samples or make predictions for the future. In many practical applications, the input data is very large and must be processed by multiple computing devi...

AI Technical Summary

Problems solved by technology

However, in doing so, a large number of experimental comparisons and verifications are required, and existing products need to be modified to be compatible with the data and model formats of the platform.

In addition, there is no guarantee that the existing implementation of the platform can meet the business requirements

At the same time, it is necessary to have an in-depth understanding of the implementation of the platform, and it takes a lot of time to realize the data format, model format and training method, which has high requirements for users

Images