Large-batch decentralized distributed image classifier training method and system

Abstract

The invention discloses a large-batch decentralized distributed image classifier training method and system, and the method comprises the steps: enabling each working node to use a local image classifier parameter, calculating a random gradient according to a locally stored image sample, carrying out the normalization processing of the gradient, and updating the momentum and the local parameter through employing the normalized gradient. And each node communicates with the neighbor node to obtain the latest image classifier parameter, and the latest image classifier parameter and the local image classifier parameter of the node are subjected to weighted averaging to participate in the next round of updating as a new local parameter. And continuously repeating the training steps until a stop condition is reached, stopping each node, and taking a parameter average value on each node as a final output parameter. According to the method, a central node is cancelled, the problem of congestion at the central node is avoided, meanwhile, the method is suitable for large-batch image classifier training, and the number of parameter updating and communication times can be reduced through large-batch training, so that computing resources such as a GPU can be fully utilized, and the training efficiency is greatly improved.

Description

technical field [0001] The invention relates to a large-scale decentralized distributed image classifier training method and system, belonging to the technical fields of image classification and machine learning. Background technique [0002] The training of many image classifiers can be formalized as solving an optimization problem of the following finite sum form: [0003] [0004] Where x is the parameter of the model, d represents the dimension of the model parameter, n represents the total number of training samples, ξ i Refers to the i-th sample, f(x; ξ i ) is the loss function corresponding to the i-th sample. [0005] In recent years, deep learning has developed vigorously, and the continuous emergence of large data sets and large models has made the computing power of a single machine no longer meet the demand. Distributed machine learning technology that uses multiple machines to work together to complete training tasks has become an important solution to this...

AI Technical Summary

Problems solved by technology

However, blindly increasing the batch size will lead to a decline in the generalization performance of the final image classifier trained, so it is necessary to design a special image classifier training method suitable for large batch training.

Images