Model processing method and device, program product and equipment

Abstract

The invention provides a model processing method and device, a program product and equipment, and relates to the technical field of computers. The model processing method comprises the following steps: dividing a training data set for training a first model into multiple batches of training data, and distributing the multiple batches of training data to each distributed computing node, so that each distributed computing node accumulates multiple batches of gradients based on the multiple batches of training data, and determining a local average gradient based on the accumulated gradients of the plurality of batches; and synchronizing the local average gradient of each distributed computing node among the distributed computing nodes to enable each distributed computing node to determine a global average gradient based on the local average gradient of each distributed computing node, and updating model parameters of the first model based on the global average gradient. According to the invention, efficient gradient synchronization between the nodes is realized, and the communication frequency and overhead can be reduced.

Images