Data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a storage medium and electronic equipment. Wherein, the method includes: obtaining a data processing request, wherein the data processing request is used to request to update the weights of the network parameters in the deep learning network; responding to the data processing request, determining the matrix structure corresponding to the weight matrix where the weights of the network parameters are respectively located Information; according to the matrix structure information, the weight matrix is ​​assigned to each data processing process, wherein, the object weight matrix assigned to each data processing process will reach the uniform distribution condition; each data processing process determines the reference based on the corresponding object weight matrix In the case of the gradient value, the object weight matrix allocated in the data processing process is updated according to the reference gradient value. The invention solves the technical problem that the zero technology cannot be compatible with the lamb optimizer for model training to save video memory.

Description

technical field [0001] The present invention relates to the field of computers, in particular to a data processing method and device, a storage medium and electronic equipment. Background technique [0002] With the increase in the complexity of deep learning models and the size of datasets, graphics processing unit (GPU) memory has always been considered to be the biggest bottleneck for training deep learning models, which means that large-scale training often encounters model parameters. If there are too many and cannot be accommodated by the video memory, the video memory of the GPU card is very limited, which is generally much lower than the system memory. Therefore, when the parameters of the deep learning model are large, the deep learning model may not be able to be trained in GPU mode. At present, the implementation of zero technology is usually used to completely "flatten" the split of the optimizer state and gradient of the deep learning model (that is, reduce the ...

AI Technical Summary

Problems solved by technology

[0004] Embodiments of the present invention provide a data processing method and device, a storage medium, and electronic equipment to at least solve the technical problem that the zero technology cannot be compatible with the lamb optimizer for model training to save video memory

Images