Gradient Quantization Method and System for Distributed Deep Learning
A technology of deep learning and quantization methods, applied in the computer field, can solve problems such as insufficient adaptability of gradient values and low gradient quantization efficiency, and achieve the effect of improving adaptability and processing efficiency
Active Publication Date: 2022-06-07
NAT UNIV OF DEFENSE TECH
View PDF4 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0005] Based on this, it is necessary to address the above technical problems and provide a distributed deep learning gradient quantization method and system, computer equipment and storage medium
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0050] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.
[0051] The gradient quantization method of distributed deep learning provided by this application can be applied to figure 1 in the application environment shown. The worker node 102 and the terminal 104 can be, but are not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices, and the parameter server 106 can be an independent server or a server cluster composed of multiple servers. accomplish.
[0052] Specifically, the terminal 104 communicates with each worker node 102, so as to obtain the gradien...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
This application relates to a gradient quantization method and system for distributed deep learning. The method includes: obtaining gradient values calculated by each distributed worker node, establishing a one-dimensional gradient array according to the gradient values, randomly sampling the one-dimensional gradient array, clustering the gradient values obtained by sampling, and clustering the obtained gradient values according to the clustering As a result, multiple hash tables are established, and according to the clusters corresponding to the gradient values, the gradient values in the one-dimensional gradient array are inserted into the hash buckets of multiple hash tables, and stored in The class cluster of the hash bucket generates a mapping relationship table, and sends multiple hash tables and mapping relationship tables inserted into the gradient value to the parameter server to complete the gradient quantization of deep learning. The method can improve the efficiency of gradient quantization.
Description
technical field [0001] The present application relates to the field of computer technology, and in particular, to a gradient quantization method and system for distributed deep learning. Background technique [0002] In recent years, deep learning technology has brought another revival to artificial intelligence, and has driven the industrialization of emerging technologies such as computer vision, autonomous driving, and neural translation. At the same time, for deep learning, academia and industry have developed a variety of deep learning system frameworks, such as TensorFlow, MxNet, Pytorch, etc., which greatly facilitates researchers to develop and deploy deep learning applications. These deep learning frameworks can support data parallelism and model parallelism. Many applications benefit from this parallelism and can handle the training of large data and large models. For example, in terms of the number of model parameters, the Google BERT model has reached 30 paramete...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06K9/62G06F16/906
CPCG06F16/906G06F18/23213
Inventor 李东升葛可适符永铨
Owner NAT UNIV OF DEFENSE TECH