Sliding window sampling-based distributed machine learning training method and system thereof

A machine learning and sliding window technology, applied in machine learning, instruments, computing models, etc., can solve the problem of poor stability and convergence effect of distributed asynchronous training, inability to perceive the context information of the expired degree of the learner gradient, and expired gradient processing Too simple and other problems to achieve the effect of alleviating poor training convergence, reducing training fluctuations, and improving robustness
CN106779093AInactive Publication Date: 2017-05-31SHANGHAI ADVANCED RES INST CHINESE ACADEMY OF SCI
0 Cites 26 Cited by

Patent Information

Authority / Receiving Office
CN Ā· China
Patent Type
Applications(China)
Current Assignee / Owner
SHANGHAI ADVANCED RES INST CHINESE ACADEMY OF SCI
Publication Date
2017-05-31
Estimated Expiration
Not applicable Ā· inactive patent

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
Patent Text Reader

Abstract

The invention provides a sliding window sampling-based distributed machine learning training method and system thereof. The method comprises the steps of initializing parameters of a machine learning model; obtaining a data fragment of all data and independently carrying out model training; collecting multiple rounds of historical gradient expiration degree samples, sampling the samples through sliding, calculating a gradient expiration degree context value, adjusting the learning rate and then initiating a gradient update request; asynchronously collecting the multiple gradient expiration degree samples, updating global model parameters by using the adjusted learning rates and pushing updated parameters; asynchronously obtaining pushed global parameters for updating, and further carrying out next training; checking the model convergence, if the model is not convergent, carrying out model training cycle; and if the model is convergent, obtaining model parameters. The learning rate of a learning device is controlled by using the expiration gradient, the stability and the convergence effect of distributed training are improved, the training fluctuation caused by the distributed system is reduced and the robustness of distributed training is improved.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The present invention relates to large-scale machine distributed training, in particular to a distributed machine learning training method and system based on sliding window sampling. Background technique

[0002] Modern neural network architectures trained on large data sets can achieve impressive results across a wide variety of domains, ranging from speech and image recognition, natural language processing, to industry-focused applications such as fraud detection and recommendation systems and other aspects. However, training these neural network models has strict computational requirements. Although significant progress has been made in GPU hardware, network architecture, and training methods in recent years, the fact is that on a single machine, the time required for network training is still long. unrealistic. Fortunately, we are not limited to a single machine: a great deal of work and research has made efficient distributed training of neural...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More