Unlock instant, AI-driven research and patent intelligence for your innovation.

Machine Deep Learning Method Based on Data Distribution

A deep learning and data distribution technology, applied in data exchange networks, instruments, digital transmission systems, etc., can solve the problems of unbalanced data distribution, large network overhead, easy to produce single point of failure, etc., and achieve the effect of reducing network transmission.

Active Publication Date: 2020-10-23
SHENZHEN GRADUATE SCHOOL TSINGHUA UNIV
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The traditional distributed deep learning method transfers the intermediate results of the model training process between each node, such as model parameters, gradients, etc., and does not transmit the user's original data. This algorithm has the following problems in practical applications : 1. Coordination of parameter servers is required, which is prone to single point of failure; 2. The data of each node is required to be independent and distributed with all data, which is almost impossible in reality, and the data distribution is almost always unbalanced; 3. The training process brings huge network overhead
[0005] At present, there is no efficient deep learning method for different data distributions in the wide area network that can effectively solve these problems

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Machine Deep Learning Method Based on Data Distribution
  • Machine Deep Learning Method Based on Data Distribution
  • Machine Deep Learning Method Based on Data Distribution

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0032] Such as figure 1 As shown, the learning method of this embodiment is as follows:

[0033] 1. The nodes participating in the training establish a connection, determine the maximum number of training rounds T, and exchange data information. Assume that there are a total of n nodes participating in the training, and the number of training data owned by the i-th node is q i , covering v i category. The number of training data of all nodes combined is recorded as Q, and the coverage category is V, then the importance p of node i can be calculated i for:

[0034]

[0035] in Represents the proportion of the data quantity of node i to the whole, It represents the coverage degree of the data type of node i. α is used to balance the influence of quantity and type, and prevent nodes with a single type and a large number, or nodes with complete types but a small number, from gaining higher importance. The value of α ranges from 0 to 1. It can be seen from this formula...

Embodiment 2

[0049] The main idea of ​​this embodiment: As found in the experiment, in the scheme of Embodiment 1, the training effect is much more sensitive to the distribution of data types than the distribution of data quantity. When the data types covered by a certain node are extremely rare, let it Sending data separately will seriously affect the effect of training. Therefore, the idea of ​​grouping is adopted, and each worker node (Worker) sends its own data quantity and type information to the group manager (GroupController), and the group manager uses this information to combine several nodes to form a training group (Group), so that each The data of all nodes in a group will not deviate too much from the overall data type. The group manager can be assumed by any worker node. When a new worker node joins or an existing node goes offline, the group manager will regroup. Nodes in the same group have the same transmission interval, making each transmission valid. The characteristic...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The present invention proposes a machine deep learning method based on data distribution, evaluates the importance of the node in the training process according to the amount and type of data owned by each node, and is used to guide the integration of transmission and models; and according to the participation The data distribution of the nodes makes the transmission frequency of each node inconsistent during the training process, so that the nodes with better data distribution spread their own models as much as possible, and vice versa, they receive more models from other nodes. It not only reduces the impact of unbalanced data distribution on training results, but also reduces network transmission without affecting the training effect because nodes with poor data distribution disseminate their own models as little as possible.

Description

technical field [0001] The invention relates to a machine deep learning method based on data distribution. Background technique [0002] With the continuous development of neural network and deep learning technology, the industry is increasingly inclined to use deep network models to solve problems such as image classification, face comparison, and speech recognition. In order to obtain a satisfactory deep model, it is often necessary to provide a large amount of data for training. Traditional deep learning methods require all training data to be concentrated on one machine, and high-speed parallel computing devices such as GPUs are used for model training. [0003] Since the speed of data growth is far greater than the improvement of the computing power of a single machine, people have proposed a distributed deep learning method to use multiple machines to cooperate for training. Share computing power within a small local area network. In reality, data not only contains ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06K9/62H04L12/24
CPCH04L41/142G06F18/214
Inventor 王智胡成豪
Owner SHENZHEN GRADUATE SCHOOL TSINGHUA UNIV