Collaborative heterogeneous processing of training data for deep neural networks

Abstract

Systems and methods for training neural networks. One embodiment is a system comprising: a memory configured to store samples of training data for a deep neural network (DNN); and a distributor. The distributor identifies a plurality of worker servers configured to train the DNN by processing the samples through a model of the DNN, receives information indicating processing capabilities of a graphics processing unit (GPU) at the worker servers, determines differences in GPU processing capabilities between the worker servers based on the information, and distributes the samples among the worker servers based on the differences.

Description

Technical Field

[0001] The present disclosure relates to the field of machine learning, and in particular to deep neural networks (DNNs). Background Art

[0002] DNNs are used for a variety of tasks, such as automatic speech recognition (ASR), computer vision, natural language processing (NLP), optical character recognition (OCR), etc. It is not uncommon for DNNs to be trained via a graphics processing unit (GPU). For example, when a DNN is trained via supervised learning, a large number of parameters for the DNN are determined by iteratively processing a large number of samples (e.g., millions of samples) via a GPU. The training process may take months or years for a machine to complete. Therefore, those who design DNNs often use distributed training techniques, in which multiple GPUs residing on multiple machines process training in parallel. In distributed training, GPUs may become out of sync during the training process, which causes different samples within the training data to be processed different times before the end of training. This is referred to as "epoch desynchronization" in this article. Epoch desynchronization is undesirable because it may cause the parameters determined for the DNN to become inaccurate. If the parameters are incorrect, additional processing time is required to retrain the DNN. This increases both the time and cost involved in training the DNN. Summary of the invention

[0003] Embodiments described herein enhance processing by machines that perform distributed parallel processing of training data for use with DNNs. Specifically, samples within a training data set can be assigned to machines based on the GPU processing capabilities of those machines so that the training data is processed over a desired number of epochs. One technical advantage is that samples within the training data are processed the same number of times, which prevents the DNN from being mistrained or otherwise needing to undergo retraining. These techniques can also help increase the overall speed of the training process.

[0004] One embodiment is a system that includes: a memory configured to store samples of training data for a deep neural network (DNN); and a distributor. The distributor identifies a plurality of worker servers that are provisioned to train the DNN by processing the samples through a model of the DNN, receives information indicating processing capabilities of a graphics processing unit (GPU) at the worker servers, determines differences in GPU processing capabilities between the worker servers based on the information, and distributes the samples among the worker servers based on the differences.

[0005] In another embodiment, the distributor is configured to determine the throughput of the worker servers, select a batch size for the worker servers based on the throughput, and report the batch size. Each batch size defines the number of samples in a batch that are processed simultaneously by one of the worker servers.

[0006] In further embodiments, the distributor is configured to dynamically determine the throughput, and to adjust the batch size based on the throughput during training of the DNN.

[0007] In another embodiment, the system further comprises at least one revision element configured to receive input from one of the work servers after completing processing of a batch of samples at one of the work servers, determine adjustments to the DNN based on the input, and report the adjustments to the work server for updating the model of the DNN.

[0008] In another embodiment, the distributor is configured to determine an integer number of epochs of training to be performed on the DNN. The distributor is configured to distribute samples among the worker servers so that the integer number of epochs will be completed but not exceeded during training.

[0009] In further embodiments, at least one of the samples comprises an image.

[0010] In further embodiments, at least one of the samples comprises a sound file.

[0011] An additional embodiment is a method comprising: identifying a plurality of work servers configured to train a deep neural network (DNN) by processing samples via a model of the DNN, receiving information indicating a graphics processing unit (GPU) processing capability at the work servers, determining differences in GPU processing capability between the work servers based on the information, and allocating samples among the work servers based on the differences.

[0012] Another additional embodiment is a non-transitory computer-readable medium embodying programming instructions that, when executed by a processor, are operable to perform a method. The method includes identifying a plurality of worker servers configured to train a deep neural network (DNN) by processing samples through a model of the DNN, receiving information indicating a graphics processing unit (GPU) processing capability at the worker servers, determining a difference in GPU processing capability between the worker servers based on the information, and allocating samples among the worker servers based on the difference.

[0013] Another additional embodiment is a system that includes means for storing samples of training data for a deep neural network (DNN) and means for distributing. The means for distributing identifies a plurality of worker servers configured to train the DNN by processing the samples through a model of the DNN, receives information indicating processing capabilities of a graphics processing unit (GPU) at the worker servers, determines differences between GPU processing capabilities between the worker servers based on the information, and distributes the samples among the worker servers based on the differences.

[0014] Other illustrative embodiments may be described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Some embodiments will now be described by way of example only and with reference to the drawings.On all drawings, the same reference number represents the same element or the same type of element.

[0016] Figure 1 is a block diagram of a neural network training system in an illustrative embodiment.

[0017] Figure 2 is a flowchart illustrating a method for training a DNN in an illustrative embodiment.

[0018] Figure 3 is a block diagram of a training data set in an illustrative embodiment.

[0019] Figures 4 to 8 is a flow chart showing additional details for training a DNN in an illustrative embodiment.

[0020] Fig. 9 is a message diagram illustrating communications between various computing elements during training of a DNN in an illustrative embodiment.

[0021] Fig.10 is a block diagram of a group of worker servers in an illustrative embodiment. DETAILED DESCRIPTION

[0022] The accompanying drawings and the following descriptions illustrate specific illustrative embodiments of the present invention. Therefore, it should be appreciated that those skilled in the art will be able to design various arrangements that implement the principles of the present invention and are included within the scope of the present invention, although not explicitly described or shown herein. In addition, any examples described herein are intended to help understand the principles of the present invention and should be interpreted as not being limited to such specifically elaborated examples and conditions. Therefore, the present invention is not limited to the specific embodiments or examples described below, but is limited by the claims and their equivalents.

[0023] Figure 1is a block diagram of a neural network training system 100 in an illustrative embodiment. The neural network training system 100 may include any suitable combination of computers for training a DNN based on a set of training data. Training of the DNN may be performed by processing samples of the training data through the DNN to make determinations. For example, a DNN examining samples in the form of images may attempt to assign categories to images (e.g., "animal," "car," "boat," etc.), or a DNN examining samples in the form of sounds may attempt to assign categories to sounds (e.g., "speech," "music," "nature," etc.). The DNN is then modified based on the accuracy of its determinations. As part of the iterative process, the DNN makes determinations for new samples after it has been modified. In this way, the DNN is adjusted over time until a desired level of accuracy is achieved. For example, the training process may involve processing millions of samples, and may involve iteratively adjusting the DNN thousands of times to improve the accuracy of the determinations made by the DNN.

[0024] Figure 1 The figure shows a neural network training system 100, which includes a work server 130, a parameter server 110, and an advisor server 140. The work server 130 is an element that includes any suitable circuit system, hardware, or component for processing samples and making determinations for those samples based on the DNN. In this embodiment, each work server 130 includes a controller 132, a GPU cluster 134, and a memory 136. The controller 132 includes any suitable circuit system, hardware, or device for controlling the manner in which the sample is processed. The GPU cluster 134 includes any suitable circuit system, hardware, or component for utilizing a neural network model 137 of the DNN in order to make determinations about the sample. Each GPU cluster 134 includes one or more GPUs 135. GPU 135 includes any suitable graphics processing unit. The GPU cluster 134 can use the neural network model 137 and the GPU 135 to classify the sample or assign a label to the sample. The neural network model 137 includes suitable information in the memory that describes how the DNN makes determinations for the sample. The properties of these samples (e.g., their actual classes or labels) are known a priori, so the accuracy of each determination made by GPU cluster 134 for these samples can be calculated. As the samples are being processed, worker servers 130 score the accuracy of the determinations made by neural network model 137 for the DNN. Memory 136 includes any suitable circuitry, hardware, or components for storing instructions for the operation of worker servers 130.

[0025] Although three worker servers 130 are shown in this embodiment, any number of worker servers 130 may be utilized. Additionally, worker servers 130 may have different numbers and types of GPUs 135 within their respective GPU clusters 134.

[0026] Advisory server 140 includes any suitable circuitry, hardware, or components for distributing samples of training data among worker servers 130. In this embodiment, advisory server 140 is configured to manage the training process of the DNN by distributing samples of training data to individual worker servers 130. Samples are distributed because they are reserved for processing by specific worker servers 130 only. Advisory server 140 distributes samples among worker servers 130 so that each worker server 130 completes processing of its assigned samples at the same time. Specifically, advisory server 140 distributes samples among worker servers 130 based on the GPU processing capabilities of those worker servers 130. This prevents epoch desynchronization from occurring, as described below.

[0027] The advisor server 140 may also determine a batch size for each worker server 130. A "batch" is a collection of samples that have been assigned to a worker server 130 for the purpose of making a determination and / or calculating accuracy and are processed simultaneously at the GPU cluster 134 of the worker server 130. The "batch size" is the number of assigned samples that will be processed simultaneously by a given worker server 130 at a point in time. For example, a worker server 130 with a batch size of eight may process eight of its assigned samples at a time. A worker server 130 may continue to process batches until all of its assigned samples have been processed, and may then optionally initiate another round of processing for its assigned samples.

[0028] In this embodiment, the advisor server 140 includes a distributor 142 and a memory 144. The distributor 142 includes any suitable circuitry, hardware, or components for managing the operation of the advisor server 140 and distributing samples among the worker servers 130. The memory 144 includes any suitable circuitry, hardware, or components for storing instructions for the operation of the advisor server 140.

[0029] The neural network training system 100 also includes a parameter server 110. The parameter server 110 includes a correction element 112 and a memory 116. The correction element 112 includes any suitable circuitry, hardware, or components for adjusting how the DNN makes determinations based on input from the worker server 130. The memory 116 includes any suitable circuitry, hardware, or components for storing neural network parameters 118. The parameter server 110 is coupled to the worker server 130 via a communication channel 120. The parameter server 110 can be implemented on the same or different machine / computer as the advisor server 140 or the worker server 130.

[0030] The correction element 112 adjusts how the DNN makes determinations by changing the neural network parameters 118. Examples of neural network parameters 118 include connection weights between nodes in the DNN. The parameter server 110 also includes a memory 116 that stores the neural network parameters 118 for access by the work servers 130 via the interface (I / F) 114. For example, these neural network parameters 118 can be used by each work server 130 to update the locally stored neural network model 137.

[0031] One method of training a DNN is known as synchronous training. In synchronous training, worker servers 130 process multiple batches of samples to make a determination. After a batch is completed, worker servers 130 pause to wait for other worker servers 130 to complete their respective batches. However, this means that valuable processing cycles may be wasted and the overall training time for the DNN may be undesirably increased.

[0032] An alternative method of training a DNN is called asynchronous training. In asynchronous training, worker servers 130 process batches of samples to make a determination. However, worker servers 130 do not pause for other worker servers 130 before continuing to process the next batch of samples. Parameters for a DNN are not guaranteed to converge during asynchronous training.

[0033] Processing samples in existing systems will cause problems related to the training process. Even in synchronous training, epoch desynchronization may occur. An "epoch" means that all samples in the training set have been processed once. It is generally expected that a specified number of epochs will be trained for the DNN. However, when samples are evenly distributed or siloed (or not distributed at all) to different worker servers 130 during training, then if one worker server 130 is faster than another worker server 130, it may process its samples more times than the slower worker server 130 when synchronous training is performed. This leads to a problem known as "epoch desynchronization", in which the exact number of epochs of training is not performed for the DNN because the samples are not processed the same number of times during training.

[0034] Further complicating this problem is that processing samples in existing systems may weight some samples more highly. For example, because some samples are processed more times than others, they may be used more frequently to adjust the neural network parameters 118 and therefore have a greater influence in adjusting the DNN. This is also a problem because such adjustments may be irreversible. Therefore, epoch desynchronization may require the DNN to be retrained and may result in the need to restart training from scratch.

[0035] The neural network training system 100 advantageously overcomes the above-mentioned problems associated with epoch desynchronization because it uses advisor servers 140 to dynamically allocate different numbers of samples to worker servers 130 based on the GPU processing power at each worker server 130. In addition, advisor servers 140 can assign different batch sizes to worker servers 130 based on the GPU processing power of worker servers 130 so that worker servers 130 complete their respective batches at approximately the same time. This allows worker servers 130 to continue without pausing, thereby advantageously increasing the training speed for DNNs. As described above, the size of each batch can be determined statically or dynamically by advisor servers 140. In one embodiment, advisor servers 140 allocate samples and assign batch sizes so that the number of samples allocated to worker servers 130 is evenly divisible by their batch sizes, with no remainders.

[0036] About Figure 2 Discuss more details of the operation of the neural network training system 100. For this embodiment, it is assumed that a DNN has been created and is awaiting training. In this embodiment, the DNN includes multiple layers of nodes, and the neural network parameters 118 indicate the numerical values ​​of the connection weights between the nodes at the DNN. In order to train the DNN to accurately perform the process, the weights are adjusted in an iterative process.

[0037] Figure 2 is a flow chart illustrating a method 200 for training a DNN in an illustrative embodiment. Figure 1 The steps of method 200 are described with reference to the neural network training system 100 of FIG. 1 , but those skilled in the art will appreciate that method 200 may be performed in other systems. The steps of the flowchart described herein are not all inclusive and may include other steps not shown. The steps described herein may also be performed in an alternative order.

[0038] In step 202, the distributor 142 identifies a plurality of worker servers 130 that have been configured for training a DNN. For example, the worker servers 130 may be selected by a user or indicated in the memory 144. In step 204, the distributor 142 receives information indicating the GPU processing power at the worker servers 130 (e.g., the processing power of the GPUs 135 within the GPU cluster 134 at each worker server 130). The GPU processing power of the GPU cluster may be based on the speed of individual processor cores, the number of processor cores, etc. The GPU processing power may be measured in raw cycles per second, etc. In this case, the GPU processing power is the processing power of the GPU 135 that will make a determination based on the sample. Therefore, in at least one embodiment, the speed of the CPU for the controller 132 is not considered (i.e., because the controller 132 does not directly process samples to make a determination based on the DNN). The GPU processing power may correspond to an expected throughput or a previously measured throughput expressed in samples per second. The advisor server 140 may also send a query to the worker server 130 to determine its specifications.

[0039] In step 206, the distributor 142 determines the difference in GPU processing capabilities between the worker servers 130 based on the information received in step 204. For example, one worker server 130 may have a GPU cluster 134 that is capable of faster processing than the GPU cluster 134 of another worker server 130. In step 208, the distributor 142 distributes the samples among the worker servers 130 based on the difference in GPU processing capabilities. This is performed so that worker servers 130 with different amounts of GPU processing capabilities are allocated different numbers of samples. For example, if a worker server 130 has twice the GPU processing capabilities of another worker server 130, the distributor 142 may allocate twice as many samples to the worker server 130. Allocating samples to the worker servers 130 may include transmitting the allocated samples to the worker servers 130, or transmitting instructions to the worker servers 130 that identify the allocated samples.

[0040] Dispatcher 142 may also select a batch size for each worker server 130 based on its GPU processing power and / or amount of memory. For example, advisor server 140 may consider available memory (e.g., more than ninety percent) at GPU cluster 134 as a constraint when determining batch size.

[0041] Work servers 130 process samples in the batch to make determinations, calculate the accuracy of those determinations and report the accuracy information to parameter servers 110. Parameter servers 110 receive this accuracy information and update neural network parameters 118. These neural network parameters 118 are used by work servers 130 to update their neural network models 137, and processing of the next batch is initiated.

[0042] Because the batch sizes assigned to worker servers 130 vary based on GPU processing power, worker servers 130 are also expected to complete their batches at the same time. If the batch sizes are equal, a GPU cluster 134 with more processing power will complete its batch faster, while a GPU cluster 134 with less processing power will process its batch more slowly. However, the batch size for each worker server 130 has been determined by advisor server 140 so that GPU clusters 134 complete processing the batches at approximately the same time. In other embodiments, advisor server 140 may determine the batch size prior to training and / or dynamically.

[0043] Completion of processing of a batch at each worker server 130 and updating of any parameters at the DNN based on that processing is called an iteration. Upon completion of one iteration, processing continues to the next iteration as worker servers 130 each process the next batch of samples. Processing can continue in this manner such that a sample of the training data has been processed once. Completion of processing for a set of training data (i.e., such that the samples in the training data have been processed the same number of times) is called an epoch.

[0044] To improve the accuracy of the DNN, training can be continued for multiple epochs as needed. Depending on the desired number of epochs, the number of worker servers 130, and the number of samples used for training, the steps described herein can be iterated multiple times (e.g., hundreds or even thousands of times). Eventually, the advisor server 140 completes training for an integer number of epochs, so that each sample in the training data has been processed an integer number of times (e.g., an integer number of times, such as ninety times). At this point, training has been completed and the weights at the DNN have been set to the desired values.

[0045] Figures 4 to 8 FIG. 1 is a flow chart showing additional details for training a DNN in an illustrative embodiment. Specifically, Figure 4 4 is a flow chart illustrating a method for selecting a batch size based on throughput in an illustrative embodiment. According to method 400, in step 402, distributor 142 determines the throughput of worker servers 130 (e.g., expressed in terms of the number of samples per unit time). In step 404, distributor 142 further selects a batch size for worker servers 130 based on the throughput. For example, worker servers 130 with higher throughput may be assigned higher batch sizes accordingly. In step 406, distributor 142 reports the batch size to worker servers 130. As previously mentioned, each batch size defines the number of samples in a batch for processing simultaneously by one of the worker servers.

[0046] Figure 55 is a flow chart illustrating dynamically selecting a batch size based on throughput in an illustrative embodiment. According to method 500, distributor 142 dynamically determines throughput in step 502. For example, distributor 142 may determine the throughput of worker servers 130 at the end of each iteration. In step 504, during DNN training, distributor 142 adjusts the batch size based on throughput. Thus, throughput may be adjusted by advisor server 140 while the DNN is actively trained.

[0047] Figure 6 6 is a flow chart illustrating adjustments performed by parameter server 110 in an illustrative embodiment. According to method 600, in step 602, after processing of samples of a batch of samples is completed at one of the worker servers, correction element 112 receives input from one of the worker servers 130. For example, correction element 112 may receive gradients from each worker server 130 upon completion of an iteration. According to step 604, correction element 112 determines adjustments to the DNN based on the input. For example, correction element 112 may adjust weighted connections between nodes at the DNN (as indicated by neural network parameters 118) based on the input. In step 606, correction element 112 reports the adjustments to worker server 130 via I / F 114 for updating the model of the DNN.

[0048] Figure 7 7 is a flow chart illustrating sample distribution in an illustrative embodiment to provide a desired number of epochs of training. According to method 700, distributor 142 determines an integer number of epochs of training to be performed on the DNN in step 702. In step 704, distributor 142 also distributes samples among worker servers 130 so that the integer number of epochs will be completed but not exceeded during training.

[0049] Figure 8 8 is a flow diagram illustrating a method for selecting a batch size for training a DNN in an illustrative embodiment. For example, in an illustrative embodiment, method 800 illustrates additional features of step 204 of method 200. According to method 800, in step 802, advisor server 140 determines the total number of samples (also referred to as the "total batch" size) N that will be used to train the DNN. For example, N can be hundreds of thousands or millions of samples to allow a large number of worker servers to train the DNN in parallel. In step 804, advisor server 140 selects a worker server 130. Advisor server 140 proceeds to select a batch size for that worker server 130. This can be performed according to the following equation, where w i is the throughput of the selected worker server 130 (i.e., the number of samples processed in a given time period), b iis the number of samples determined for the batch, and j is the number of iterations performed on the plurality of worker servers 130 used to train the DNN.

[0050] (1)

[0051] In other words, the batch size for the worker server 130 may be set equal to N times the measured throughput of the worker server 130 divided by an amount equal to the sum of the throughput for all worker servers (step 806). In step 808, if not all worker servers 130 have been reviewed for the batch size, a new worker server 130 may be selected. This process may be performed for all worker servers 130 and may be performed at the beginning of training. This process may also be performed after each iteration, periodically over time, etc.

[0052] Fig. 9 9 is a message diagram 900 illustrating the communication between various computing elements during the training of a DNN in an illustrative embodiment. Figure 5 As shown in , advisory server 140 can initiate this process by requesting GPU configuration from each worker server 130. Controller 132 at each worker server 130 proceeds to query GPU cluster 134 for data describing the speed, amount of memory, model, etc. of each GPU 135 in the GPU cluster. Upon receiving this data, worker server 130 reports the GPU configuration data back to advisory server 140. At this point in time, advisory server 140 estimates the throughput of each GPU cluster 134 based on the memory size for each sample, GPU processing power, and other factors. Advisory server 140 also allocates samples to each worker server 130 based on the GPU processing power of that worker server 130.

[0053] The advisor server 140 may also determine a batch size for each worker server 130. The sample allocation and / or batch size is reported to the controller 132, which obtains or otherwise tracks the location of its assigned samples. The parameter server 110 continues to determine initial parameters for the DNN and reports these parameters to the controller 132. The controller 132 then operates the GPU cluster 134 on a batch of samples to make a determination for the batch of samples based on the neural network model 137. The controller 132 also calculates the accuracy of the determination. For example, the controller 132 may determine the value of each determination (or sample) via a loss function. The controller 132 may also determine gradients based on these loss function values ​​and report the gradients to the parameter server 110, which updates its parameters. Although not illustrated, the controller 132 may also report throughput parameters (e.g., idle time, duration spent processing a batch, free GPU memory, etc.) to the advisor server 140. The advisor server 140 may then update the batch size for the worker servers 130 as needed. The process continues with another iteration as the worker server 130 and any other worker servers 130 process another batch of samples tailored to the amount of their throughput.

[0054] Example

[0055] In the following examples, additional processes, systems, and methods are described in the context of systems for training DNNs.

[0056] Fig.10 is a block diagram of a group of worker servers in an illustrative embodiment. Fig.10 , worker server 10 has one GTX 1080Ti GPU ("GTX 1012"). Worker servers 1020 and 1030 each have two such GPUs. In addition, worker server 1040 has eight Tesla P100 GPUs ("TESLA 642"). In order to fully utilize all 13 GPUs, it is possible to implement 13 separate subroutines, each assigned a separate GPU. However, such an implementation necessarily increases overhead. Implementing multiple subroutines on one machine may also cause the subroutines to adversely affect or otherwise interfere with each other. For example, if eight subroutines were run independently on worker server 640, there would be eight models (as opposed to one model) whose parameter and gradient updates are transmitted to the parameter server at each iteration.

[0057] according to Fig.10In the example of , worker servers 1010-1040 each implement a worker server subroutine and are assigned samples heterogeneously. For example, worker server 1010 may use one GPU and have a batch size of 64, worker server 1020 and worker server 1030 may each use two GPUs and have a batch size of 128, and worker server 1040 may use eight GPUs and have a batch size of 512. The heterogeneous training process achieves a substantial improvement (e.g., 71%) in completion time compared to the base case where each subroutine on each of the 13 GPUs uses the same batch size of 64 homogeneously.

[0058] When performing training that is expected to terminate after a specified number of epochs, the workers together cover a sample of the training data once (e.g., exactly once) in an epoch. In homogeneous distributed training, the dataset can simply be divided equally among the workers. This means that the workers complete their samples with the same number of iterations in each epoch. To guarantee this property for heterogeneous distributed training, for each epoch, the training data is divided among the workers in proportion to the batch size of the workers. New workers and / or new GPU devices can be introduced at the boundaries of the epochs, or alternatively, the partitions can be dynamically adjusted to reflect changes in the ratio of worker batch sizes.

[0059] Any of the various elements or modules shown in the drawings or described herein can be implemented as hardware, software, firmware or some combination of these. For example, an element can be implemented as special hardware. Special hardware elements can be referred to as "processors", "controllers" or some similar terms. When provided by a processor, functions can be provided by a single special processor, a single shared processor or multiple individual processors, some of which can be shared. In addition, the explicit use of the term "processor" or "controller" should not be interpreted as specifically referring to hardware capable of executing software, and may implicitly include but are not limited to digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC) or other circuit systems, field programmable gate array (FPGA), read-only memory (ROM) for storing software, random access memory (RAM), non-volatile memory, logic or some other physical hardware components or modules.

[0060] Moreover, the element can be implemented as an instruction, which is executable by a processor or computer to perform the function of the element. Some examples of instructions are software, program code and firmware. The instruction is operable when executed by the processor to direct the processor to perform the function of the element. The instruction can be stored on a storage device readable by the processor. Some examples of storage devices are digital or solid-state memory, magnetic storage media (such as disks and tapes), hard drives or optically readable digital data storage media.

[0061] As used in this application, the term "circuitry" may refer to one or more or all of the following:

[0062] (a) hardware circuit implementation only (such as implementation only in analog and / or digital circuitry), and

[0063] (b) a combination of hardware circuitry and software such as (where applicable):

[0064] (i) a combination of analog and / or digital hardware circuits and software / firmware, and

[0065] (ii) having software (including digital signal processor(s))

[0066] Any portion of a hardware processor, software, and memory(s) that work together to enable a device such as a mobile phone or server to perform various functions, and (c) hardware circuit(s) and / or processor(s), such as microprocessor(s) or portion(s) of microprocessor(s), that requires software (e.g., firmware) to operate, but that software may be absent when not required for operation.

[0067] This definition of circuitry applies to all uses of the term in this application, including in any claims. As a further example, as used in this application, the term "circuitry" also covers only an implementation of a hardware circuit or processor (or multiple processors) or a portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers (for example and if applicable to a particular claim element) a baseband integrated circuit or processor integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or network device.

[0068] Although specific embodiments are described herein, the scope of the present disclosure is not limited to those specific embodiments.The scope of the present disclosure is defined by the following claims and any equivalents thereof.

Claims

1. A system for machine learning, comprising: A memory configured to store samples of training data for a deep neural network DNN; as well as a distributor configured to identify a plurality of worker servers configured to train the DNN by processing the samples through a model of the DNN, receive information indicating processing capabilities of a graphics processing unit (GPU) at the worker servers, determine differences in the GPU processing capabilities between the worker servers based on the information, and distribute the samples among the worker servers based on the differences, and wherein the distributor is configured to determine a throughput of the worker servers during training of the DNN, dynamically select a batch size for individual ones of the worker servers based on the throughput between epochs during training of the DNN, and report the selected batch size to the worker servers so as to control how many samples are processed by the worker servers, wherein an epoch in an epoch is when all samples in a training set have been processed once; and Each batch size defines the number of samples to be processed by a worker server.

2. The system of claim 1, wherein: The distributor is configured to dynamically determine the throughput and adjust the batch size based on the throughput during training of the DNN.

3. The system according to claim 1, further comprising: At least one correction element is configured to: receive input from one of the working servers after processing of a batch of the samples is completed at the one of the working servers; determine adjustments to the DNN based on the input; and report the adjustments to the working server for updating the model of the DNN.

4. The system of claim 1, wherein: The distributor is configured to determine an integer number of epochs of training to be performed on the DNN; and The distributor is configured to distribute the samples among the worker servers such that during training the integer number of the epochs will be completed but not exceeded.

5. A system according to any one of claims 1 to 4, wherein: At least one of the samples comprises an image.

6. A system according to any one of claims 1 to 4, wherein: At least one of the samples comprises a sound file.

7. A machine learning method comprising: Identifying a plurality of worker servers configured to train a deep neural network (DNN) by processing samples through a model of the DNN; receiving information indicating processing capabilities of a graphics processing unit (GPU) at the worker server; determining a difference in the GPU processing capabilities between the worker servers based on the information; allocating the samples among the worker servers based on the differences; determining a throughput of the worker server during training of the DNN; dynamically selecting a batch size for individual ones of the worker servers based on the throughput between epochs during training of the DNN, wherein an epoch in an epoch is when all samples in a training set have been processed once; and reporting the selected batch size to the worker server to control how many samples are processed by the worker server; and Each batch size defines the number of samples to be processed by a worker server.

8. The method according to claim 7, further comprising: dynamically determining the throughput; as well as The batch size is adjusted based on the throughput during training of the DNN.

9. The method according to claim 7, further comprising: receiving input from one of the worker servers after processing of a batch of the samples is completed at the one of the worker servers; determining an adjustment to the DNN based on the input; as well as The adjustment is reported to the worker server for updating the model of the DNN.

10. The method according to claim 7, further comprising: determining an integer number of epochs of training to be performed on the DNN; as well as The samples are distributed among the worker servers such that during training the integer number of epochs will be completed but not exceeded.

11. The method according to any one of claims 7 to 10, wherein: At least one of the samples comprises an image.

12. The method according to any one of claims 7 to 10, wherein: At least one of the samples comprises a sound file.

13. A non-transitory computer readable medium embodied with programming instructions, the programming instructions being operable when executed by a processor to perform a method comprising: Identifying a plurality of worker servers configured to train a deep neural network (DNN) by processing samples through a model of the DNN; receiving information indicating processing capabilities of a graphics processing unit (GPU) at the worker server; determining a difference in the GPU processing capabilities between the worker servers based on the information; allocating the samples among the worker servers based on the differences; determining a throughput of the worker server during training of the DNN; dynamically selecting a batch size for individual ones of the worker servers based on the throughput between epochs during training of the DNN, wherein an epoch in an epoch is when all samples in a training set have been processed once; and reporting the selected batch size to the worker server to control how many samples are processed by the worker server; and Each batch size defines the number of samples to be processed by a worker server.

14. The medium of claim 13, wherein: dynamically determining the throughput; and The batch size is adjusted based on the throughput during training of the DNN.

15. The medium of claim 13, wherein the method further comprises: receiving input from one of the worker servers after processing of a batch of the samples is completed at the one of the worker servers; determining an adjustment to the DNN based on the input; as well as The adjustment is reported to a worker server for updating the model of the DNN.

16. The medium of claim 13, wherein the method further comprises: determining an integer number of epochs of training to be performed on the DNN; as well as The samples are distributed among the worker servers such that during training the integer number of epochs will be completed but not exceeded.

17. A medium according to any one of claims 13 to 16, wherein: At least one of the samples comprises an image.

Images