dynamic mini-batch size

Abstract

The invention relates to dynamic mini-batch sizes. Methods, systems, and devices for using dynamic mini-batch sizes during neural network training, including computer programs encoded on computer storage media. One of the methods includes receiving, by each of a plurality of host computers, a respective batch of training examples, each training example having zero or more features; computing a minimum number of mini-batches into which the host computer can divide the respective batch of training examples such that the host computer can process each mini-batch using an embedding layer of a neural network without exceeding available computing resources; determining a maximum minimum number of mini-batches (N) into which any host computer can divide its respective batch of training examples; generating N mini-batches from the respective batch of training examples received by the host computer; and processing, by each host computer, the N mini-batches using the embedding layer.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201980058339.8, filed on December 16, 2019. Background Art

[0003] This specification relates to artificial neural networks.

[0004] A neural network is a machine learning model that uses multiple operating layers to predict one or more outputs from one or more inputs. A neural network typically includes one or more hidden layers between the input layer and the output layer. The output of each hidden layer serves as the input to the next layer (e.g., the next hidden layer or the output layer).

[0005] Each layer of a neural network specifies one or more transformation operations to be performed on the inputs to that layer. Some neural network layers include units called neurons. Each neuron receives one or more inputs and generates outputs that are received by other neural network layers. Typically, each neuron receives inputs from one or more other neurons, and each neuron provides outputs to one or more other neurons.

[0006] Each layer generates one or more outputs using the current values ​​of a set of parameters for that layer. Training a neural network involves continuously performing a forward pass on the input, computing gradient values, and updating the current values ​​of each layer's parameter set. Once the neural network is trained, the final set of parameters can be used to make predictions.

[0007] Some neural networks have hidden layers that act as embedding layers. Typically, an embedding layer transforms a sparse neural network input (i.e., one that has only a small number of features with non-zero values ​​relative to the total number of features) into a low-dimensional vector called an embedding. Summary of the Invention

[0008] This specification describes a distributed computing system that can process the embedding layer of a neural network during training. The system includes host computers that each store a partition of a trainable embedding lookup table. To process a particular training example, for each feature of the training example, the host computer obtains an embedding from a corresponding partition of the lookup table stored on the corresponding host computer. Obtaining the embedding involves sending a lookup request to the host computer storing the partition of the lookup table to which the embedding belongs. After obtaining the embedding for each feature, the host computer processes the embeddings to generate an embedding layer output for the training example.

[0009] The host computer processes multiple training examples at a time. Specifically, each host computer receives a corresponding batch of training examples for processing during an iteration of training. Typically, each batch of training examples has the same number of training examples, and each training example has one or more features. Each host computer calculates the minimum number of embedding mini-batches into which the host computer can divide its batch of training examples, so that the host computer can use the embedding layer of the neural network to process each embedding mini-batch without exceeding the available computing resources on the host computer. In this specification, the embedding mini-batch will be referred to as a "mini-batch" for short. The system determines the maximum minimum number of mini-batches (N) into which any host computer can divide its batch of training examples. Each host computer divides its batch of training examples into N mini-batches and processes the N mini-batches using the embedding layer of the neural network.

[0010] Distributed computing systems operate under several constraints. First, each host computer must typically process the same number of mini-batches as every other host computer. This is because (i) the host computers receive synchronous, parallel instructions, and (ii) the host computers must exchange embeddings with each other during each iteration of training. Second, the host computers each have a limited amount of memory to store embeddings obtained from other host computers. More specifically, host computers are specialized computers used to process neural networks and have only a small amount of scratch memory to store embeddings, while traditional CPUs have large amounts of main memory or cache memory. Scratch memory is high-speed internal memory used to temporarily store computations, data, and other ongoing work. Therefore, the number of embeddings a host computer can obtain at one time is limited, which in turn limits the maximum mini-batch size. Third, the number of features in each training example, and the amount of scratch memory required to store the embedding for each training example, varies.

[0011] In some implementations, a distributed computing system is configured to perform operations including: receiving, by each of a plurality of host computers, a respective batch of training examples, each training example having zero or more features; computing, by each host computer, a minimum number of mini-batches into which the host computer can divide the respective batch of training examples, such that the host computer can process each mini-batch using an embedding layer of a neural network without exceeding available computing resources; determining, based on the computed minimum number of mini-batches, a maximum minimum number of mini-batches (N) into which any host computer can divide its respective batch of training examples; generating, by each host computer, N mini-batches from the respective batch of training examples received by the host computer; and processing, by each host computer, the N mini-batches using the embedding layer.

[0012] Given these constraints, the subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages. Dividing a batch of training examples into a large number of mini-batches increases the number of times each host computer must obtain embeddings from other host computers, thereby generating avoidable read / write latency. On the other hand, due to the available resources on each host computer, such as temporary memory, it may be impossible to divide a batch of training examples into a small number of mini-batches with a large number of training examples. The system described in this specification dynamically adjusts the number of mini-batches before each iteration of training. Specifically, the system calculates the maximum minimum number of mini-batches (N) into which any host computer can divide its corresponding batch of training examples without exceeding the available computing resources on any host computer. Each host computer then divides its batch of training examples into N mini-batches. This process minimizes the number of mini-batches into which each group of unique training examples is divided in each iteration of training. This in turn reduces the latency and overhead associated with each iteration of training, which reduces the total training time and the amount of computing resources, such as processing power, consumed by the training process.

[0013] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a diagram of an exemplary distributed computing system.

[0015] Figure 2 is a flowchart of an exemplary process for partitioning a batch of neural network training examples into mini-batches.

[0016] Figure 3 is a flow chart of an exemplary process for processing small batches.

[0017] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0018] Figure 1 is a diagram of an exemplary distributed computing system 100. The distributed computing system 100 is configured to process an embedding layer of a neural network.

[0019] This specification uses the term "configured to" in connection with systems, devices, and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform the operation or action.

[0020] An embedding layer is a neural network layer that transforms sparse neural network inputs into embeddings. An embedding is a dense, multi-dimensional vector representation of the input. Intuitively, the value of each dimension of a particular embedding encodes different characteristics of the input represented by the embedding.

[0021] Typically, an embedding layer is trained by completing a forward pass through the entire neural network to which the embedding layer belongs; determining whether there is a difference between the actual output of the neural network and the expected output for a given input; and updating the weights of the neural network, including the weights of the embedding layer, for example, by using conventional backpropagation techniques or similar techniques known in the art. In other words, the embedding layer can be trained during the normal training process of the neural network.

[0022] The embedding layer can be simplified to a lookup table that stores an embedding for each latent feature input to the neural network. For example, if a particular neural network is configured to generate movie recommendations for users of a streaming service by modeling the user's viewing history, the neural network's lookup table might store a separate embedding for each movie available on the streaming service. As another example, if a particular neural network is configured to generate search engine results based on queries written in English, the neural network's lookup table might store a separate embedding for each word in the English language.

[0023] Each feature is represented by a different index, such as a different integer. Features can be arranged in a lookup table by index. Processing an input using an embedding layer involves looking up the index of each feature in the input in the lookup table; retrieving the embedding at each index; and combining the embeddings, such as by adding them together. In a lookup table implementation of the embedding layer, training the embedding layer involves updating the values ​​in the lookup table during the normal training process.

[0024] The distributed computing system 100 includes host computers 110a-110n. Each of the host computers 110a-110n stores a partition of a lookup table for an embedding layer in an embedding memory 112a-112n. The lookup table is distributed among the host computers because the number of input features and the dimensionality of each feature make the entire lookup table inadequate for temporary storage on a single host computer. Additionally, obtaining embeddings from multiple host computers in parallel is faster than obtaining all embeddings from a single host computer. In some implementations, the embedding memory 112a-112n stores partitions of multiple lookup tables, each of which can store embeddings for different types of features. For example, referring again to a neural network configured to generate movie recommendations for users of a streaming service by modeling the user's viewing history, the neural network can additionally receive input specifying the user's genre preferences. A separate lookup table can store the genre embeddings.

[0025] During an iteration of training, each host computer in distributed computing system 100 receives a batch of training examples. Each training example includes one or more features. To process a particular training example, the host computer obtains the embedding for each feature in the training example from a partition of the lookup table (i.e., from one or more of embedding memories 112a-112n). The host computers exchange embeddings using input / output units 114a-114n. The input / output units are hardware components that include temporary register space and communication channels (e.g., buses that connect the host computers to each other). The host computers store the obtained embeddings and metadata about those embeddings, such as deduplication information and the number of embeddings in the training example, in scratch memory 118a-118n. Scratch memory 118a-118n is a high-speed internal memory that is physically very close to processing units 116a-116n and is used for computation and temporary storage of data. Scratch memory 118a-118n can be any suitable memory structure, such as SRAM.

[0026] Using separate embeddings, the host computer uses processing units 116a-116n to compute the embedding layer outputs for their respective training examples. Processing units 116a-116n may include dedicated hardware components to compute the embedding layer outputs. The host computer may include Figure 1 Additional processing components not depicted in the diagram, such as a general-purpose CPU, may be used to perform some or all of the techniques described in this specification for determining mini-batch size.

[0027] Each host computer in the distributed computing system can provide the embedding layer output to the system or subsystem that implements the remaining layers of the neural network. Such system or subsystem can be implemented on the host computer or on separate physical hardware. Training proceeds as described above. Updating the embedding layer involves, by each host computer, updating the values ​​in a partition of the lookup table stored on the host computer based on the gradients received from the other systems or subsystems that implement the remaining layers of the neural network.

[0028] Because each host computer has a limited amount of scratch memory to store the acquired embeddings, each host computer can only acquire embeddings for a limited number of training examples at a time. However, the amount of scratch memory used by a training example varies depending on the number of features the training example has. For example, referring again to a neural network configured to generate movie recommendations for users of a streaming service by modeling their viewing histories, some users have larger viewing histories than others. For example, a training example for a first user may include hundreds of features (i.e., movies), while a training example for a second user may include only a few dozen features. Because the first user's training example has more features, the host computer will have to acquire and store more embeddings for the first training example, and those embeddings will take up more space in the scratch memory. As another example, referring again to a neural network configured to generate search engine results based on queries written in English, some queries have more words than others. The host computer will have to acquire and store more embeddings for queries with more words, and those embeddings will take up more space in the scratch memory than embeddings for queries with fewer words. Additionally, the neural network can take into account a user's query history, but the number of queries in the query history can vary from user to user. Query histories with a larger number of queries have more embeddings and therefore take up more space in the temporary memory. Conversely, some queries include repeated words. The host computer only needs to obtain a single copy of the embedding for any repeated words, so queries with repeated words will have fewer embeddings and therefore take up less space in the temporary memory. Similarly, two training examples in a single mini-batch may have common features. The host computer only needs to obtain a single copy of the embedding for any features that are repeated between training examples.

[0029] In summary, the number of training examples that a host computer can process at one time varies widely based on the nature of the training examples. Specifically, the more unique features that appear in a given set of training examples, the more embeddings the host computer must obtain from a lookup table. More embeddings take up more space in temporary memory than fewer embeddings. The distributed computing system described in this specification can optimize the number of training examples that a host computer processes at one time, i.e., the size of each mini-batch, and therefore the number of mini-batches of training examples per batch. Specifically, each host computer receives a batch of training examples, divides the batch of training examples into an optimal number of mini-batches N, and processes one mini-batch at a time until it has generated an embedding for each training example. Figure 2 This method is described in more detail.

[0030] Figure 2FIG. 2 is a flow chart of an exemplary process 200 for dividing a batch of neural network training examples into an appropriate number of mini-batches. The exemplary process 200 may be performed by the distributed computing system 100 .

[0031] Each host computer in the system receives a corresponding batch of training examples for processing during an iteration of a training process for training a neural network (210). Each batch of training examples typically has the same number of training examples. Each training example has zero or more features. The number of features in each training example varies.

[0032] Each host computer calculates the minimum number of mini-batches into which it can partition the corresponding batch of training examples received so that it can process each mini-batch using the embedding layer of the neural network without exceeding the host computer's computational resources (220). Typically, the host computer is limited by the amount of scratch memory (e.g., SRAM) on the host computer because, to process the training examples, the host computer obtains, for each feature in each training example, an embedding from a partition of a lookup table stored on one of the host computers in the distributed computing system. The host computer stores the obtained embeddings in scratch memory on the host computer in preparation for processing those embeddings.

[0033] Computing the minimum number of mini-batches into which a host computer can divide its batch of training examples involves, for values ​​of M from an upper bound to a lower bound, repeatedly: (i) splitting a corresponding batch of training examples received by the host computer into M mini-batches; and (ii) for each mini-batch, determining whether an embedding corresponding to each input feature in each training example in the mini-batch fits in a temporary memory on the host computer.

[0034] The following pseudocode illustrates this process:

[0035]

[0036] In the pseudocode, i is the training example index, and len is the total number of training examples in the batch of training examples. The first conditional statement tests whether the minibatch containing training examples 0 through i-1 fits in the scratch memory on the host computer. If so, the host computer increments i. The host computer repeats this process until the minibatch containing training examples 0 through i-1 does not fit in the scratch memory. If the last minibatch that fits in the scratch memory on the host computer has no training examples, the host computer returns an error. Otherwise, the host computer outputs the next largest minibatch that fits in the scratch memory, which is the minibatch containing training examples 0 through i-2. Assume that i equals 7. This means that the minibatch containing training examples 0 through 5 fits in the scratch memory, but the minibatch containing training examples 0 through 6 does not. The host computer then sets the variable starting at 6 and repeats this process for the minibatch containing training examples 6 through i-1. The result of this iterative process is a number of minibatches that each fit in the scratch memory on the host computer.

[0037] In some implementations, the above algorithm is modified to test non-sequential training examples. For example, while a mini-batch containing training examples 0 through 6 may not fit in the scratchpad memory on the host computer, a mini-batch containing training examples 0 through 5 and 7 may fit. As mentioned above, this is because the size of each training example varies. In some cases, modifying the algorithm in this way can reduce the number of mini-batches obtained. The above algorithm is illustrative only, and other suitable algorithms can be used.

[0038] Determining whether a particular set of training examples (i.e., a particular mini-batch) fits into scratch memory on a host computer generally involves: (i) calculating the amount of scratch memory used by each training example, for example, by multiplying the number of features in each training example by the size of the features, (ii) summing the results of step (i), and (iii) determining whether the sum exceeds the total available scratch memory on the host computer. When calculating the amount of scratch memory used by a particular training example, the host computer takes into account any embeddings that have already been considered, such as repeated embeddings from previous training examples. In this example, the amount of scratch memory required is reduced by the amount corresponding to the repeated embeddings from previous training examples.

[0039] The host computer also considers the amount of scratch memory it uses for processing (eg, for executing the optimization algorithm described above).

[0040] Using the results computed in the previous step, the system determines the largest minimum number of mini-batches (N) into which any one of the host computers can divide its corresponding batch of training examples (230). In some implementations, one of the host computers is a designated master host, and the designated master host makes the determination. In such implementations, each other host computer sends its computed minimum number of mini-batches to the designated master host, which determines the maximum of the minimum values. The designated master host can be rotated among the host computers for different iterations of the training process. This is advantageous because the system can compute the number of mini-batches for multiple batches of training examples simultaneously. After computing N, the designated master host broadcasts the value N to all other host computers. In some other implementations, each host sends its computed minimum number to a separate component of the distributed computing system, which determines the maximum of the minimum values. Other techniques for determining the largest minimum number of mini-batches are also possible.

[0041] Each host computer divides the corresponding batch of training examples received by the host computer into N mini-batches (240). In some implementations, when the host computer calculates the minimum number of mini-batches into which the host computer can divide the corresponding batch of training examples, it also divides the training examples of the batch into that number of mini-batches. In such implementations, dividing the corresponding batch of training examples into N mini-batches involves re-dividing or re-dividing the mini-batches already existing on the host computer into N mini-batches. In some cases, N and the minimum number of mini-batches for the host computer may be the same. In other cases, N may be greater than the minimum number of mini-batches for the host computer. In such cases, the host computer may generate a mini-batch without training examples. In other implementations, the host computer may split a mini-batch into multiple mini-batches, which may involve splitting the mini-batch into two mini-batches with the same number of training examples. Other splitting methods are possible.

[0042] Finally, each host computer processes N mini-batches (250) using the embedding layer of the neural network. Each host computer processes only one mini-batch at a time. Figure 3 Describe this process in more detail.

[0043] Figure 3 is a flow chart of an exemplary process 300 for processing mini-batches. Process 300 may be performed by distributed computing system 100.

[0044] The host computers of the distributed computing system each receive instructions to execute a batch of training examples (310). Typically, the number of training examples in each batch is the same, but the training examples in each batch are different.

[0045] Host computer execution reference Figure 2 The process described is as follows, i.e., each host computer divides its corresponding batch of training examples into N mini-batches (320). The host computers process these mini-batches in parallel. However, due to the computational constraints described previously, a particular host computer processes only one of its N mini-batches at a time.

[0046] Processing the mini-batch by the host computer involves, for each input feature in each training example in the mini-batch, obtaining an embedding corresponding to the input feature from a trainable lookup table (330). The trainable lookup table is distributed among the host computers, so obtaining the embedding includes sending a lookup request to other host computers. The lookup request includes an index of the desired embedding. The host computer stores the obtained embedding in a temporary memory on the host computer.

[0047] The host computer processes these embeddings to generate an embedding layer output for each training example (340). Generating the embedding layer output for a particular training example involves adding or concatenating the embeddings for each feature of the particular training example. Specific techniques are described in more detail in U.S. Patent No. 9,898,441, which is incorporated herein by reference.

[0048] The host computer provides the embedding layer output for each training example to a system configured to implement the remaining layers of the neural network (350). Such a system can be implemented on the host computer or on separate physical hardware.

[0049] In some cases where embeddings are updated as part of a training process for training a neural network, the host computer receives backpropagation data from other systems. The host computer uses the backpropagation data to update the embeddings stored in the distributed lookup table (360). Specifically, using the backpropagation data for the batch of training examples, each host computer can calculate the gradient of the loss relative to the current value of each dimension of each embedding stored by the host computer. To reduce the loss contribution of each dimension, the host computer can adjust the value of the dimension by the ratio of the gradient. This is called stochastic gradient descent. Other suitable methods known in the art can be used to update these embeddings. This completes one iteration of training. By simply omitting step 360, the exemplary process 300 can also be used to process batches of inputs during inference.

[0050] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs (i.e., one or more modules of computer program instructions) encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof. Alternatively or in addition, the program instructions may be encoded on an artificially generated propagating signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0051] The term "data processing equipment" refers to data processing hardware and includes all kinds of equipment, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The equipment may also be or also include special-purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits). In addition to hardware, the equipment may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof.

[0052] A computer program (which may also be referred to or described as a program, software, software application, app, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may (but need not) correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store portions of one or more modules, subroutines, or code). A computer program may be deployed to execute on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communications network.

[0053] A system of one or more computers configured to perform a specific operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when operated, causes the system to perform the operation or action. A system of one or more computer programs configured to perform a specific operation or action means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0054] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry (e.g., FPGA or ASIC) or by a combination of special purpose logic circuitry and one or more programmed computers.

[0055] A computer suitable for executing a computer program can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, a central processing unit will receive instructions and data from a read-only memory or random access memory, or both. The basic elements of a computer are a central processing unit for executing or implementing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory can be supplemented by or incorporated into dedicated logic circuits. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or be operatively coupled to receive data from or transfer data to the one or more mass storage devices, or both. However, a computer need not have such devices. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive.

[0056] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM optical disks.

[0057] To provide for interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse, trackball, or presence-sensitive display or other surface) through which the user can provide input to the computer. Other types of devices may also be used to provide for interaction with the user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. Furthermore, a computer may interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smartphone), running a messaging application, and receiving a response message back from the user.

[0058] Although this specification includes many specific implementation details, these details should not be understood as limitations on the scope of any invention or on the scope of what is claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable subcombination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed as such, in some cases, one or more features from a claimed combination may be deleted from the combination, and a claimed combination may involve a subcombination or a variant of a subcombination.

[0059] Similarly, although operations are depicted in a particular order in the figures, this should not be construed as requiring that such operations be performed in the particular order shown, or in sequential order, or that all illustrated operations be performed, in order to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0060] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

Claims

1. A system comprising a plurality of host computers configured to process an embedding layer of a neural network during training of the neural network, wherein the system is configured to perform operations comprising: receiving, by each host computer, a respective batch of training examples for processing during an iteration of a training process for training the neural network, each training example having zero or more features, and wherein at least two different training examples in the respective batches received by the host computer have a different number of features; computing, based on the number of features in each of the training examples in the respective batch received by the host computer, a minimum number of mini-batches into which the host computer can divide the respective batch of training examples such that the host computer can process each mini-batch using the embedding layer of the neural network without exceeding available computing resources on the host computer, wherein at least two of the host computers compute different minimum numbers of mini-batches as a result of the at least two different training examples having different numbers of features; Determining, based on the minimum number of mini-batches, a maximum minimum number of mini-batches N among the minimum number of mini-batches calculated for each host computer; generating N mini-batches from the corresponding batches of training examples received by the host computer; and The N mini-batches of the training examples are processed by each host computer using the embedding layer of the neural network.

2. The system according to claim 1, wherein: Examples of training using the embedding layer of the neural network to process corresponding mini-batches include: For each input feature in each training example in the mini-batch, obtaining a vector corresponding to the input feature from a trainable lookup table; and For each training example, the obtained vector is processed to generate a vector representation.

3. The system according to claim 2, wherein: The trainable lookup table is distributed in the host computer.

4. The system according to claim 3, wherein: Computing the minimum number of mini-batches M into which the host computer can divide the corresponding batch of training examples comprises repeatedly, for values ​​of M starting from an upper bound: splitting the corresponding batch of training examples received by the host computer into M mini-batches; for each of the M mini-batches, determining whether the vector corresponding to each input feature of each training example in the mini-batch fits into memory on the host computer; as well as Until any of the M mini-batches does not fit into the memory on the host computer, M is decremented.

5. The system according to claim 1, wherein The operations further include: dividing the corresponding batch of training examples received by the host computer into a minimum number of mini-batches calculated by the host computer; and If N is greater than the host computer's calculated minimum number of mini-batches, then a mini-batch is re-divided into multiple mini-batches.

6. The system according to claim 5, wherein: Splitting a mini-batch into multiple mini-batches involves generating mini-batches with no training examples.

7. The system according to claim 1, wherein: One of the host computers is a designated master host, wherein the designated master host is configured to perform operations comprising: receiving from each host computer the minimum number of mini-batches into which the host computer is capable of dividing the corresponding batch of training examples; determining the largest minimum number of mini-batches N into which any host computer can divide its corresponding batch of training examples; and Sends data specifying N to each host computer.

8. The system according to claim 7, wherein: The designated master host is rotated among the host computers for different iterations of the training process.

9. A method performed by a plurality of host computers configured to process an embedding layer of a neural network during training of the neural network, the method comprising: receiving, by each host computer, a respective batch of training examples for processing during an iteration of a training process for training the neural network, each training example having zero or more features, and wherein at least two different training examples in the respective batches received by the host computer have a different number of features; computing, based on the number of features in each of the training examples in the respective batch received by the host computer, a minimum number of mini-batches into which the host computer can divide the respective batch of training examples such that the host computer can process each mini-batch using the embedding layer of the neural network without exceeding available computing resources on the host computer, wherein at least two of the host computers compute different minimum numbers of mini-batches as a result of the at least two different training examples having different numbers of features; Determining, based on the minimum number of mini-batches, a maximum minimum number of mini-batches N among the minimum number of mini-batches calculated for each host computer; generating N mini-batches from the corresponding batches of training examples received by the host computer; and The N mini-batches of the training examples are processed by each host computer using the embedding layer of the neural network.

10. The method according to claim 9, wherein: Examples of training using the embedding layer of the neural network to process corresponding mini-batches include: For each input feature in each training example in the mini-batch, obtaining a vector corresponding to the input feature from a trainable lookup table; and For each training example, the obtained vector is processed to generate a vector representation.

11. The method according to claim 10, wherein: The trainable lookup table is distributed in the host computer.

12. The method according to claim 11, wherein Computing the minimum number of mini-batches M into which the host computer can divide the corresponding batch of training examples comprises repeatedly, for values ​​of M starting from an upper bound: splitting the corresponding batch of training examples received by the host computer into M mini-batches; for each of the M mini-batches, determining whether the vector corresponding to each input feature of each training example in the mini-batch fits into memory on the host computer; as well as Until any of the M mini-batches does not fit into the memory on the host computer, M is decremented.

13. The method according to claim 9, further comprising: dividing the corresponding batch of training examples received by the host computer into a minimum number of mini-batches calculated by the host computer; as well as If N is greater than the host computer's calculated minimum number of mini-batches, then a mini-batch is re-divided into multiple mini-batches.

14. The method according to claim 13, wherein Splitting a mini-batch into multiple mini-batches involves generating mini-batches with no training examples.

15. The method according to claim 9, wherein One of the host computers is a designated master host, wherein the designated master host is configured to perform operations comprising: receiving from each host computer the minimum number of mini-batches into which the host computer is capable of dividing the corresponding batch of training examples; determining the largest minimum number of mini-batches N into which any host computer can divide its corresponding batch of training examples; and Sends data specifying N to each host computer.

16. The method according to claim 15, wherein The designated master host is rotated among the host computers for different iterations of the training process.

17. One or more non-transitory computer-readable storage media storing instructions that, when executed by a processor of a system comprising a plurality of host computers configured to process an embedding layer of a neural network during training of the neural network, cause the system to perform operations comprising: receiving, by each host computer, a respective batch of training examples for processing during an iteration of a training process for training the neural network, each training example having zero or more features, and wherein at least two different training examples in the respective batches received by the host computer have a different number of features; computing, based on the number of features in each of the training examples in the respective batch received by the host computer, a minimum number of mini-batches into which the host computer can divide the respective batch of training examples such that the host computer can process each mini-batch using the embedding layer of the neural network without exceeding available computing resources on the host computer, wherein at least two of the host computers compute different minimum numbers of mini-batches as a result of the at least two different training examples having different numbers of features; Determining, based on the minimum number of mini-batches, a maximum minimum number of mini-batches N among the minimum number of mini-batches calculated for each host computer; generating N mini-batches from the corresponding batches of training examples received by the host computer; and The N mini-batches of the training examples are processed by each host computer using the embedding layer of the neural network.

18. The computer-readable storage medium of claim 17, wherein: Examples of training using the embedding layer of the neural network to process corresponding mini-batches include: For each input feature in each training example in the mini-batch, obtaining a vector corresponding to the input feature from a trainable lookup table; and For each training example, the obtained vector is processed to generate a vector representation.

19. The computer-readable storage medium of claim 18, wherein: The trainable lookup table is distributed in the host computer.

20. The computer-readable storage medium of claim 19, wherein: Computing the minimum number of mini-batches M into which the host computer can divide the corresponding batch of training examples comprises repeatedly, for values ​​of M starting from an upper bound: splitting the corresponding batch of training examples received by the host computer into M mini-batches; for each of the M mini-batches, determining whether the vector corresponding to each input feature of each training example in the mini-batch fits into memory on the host computer; as well as Until any of the M mini-batches does not fit into the memory on the host computer, M is decremented.

Images