Dual-Momentum Gradient Optimization with Reduced Memory Requirements

Abstract

Systems and methods related to dual momentum gradient optimization with reduced memory requirements are described. An exemplary method is described in a system including a gradient optimizer and a memory configured to store momentum values ​​associated with a neural network model including L layers. The method includes retrieving from the memory a first set of momentum values ​​and a second set of momentum values ​​corresponding to layers of the neural network model having a selected storage format. The method further includes converting the first set of momentum values ​​to a third set of momentum values ​​having a training format associated with the gradient optimizer, and converting the second set of momentum values ​​to a fourth set of momentum values ​​having a training format associated with the gradient optimizer. The method further includes performing gradient optimization using the third set of momentum values ​​and the fourth set of momentum values.

Description

[Technical field]

[0001] One embodiment of the present invention, for example, relates to dual momentum gradient optimization with reduced memory requirements. [Background technology]

[0002]

[0001] Trained models, including neural network models, are used to perform complex tasks such as reading comprehension, language translation, image recognition, or speech recognition. Machine learning services, such as those based on natural language processing (NLP), recurrent neural networks (RNNs), convolutional neural networks (CNNs), long short-term memory (LSTM) neural networks, or gated recurrent units (GRUs), are deployed to perform such complex tasks. These types of models are trained using a variety of architectures and techniques, but the underlying architectures and techniques for training continue to require improvement. Summary of the Invention [Problem to be solved by the invention]

[0003] One embodiment of the present invention, for example, relates to dual momentum gradient optimization with reduced memory requirements. [Means for solving the problem]

[0004] In one example, the disclosure relates to a method in a system including a gradient optimizer and a memory configured to store momentum values ​​associated with a neural network model including L layers, where L is an integer greater than 1. The method may include retrieving from the memory a first set of momentum values ​​corresponding to a layer of the neural network model having a selected storage format, and retrieving from the memory a second set of momentum values ​​corresponding to the layer of the neural network model having the selected storage format. The method may further include converting the first set of momentum values ​​having the selected storage format to a third set of momentum values ​​having a training format associated with the gradient optimizer, and converting the second set of momentum values ​​having the selected storage format to a fourth set of momentum values ​​having a training format associated with the gradient optimizer. The method may further include performing a gradient optimization using the third set of momentum values ​​and the fourth set of momentum values ​​with the gradient optimizer.

[0005] In another example, the disclosure relates to a system including a gradient optimizer. The system may include a memory configured to store momentum values ​​associated with a neural network model including L layers, where L is an integer greater than 1. The system may be configured to: (1) retrieve from the memory a first set of momentum values ​​corresponding to layers of the neural network model having a selected storage format, and retrieve from the memory a second set of momentum values ​​corresponding to layers of the neural network model having the selected storage format, (2) convert the first set of momentum values ​​having the selected storage format to a third set of momentum values ​​having a training format associated with the gradient optimizer, and convert the second set of momentum values ​​having the selected storage format to a fourth set of momentum values ​​having a training format associated with the gradient optimizer, and (3) perform a gradient optimization using the third set of momentum values ​​and the fourth set of momentum values ​​using the gradient optimizer.

[0006] In yet another example, the disclosure relates to a method in a system including a gradient optimizer and a memory configured to store weights and momentum values ​​associated with a neural network model including L layers, where L is an integer greater than 1. The method may further include retrieving from the memory a first set of momentum values ​​corresponding to the layers of the neural network model having a selected storage format, and retrieving from the memory a second set of momentum values ​​corresponding to the layers of the neural network model having the selected storage format. The method may further include converting the first set of momentum values ​​having the selected storage format to a third set of momentum values ​​having a training format associated with the gradient optimizer, and converting the second set of momentum values ​​having the selected storage format to a fourth set of momentum values ​​having a training format associated with the gradient optimizer. The method may further include performing a gradient optimization using the third set of momentum values ​​and the fourth set of momentum values ​​with the gradient optimizer. The method may further include generating a fifth set of momentum values ​​and a sixth set of momentum values ​​for a next iteration of the gradient optimization. The method may further include converting the fifth set of momentum values ​​to a seventh set of momentum values ​​having a storage format by storing only a sign bit and truncating a predetermined number of bits associated with the mantissa of the fifth set of momentum values. The method may further include converting the sixth set of momentum values ​​to an eighth set of momentum values ​​having a storage format by storing only a sign bit and truncating a predetermined number of bits associated with the mantissa of the sixth set of momentum values.

[0007]

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0008]

[0006] The present disclosure is illustrated by way of example and not limitation in the accompanying figures, in which like references indicate like elements and in which elements are illustrated for simplicity and clarity and have not necessarily been drawn to scale. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an artificial intelligence training system (AITS) including a dual momentum gradient optimizer (DMGO) according to one embodiment. [Diagram 2] FIG. 1 illustrates a dual momentum gradient optimizer (DMGO) according to one embodiment. [Diagram 3] 1 is a graph containing error bound curves (EB(M)) for weight update errors as compressed momentum values ​​are swept from a truncation of 1 mantissa bit to a truncation of 23 mantissa bits. [Figure 4]

[0010] FIG. 4 shows a graph 400 with simulated convergence curves illustrating the effect of reduced precision momentum values ​​on training accuracy. [Diagram 5]

[0011] FIG. 1 illustrates a training system including a hardware-assisted gradient optimizer (HAGO), according to one embodiment. [Figure 6]

[0012] FIG. 1 is a block diagram of a hardware-assisted gradient optimizer (HAGO) including a dual momentum gradient optimizer (DMGO), according to one embodiment. [Figure 7]

[0013] 7 illustrates a timing diagram associated with the HAGO of FIG. 6, according to one embodiment. [Figure 8]

[0014] FIG. 1 illustrates a gradient optimization system according to another embodiment. [Figure 9]

[0015] FIG. 1 is a flow diagram of a method for a dual momentum gradient optimizer (DMGO) according to one embodiment. [Figure 10]

[0016] FIG. 1 is a flow diagram of another method for a dual momentum gradient optimizer (DMGO) according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010]

[0017] Examples disclosed in this disclosure relate to dual momentum gradient optimization with reduced memory requirements. A particular example relates to distributed training of models, including neural network models, requiring less memory. In a particular example training system, each worker (e.g., GPU, DSP, or ASIC accelerator) takes a subset (e.g., mini-batch) of the training data and calculates gradients, which are further averaged / reduced to update the parameters (e.g., weights) of the model. The weight updates are applied after a backpropagation step that calculates the gradients. As an example, the gradient of a loss function with respect to the weights of a network (or a portion of a network) is calculated. The gradient is then fed into an optimization technique that uses the gradient to update the weights to minimize the loss function. The goal of backpropagation is to update each weight (or at least a portion of the weights) in the network to move the actual output closer to the target output, thereby minimizing the error of each output neuron and the network as a whole.

[0011]

[0018] In certain instances, the trained model can be deployed in a cloud computing environment. Cloud computing may refer to a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing may be adopted in the market to provide ubiquitous and convenient on-demand access to a shared pool of configurable computing resources. The shared pool of configurable computing resources is rapidly provisioned through virtualization, released with little administrative effort or service provider interaction, and then scaled accordingly. Cloud computing models can be configured with various characteristics, for example, on-demand self-service, broad network access, resource pooling, rapid elasticity, metered services, etc. Cloud computing models can be used to expose various service models, for example, Hardware as a Service ("HaaS"), Software as a Service ("SaaS"), Platform as a Service ("PaaS"), Infrastructure as a Service ("IaaS"). Cloud computing models can also be deployed using different deployment models, such as private cloud, community cloud, public cloud, hybrid cloud, etc.

[0012]

[0019] Various machine learning services, such as those based on recurrent neural networks (RNN), convolutional neural networks (CNN), long short-term memory (LSTM) neural networks, or gated recurrent units (GRU), may be implemented using the examples described in this disclosure. In one example, service-related content or other information, such as words, sentences, images, videos, or other such content / information, may be converted into a vector representation. The vector representation may correspond to techniques such as RNN, CNN, LSTM, or GRU. In one example, a neural network model may include many layers, and each layer may be encoded as a matrix or vector of weights represented in the form of coefficients or constants obtained via offline training of the neural network.

[0013]

[0020] Taking LSTM as an example, an LSTM network may include a repeated sequence of RNN layers or other types of layers. Each layer of an LSTM network may consume input at a given time step, e.g., the state of the layer from the previous time step, and may generate a new set of outputs or states. When using LSTM, a single chunk of content may be encoded into a single vector or multiple vectors. As an example, a word or combination of words (e.g., a phrase, sentence, or paragraph) may be encoded as a single vector. Each chunk may be encoded into an individual layer (e.g., a particular time step) of the LSTM network. An LSTM layer may be described using a set of equations such as:

[0014] i t =ρ(W xi xt+W hi h t-1 +W ci c t-1 +b i f t =ρ(W xf x t +W hf h t-1 +W cf c t-1 +b f ) ct =f t c t-1 i t tanh(W xc x t +W hc h t-1 +b c ) o t =ρ(W xo x t +W ho h t-1 +W co c t +b o ) h t =o t tanh(c t )

[0021] In this example, within each LSTM layer, the inputs and hidden states may be processed using a combination of vector operations (e.g., dot products, inner products, or vector addition) and nonlinear functions (e.g., sigmoids, hyperbolic, and tangents). In some cases, the most computationally intensive operations may result from dot products, which may be implemented using dense matrix-vector and matrix-matrix multiplication routines. In one embodiment, the processing of vector operations and nonlinear functions may be performed in parallel.

[0015]

[0022] Gradient optimizers are one of the key components in training systems used for artificial intelligence. Gradient optimizers are designed to update a model by applying gradients on the parameters (weights) of the model so that the prediction of the next iteration can be closer to the ground truth. A specific example of the present disclosure relates to an AI training framework that supports a dual momentum gradient optimizer in which two single-precision (FP32) momentums are buffered along with the model parameters. Each update step calculates a new momentum value, which is then used to generate new weights. Both the new momentum value and the new weights are saved for the next iteration. As an example, the dual momentum adaptive momentum estimation (Adam) algorithm can be expressed as the following equation:

[0016] vt :=β1·v t +(1-β1)·dw s t :=β2·s t +(1-β2)·dw 2

[0017]

number

[0018] where v t ,s t are the two momentums, w is the model weight, dw is the gradient, a is the initial learning rate, and ε is a small value for better numerical stability.

[0023] Advantageously, the use of a dual momentum optimizer can result in neural network models that converge faster than zero / single momentum optimizers such as SGD (stochastic gradient descent). However, the use of a dual momentum optimizer may require at least twice as much memory as a zero / single momentum optimizer would require to buffer momentum other than the model weights. This tendency may be exacerbated, especially for large models that contain millions to billions of parameters. If the optimizer runs on a memory-limited GPU / ASIC, the memory space required by the optimizer will in turn reduce the maximum batch size. On the other hand, moving the optimizer to the host CPU may eliminate the memory size bottleneck. However, the memory bandwidth bottleneck may slow down the optimizer, as the processor must extra fetch two FP32 momentums from memory for each weight / gradient pair and store them.

[0019]

[0024] Furthermore, given that a model such as the Bidirectional Encoded Representation of Transformers (BERT)-large model has 1.36 gigabytes (GBs) of gradients generated per worker, buffering gradients for the entire model requires a significant amount of memory. In the context of this model, 5.44 GB of memory is required to buffer gradients in a four-worker training cluster. The memory requirement scales linearly with the number of workers. Furthermore, when coupled with algorithms such as adaptive moment estimation (Adam), which require buffering two floating-point 32-bit (FP32) momentum terms per gradient, the memory requirements for each worker can be three times that required for BERT.

[0020]

[0025] FIG. 1 illustrates an artificial intelligence training system (AITS) 100 including a dual momentum gradient optimizer (DMGO) 150 according to one embodiment. The AITS 100 may include a memory 110 (e.g., DRAM or another type of memory), a memory interface 120 (e.g., a DDR interface or another type of interface), a momentum format converter (MFC) 130, another momentum format converter (MFC) 140, and a DMGO 150. The memory 110 may be coupled to the memory interface 120 via a bus 112. Alternatively, the memory 110 may be integrated with the memory interface 120 via a chip-on-chip package, a system-in-chip package, a 3D-memory system, or a stacked die system. The memory interface 120 may be coupled to the DMGO 150 via a bus 122 such that model weights (W) may be transferred from the memory 110 to the DMGO 150 via the memory bus 122. The memory interface 120 may be coupled to the MFC 130 via buses 132 and 134. The bus 132 may be used to transfer momentum values ​​in a storage format associated with the first momentum (e.g., vt_fp8 format) from the memory 110 to the MFC 130. The bus 134 may be used to transfer momentum values ​​in a storage format associated with the second momentum (e.g., st_fp8 format) from the memory 110 to the MFC 130. In one example, the storage format may correspond to an 8-bit reduced precision format for momentum values. The MFC 130 may be configured to process both the first momentum values ​​and the second momentum values ​​to change their format from the storage format to a training format associated with the DGO 150. In one example, the training format may correspond to a full 32-bit single precision format. The MFC 130 may be further configured to transmit the first and second momentum values ​​to the DGO 150 via the buses 136 and 138, respectively. In this example, MFC 140 may be coupled to DMGO 150 via buses 142 and 144 .MFC 140 may further be coupled to memory interface 110 via buses 146 and 148. MFC 140 may convert the momentum values ​​from a training format to a storage format.

[0021]

[0026] 1, the AITS 100 can enable the implementation of new methods that use reduced precision momentum for Adam without introducing performance degradation. As an example, the dual momentum Adam algorithm can be expressed as follows:

[0022] v t :=β1·v t +(1-β1)·dw s t :=β2·s t +(1-β2)·dw 2

[0023]

number

[0024] where v t ,s t are the two momentums, w is the model weight, dw is the gradient, a is the initial learning rate, and ε is a small value for better numerical stability. In one example, one of the momentum values ​​(v t ) corresponds to the expected value of the gradient value, and the other one of the momentum values ​​(s t ) can correspond to the expected variance of the gradient values. In this example, instead of using the full 32-bit single precision momentum, we use only 8 bits: 1 bit for the sign and 1 bit for the momentum {v t ,s t} has a dynamic range of 7 bits. In this example, v t ,s tSince both are less than 1.0 by multiplication with (1-β), where β is close to 1.0, the most significant bit (MSB) of the 8-bit exponent is a zero constant and can be safely discarded before storage. In this example, the momentum value storage format is 8-bit, but the calculation format is still more precise to achieve higher accuracy. Although FIG. 1 shows AITS 100 including a certain number of components arranged in a certain manner, AITS 100 may include additional or fewer components arranged in a different manner. As an example, MFC 130 and MFC 140 may be integrated with DMGO 150 via a chip-on-chip package, a system-in-chip package, a 3D-memory system, or a stacked die system. Alternatively, MFC 130 and MFC 140 may be integrated with DMGO 150 on the same chip (e.g., as a system-on-chip (SoC)). In another example, each of memory 110, memory interface 120, MFC 130, MFC 140, and DMGO 150 may be integrated via a chip-on-chip package, a system-in-chip package, a 3D-memory system, or a stacked die system.

[0025]

[0027] FIG. 2 illustrates a dual momentum gradient optimizer (DMGO) 200 according to one embodiment. The DMGO 200 may include a momentum expander 202 and another momentum expander 204. The DMGO 200 may further include a momentum compressor 262 and another momentum compressor 264. The DMGO 200 may further include logic configured to perform various operations associated with implementing a dual momentum adaptive momentum estimation (Adam) algorithm. In this example, the momentum expander 202 and the momentum expander 204 may be used to provide functionality similar to that described above with respect to the MFC 130 of FIG. 1. Similarly, the momentum compressor 262 and the momentum compressor 264 may be used to implement functionality similar to that described above with respect to the MFC 140 of FIG. 1. A momentum value associated with a first momentum (e.g., vt_fp8) may be retrieved from memory and provided to the momentum expander 202. A momentum value associated with a second momentum (e.g., st_fp8) may be retrieved from memory and provided to the momentum expander 204. In one example, the momentum values ​​may have a storage format (e.g., an 8-bit reduced precision format). The momentum expander 202 may be configured to process the first momentum value and change its format from the storage format to a training format associated with the DMGO 200. The momentum expander 204 may be configured to process the second momentum value and change its format from the storage format to a training format associated with the DMGO 200.

[0026]

[0028] With continued reference to FIG. 2, logic included as part of the DMGO 200 may include logic configured to perform operations such as addition, multiplication, and fused multiply-add operations. The logic may be configured to implement a dual momentum optimization algorithm (e.g., a Dual Momentum Adaptive Momentum Estimation (Adam) algorithm). As an example, the logic may be configured to calculate a new momentum value (v t_new ,s t_newThe calculation corresponding to the dual momentum Adam algorithm for the calculation of ) can be expressed as follows:

[0027] v t_new :=β1·v t_fp32 +(1-β1)·dw s t_new :=β2·s t_fp32 +(1-β2)·dw 2

[0028]

number

[0029] where v t_new ,s t_new is the new momentum value, w is the model weight, dw is the gradient, a is the initial learning rate, and ε is a small value used for better numerical stability. Thus, in this example, momentum extenders 202 and 204 each increase their momentum value (v t_fp8 ,s t_fp8 ) storage format (e.g., 8-bit compressed format) to a training format (e.g., 32-bit full precision format) associated with the DMGO 200. As an example, before the gradient optimization is performed, a momentum expander (e.g., momentum expander 202 or 204) will convert the compressed 8-bit momentum value back to 32-bit single precision by inserting one "0" bit between the sign bit and the most significant bit (MSB) of the exponent and padding 23 "0" bits immediately after the least significant bit (LSB) of the exponent. The padded bits do not have to be "0" bits. In one example, at least a small number of bits may be non-zero bits. As an example, two or three LSBs may be non-zero bits.

[0030]

[0029] Still referring to FIG. 2, the logic associated with DMGO 200, including multipliers 210 and 214 and adder 212, calculates the new momentum value (v t_newSimilarly, logic associated with the DMGO 200, including multipliers 216, 218 and adder 222, may perform calculations associated with generating one of the new momentum values ​​(s t_new ) may be performed. Additionally, in this example, t_new ,s t_new Before storing the new momentum values ​​(v) in memory, momentum compressors 262 and 264 convert the format of the new momentum values ​​from the training format to the storage format. tc_new ,s tc_new 2 as an 8-bit storage format value. In this example, each of the momentum compressors 262 and 264 removes the most significant bit (MSB) of the exponent associated with the momentum value and truncates all 23 mantissa bits to produce a 32-bit full precision momentum value (v t_new ,s t_new ) into 8-bit compressed momentum value (v tc_new ,s tc_new ), which are to be stored in memory. Also, the mantissa bits less than 23 bits may be truncated. Advantageously, the compression of the momentum values ​​can reduce the buffer size required to store the momentum values ​​by a factor of four. Furthermore, the use of least significant bit (LSB) truncation can reduce accuracy without loss of training performance by introducing the same rounding error direction such that the errors can cancel each other.

[0031]

[0030] With continued reference to FIG. 2, the DMGO 200 may generate new weight values ​​(e.g., w _new ) in this example, this logic may include multipliers 232 and 234, adder 242, reciprocal square root (RSQRT) 246, and fused multiplier (FMA) 250. Although FIG. 2 shows DMGO 200 including a certain number of components arranged in a certain manner, DMGO 200 may include additional or fewer components arranged in a different manner.

[0032] When the momentum is compressed by the momentum compressor, its precision bits are tc_new ,s tc_new ) are truncated to introduce relative errors of δ1 and δ2 with respect to δ. Taking this into account, the gradient optimization equation can be rewritten as

[0033]

number

[0034] For 32-bit single precision, |δ1|≦2 m-24 , |δ2|≦2 m-24 where m is the number of bits of precision truncated. In this example, we can safely approximate 1-δ1=1-δ2≈1-δ, provided m is not too large. This error bound is only reached if the truncated m bits are all "1", which is very unlikely in real applications. Nevertheless, even if this extreme case does occur, the momentary peak error cannot adversely affect the overall performance. The relative error of the updated weights after truncating both momentums is

[0035]

number

[0036] For any δ∈[0,1], when the value of δ is small,

[0037]

number

[0038] is closer to 1 than 1-δ. This shows that reducing the precision of both momentums does not linearly reduce the precision of the updated parameters. This "square root of error" effect compensates for some of the "precision loss" due to truncation.

[0039] 3 is a graph 300 illustrating the error bound curve (EB(M)) of the weight update error as the compressed momentum value is swept from a truncation of 1 mantissa bit to a truncation of 23 mantissa bits. As shown in graph 300, in this example, when the number of truncated mantissa bits is less than 16, the error bound is close to zero compared to the full single precision (FP32) momentum value. In this example, as the number of truncated mantissa bits exceeds 17, the error bound increases rapidly, with the maximum error bound being 0.13 when all of the mantissa bits are truncated.

[0040]

[0033] Figure 4 shows a graph 400 with simulated convergence curves 410, 420, and 420 illustrating the effect of reduced precision momentum values ​​on training accuracy. As an example, simulated convergence curve 410 corresponds to a 1-bit mantissa of the momentum value, simulated convergence curve 420 corresponds to a 4-bit mantissa of the momentum value, and simulated convergence curve 430 corresponds to a 24-bit mantissa of the momentum value. As is evident from the simulated convergence curves, truncating the mantissa bits of the momentum value does not degrade performance in terms of training accuracy. However, reduced precision momentum values ​​save at least 50% of the memory footprint, so that only half the cycles may be required to read the memory.

[0041]

[0034] The techniques described as part of this disclosure may be implemented using a combination of hardware and / or software. In a particular example, the DGMO 200 of FIG. 2 may be deployed as part of a training system 500 as shown in FIG. 5. In this example, the DGMO 200 may be included as part of a Hardware Assisted Gradient Optimizer (Dual Momentum) 550 (referred to as HAGO) for training a model, according to one example. The gradient optimization system 500 may include multiple workers (e.g., Worker_1 502, Worker_2 504, and Worker_N 506) coupled to the HAGO 550 via a high-speed bus bridge 520. Each of the workers (e.g., worker_1 502, worker_2 504, and worker_N 506) may be implemented using any number of graphics processing units (GPUs), computer processing units (CPUs), memory (e.g., SRAM or other types of memory), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), erasable and / or complex programmable logic devices (PLDs), programmable array logic (PAL) devices, or general array logic (GAL) devices. Each worker may carry an entire copy of the neural network model that has been trained using the gradient optimization system 500. Although each worker may carry an entire copy of the neural network model, each worker may operate on only a subset of the training data associated with the model (e.g., images or other data samples for classification). Each worker may perform a forward pass on the data to calculate the error, and then a backward pass to calculate the gradient for each parameter (e.g., each weight). The gradient optimization system 500 may further include a host 560 coupled to the high speed bus bridge 520 via a bus 528 .

[0042]

[0035] With continued reference to FIG. 5, in one example, the high-speed bus bridge 520 may be implemented as a PCI Express (PCIe) bridge. Worker_1 502 may be connected to the high-speed bus bridge 520 via PCIe link 510. Worker_2 504 may be connected to the high-speed bus bridge 520 via PCIe link 512. Worker_N 506 may be connected to the high-speed bus bridge 520 via PCIe link 514. HAGO 550 may be coupled to the high-speed bus bridge 520 via bus links 522 and 524. Other types of bus bridges and bus links may also be used. Furthermore, instead of a bus bridge, a switching fabric may be used. Each worker may compute gradients on a subset (e.g., a mini-batch) of the training data and provide updated gradients to HAGO 550 via the high-speed bus bridge 520. HAGO 550 may perform reduction and optimization on the gradients collected from each worker (e.g., worker_1 502, worker_2 504, and worker_N 506). HAGO 550 may operate on data bursts, and in one example, each burst may consist of a fixed number of received gradients. The current gradient burst may be buffered while the previous gradient burst is optimized along with its corresponding weights. Although FIG. 5 shows gradient optimization system 500 including a certain number of components arranged in a certain manner, gradient optimization system 500 may include additional or fewer components arranged in a different manner.

[0043]

[0036] Figure 6 is a block diagram of HAGO 600 including a dual momentum gradient optimizer, according to one embodiment. In one embodiment, HAGO 550 of Figure 5 may be implemented as HAGO 600 of Figure 6. HAGO 600 may receive updated gradients from workers via a high-speed bus. The example of Figure 6 assumes that there are four workers coupled to HAGO 600. An updated gradient from one of the workers (DW1) may be received via a first-in-first-out block (e.g., GFIFO_1 602). An updated gradient from another worker (DW2) may be received via another first-in-first-out block (e.g., GFIFO_2 604). An updated gradient from another worker (DW3) may be received via another first-in-first-out block (e.g., GFIFO_3 606). An updated gradient (DW4) from another worker may be received via another first-in-first-out block (e.g., GFIFO_4 608). The FIFO may be implemented as a single multi-entry FIFO or multiple single-entry FIFOs.

[0044]

[0037] With continued reference to FIG. 6, HAGO 600 may further include a floating-point averaging block (e.g., FP-AVG block 620). The FP-AVG block 620 may receive the buffered updated gradients from the first-in-first-out block. As an example, the FP-AVG block 620 may receive the buffered updated gradients (BDW1) from GFIFO_1 602. The FP-AVG block 620 may also receive the buffered updated gradients (BDW2) from GFIFO_2 604. The FP-AVG block 620 may also receive the buffered updated gradients (BDW3) from GFIFO_3 606. The FP-AVG block 620 may also receive the buffered updated gradients (BDW4) from GFIFO_4 608. The FP-AVG block 620 may perform a reduction on the received updated gradients. In this example, the reduction operation may include averaging the received gradients to generate DW_AVG as an output. The reduction operation is not limited to averaging. Other operations such as addition or scaled addition may be used to perform the reduction. The output from the FP-AVG block 620 may be provided to a gradient optimization data path (GODP) block 650 that performs floating point operations to implement a gradient optimization algorithm such as those described above with respect to Figures 1 and 2. In one example, the FP-AVG block 620 and the GODP block 650 may be implemented as part of an FPGA.

[0045]

[0038] Still referring to FIG. 6, a master copy of the weights (W) and optional momentum (M) may be stored in a memory 660. In one example, the memory 660 may be implemented using double data rate (DDR) dynamic random access memory (DRAM). The memory 660 may be coupled to a first-in-first-out block (e.g., PFIFO block 670) via a memory interface 662. The memory interface 662 may be a DDR interface. As shown in FIG. 6, the weights (W) of the model may be loaded into the PFIFO 670 under the control of a control finite state machine (FSM) (e.g., control FSM 640). Additionally, the memory 660 may be coupled to a momentum format converter (MFC) 684 via the memory interface 662. The MFC 684 may provide functionality similar to that previously described with respect to the MFC 130 of FIG. 1. An output of the MFC 684 may be coupled to the PFIFO 670 (or another FIFO).

[0046]

[0039] In one example, simultaneously, control FSM 640 enables both (1) each worker to send its locally computed gradients (e.g., DW1, DW2, DW3, and DW4) to its respective GFIFO (e.g., GFIFO_1 602, GFIFO_2 604, GFIFO_3 606, and GFIFO_4 608) and (2) memory interface 662 to fetch the same number of parameters (e.g., weights (W)) along with optional momentum from memory 660 to PFIFO 670. Thus, in this example, control FSM 640 controls memory interface 662 to fetch the same number of weights to PFIFO 670 as the number of gradients injected into GFIFO during a single burst cycle. When each of these FIFOs reaches the burst size, their respective read ports are activated by control FSM 640 to output the contents of their respective FIFOs with a single cycle latency. In this example, the control FSM 640 enables weighted reads from the PFIFO 670 and GFIFO such that data from these buffers is flushed with the same latency. In this way, data reads are perfectly coordinated.

[0047]

[0040] With continued reference to FIG. 6, this implementation ensures that the four gradients output from each FIFO (e.g., BDW1, BDW2, BDW3, and BDW4) have the same alignment index. The FP-AVG block 620 then calculates the average of these gradients (e.g., BDW1, BDW2, BDW3, BDW4) to generate a reduced gradient (e.g., DW_AVG), which is then processed by the GODP block 650 along with the buffered weights and momentum. In this example, the output of the GODP block 650 may include two parts: new weights (NW) and new momentum (NM), both of which are written back to the memory 660 via the memory interface 662. The new momentum values ​​may be provided to the MFC 682, which may implement similar functionality as described above with respect to the MFC 140 of FIG. 1. Thus, in this embodiment, the compressed momentum values ​​may be stored in a storage format that requires less storage space.

[0048]

[0041] Additionally, at the same time, new weights (NW) are injected into a transmit FIFO (TXFIFO) that sends the new weights back to the workers. In this embodiment, the control FSM 640 may control the flow of weights to the workers via the high-speed bus bridge 520 of FIG. 5. As an example, the control / status registers 630 may be used to specify timing associated with various operations. Additionally, the control / status registers 630 may be used to configure hyperparameters (e.g., learning rate) of the gradient optimizer.

[0049]

[0042] During each read from the GFIFO and weight update, the next burst of gradients is written to the respective GFIFO. Similarly, the next set of weights and momentum are fetched from memory 660 to PFIFO 670 (e.g., via MFC 684) during each read from PFIFO 670 and weight update. In this example, since memory 660 supports only a single data port shared by both read and write operations, the data width of PFIFO 670 is configured to be twice the data width of each GFIFO. Thus, in this example, all data bursts except the first and last will have an overlap between receiving gradients and sending new weights.

[0050]

[0043] With further reference to Figure 6, certain aspects of the operation of HAGO 600 may be controlled using control / status registers in conjunction with control FSM 640. Table 1 below shows exemplary contents of control / status registers 630.

[0051] [Table 1]

[0052] As shown in Table 1, the ctrl_opt mode register allows for specifying an optimization algorithm type, which may be any dual momentum optimization algorithm. The ctrl_burst_size register may select the burst size associated with each burst cycle. In this example, the burst size (e.g., specified in number of gradients) may be 256 gradients, 512 gradients, 1024 gradients, 2048 gradients, or 4096 gradients. The ctrl_param_num register may specify the size of the model's parameters. The ctrl_init_lr register may specify the initial learning rate of the model. In one embodiment, the initial learning rate may be related to the model's step size. As previously described, in one embodiment, the optimization algorithm may be the dual momentum Adam algorithm, which may be expressed as follows:

[0053] v t :=β1·v t +(1-β1)·dw s t :=β2·s t +(1-β2)·dw 2

[0054]

number

[0055] where v t ,s t are two momentums, w are the model weights, dw are the gradients, a is the initial learning rate, and ε is a small value for better numerical stability. The ctrl_beta1 register allows for the specification of a coefficient of momentum that can control the acceleration associated with the rate at which weights are updated. The ctrl_beta2 register allows for the specification of a coefficient of a second momentum that can further control the acceleration associated with the rate at which weights are updated. The ctrl_is_fp16 register is set if the format of the input gradient is the FP16 format. Although Table 1 illustrates a particular set of registers included as part of the control / state registers 630, additional or fewer registers may be used with HAGO 600. Additionally, at least some or all of the control information included in the control / state registers 630 may be specified via fields or operands associated with instructions associated with gradient optimization using HAGO 600. Although FIG. 6 illustrates HAGO 600 including a certain number of components arranged in a certain manner, HAGO 600 may include additional or fewer components arranged in a different manner. As an example, five or more workers may be coupled to HAGO 600.

[0056]

[0045] Figure 7 shows a timing diagram 700 associated with HAGO 600 of Figure 6, according to one embodiment. The timing diagram 700 describes the operation of HAGO 600 in terms of bursts. The timing diagram 700 shows K+1 burst cycles, where K is a positive integer. The first burst cycle (Burst_1) and the last burst cycle (Burst_K+1) contain less overlap than the other burst cycles: Burst_2 cycle through Burst_K cycle. The Burst_1 cycle contains the timing (GW) involved from the start of sending gradients (e.g., DW1) from the workers to injecting the gradients into the FIFO (e.g., GFIFO_1 602 of Figure 6) up to the burst size of the FIFO. Multiple workers (e.g., worker_1 502, worker_2 504, worker_N 506 in FIG. 5) can inject gradients into the FIFO up to the burst size of the FIFO during a burst_1 cycle. The burst_1 cycle further includes a timing (PW) involved in starting the transfer of parameters (e.g., model weights) stored in a memory (e.g., memory 660 in FIG. 6) until the injection of weights into the PFIFO 670 in FIG. 6. As shown in FIG. 7, the transfer of weights is delayed by the read latency (L) of the memory (e.g., memory 660 in FIG. 6). DDR ) Therefore, in a burst_1 cycle, HAGO600 overlaps two operations: injecting gradients into the FIFO and fetching weights from memory.

[0057]

[0046] With continued reference to FIG. 7, the burst_2 cycle includes the timing involved in starting to send the next set of gradients (e.g., DW1) from a worker until the injection of the gradients into the FIFO (e.g., GFIFO_1 602 in FIG. 6) up to the burst size of the FIFO (GW). Multiple workers (e.g., worker_1 502, worker_2 504, worker_N 506 in FIG. 5) can inject gradients into the FIFO up to the burst size of the FIFO during the burst_2 cycle. While the next set of gradients is being injected into the FIFO, the burst_2 cycle also includes a computation cycle including the computation of the average of the gradients by the FP-AVG block 620, and the processing of the reduced gradients along with the buffered momentum and buffered weights (BM, BW) by the GODP block 650. The timing involved as part of this operation is labeled as GPR in FIG. 7. The timing involved in writing the new momentum and new weights (NM, NW) output from the GODP block 650 back to the memory 660 is labeled as NP. The Burst_2 cycle further includes an overlapping timing involved in sending the new weights to the workers via the TX FIFO 680. As shown in the timing diagram 700 of FIG. 7, this operation reduces the processing latency (L) of the GODP block 650. DP ) The burst_2 cycle further includes the timing (PW) involved from the start of the transfer of parameters (e.g., model weights) stored in a memory (e.g., memory 660 in FIG. 6) to the injection of the weights into the PFIFO 670 in FIG. 6. As shown in FIG. 7, the transfer of the weights is affected by the read latency (L DDR ). Thus, in a burst_2 cycle, HAGO 600 at least partially overlaps four operations as part of the gradient optimization: (1) injecting gradients into a FIFO (e.g., one of the GFIFOs), (2) reading gradients and weights from the GFIFO and PFIFO, (3) writing new weights and new momentum back to memory, and (4) transferring weights to the workers.

[0058]

[0047] With further reference to Figure 7, the control FSM 640 continues to operate HAGO 600 in a streaming manner such that successive burst cycles result in the streaming of gradients through HAGO 600. Thus, the burst_3 cycle includes additional streaming of gradients and overlapping of multiple operations as part of gradient optimization, including at least partial overlapping of the following four operations: (1) injecting gradients into a FIFO (e.g., either the GFIFO); (2) reading gradients and weights from the GFIFO and PFIFO; (3) writing new weights and new momentum back to memory; and (4) transferring new weights to the workers. Similarly, the burst_K cycle includes at least partial overlapping of the following four operations as part of gradient optimization: (1) injecting gradients into a FIFO (e.g., either the GFIFO); (2) reading gradients and weights from the GFIFO and PFIFO; (3) writing new weights and new momentum back to memory; and (4) transferring new weights to the workers. The burst_K+1 cycle corresponds to the last burst cycle. Therefore, the burst_K+1 cycle has less overlap compared to the other burst cycles: burst_2 cycle to burst_K cycle. As part of the burst_K+1 cycle, three operations overlap: (1) reading gradients and weights from GFIFO and PFIFO, (2) writing new weights and new momentum back to memory, and (3) sending new weights to workers.

[0059]

[0048] With continued reference to Figure 7, during each single burst cycle, HAGO 700 may operate in gradient bursts, with each gradient burst including a fixed number of gradients. The number of gradients operated during a single burst cycle corresponds to a burst size, which may be configured via the control / status register 630, as described above with respect to Figure 6.

[0060] FIG. 8 illustrates a gradient optimization system 800, including a software version of a dual momentum gradient optimizer (DMGO), according to another embodiment. In this example, an implementation of logic corresponding to the DMGO (e.g., DMGO 200 of FIG. 2) may be implemented using instructions stored in a memory when executed by a processor. The gradient optimization system 800 includes a processor 802, an I / O component 804, a memory 806, a presentation component 808, a sensor 810, a database 812, a networking interface 814, and an I / O port 816, which may be interconnected via a bus 820. The processor 802 may execute instructions stored in the memory 806. The I / O component 804 may include components such as a keyboard, a mouse, a voice recognition processor, or a touch screen. The memory 806 may be any combination of non-volatile or volatile storage (e.g., flash memory, DRAM, SRAM, or other types of memory). Instructions corresponding to methods related to the DMGO may be stored in the memory 806 or in another memory. These instructions, when executed by processor 802, or another processor (e.g., another CPU or GPU), may provide functionality related to these methods. As an example, instructions including right shift / left shift and packing / unpacking instructions may be used to compress or expand momentum values. Table 2 below shows an example set of instructions that, when executed by processor 802, may perform functionality corresponding to a DMGO (e.g., DMGO 200 of FIG. 2).

[0061] [Table 2]

[0062]

[0050] Still referring to FIG. 8, the presentation component 808 may include a display, a holographic device, or other presentation device. The display may be any type of display, such as an LCD, LED, or other type of display. The database 812 may be used to store data as needed for the execution of the methods described herein. The database 812 may be implemented as a collection of distributed databases or as a single database. The network interface 814 may include a communication interface, such as an Ethernet, cellular wireless, Bluetooth wireless, UWB wireless, or other type of wireless or wired communication interface. The I / O port 816 may include an Ethernet port, a fiber optic port, a wireless port, or other communication port. Although FIG. 8 illustrates the gradient optimization system 800 as including a certain number of components arranged and coupled in a certain manner, it may include fewer or additional components arranged and coupled in a different manner. Additionally, the functionality associated with the system 800 may be distributed as needed.

[0063]

[0051] Figure 9 shows a flow diagram 900 of a method for gradient optimization according to one embodiment. The method may be performed using the AITS 100 of Figure 1, the DMGO 200 of Figure 2, or the gradient optimization system 800 of Figure 8. Thus, as an example when the DMGO 200 is implemented as part of the HAGO 600, steps corresponding to the method may be performed based on control signals received from the control FSM 640. Additionally, although not shown in Figure 6, a clock signal may be used to synchronize timing associated with operations performed as part of the method. Alternatively, instructions stored in the memory 806, when executed by the processor 802, may be used to perform steps associated with the method. Additionally, modified combinations of the DMGO 200 and the gradient optimization system 800 may also be used. Thus, as an example, the momentum compression and expansion steps may be performed using instructions stored in the memory, while the gradient optimization may be performed using hardware logic associated with the DMGO 200. Step 910 may include retrieving from memory a first set of momentum values ​​corresponding to layers of the neural network model having a selected storage format, and retrieving from memory a second set of momentum values ​​corresponding to layers of the neural network model having a selected storage format, As discussed above, the storage format may correspond to a reduced precision format.

[0064]

[0052] Step 920 may include converting the first set of momentum values ​​having the selected storage format to a third set of momentum values ​​having a training format associated with the gradient optimizer, and converting the second set of momentum values ​​having the selected storage format to a fourth set of momentum values ​​having a training format associated with the gradient optimizer. As previously described, this step may include expanding the momentum values ​​by padding the momentum values ​​in the storage format with extra zero bits. The extra zero bits added may depend on the number of bits truncated as part of the compression of the momentum values.

[0065]

[0053] Step 930 may include performing a gradient optimization using the third set of momentum values ​​and the fourth set of momentum values ​​with a gradient optimizer. After the calculations associated with the gradient optimization, the momentum values ​​may be converted to a storage format. As previously described, this step may include storing only the sign bit and the seven most significant bits (MSBs) of the momentum values ​​in a single precision format. Although FIG. 9 describes a number of steps performed in a certain order, additional or fewer steps in a different order may be performed.

[0066]

[0054] Figure 10 shows a flow diagram 1000 of a method for gradient optimization according to one embodiment. The method may be performed using the AITS 100 of Figure 1, the DMGO 200 of Figure 2, or the gradient optimization system 800 of Figure 8. Thus, as an example when the DMGO 200 is implemented as part of the HAGO 600, steps corresponding to the method may be performed based on control signals received from the control FSM 640. Additionally, although not shown in Figure 6, a clock signal may be used to synchronize timing associated with operations performed as part of the method. Alternatively, instructions stored in the memory 806, when executed by the processor 802, may be used to perform steps associated with the method. Additionally, modified combinations of the DMGO 200 and the gradient optimization system 800 may also be used. Thus, as an example, the momentum compression and expansion steps may be performed using instructions stored in the memory, while the gradient optimization may be performed using hardware logic associated with the DMGO 200. Step 1010 may include retrieving from memory a first set of momentum values ​​corresponding to layers of the neural network model having a selected storage format, and retrieving from memory a second set of momentum values ​​corresponding to layers of the neural network model having a selected storage format, As discussed above, the storage format may correspond to a reduced precision format.

[0067]

[0055] Step 1020 may include converting the first set of momentum values ​​having the selected storage format to a third set of momentum values ​​having a training format associated with a gradient optimizer, and converting the second set of momentum values ​​having the selected storage format to a fourth set of momentum values ​​having a training format associated with a gradient optimizer. As previously described, this step may include expanding the momentum values ​​by padding the momentum values ​​in the storage format with extra zero bits. The extra zero bits added may depend on the number of bits truncated as part of the compression of the momentum values. As previously described, this step, when performed by a processor (e.g., processor 802 of FIG. 8), may be performed using instructions stored in (e.g., memory 806 of FIG. 8). Alternatively, this step may be performed using logic associated with DMGO 200 of FIG. 2.

[0068]

[0056] Step 1030 may include performing a gradient optimization using the third set of momentum values ​​and the fourth set of momentum values ​​with a gradient optimizer. As previously described, this step, when performed by a processor (e.g., processor 802 of FIG. 8), may be performed using instructions stored in (e.g., memory 806 of FIG. 8). Alternatively, this step may be performed using logic associated with DMGO 200 of FIG. 2.

[0069]

[0057] Step 1040 may include generating a fifth set of momentum values ​​and a sixth set of momentum values ​​for the next iteration of the gradient optimization. As previously described, this step, when performed by a processor (e.g., processor 802 of FIG. 8), may be performed using instructions stored in (e.g., memory 806 of FIG. 8). Alternatively, this step may be performed using logic associated with DMGO 200 of FIG. 2.

[0070]

[0058] Step 1050 may include converting the fifth set of momentum values ​​to a seventh set of momentum values ​​having a storage format by storing only a sign bit and truncating a predetermined number of bits associated with the mantissa of the fifth set of momentum values. As previously described, this step may include storing only the sign bit and the seven most significant bits (MSBs) of the momentum values ​​in a single precision format. As previously described, this step may be performed using instructions stored in (e.g., memory 806 of FIG. 8) when performed by a processor (e.g., processor 802 of FIG. 8). Alternatively, this step may be performed using logic associated with DMGO 200 of FIG. 2.

[0071]

[0059] Step 1060 may include converting the sixth set of momentum values ​​to an eighth set of momentum values ​​having a storage format by storing only the sign bit and truncating a predetermined number of bits associated with the mantissa of the sixth set of momentum values. As previously described, this step may include storing only the sign bit and the seven most significant bits (MSBs) of the momentum values ​​in a single precision format. As previously described, this step may be performed using instructions stored in (e.g., memory 806 of FIG. 8) when performed by a processor (e.g., processor 802 of FIG. 8). Alternatively, this step may be performed using logic associated with DMGO 200 of FIG. 2. Although FIG. 10 describes a number of steps performed in a certain order, additional or fewer steps may be performed in a different order.

[0072] In conclusion, the present disclosure relates to a method in a system including a gradient optimizer and a memory configured to store momentum values ​​associated with a neural network model including L layers, where L is an integer greater than 1. The method may include retrieving from the memory a first set of momentum values ​​corresponding to a layer of the neural network model having a selected storage format, and retrieving from the memory a second set of momentum values ​​corresponding to the layer of the neural network model having the selected storage format. The method may further include converting the first set of momentum values ​​having the selected storage format to a third set of momentum values ​​having a training format associated with the gradient optimizer, and converting the second set of momentum values ​​having the selected storage format to a fourth set of momentum values ​​having a training format associated with the gradient optimizer. The method may further include performing a gradient optimization using the third set of momentum values ​​and the fourth set of momentum values ​​with the gradient optimizer.

[0073]

[0061] The selected storage format may include a reduced single precision format, and the training format may include a single precision format or a double precision format.

[0074]

[0062] Converting the first set of momentum values ​​having the selected storage format to the third set of momentum values ​​having the training format may include padding extra zero bits to form the momentum values ​​in a single precision format. The method may further include generating a fifth set of momentum values ​​and a sixth set of momentum values ​​for a next iteration of the gradient optimization, and converting each of the fifth set of momentum values ​​and the sixth set of momentum values ​​to the storage format by storing only a sign bit and the seven most significant bits associated with each respective momentum value before storing each of the fifth set of momentum values ​​and the sixth set of momentum values.

[0075]

[0063] Performing gradient optimization includes implementing an adaptive momentum estimation algorithm. The gradient optimizer may be implemented using a field programmable gate array (FPGA), and the gradient optimizer may be configured to operate in a burst mode, such that successive burst cycles result in streaming of gradients through the gradient optimizer.

[0076] In another example, the disclosure relates to a system including a gradient optimizer. The system may include a memory configured to store momentum values ​​associated with a neural network model including L layers, where L is an integer greater than 1. The system may be configured to: (1) retrieve from the memory a first set of momentum values ​​corresponding to layers of the neural network model having a selected storage format, and retrieve from the memory a second set of momentum values ​​corresponding to layers of the neural network model having the selected storage format, (2) convert the first set of momentum values ​​having the selected storage format to a third set of momentum values ​​having a training format associated with the gradient optimizer, and convert the second set of momentum values ​​having the selected storage format to a fourth set of momentum values ​​having a training format associated with the gradient optimizer, and (3) perform a gradient optimization using the third set of momentum values ​​and the fourth set of momentum values ​​using the gradient optimizer.

[0077]

[0065] The selected storage format may include a reduced single precision format, and the training format may include a single precision format or a double precision format.

[0078]

[0066] The system may be further configured to pad extra zero bits to form the momentum values ​​in single precision format or the momentum values ​​in double precision format. The system may be further configured to generate a fifth set of momentum values ​​and a sixth set of momentum values ​​for a next iteration of the gradient optimization, and convert each of the fifth set of momentum values ​​and the sixth set of momentum values ​​to a storage format by storing only a sign bit and the seven most significant bits associated with each respective momentum value before storing each of the fifth set of momentum values ​​and the sixth set of momentum values.

[0079]

[0067] The system may be configured to implement an adaptive momentum estimation algorithm. The gradient optimizer may be implemented using a field programmable gate array (FPGA), and the gradient optimizer may be configured to operate in a burst mode, such that successive burst cycles result in streaming of gradients through the gradient optimizer.

[0080] In yet another example, the disclosure relates to a method in a system including a gradient optimizer and a memory configured to store weights and momentum values ​​associated with a neural network model including L layers, where L is an integer greater than 1. The method may further include retrieving from the memory a first set of momentum values ​​corresponding to the layers of the neural network model having a selected storage format, and retrieving from the memory a second set of momentum values ​​corresponding to the layers of the neural network model having the selected storage format. The method may further include converting the first set of momentum values ​​having the selected storage format to a third set of momentum values ​​having a training format associated with the gradient optimizer, and converting the second set of momentum values ​​having the selected storage format to a fourth set of momentum values ​​having a training format associated with the gradient optimizer. The method may further include performing a gradient optimization using the third set of momentum values ​​and the fourth set of momentum values ​​with the gradient optimizer. The method may further include generating a fifth set of momentum values ​​and a sixth set of momentum values ​​for a next iteration of the gradient optimization. The method may further include converting the fifth set of momentum values ​​to a seventh set of momentum values ​​having a storage format by storing only a sign bit and truncating a predetermined number of bits associated with the mantissa of the fifth set of momentum values. The method may further include converting the sixth set of momentum values ​​to an eighth set of momentum values ​​having a storage format by storing only a sign bit and truncating a predetermined number of bits associated with the mantissa of the sixth set of momentum values.

[0081]

[0069] The selected storage format may include a reduced single precision format. The training format may include a single precision format or a double precision format. In the method, performing gradient optimization may include implementing an adaptive momentum estimation algorithm.

[0082]

[0070] The training format may include a 32-bit floating point format and the storage format may include an 8-bit floating point format. The gradient optimizer may be implemented using a field programmable gate array (FPGA), and the gradient optimizer may be configured to operate in a burst mode, such that successive burst cycles result in streaming of gradients through the gradient optimizer.

[0083]

[0071] It should be understood that the methods, modules, and components depicted herein are merely exemplary. Alternatively, or in addition, the functions described herein may be performed, at least in part, by one or more hardware logic components. For example, but not limited to, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like. In an abstract, but definite sense, any arrangement of components to achieve the same function is effectively "associated" such that the desired function is achieved. Thus, any two components herein that are combined to achieve a particular function may be considered to be "associated" with each other such that the desired function is achieved, regardless of the architecture or intermediary components. Similarly, any two components so associated may also be considered to be "operably connected" or "coupled" with each other to achieve the desired function.

[0084]

[0072] Functions related to some examples described in this disclosure may also include instructions stored on non-transitory media. The term "non-transitory media" as used herein refers to any medium that stores data and / or instructions that cause a machine to operate in a particular manner. Exemplary non-transitory media include non-volatile media and / or volatile media. Non-volatile media include, for example, hard disks, solid state drives, magnetic disks or tapes, optical disks or tapes, flash memory, EPROM, NVRAM, PRAM, or other such media, or networked versions of these media. Volatile media include, for example, dynamic memory such as DRAM, SRAM, cache, or other such media. Non-transitory media are distinct from transmission media, but may be used in conjunction with transmission media. Transmission media are used to transmit data and / or instructions to or from a machine. Exemplary transmission media include coaxial cables, fiber optic cables, copper wire, and wireless media such as radio waves.

[0085]

[0073] Furthermore, those skilled in the art will recognize that the boundaries between the functions of the operations described above are merely exemplary. The functions of multiple operations may be combined into a single operation and / or the functions of a single operation may be distributed among additional operations. Moreover, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be changed in various other embodiments.

[0086]

[0074] While the present disclosure provides specific examples, various modifications and changes can be made without departing from the scope of the disclosure, as defined in the following claims. Accordingly, the specification and drawings are to be regarded in an illustrative rather than restrictive sense, and all such modifications are intended to be included within the scope of the disclosure. Any benefits, advantages, or solutions to problems described herein with respect to specific embodiments are not intended to be construed as critical, necessary, or essential features or elements of any or all claims.

[0087]

[0075] Furthermore, the terms "a" or "an" as used herein are defined as one or more. Also, the use of introductory phrases such as "at least one" and "one or more" in a claim should not be construed as implying that the introduction of another claim element with the indefinite article "a" or "an" limits the invention to only one of that element in any particular claim containing the introduced claim element, even if the same claim contains the introductory phrase "one or more" or "at least one" and the indefinite article "a" or "an", etc. Similarly, the use of definite articles.

[0088]

[0076] Unless otherwise noted, terms such as "first" and "second" are used to arbitrarily distinguish between the elements such terms describe, and as such, these terms are not necessarily intended to indicate a temporal or other priority of such elements.

Claims

1. A method in a system including a gradient optimizer and a memory configured to store momentum values ​​associated with a neural network model including L layers, where L is an integer greater than 1; retrieving from the memory a first set of momentum values ​​corresponding to a layer of the neural network model having a selected storage format; retrieving from the memory a second set of momentum values ​​corresponding to a layer of the neural network model having the selected storage format; converting the first set of momentum values ​​having the selected storage format into a third set of momentum values ​​having a training format associated with the gradient optimizer; converting the second set of momentum values ​​having the selected storage format into a fourth set of momentum values ​​having a training format associated with the gradient optimizer; performing a gradient optimization using the third set of momentum values ​​and the fourth set of momentum values ​​with the gradient optimizer; wherein the gradient optimizer is configured to operate in a burst mode such that successive burst cycles result in streaming of gradients through the gradient optimizer. method.

2. the selected storage format includes a reduced single precision format; The method of claim 1.

3. The training format includes a single precision format or a double precision format. The method of claim 1.

4. A method in a system including a gradient optimizer and a memory configured to store momentum values ​​associated with a neural network model including L layers, where L is an integer greater than 1; retrieving from the memory a first set of momentum values ​​corresponding to a layer of the neural network model having a selected storage format; retrieving from the memory a second set of momentum values ​​corresponding to a layer of the neural network model having the selected storage format; converting the first set of momentum values ​​having the selected storage format into a third set of momentum values ​​having a training format associated with the gradient optimizer; converting the second set of momentum values ​​having the selected storage format into a fourth set of momentum values ​​having a training format associated with the gradient optimizer; performing a gradient optimization using the third set of momentum values ​​and the fourth set of momentum values ​​with the gradient optimizer; and converting the first set of momentum values ​​having the selected storage format to the third set of momentum values ​​having the training format includes padding extra zero bits to form momentum values ​​in a single precision format. method.

5. A method in a system including a gradient optimizer and a memory configured to store momentum values ​​associated with a neural network model including L layers, where L is an integer greater than 1; retrieving from the memory a first set of momentum values ​​corresponding to a layer of the neural network model having a selected storage format; retrieving from the memory a second set of momentum values ​​corresponding to a layer of the neural network model having the selected storage format; converting the first set of momentum values ​​having the selected storage format into a third set of momentum values ​​having a training format associated with the gradient optimizer; converting the second set of momentum values ​​having the selected storage format into a fourth set of momentum values ​​having a training format associated with the gradient optimizer; performing a gradient optimization using the third set of momentum values ​​and the fourth set of momentum values ​​with the gradient optimizer; Including, generating a fifth set of momentum values ​​and a sixth set of momentum values ​​for a next iteration of gradient optimization; and converting each of the fifth set of momentum values ​​and the sixth set of momentum values ​​into the storage format by storing only a sign bit and the seven most significant bits associated with each respective momentum value prior to storing each of the fifth set of momentum values ​​and the sixth set of momentum values; The method further comprising:

6. performing gradient optimization includes implementing an adaptive momentum estimation algorithm; The method of claim 1.

7. The gradient optimizer is implemented using a field programmable gate array (FPGA). The method according to claim 6.

8. A system including a gradient optimizer, comprising: a memory configured to store momentum values ​​associated with a neural network model including L layers, where L is an integer greater than 1; Retrieving from the memory a first set of momentum values ​​corresponding to a layer of the neural network model having a selected storage format; Retrieving from the memory a second set of momentum values ​​corresponding to a layer of the neural network model having the selected storage format; converting the first set of momentum values ​​having the selected storage format into a third set of momentum values ​​having a training format associated with the gradient optimizer; converting the second set of momentum values ​​having the selected storage format into a fourth set of momentum values ​​having a training format associated with the gradient optimizer; performing a gradient optimization using the third set of momentum values ​​and the fourth set of momentum values ​​using the gradient optimizer; wherein the gradient optimizer is configured to operate in a burst mode such that successive burst cycles result in streaming of gradients through the gradient optimizer. The system described.

9. the selected storage format includes a reduced single precision format; The system of claim 8.

10. A system including a gradient optimizer, comprising: a memory configured to store momentum values ​​associated with a neural network model including L layers, where L is an integer greater than 1; Retrieving from the memory a first set of momentum values ​​corresponding to a layer of the neural network model having a selected storage format; Retrieving from the memory a second set of momentum values ​​corresponding to a layer of the neural network model having the selected storage format; converting the first set of momentum values ​​having the selected storage format into a third set of momentum values ​​having a training format associated with the gradient optimizer; converting the second set of momentum values ​​having the selected storage format into a fourth set of momentum values ​​having a training format associated with the gradient optimizer; performing a gradient optimization using the third set of momentum values ​​and the fourth set of momentum values ​​using the gradient optimizer; It is configured as follows: the training format includes a single precision format or a double precision format; and further configured to pad the momentum values ​​with extra zero bits to form momentum values ​​in single precision format or momentum values ​​in double precision format. system.

11. A system including a gradient optimizer, comprising: a memory configured to store momentum values ​​associated with a neural network model including L layers, where L is an integer greater than 1; Retrieving from the memory a first set of momentum values ​​corresponding to a layer of the neural network model having a selected storage format; Retrieving from the memory a second set of momentum values ​​corresponding to a layer of the neural network model having the selected storage format; converting the first set of momentum values ​​having the selected storage format into a third set of momentum values ​​having a training format associated with the gradient optimizer; converting the second set of momentum values ​​having the selected storage format into a fourth set of momentum values ​​having a training format associated with the gradient optimizer; performing a gradient optimization using the third set of momentum values ​​and the fourth set of momentum values ​​using the gradient optimizer; It is configured as follows: generating a fifth set of momentum values ​​and a sixth set of momentum values ​​for a next iteration of gradient optimization, and before storing each of the fifth set of momentum values ​​and the sixth set of momentum values, converting each of the fifth set of momentum values ​​and the sixth set of momentum values ​​into the storage format by storing only a sign bit and the seven most significant bits associated with each respective momentum value. system.

12. configured to implement an adaptive momentum estimation algorithm; The system of claim 8.

13. The gradient optimizer is implemented using a field programmable gate array (FPGA). The system of claim 8.

14. 1. A method in a system including a gradient optimizer and a memory configured to store weights and momentum values ​​associated with a neural network model including L layers, where L is an integer greater than 1; retrieving from the memory a first set of momentum values ​​corresponding to layers of the neural network model having a selected storage format; retrieving from the memory a second set of momentum values ​​corresponding to layers of the neural network model having the selected storage format; converting the first set of momentum values ​​having the selected storage format into a third set of momentum values ​​having a training format associated with the gradient optimizer; converting the second set of momentum values ​​having the selected storage format into a fourth set of momentum values ​​having a training format associated with the gradient optimizer; performing a gradient optimization using the third set of momentum values ​​and the fourth set of momentum values ​​with the gradient optimizer; generating a fifth set of momentum values ​​and a sixth set of momentum values ​​for a next iteration of the gradient optimization; converting said fifth set of momentum values ​​to a seventh set of momentum values ​​having said storage format by storing only a sign bit and discarding a predetermined number of bits associated with the mantissas of said fifth set of momentum values; converting said sixth set of momentum values ​​to an eighth set of momentum values ​​having said storage format by storing only a sign bit and truncating a predetermined number of bits associated with the mantissas of said sixth set of momentum values; A method comprising:

15. The gradient optimizer is configured to operate in a burst mode such that successive burst cycles result in streaming of gradients through the gradient optimizer. The method of claim 14.

16. the selected storage format includes a reduced single precision format; 16. The method according to claim 14 or 15.

17. The training format includes a single precision format or a double precision format.

16. The method according to claim 14 or 15.

18. performing gradient optimization includes implementing an adaptive momentum estimation algorithm; 16. The method according to claim 14 or 15.

19. The method of claim 18, wherein the training format comprises a 32-bit floating point format and the storage format comprises an 8-bit floating point format.

16. The method according to claim 14 or 15.

20. The gradient optimizer is implemented using a field programmable gate array (FPGA).

16. The method according to claim 14 or 15.

Images