Dual momentum gradient optimization method and system with reduced memory requirements

Abstract

Systems and methods related to double momentum gradient optimization with reduced memory requirements are described. An example method in a system including a gradient optimizer and a memory configured to store momentum values associated with a neural network model including L layers is described. The method includes retrieving, from the memory, a first set of momentum values and a second set of momentum values having a selected storage format corresponding to a layer of the neural network model. The method also includes converting the first set of momentum values to a third set of momentum values having a training format associated with the gradient optimizer and converting the second set of momentum values to a fourth set of momentum values having the training format associated with the gradient optimizer. The method further includes performing gradient optimization using the third set of momentum values and the fourth set of momentum values.

Description

Background Technology

[0001] Trained models, including neural network models, are used to perform complex tasks such as reading comprehension, language translation, image recognition, or speech recognition. Machine learning services have been deployed to perform such complex tasks, including those based on Natural Language Processing (NLP), Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) neural networks, or Gated Recurrent Units (GRUs). While these types of models have been trained using various architectures and methods, there is a need to continue improving the underlying architectures and methods for performing training. Summary of the Invention

[0002] In one example, this disclosure relates to a method in a system including a gradient optimizer and a memory configured to store momentum values ​​associated with a neural network model comprising L layers, where L is an integer greater than one. The method may include: retrieving from the memory a first set of momentum values ​​corresponding to layers of the neural network model, the first set of momentum values ​​having a selected storage format; and retrieving from the memory a second set of momentum values ​​corresponding to layers of the neural network model, the second set of momentum values ​​having the selected storage format. The method may further include: converting the first set of momentum values ​​having the selected storage format into a third set of momentum values ​​having a training format associated with the gradient optimizer; and converting the second set of momentum values ​​having the selected storage format into a fourth set of momentum values ​​having a training format associated with the gradient optimizer. The method may further include: performing gradient optimization using the gradient optimizer, using the third and fourth sets of momentum values.

[0003] In another example, this disclosure relates to a system including a gradient optimizer. The system may include a memory configured to store momentum values ​​associated with a neural network model comprising L layers, where L is an integer greater than one. The system may be configured to: (1) retrieve a first set of momentum values ​​from the memory, the first set of momentum values ​​corresponding to layers of the neural network model having a selected storage format, and retrieve a second set of momentum values ​​from the memory, the second set of momentum values ​​corresponding to layers of the neural network model having a selected storage format; (2) convert the first set of momentum values ​​having the selected storage format into a third set of momentum values ​​having a training format associated with the gradient optimizer; and convert the second set of momentum values ​​having the selected storage format into a fourth set of momentum values ​​having a training format associated with the gradient optimizer; and (3) perform gradient optimization using the gradient optimizer, using the third set of momentum values ​​and the fourth set of momentum values.

[0004] In yet another example, this disclosure relates to a method in a system including a gradient optimizer and a memory configured to store weights and momentum values ​​associated with a neural network model comprising L layers, where L is an integer greater than one. The method may further include: retrieving from the memory a first set of momentum values ​​corresponding to layers of the neural network model, the first set of momentum values ​​having a selected storage format; and retrieving from the memory a second set of momentum values ​​corresponding to layers of the neural network model, the second set of momentum values ​​having a selected storage format. The method may further include: converting the first set of momentum values ​​having the selected storage format into a third set of momentum values ​​having a training format associated with the gradient optimizer; and converting the second set of momentum values ​​having the selected storage format into a fourth set of momentum values ​​having a training format associated with the gradient optimizer. The method may further include: performing gradient optimization using the gradient optimizer, using the third and fourth sets of momentum values. The method may further include: generating a fifth and a sixth set of momentum values ​​for the next iteration of gradient optimization. The method may further include: converting the fifth set of momentum values ​​into a seventh set of momentum values ​​with a storage format by storing only the sign bit and truncating a predetermined number of bits associated with the mantissa of the fifth set of momentum values. The method may further include: converting the sixth set of momentum values ​​into an eighth set of momentum values ​​with a storage format by storing only the sign bit and truncating a predetermined number of bits associated with the mantissa of the sixth set of momentum values.

[0005] This overview is provided to introduce a selection of concepts in a simplified form, which will be further described in the detailed embodiments below. This summary is not intended to represent key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0006] This disclosure is illustrated by way of example and is not limited to the accompanying drawings, wherein similar reference numerals indicate similar elements. The elements in the drawings are illustrated for simplicity and clarity and are not necessarily drawn to scale.

[0007] Figure 1 An Artificial Intelligence Training System (AITS) including a Dual Momentum Gradient Optimizer (DMGO) is shown as an example. Figure 2 The dual momentum gradient optimizer (DMGO) is shown according to an example. Figure 3 A graph showing the error bound curve (EB(M)) for weight update error is shown, where the compressive momentum value is scanned from truncated mantissa to truncated twenty-three mantissa. Figure 4A graph 400 with simulated convergence curves is shown, illustrating the effect of momentum values ​​that reduce accuracy on training accuracy. Figure 5 A training system including a hardware-assisted gradient optimizer (HAGO) is shown according to an example; Figure 6 A block diagram of a hardware-assisted gradient optimizer (HAGO) including a dual-momentum gradient optimizer (DMGO) is shown according to an example. Figure 7 An example is shown with Figure 6 The HAGO-related time series diagram; Figure 8 A gradient optimization system based on another example is shown; Figure 9 A flowchart illustrating a method for a dual-momentum gradient optimizer (DMGO) based on an example is shown; and Figure 10 A flowchart of another approach for the dual momentum gradient optimizer (DMGO) is shown, based on an example. Detailed Implementation

[0008] The examples disclosed in this disclosure relate to two-momentum gradient optimization with reduced memory requirements. Some examples involve distributed training of models, including neural network models that require less memory. In some example training systems, each worker (e.g., a GPU, DSP, or ASIC accelerator) takes a subset of the training data (e.g., a mini-batch) and computes a gradient, which is further averaged / reduced to update the model's parameters (e.g., weights). The weight update is applied after the backpropagation step that computes the gradient. For example, the gradient of the loss function with respect to the weights in the network (or a portion of the network) is computed. The gradient is then fed into an optimization method that uses the gradient to update the weights to minimize the loss function. The goal of backpropagation is to update each weight (or at least some weights) in the network so that these weights bring the actual output closer to the target output, thereby minimizing the error for each output neuron and the entire network.

[0009] In some examples, trained models can be deployed in cloud computing environments. Cloud computing can refer to a model used to enable on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be used in the marketplace to provide ubiquitous and convenient on-demand access to a shared pool of configurable computing resources. A shared pool of configurable computing resources can be rapidly configured through virtualization, released with low management effort or service provider interaction, and then scaled accordingly. Cloud computing models can be characterized by various features such as on-demand self-service, broad network access, resource pooling, rapid elasticity, and customized services. Cloud computing models can be used to expose various service models, such as Hardware as a Service (“HaaS”), Software as a Service (“SaaS”), Platform as a Service (“PaaS”), and Infrastructure as a Service (“IaaS”). Cloud computing models can also be deployed using different deployment models such as private cloud, community cloud, public cloud, hybrid cloud, etc.

[0010] Various machine learning services, such as those based on recurrent neural networks (RNNs), convolutional neural networks (CNNs), long short-term memory (LSTM) neural networks, or gated recurrent units (GRUs), can be implemented using the examples described in this disclosure. In one example, service-related content or other information, such as words, sentences, images, videos, or other such content / information, can be converted into a vector representation. The vector representation can correspond to techniques such as RNNs, CNNs, LSTMs, or GRUs. In one example, the neural network model can include many layers, and each layer can be encoded as a weight matrix or weight vector, which is represented by coefficients or constants obtained through offline training of the neural network.

[0011] Taking LSTM as an example, an LSTM network can include a series of repeating RNN layers or other types of layers. Each layer of an LSTM network can consume an input at a given time step, such as the state from the layer at the previous time step, and can produce a new set of outputs or states. When using LSTM, a single block of content can be encoded as a single vector or multiple vectors. For example, a word or combination of words (e.g., a phrase, sentence, or paragraph) can be encoded as a single vector. Each block can be encoded into a single layer of the LSTM network (e.g., at a specific time step). An LSTM layer can be described using a set of formulas, such as the following:

[0012] In this example, within each LSTM layer, the input and hidden states can be processed using vector operations (e.g., dot products, inner products, or vector addition) and nonlinear functions (e.g., sigmoids, hyperbolas, and tangents). In some cases, the most computationally intensive operation may come from the dot product, which can be implemented using dense matrix vectors and matrix-matrix multiplication routines. In one example, the processing of vector operations and nonlinear functions can be performed in parallel.

[0013] Gradient optimizers are a key component in training systems for artificial intelligence. They aim to apply gradients to model parameters (weights) to update the model so that the predictions in the next iteration are closer to the truth. Some examples in this disclosure relate to AI training frameworks that support dual-momentum gradient optimizers, where two single-precision (FP32) momentum values ​​are cached along with the model parameters. At each update step, a new momentum value is computed and then used to generate new weights. Both the new momentum value and the new weights are stored for use in the next iteration. For example, the dual-momentum adaptive moment estimation (Adam) algorithm can be represented by the following equation:

[0014] in There are two momentum variables: w is the model weight, dw is the gradient, and α is the initial learning rate. It is a very small value used to improve numerical stability.

[0015] Advantageously, the use of a dual-momentum optimizer can make neural network models converge faster than zero-momentum / single-momentum optimizers such as SGD (Stochastic Gradient Descent). However, using a dual-momentum optimizer may require at least twice the memory of a zero-momentum / single-momentum optimizer to cache momentum other than model weights. This tendency can become particularly pronounced for large models containing millions or billions of parameters. If the optimizer runs on a memory-constrained GPU / ASIC, the memory space occupied by the optimizer will adversely reduce the maximum batch size. On the other hand, relocating the optimizer to a host CPU can address the memory size bottleneck. However, the memory bandwidth bottleneck may slow down the optimizer because the processor needs to fetch two additional FP32 momentum from memory for each weight / gradient pair and store them back.

[0016] Furthermore, considering that models such as Transformer-Based Bidirectional Encoder Representation (BERT) – a large model – generate 1.36 gigabytes (GB) of gradients per worker, caching the gradients of the entire model requires a very large amount of memory. In the context of this model, 5.44 GB of memory is needed to cache gradients in a four-worker training cluster. The memory requirement increases linearly with the number of workers. Moreover, when used with algorithms such as Adaptive Moment Estimation (Adam), which require caching two floating-point 32-bit (FP32) momentum terms per gradient, the memory requirement per worker can be three times that of BERT.

[0017] Figure 1 Artificial Intelligence Training System (AITS) 100, including a Dual Momentum Gradient Optimizer (DMGO) 150, is illustrated according to an example. AITS 100 may include memory 110 (e.g., DRAM or other type of memory), memory interface 120 (e.g., DDR interface or other type of interface), momentum format converter (MFC) 130, another momentum format converter (MFC) 140, and DMGO 150. Memory 110 may be coupled to memory interface 120 via bus 112. Alternatively, memory 110 may be integrated with memory interface 120 via on-chip package, system-on-chip package, 3D memory system, or stacked die system. Memory interface 120 may be coupled to DMGO 150 via bus 122, such that model weights (W) can be transferred from memory 110 to DMGO 150 via memory bus 122. Memory interface 120 may be coupled to MFC 130 via buses 132 and 134. Bus 132 can be used to transfer a stored-format momentum value (e.g., v) associated with the first momentum from memory 110 to MFC 130. t _fp8 format). Bus 134 can be used to transfer momentum values ​​(e.g., s) in a stored format associated with the second momentum from memory 110 to MFC 130. t (_fp8 format). In one example, the storage format may correspond to an 8-bit reduced-precision format of the momentum values. MFC 130 can be configured to process the first and second momentum values ​​to change their format from the storage format to the training format associated with DMGO 150. In one example, the training format may correspond to a full 32-bit single-precision format. MFC 130 can also be configured to transmit the first and second momentum values ​​to DMGO 150 via buses 136 and 138, respectively. In this example, MFC 140 can be coupled to DMGO 150 via buses 142 and 144. MFC 140 can be further coupled to memory interface 110 via buses 146 and 148. MFC 140 can convert the momentum values ​​from the training format to the storage format.

[0018] Still referencing Figure 1 AITS 100 allows for the implementation of a new method for using reduced-precision momentum for Adam without introducing any performance degradation. For example, the two-momentum Adam algorithm can be represented by the following equation:

[0019] in There are two momentum variables: w is the model weight, dw is the gradient, and α is the initial learning rate. It is a very small value used to improve numerical stability. In one example, one momentum value ( ) can correspond to the expectation of the gradient value, and another momentum value in the momentum value ( This can correspond to the expected value of the gradient value variance. In this example, instead of using a full 32-bit single-precision momentum, only 8 bits are used: 1 bit for the sign and 7 bits for the dynamic range of the momentum. In this example, because Since multiplying by (1-β) results in all values ​​being less than 1.0, where β is a value close to 1.0, The most significant bit (MSB) of the 8-bit exponent is a constant zero and can be safely discarded before storage. In this example, although the momentum value is stored in 8-bit format, the calculation format is still of higher precision for greater accuracy. Despite Figure 1 AITS 100 is shown, comprising a number of components arranged in a specific manner; however, AITS 100 may include additional or fewer components arranged in a different manner. As an example, MFC 130 and MFC 140 may be integrated with DMGO 150 via on-chip package, system-on-chip package, 3D memory system, or stacked die system. Alternatively, MFC 130 and MFC 140 may be integrated with DMGO 150 on the same chip (e.g., as a system-on-chip (SoC)). In another example, each of memory 110, memory interface 120, MFC 130, MFC 140, and DMGO 150 may be integrated via on-chip package, system-on-chip package, 3D memory system, or stacked die system.

[0020] Figure 2A dual momentum gradient optimizer (DMGO) 200 according to an example is shown. DMGO 200 may include a momentum decompressor 202 and another momentum decompressor 204. DMGO 200 may also include a momentum compressor 262 and another momentum compressor 264. DMGO 200 may also include logic configured to perform various operations associated with implementing the dual momentum adaptive moment estimation (Adam) algorithm. In this example, momentum decompressors 202 and 204 can be used to provide the previously mentioned... Figure 1 Similar functionality to that described in MFC 130. Similarly, momentum compressors 262 and 264 can be used to implement the same functionality as previously described. Figure 1 Similar functionality described in MFC 140. The momentum value associated with the first momentum (e.g., v). t_fp8 The momentum value associated with the second momentum (e.g., s) can be retrieved from memory and provided to the momentum decompressor 202. t_fp8 The momentum values ​​can be retrieved from memory and provided to momentum decompressor 204. In one example, the momentum values ​​may have a storage format (e.g., an 8-bit reduced-precision format). Momentum decompressor 202 can be configured to process the first momentum values ​​to change their format from the storage format to the training format associated with DMGO 200. Momentum decompressor 204 can be configured to process the second momentum values ​​to change their format from the storage format to the training format associated with DMGO 200.

[0021] Continue to refer to Figure 2 The logic included as part of DMGO 200 may include logic configured to perform operations such as addition, multiplication, and fused multiplication-addition operations. This logic may be configured to implement a dual-momentum optimization algorithm (e.g., the dual-momentum adaptive moment estimation (Adam) algorithm). For example, regarding new momentum values ​​( The calculation of ), corresponding to the two-momentum Adam algorithm, can be expressed by the following formula:

[0022] in Here, is the new momentum value, w is the model weight, dw is the gradient, and α is the initial learning rate. This is a very small value used to improve numerical stability. Therefore, in this example, momentum decompressors 202 and 204 will decompress the corresponding momentum values ​​( The storage format of the momentum (e.g., 8-bit compressed format) is converted to the training format associated with DMGO 200 (e.g., 32-bit full-precision format). For example, before performing gradient optimization, the momentum decompressor (e.g., momentum decompressor 202 or 204) converts the compressed 8-bit momentum value back to 32-bit single precision by inserting a "0" bit between the sign bit and the most significant bit (MSB) of the exponent, and padding with twenty-three "0" bits after the least significant bit (LSB) of the exponent. The padded bits do not have to be "0" bits. In one example, at least a few bits can be non-zero bits. For example, two or three LSBs can be non-zero bits.

[0023] Still referencing Figure 2 The logic associated with DMGO 200, including multipliers 210 and 214 and adder 212, can be executed to generate a new momentum value in the new momentum value. The calculations associated with DMGO 200, including multipliers 216, 218, and 218, and adder 222, can be performed to generate other new momentum values. The calculation is associated with ). Furthermore, in this example, the new momentum value ( Before storing the new momentum values ​​back into memory, momentum compressors 262 and 264 convert the format of the new momentum values ​​from the training format to the storage format. )exist Figure 2 The value is shown as an 8-bit storage format. In this example, each of the momentum compressors 262 and 264 can be configured to remove the most significant bit (MSB) of the exponent associated with the momentum value and truncate all 23 mantissa bits to produce a 32-bit full-precision momentum value. Convert to 8-bit compressive momentum value ( The 8-bit compressed momentum value will be stored in memory. Alternatively, fewer than 23 mantissa bits can be truncated. Advantageously, compression of the momentum value can reduce the buffer size required to store the momentum value by a factor of four. Furthermore, using the least significant bit (LSB) for truncation can reduce accuracy without sacrificing training performance by introducing the same rounding error direction, allowing errors to cancel each other out.

[0024] Continue to refer to Figure 2 DMGO 200 may also include logic configured to generate new weight values ​​(e.g., w_new). In this example, this logic may include multipliers 232 and 234, adder 242, inverse square root (RSQRT) 246, and fused multiplication-addition (FMA) 250. Although Figure 2The DMGO 200 is shown as including a certain number of components arranged in a particular manner, but the DMGO 200 may include additional or fewer components arranged in a different manner.

[0025] When momentum is compressed by a momentum compressor, the precision bit of momentum is truncated to introduce a value relative to the new momentum value (e.g., respectively). and The relative errors δ1 and δ2 are given. Therefore, the gradient optimization formula can be rewritten as:

[0026] For 32-bit single-precision numbers , where m is the number of bits of precision truncated. In this example, if m is not too large, it is safe to approximate 1-δ1=1-δ2≈1-δ. The error bound is only reached when all truncated m bits are "1", and this situation rarely occurs in practical applications. Even so, if such an extreme case occurs, the instantaneous peak error may not negatively impact the overall performance. The relative error of the updated weights after truncating the two momentums can be expressed as: For any For small δ values, This will be closer to 1 than 1-δ. This indicates that reducing the precision of the two momentums does not linearly reduce the precision of the updated parameters. This "square root of error" effect compensates for some of the "precision loss" caused by the truncation.

[0027] Figure 3 Plot 300 shows the error limit curve (EB(M)) for the weight update error, where the compressive momentum value scans from truncating one mantissa to truncating twenty-three mantissas. As shown in Plot 300, in this example, when the number of truncated mantissas is less than sixteen, the error limit value is close to zero compared to the full single precision (FP32) momentum value. In this example, when the number of truncated mantissas exceeds seventeen, the error limit increases rapidly, reaching a maximum of 0.13 when all mantissas have been truncated.

[0028] Figure 4 Graph 400, showing simulated convergence curves 410, 420, and 420, illustrates the impact of truncating the mantissa of the momentum value on training accuracy. As an example, simulated convergence curve 410 corresponds to a 1-bit mantissa of the momentum value, simulated convergence curve 420 corresponds to a 4-bit mantissa of the momentum value, and simulated convergence curve 430 corresponds to a 24-bit mantissa of the momentum value. It is clear from the simulated convergence curves that truncating the mantissa of the momentum value does not degrade performance in terms of training accuracy. However, since truncating the momentum value saves at least 50% of memory usage, only half the number of cycles required to read memory can be used.

[0029] The techniques described as part of this disclosure can be implemented using a combination of hardware and / or software. In some examples, Figure 2 The DGMO 200 can be deployed as part of the training system 500, such as Figure 5 As shown. In this example, according to one example, the DGMO 200 can be included as part of a hardware-assisted gradient optimizer (dual momentum) 550 (referred to as HAGO) for training a model. The gradient optimization system 500 can include multiple workers (e.g., worker_1 502, worker_2 504, and worker_N 506) coupled to the HAGO 550 via a high-speed bus bridge 520. Each worker (e.g., worker_1 502, worker_2 504, and worker_N 506) can be implemented using any number of graphics processing units (GPUs), computer processing units (CPUs), memory (e.g., SRAM or other types of memory), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), erasable and / or complex programmable logic devices (PLDs), programmable array logic (PAL) devices, or general-purpose array logic (GAL) devices. Each worker can carry a complete copy of the neural network model being trained using the gradient optimization system 500. While each worker can carry a complete copy of the neural network model, each worker can operate only on a subset of the training data associated with the model (e.g., images or other data samples used for classification). Each worker can perform a forward pass on the data to compute the error, and then perform a backward pass to compute the gradient of each parameter (e.g., each weight). The gradient optimization system 500 may also include a host 560 coupled to a high-speed bus bridge 520 via a bus 528.

[0030] Continue to refer to Figure 5In one example, the high-speed bus bridge 520 can be implemented as a Peripheral Component Interconnect Fast (PCIe) bridge. Worker_1 502 can be connected to the high-speed bus bridge 520 via PCIe link 510. Worker_2 504 can be connected to the high-speed bus bridge 520 via PCIe link 512. Worker_N 506 can be connected to the high-speed bus bridge 520 via PCIe link 514. HAGO 550 can be coupled to the high-speed bus bridge 520 via bus links 522 and 524. Other types of bus bridges and bus links can also be used. Furthermore, a swap structure can be used instead of a bus bridge. Each worker can compute gradients on a subset of the training data (e.g., mini-batch) and provide updated gradients to HAGO 550 via the high-speed bus bridge 520. HAGO 550 can perform reduction and optimization on the gradients collected from each worker (e.g., worker_1 502, worker_2 504, and worker_N 506). HAGO 550 can operate on data bursts, and in one example, each burst can consist of a constant number of incoming gradients. The current gradient burst can be cached, while previous gradient bursts, along with their corresponding weights, are optimized. Although Figure 5 A gradient optimization system 500 is shown, comprising a number of components arranged in a particular manner, but the gradient optimization system 500 may include additional or fewer components arranged in a different manner.

[0031] Figure 6 A block diagram of the HAGO 600, including a dual-momentum gradient optimizer, is shown according to an example. In one example, Figure 5 The HAGO 550 can be implemented as Figure 6 The HAGO 600 can receive updated gradients from the workers via a high-speed bus. Figure 6 The example assumes four workers coupled to the HAGO 600. An update gradient (DW1) from one of the workers can be received via a FIFO block (e.g., GFIFO_1 602). An update gradient (DW2) from another worker can be received via another FIFO block (e.g., GFIFO_2 604). An update gradient (DW3) from another worker can be received via another FIFO block (e.g., GFIFO_3 606). An update gradient (DW4) from another worker can be received via another FIFO block (e.g., GFIFO_4 608). The FIFO can be implemented as a single multi-entry FIFO or multiple single-entry FIFOs.

[0032] Continue to refer to Figure 6HAGO 600 may also include a floating-point averaging block (e.g., FP-AVG block 620). FP-AVG block 620 may receive cached updated gradients from a first-in-first-out block. As an example, FP-AVG block 620 may receive cached updated gradients (BDW1) from GFIFO_1 602. FP-AVG block 620 may also receive cached updated gradients (BDW2) from GFIFO_2 604. FP-AVG block 620 may also receive cached updated gradients (BDW3) from GFIFO_3 606. FP-AVG block 620 may also receive cached updated gradients (BDW4) from GFIFO_4 608. FP-AVG block 620 may perform reduction on the received updated gradients. In this example, the reduction operation may include averaging the received gradients and generating DW_AVG as output. The reduction operation is not limited to averaging. Other operations such as summation or scaling summation may be used to perform reduction. The output from FP-AVG block 620 can be provided to gradient optimizer data path (GODP) block 650, which performs floating-point arithmetic to implement gradient optimization algorithms, such as those previously discussed. Figure 1 and Figure 2 The gradient optimization algorithm is described. In one example, FP-AVG block 620 and GODP block 650 can be implemented as part of an FPGA.

[0033] Still referencing Figure 6 The master copy of the weight (W) and optional momentum (M) can be stored in memory 660. In one example, memory 660 can be implemented using double data rate (DDR) dynamic random access memory (DRAM). Memory 660 can be coupled to a first-in-first-out block (e.g., PFIFO block 670) via memory interface 662. Memory interface 662 can be a DDR interface. Figure 6 As shown, the model's weights (W) can be loaded into the PFIFO 670 under the control of a finite state machine (FSM) (e.g., control FSM 640). Furthermore, the memory 660 can be coupled to a momentum format converter (MFC) 684 via a memory interface 662. The MFC 684 can provide the previously mentioned... Figure 1 Similar functionality to that described in MFC 130. The output of MFC684 can be coupled to PFIFO 670 (or another FIFO).

[0034] In one example, simultaneously, control FSM 640: (1) enables each worker to transfer its locally computed gradients (e.g., DW1, DW2, DW3, and DW4) to the corresponding GFIFOs (e.g., GFIFO_1 602, GFIFO_2 604, GFIFO_3 606, and GFIFO_4 608), and (2) enables memory interface 662 to retrieve the same number of parameters (e.g., weights (W)) and optional momentum from memory 660 to PFIFO 670. Thus, in this example, control FSM 640 controls memory interface 662 to retrieve the same number of weights as the number of gradients injected into the GFIFOs during a single burst cycle into PFIFO 670. As each of these FIFOs reaches its burst size, the corresponding read ports of these FIFOs are activated by control FSM 640 to output the contents of the corresponding FIFO with a single-cycle delay. In this example, the control FSM 640 is able to read weights from the PFIFO670 and GFIFO, ensuring that data from these buffers is cleared with the same latency. Therefore, the data readout is perfectly aligned.

[0035] Continue to refer to Figure 6 This implementation ensures that the four gradient outputs (e.g., BDW1, BDW2, BDW3, and BDW4) from each FIFO have the same localization index. The FP-AVG block 620 then calculates the average of these gradients (e.g., BDW1, BDW2, BDW3, and BDW4) to generate a reduced gradient (e.g., DW_AVG), which is then processed by the GODP block 650 along with cached weights and momentum. In this example, the output of the GODP block 650 may contain two parts: new weights (NW) and new momentum (NM), both of which are written back to memory 660 via memory interface 662. The new momentum value can be provided to MFC 682, which can implement the previously mentioned... Figure 1 This is similar to the functionality described in MFC 140. Therefore, in this way, compressed momentum values ​​can be stored in a storage format that requires less storage space.

[0036] In addition, the new weight (NW) is simultaneously injected into the transmit FIFO (TXFIFO), which sends the new weight back to the worker. In this example, the control FSM 640 can control the weight via... Figure 5 The high-speed bus bridge 520 supplies power to the workers. As an example, the control / status register 630 can be used to specify the timing associated with various operations. Furthermore, the control / status register 630 can also be used to configure the hyperparameters (e.g., learning rate) of the gradient optimizer.

[0037] During reads from and weight updates of the corresponding GFIFO, the next gradient burst is written to the corresponding GFIFO. Similarly, during reads from and weight updates of PFIFO 670, the next set of weights and momentum is also retrieved from memory 660 into PFIFO 670 (e.g., via MFC 684). Since memory 660 in this example only supports a single data port shared by read and write operations, the data width of PFIFO 670 is configured to be twice the data width of each GFIFO. Thus, in this example, all data bursts except the first and last data bursts will overlap between gradient reception and the transmission of new weights.

[0038] Still referencing Figure 6 Certain aspects of the operation of the HAGO 600 can be controlled using the control / status register and the control FSM 640. Table 1 below shows an example of the contents of the control / status register 630.

[0039]

[0040] Table 1 As shown in Table 1, the `ctrl_opt_mode` register allows you to specify the optimization algorithm type, which can be any two-momentum optimization algorithm. The `ctrl_burst_size` register allows you to select the burst size associated with each burst cycle. In this example, the burst size (specified in terms of the number of gradients) can be 256, 512, 1024, 2048, or 4096 gradients. The `ctrl_param_num` register allows you to specify the size of the model parameters. The `ctrl_init_lr` register allows you to specify the initial learning rate of the model. In one example, the initial learning rate can be related to the model's step size. As mentioned earlier, in one example, the optimization algorithm can be the two-momentum Adam algorithm, which can be represented by the following formula:

[0041] in There are two momentum variables: w is the model weight, dw is the gradient, and α is the initial learning rate. This is a very small value used to improve numerical stability. The ctrl_beta1 register allows specifying a momentum coefficient, which controls the acceleration associated with the weight update rate. The ctrl_beta2 register allows specifying a second momentum coefficient, which further controls the acceleration associated with the weight update rate. If the input gradient is in FP16 format, the ctrl_is_fp16 register is set. Although Table 1 shows the specific register groups included as part of the control / status register 630, additional or fewer registers can be used with the HAGO 600. Furthermore, at least some or all of the control information included in the control / status register 630 can be specified via fields or operands associated with instructions related to gradient optimization using the HAGO 600. Although Figure 6 The diagram shows a HAGO 600 comprising a certain number of components arranged in a specific manner, but the HAGO 600 may include additional or fewer components arranged in a different manner. For example, more than four workers may be coupled to the HAGO 600.

[0042] Figure 7 An example is shown with Figure 6 Timing diagram 700 is associated with the HAGO 600. Timing diagram 700 explains the operation of the HAGO 600 in terms of bursts. Timing diagram 700 shows K+1 burst cycles, where K is a positive integer. The first burst cycle (burst_1) and the last burst cycle (burst_K+1) include less overlap than the other burst cycles (burst_2 to burst_K cycles). Burst_1 cycle includes the start of gradient transfer from the worker (e.g., DW1) until the gradient is injected into the FIFO (e.g., ...). Figure 6 The timing (GW) involved in achieving the burst size of the FIFO (GFIFO_1 602). Multiple workers (e.g., Figure 5 Workers _1 502, _2 504, and _N 506 can inject gradients into the FIFO to reach the burst size of the FIFO during the burst_1 cycle. The burst_1 cycle also includes the start of passing the gradient stored in memory (e.g., ...). Figure 6 The parameters (e.g., model weights) in the memory 660) until the weights are injected into Figure 6 The timing (PW) involved in the PFIFO 670. For example... Figure 7 As shown, the transfer of weights is affected by memory (e.g., Figure 6 The read latency of the memory 660) DDR The impact of this is that, during burst _1 cycles, HAGO 600 overlaps two operations: injecting gradients into the FIFO and retrieving weights from memory.

[0043] Continue to refer to Figure 7 The burst_2 cycle includes the start of transferring the next set of gradients from the worker (e.g., DW1) until the gradients are injected into the FIFO (e.g., Figure 6 The timing (GW) involved in achieving the burst size of the FIFO (GFIFO_1 602). Multiple workers (e.g., Figure 5 Workers _1 502, _2 504, and _N 506 can inject gradients into the FIFO to reach the burst size of the FIFO during the burst 2 cycle. The burst 2 cycle also includes a computation cycle as the next set of gradients is injected into the FIFO. This computation cycle includes calculating the average gradient by the FP-AVG block 620 and processing the reduced gradient, cached momentum, and cached weights (BM, BW) by the GODP block 650. The timing involved as part of this operation is... Figure 7 The timing involved in writing the new momentum and new weights (NM, NW) output from GODP block 650 back to memory 660 is marked as GPR. The timing involved in the burst_2 cycle also includes the overlap of timing involved in transferring the new weights to the workers via TXFIFO 680. (As...) Figure 7 As shown in timing diagram 700, this operation is subject to the processing delay (L) of GODP block 650. DP The burst_2 cycle also includes the start of transferring data stored in memory (e.g., ...). Figure 6 The parameters (e.g., model weights) in the memory 660) until the weights are injected into Figure 6 The timing (PW) involved in the PFIFO 670. For example... Figure 7 As shown, the transfer of weights is affected by memory (e.g., Figure 6 The read latency of the memory 660) DDR The impact of ) Therefore, during the burst _2 cycle, HAGO 600 at least partially overlaps four operations as part of gradient optimization: (1) injecting gradients into FIFO (e.g., injecting into any GFIFO), (2) reading gradients and weights from (multiple) GFIFOs and PFIFOs, (3) writing new weights and new momentum back to memory, and (4) transferring weights to the workers.

[0044] Still referencing Figure 7The FSM 640 continues to operate the HAGO 600 in a streaming manner, causing successive burst cycles to result in gradients flowing through the HAGO 600. Therefore, the burst_3 cycle includes an additional flow of gradients and an overlap of multiple operations, including at least partial overlap of four operations as part of gradient optimization: (1) injecting gradients into a FIFO (e.g., any GFIFO), (2) reading gradients and weights from (multiple) GFIFOs and PFIFOs, (3) writing new weights and momentum back to memory, and (4) transferring new weights to the workers. Similarly, the burst_K cycle includes at least partial overlap of four operations as part of gradient optimization: (1) injecting gradients into a FIFO (e.g., any GFIFO), (2) reading gradients and weights from (multiple) GFIFOs and PFIFOs, (3) writing new weights and momentum back to memory, and (4) transferring new weights to the workers. The burst_K+1 cycle corresponds to the last burst cycle. Therefore, the burst_K+1 cycle has less overlap than other burst cycles (burst_2 to burst_K cycles). As part of the burst_K+1 cycle, three operations overlap, including: (1) reading gradients and weights from (multiple) GFIFOs and PFIFOs, (2) writing new weights and new momentum back to memory, and (3) transferring new weights to the workers.

[0045] Continue to refer to Figure 7 During each individual burst cycle, the HAGO 700 can operate on gradient bursts, and each gradient burst can include a fixed number of gradients. The number of gradients operated on during a single burst cycle corresponds to the burst size, which can be configured via the control / status register 630, as previously described. Figure 6 The explanation given.

[0046] Figure 8 A gradient optimization system 800 according to another example is shown, which includes a software version of a dual-momentum gradient optimizer (DMGO). In this example, when executed by a processor, the DMGO (e.g., Figure 2The logic of the DMGO (200) can be implemented using instructions stored in memory. The gradient optimization system 800 may include processor(s) 802, I / O components(s) 804, memory 806, presentation components(s) 808, sensors 810, database(s) 812, network interface(s) 814, and I / O ports(s) 816, all interconnected via bus 820. Processor(s) 802 can execute instructions stored in memory 806. I / O components(s) 804 may include components such as a keyboard, mouse, voice recognition processor, or touchscreen. Memory 806 may be any combination of non-volatile memory or volatile memory (e.g., flash memory, DRAM, SRAM, or other types of memory). Instructions corresponding to methods associated with the DMGO may be stored in memory 806 or another memory. When executed by processor(s) 802 or other processors (e.g., other CPUs or GPUs), these instructions can provide functionality associated with these methods. As an example, instructions including right / left shift instructions and pack / unpack instructions can be used to compress or decompress momentum values. Table 2 below shows an example instruction set that, when executed by processor(s) 802, can perform operations corresponding to DMGO (e.g., Figure 2 The functionality of DMGO 200.

[0047]

[0048] Table 2 Still referencing Figure 8 The (multiple) presentation components 808 may include a display, a holographic device, or other presentation device. The display may be any type of display, such as an LCD, LED, or other type of display. The (multiple) databases 812 may be used to store data, as required for performing the methods described herein. The (multiple) databases 812 may be implemented as a collection of distributed databases or a single database. The (multiple) network interfaces 814 may include communication interfaces, such as Ethernet, cellular radio, Bluetooth radio, UWB radio, or other types of wireless or wired communication interfaces. The (multiple) I / O ports 816 may include Ethernet ports, fiber optic ports, wireless ports, or other communication ports. Although... Figure 8 The gradient optimization system 800 is shown as including a certain number of components arranged and coupled in a particular manner, but the gradient optimization system 800 may include fewer or additional components arranged and coupled in a different manner. Furthermore, the functionality associated with the system 800 can be distributed as needed.

[0049] Figure 9 A flowchart 900 is shown illustrating a method for gradient optimization based on an example. This method can use... Figure 1AITS 100 Figure 2 DMGO 200 or Figure 8 The gradient optimization system 800 is used to perform this. Therefore, as an example, when DMGO200 is implemented as part of HAGO 600, steps corresponding to this method can be executed based on control signals received from the control FSM 640. Furthermore, although in Figure 6 Not shown, but a clock signal can be used to synchronize the timing associated with the operations performed as part of the method. Alternatively, instructions stored in memory 806 can be used to perform steps associated with the method when executed by processor(s) 802. Furthermore, a modified combination of DMGO 200 and gradient optimization system 800 can also be used. Thus, as an example, while momentum compression and decompression steps can be performed using instructions stored in memory, gradient optimization can be performed using hardware logic associated with DMGO 200. Step 910 may include: retrieving a first set of momentum values ​​from memory, the first set of momentum values ​​corresponding to layers of a neural network model having a selected storage format, and retrieving a second set of momentum values ​​from memory, the second set of momentum values ​​corresponding to layers of a neural network model having a selected storage format. As previously mentioned, the storage format may correspond to a reduced-precision format.

[0050] Step 920 may include: converting a first set of momentum values ​​with the selected storage format into a third set of momentum values ​​with a training format associated with the gradient optimizer, and converting a second set of momentum values ​​with the selected storage format into a fourth set of momentum values ​​with a training format associated with the gradient optimizer. As previously described, this step may include decompressing the momentum values ​​by padding the momentum values ​​in the storage format with additional zero bits. The additional zero bits added may depend on the number of bits truncated as part of the compressed momentum values.

[0051] Step 930 may include: performing gradient optimization using a gradient optimizer, using a third set of momentum values ​​and a fourth set of momentum values. After the computations associated with the gradient optimization, the momentum values ​​may be converted to a storage format. As previously described, this step may include storing only the sign bit and seven most significant bits (MSB) of the momentum values ​​in single-precision format. Although Figure 9 It describes a certain number of steps to be performed in a specific order, but additional or fewer steps may be performed in a different order.

[0052] Figure 10 A flowchart 1000 is shown illustrating a method for gradient optimization based on an example. This method can use... Figure 1 AITS 100 Figure 2 DMGO 200 or Figure 8The gradient optimization system 800 is used to perform this. Therefore, as an example, when DMGO200 is implemented as part of HAGO 600, steps corresponding to this method can be executed based on control signals received from the control FSM 640. Furthermore, although in Figure 6 Not shown, but a clock signal can be used to synchronize the timing associated with the operations performed as part of the method. Alternatively, instructions stored in memory 806 can be used to perform steps associated with the method when executed by processor(s) 802. Furthermore, a modified combination of DMGO 200 and gradient optimization system 800 can also be used. Thus, as an example, while momentum compression and decompression steps can be performed using instructions stored in memory, gradient optimization can be performed using hardware logic associated with DMGO 200. Step 1010 may include: retrieving a first set of momentum values ​​from memory, the first set of momentum values ​​corresponding to a layer of a neural network model having a selected storage format, and retrieving a second set of momentum values ​​from memory, the second set of momentum values ​​corresponding to a layer of a neural network model having a selected storage format. As previously mentioned, the storage format may correspond to a reduced-precision format.

[0053] Step 1020 may include: converting a first set of momentum values ​​having a selected storage format into a third set of momentum values ​​having a training format associated with a gradient optimizer, and converting a second set of momentum values ​​having a selected storage format into a fourth set of momentum values ​​having a training format associated with a gradient optimizer. As previously described, this step may include decompressing the momentum values ​​by padding the momentum values ​​in the storage format with additional zero bits. The additional zero bits added may depend on the number of bits truncated as part of the compressed momentum values. As previously described, when processed by a processor (e.g., Figure 8 When the (multiple) processors 802) execute, they can use the data stored in (e.g., Figure 8 This step can be performed using instructions from memory (806). Alternatively, instructions from memory (806) can be used. Figure 2 The logic associated with DMGO 200 is used to perform this step.

[0054] Step 1030 may include: using a gradient optimizer, performing gradient optimization using a third set of momentum values ​​and a fourth set of momentum values. As previously described, when the processor (e.g., Figure 8 When the (multiple) processors 802) execute, they can use the data stored in (e.g., Figure 8 This step can be performed using instructions from memory (806). Alternatively, instructions from memory (806) can be used. Figure 2 The logic associated with DMGO 200 is used to perform this step.

[0055] Step 1040 may include generating a fifth set of momentum values ​​and a sixth set of momentum values ​​for the next iteration of gradient optimization. As previously described, when generated by a processor (e.g., Figure 8 When the (multiple) processors 802) execute, they can use the data stored in (e.g., Figure 8 This step can be performed using instructions from memory (806). Alternatively, instructions from memory (806) can be used. Figure 2 The logic associated with DMGO 200 is used to perform this step.

[0056] Step 1050 may include: converting the fifth set of momentum values ​​into a seventh set of momentum values ​​with a storage format by storing only the sign bit and truncating a predetermined number of bits associated with the mantissa of the fifth set of momentum values. As previously described, this step may include storing only the sign bit and seven most significant bits (MSB) of the single-precision format momentum values. As previously described, when processed by a processor (e.g., Figure 8 When the (multiple) processors 802) execute, they can use the data stored in (e.g., Figure 8 This step can be performed using instructions from memory (806). Alternatively, instructions from memory (806) can be used. Figure 2 The logic associated with DMGO 200 is used to perform this step.

[0057] Step 1060 may include: converting the sixth set of momentum values ​​into an eighth set of momentum values ​​with a storage format by storing only the sign bit and truncating a predetermined number of bits associated with the mantissa of the sixth set of momentum values. As previously described, this step may include storing only the sign bit and seven most significant bits (MSB) of the single-precision format momentum values. As previously described, when processed by a processor (e.g., Figure 8 When the (multiple) processors 802) execute, they can use the data stored in (e.g., Figure 8 This step can be performed using instructions from memory (806). Alternatively, instructions from memory (806) can be used. Figure 2 The logic associated with DMGO 200 is used to perform this step. Although Figure 10 It describes a certain number of steps to be performed in a specific order, but additional or fewer steps may be performed in a different order.

[0058] In summary, this disclosure relates to a method in a system including a gradient optimizer and a memory configured to store momentum values ​​associated with a neural network model comprising L layers, where L is an integer greater than one. The method may include: retrieving from the memory a first set of momentum values ​​corresponding to layers of the neural network model, the first set of momentum values ​​having a selected storage format; and retrieving from the memory a second set of momentum values ​​corresponding to layers of the neural network model, the second set of momentum values ​​having the selected storage format. The method may further include: converting the first set of momentum values ​​having the selected storage format into a third set of momentum values ​​having a training format associated with the gradient optimizer; and converting the second set of momentum values ​​having the selected storage format into a fourth set of momentum values ​​having a training format associated with the gradient optimizer. The method may further include: performing gradient optimization using the gradient optimizer, using the third set of momentum values ​​and the fourth set of momentum values.

[0059] The selected storage format can include a simplified single-precision format. The training format can include either single-precision or double-precision formats.

[0060] Converting a first set of momentum values ​​with a selected storage format to a third set of momentum values ​​with a training format includes padding with additional zero bits to form single-precision format momentum values. The method may further include: generating a fifth and a sixth set of momentum values ​​for the next iteration of gradient optimization; and converting each set of momentum values ​​in the fifth and sixth sets of momentum values ​​to a storage format by storing only the sign bit and seven most significant bits associated with each momentum value in the respective set before storing each set of momentum values ​​in the fifth and sixth sets of momentum values.

[0061] The steps involved in performing gradient optimization include implementing an adaptive moment estimation algorithm. The gradient optimizer can be implemented using a field-programmable gate array (FPGA), and the gradient optimizer can be configured to operate in burst mode, such that successive burst cycles cause gradients to flow through the gradient optimizer.

[0062] In another example, this disclosure relates to a system including a gradient optimizer. The system may include a memory configured to store momentum values ​​associated with a neural network model comprising L layers, where L is an integer greater than one. The system may be configured to: (1) retrieve a first set of momentum values ​​from the memory, the first set of momentum values ​​corresponding to layers of the neural network model having a selected storage format, and retrieve a second set of momentum values ​​from the memory, the second set of momentum values ​​corresponding to layers of the neural network model having a selected storage format; (2) convert the first set of momentum values ​​having the selected storage format into a third set of momentum values ​​having a training format associated with the gradient optimizer; and convert the second set of momentum values ​​having the selected storage format into a fourth set of momentum values ​​having a training format associated with the gradient optimizer; and (3) perform gradient optimization using the gradient optimizer, using the third set of momentum values ​​and the fourth set of momentum values.

[0063] The selected storage format can include a simplified single-precision format. The training format can include either single-precision or double-precision formats.

[0064] The system can also be configured to pad with additional zero bits to form single-precision or double-precision momentum values. The system can also be configured to: generate a fifth and sixth set of momentum values ​​for the next iteration of gradient optimization, and, before storing each set of momentum values ​​in the fifth and sixth sets, convert each set of momentum values ​​in the fifth and sixth sets into a storage format by storing only the sign bit and seven most significant bits associated with each momentum value in the corresponding set.

[0065] The system can be configured to implement an adaptive moment estimation algorithm. The gradient optimizer can be implemented using a field-programmable gate array (FPGA), and the gradient optimizer can be configured to operate in burst mode, such that successive burst cycles cause gradients to flow through the gradient optimizer.

[0066] In yet another example, this disclosure relates to a method in a system including a gradient optimizer and a memory configured to store weights and momentum values ​​associated with a neural network model comprising L layers, where L is an integer greater than one. The method may further include: retrieving from the memory a first set of momentum values ​​corresponding to layers of the neural network model, the first set of momentum values ​​having a selected storage format; and retrieving from the memory a second set of momentum values ​​corresponding to layers of the neural network model, the second set of momentum values ​​having a selected storage format. The method may further include: converting the first set of momentum values ​​having the selected storage format into a third set of momentum values ​​having a training format associated with the gradient optimizer; and converting the second set of momentum values ​​having the selected storage format into a fourth set of momentum values ​​having a training format associated with the gradient optimizer. The method may further include: performing gradient optimization using the gradient optimizer, using the third and fourth sets of momentum values. The method may further include: generating a fifth and a sixth set of momentum values ​​for the next iteration of gradient optimization. The method may further include: converting the fifth set of momentum values ​​into a seventh set of momentum values ​​with a storage format by storing only the sign bit and truncating a predetermined number of bits associated with the mantissa of the fifth set of momentum values. The method may further include: converting the sixth set of momentum values ​​into an eighth set of momentum values ​​with a storage format by storing only the sign bit and truncating a predetermined number of bits associated with the mantissa of the sixth set of momentum values.

[0067] The selected storage format may include a simplified single-precision format. The training format may include either single-precision or double-precision formats. In this method, the step of performing gradient optimization may include implementing an adaptive moment estimation algorithm.

[0068] The training format can include 32-bit floating-point format, and the storage format can include 8-bit floating-point format. The gradient optimizer can be implemented using a field-programmable gate array (FPGA), and the gradient optimizer can be configured to operate in burst mode, such that successive burst cycles cause gradients to flow through the gradient optimizer.

[0069] It should be understood that the methods, modules, and components described herein are merely exemplary. Alternatively or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc. In an abstract but still explicit sense, any arrangement of components that implements the same function is effectively “associated” to achieve the desired function. Therefore, any two components combined herein to implement a particular function can be considered “associated” with each other to achieve the desired function, regardless of the architecture or intermediate components. Similarly, any two components so associated can also be considered “operably connected” or “coupled” with each other to achieve the desired function.

[0070] The functionality associated with some of the examples described in this disclosure may also include instructions stored in non-transitory media. As used herein, the term "non-transitory media" refers to any medium that stores data and / or instructions that cause a machine to operate in a particular manner. Exemplary non-transitory media include non-volatile media and / or volatile media. Non-volatile media include, for example, hard disks, solid-state drives, magnetic disks or magnetic tapes, optical disks or optical magnetic tapes, flash memory, EPROM, NVRAM, PRAM or other such media, or network versions of such media. Volatile media include, for example, dynamic memory, such as DRAM, SRAM, cache or other such media. Non-transitory media are distinct from transport media but may be used in conjunction with transport media. Transport media are used to transfer data and / or instructions to or from a machine. Exemplary transport media include coaxial cables, fiber optic cables, copper wires, and wireless media such as radio waves.

[0071] Furthermore, those skilled in the art will recognize that the boundaries between the functions of the above operations are merely illustrative. The functions of multiple operations can be combined into a single operation, and / or the functions of a single operation can be distributed among additional operations. Moreover, alternative embodiments may include multiple instances of a particular operation, and the order of operations can be changed in various other embodiments.

[0072] Although specific examples are provided in this disclosure, various modifications and changes may be made without departing from the scope of this disclosure as set forth in the following claims. Therefore, the specification and drawings should be viewed in an illustrative rather than restrictive sense, and all such modifications are intended to be included within the scope of this disclosure. Any benefits, advantages, or solutions to problems described herein with reference to specific examples are not intended to be construed as key, essential, or fundamental features or elements of any or all claims.

[0073] Furthermore, as used herein, the terms “a” or “an” are defined as one or more. Additionally, the use of introductory phrases such as “at least one” and “one or more” in claims should not be construed as implying that introducing another claim element through the indefinite article “a” or “an” limits any particular claim containing such an introduced claim element to inventions containing only one such element, even if the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” or “an.” The same applies to the use of definite articles.

[0074] Unless otherwise stated, terms such as “first” and “second” are used to distinguish the elements described by these terms. Therefore, these terms are not necessarily intended to indicate the time priority or other priority of these elements.

Claims

1. A method in a system, the system comprising a gradient optimizer and a memory configured to store momentum values ​​associated with a neural network model comprising L layers, wherein L is an integer greater than one, the method comprising: Retrieve a first set of momentum values ​​from the memory, the first set of momentum values ​​corresponding to a layer of the neural network model, the first set of momentum values ​​having a selected storage format; and retrieve a second set of momentum values ​​from the memory, the second set of momentum values ​​corresponding to the layer of the neural network model, the second set of momentum values ​​having the selected storage format. The first set of momentum values ​​with the selected storage format is converted into a third set of momentum values ​​with a training format associated with the gradient optimizer, and the second set of momentum values ​​with the selected storage format is converted into a fourth set of momentum values ​​with a training format associated with the gradient optimizer. as well as Gradient optimization is performed using the gradient optimizer, using the third set of momentum values ​​and the fourth set of momentum values, wherein the gradient optimizer is configured to operate in burst mode, such that consecutive bursts of periods cause gradients to flow through the gradient optimizer. The storage format described therein has lower accuracy than the training format.

2. The method of claim 1, wherein the selected storage format includes a simplified single-precision format.

3. The method according to claim 1, wherein the training format includes a single-precision format or a double-precision format.

4. The method of claim 1, wherein converting the first set of momentum values ​​having the selected storage format into the third set of momentum values ​​having the training format includes padding with additional zeros to form single-precision format momentum values.

5. The method according to claim 1, further comprising: A fifth set of momentum values ​​and a sixth set of momentum values ​​are generated for the next iteration of gradient optimization. Before storing each of the fifth set of momentum values ​​and the sixth set of momentum values, each of the fifth set of momentum values ​​and the sixth set of momentum values ​​is converted into the storage format by storing only the sign bit and seven most significant bits associated with each of the corresponding momentum values.

6. The method of claim 1, wherein performing gradient optimization includes implementing an adaptive moment estimation algorithm.

7. The method of claim 6, wherein the gradient optimizer is implemented using a field-programmable gate array.

8. A system including a gradient optimizer, the system comprising: The memory is configured to store momentum values ​​associated with a neural network model comprising L layers, where L is an integer greater than one. and The system is configured as follows: Retrieve a first set of momentum values ​​from the memory, the first set of momentum values ​​corresponding to a layer of the neural network model, the first set of momentum values ​​having a selected storage format; and retrieve a second set of momentum values ​​from the memory, the second set of momentum values ​​corresponding to the layer of the neural network model, the second set of momentum values ​​having the selected storage format. The first set of momentum values ​​with the selected storage format is converted into a third set of momentum values ​​with a training format associated with the gradient optimizer, and the second set of momentum values ​​with the selected storage format is converted into a fourth set of momentum values ​​with a training format associated with the gradient optimizer. Gradient optimization is performed using the gradient optimizer, using the third set of momentum values ​​and the fourth set of momentum values, wherein the gradient optimizer is configured to operate in burst mode, such that consecutive bursts of periods cause gradients to flow through the gradient optimizer. The storage format described therein has lower accuracy than the training format.

9. The system of claim 8, wherein the selected storage format includes a simplified single-precision format.

10. The system of claim 8, wherein the training format includes a single-precision format or a double-precision format.

11. The system of claim 10, wherein the system is further configured to fill in additional zeros to form a single-precision or double-precision momentum value.

12. The system of claim 8, wherein the system is further configured to: generate a fifth set of momentum values ​​and a sixth set of momentum values ​​for the next iteration of gradient optimization, and wherein the system is further configured to: convert each set of momentum values ​​in the fifth set of momentum values ​​and the sixth set of momentum values ​​into the storage format by storing only the sign bit and seven most significant bits associated with each of the respective momentum values ​​before storing each set of momentum values ​​in the fifth set of momentum values ​​and the sixth set of momentum values.

13. The system of claim 8, wherein the system is configured to implement an adaptive moment estimation algorithm.

14. The system of claim 8, wherein the gradient optimizer is implemented using a field-programmable gate array.

15. A method in a system, the system comprising a gradient optimizer and a memory configured to store weights and momentum values ​​associated with a neural network model comprising L layers, wherein L is an integer greater than one, the method comprising: Retrieve a first set of momentum values ​​from the memory, the first set of momentum values ​​corresponding to a layer of the neural network model, the first set of momentum values ​​having a selected storage format; and retrieve a second set of momentum values ​​from the memory, the second set of momentum values ​​corresponding to the layer of the neural network model, the second set of momentum values ​​having the selected storage format. The first set of momentum values ​​with the selected storage format is converted into a third set of momentum values ​​with a training format associated with the gradient optimizer, and the second set of momentum values ​​with the selected storage format is converted into a fourth set of momentum values ​​with a training format associated with the gradient optimizer. The gradient optimizer is used to perform gradient optimization using the third set of momentum values ​​and the fourth set of momentum values. Generate the fifth and sixth sets of momentum values ​​for the next iteration of gradient optimization; The fifth set of momentum values ​​is converted into a seventh set of momentum values ​​having the storage format by storing only the sign bit and truncating a predetermined number of bits associated with the mantissa of the fifth set of momentum values. as well as The sixth set of momentum values ​​is converted into an eighth set of momentum values ​​having the storage format by storing only the sign bit and truncating the predetermined number of bits associated with the mantissa of the sixth set of momentum values, wherein the gradient optimizer is configured to operate in burst mode such that successive bursts of time cause gradients to flow through the gradient optimizer. The storage format described therein has lower accuracy than the training format.

16. The method of claim 15, wherein the selected storage format includes a simplified single-precision format.

17. The method of claim 15, wherein the training format includes a single-precision format or a double-precision format.

18. The method of claim 15, wherein performing gradient optimization includes implementing an adaptive moment estimation algorithm.

19. The method of claim 15, wherein the training format includes a 32-bit floating-point format, and wherein the storage format includes an 8-bit floating-point format.

20. The method of claim 15, wherein the gradient optimizer is implemented using a field-programmable gate array.

Images