Training optimization for low memory occupancy

Abstract

A method is described. The method includes profiling a model of a learning network having a plurality of layers and associated memory. The layers include a weight layer and an active layer. The plurality of weight layers includes weights. The method also includes determining a training technique for learning a model on the network based on the profiling. The determination of the training technique includes optimizing latency for at least one training iteration for the capacity of the associated memory. The training iteration (one or more times) includes at least one update of the weight of the weight layer.

Description

[0001] Cross Reference to Other Applications This application claims priority to U.S. Provisional Patent Application No. 63 / 472,898, filed June 14, 2023, entitled “TRAINING GRAPH OPTIMIZATION FOR LOW MEMORY FOOTPRINT,” which is incorporated herein by reference for all purposes. BACKGROUND

[0002] A learning network (e.g., a deep neural network) typically includes weight layers interleaved with activation layers. A weight layer can be thought of as multiplying an input signal (an “activation” to that weight layer) by weights stored therein and providing a corresponding output signal. For example, the weights can be stored digital values or analog resistances that are multiplied by an input current, voltage, or bit signal. The output signal of a weight layer is input to the next activation layer, if any. A neuron in an activation layer applies some activation function to the weighted input signal and provides an output signal corresponding to the state of the neuron. The output signal from an activation layer is provided as an input signal (i.e., an activation) to the next weight layer, if any. This process can be repeated for the layers of the network. The structure of the network (e.g., the number of layers and connectivity between layers, the dimensions of the layers, the types of activation functions applied) including the values of the weights is referred to as a model.

[0003] To utilize a learning network, the learning network is trained. Training determines optimized values for the weights in each weight layer. Training thus provides parameters for the model. Supervised training can include performing an inference, determining a loss function, and updating the weights based on the loss function. An inference provides an output signal for a given activation input to the learning network. In other words, an inference provides an output signal based on propagating an initial input activation forward through the learning network. The loss function can be a measure of how much the output signal of the learning network differs from a target output for the initial input activation. The loss function can be thought of as a difference between the final output signal and the target output. The weights in one or more weight layers are updated to improve the correlation between the output signal of the learning network and the target output (i.e., to reduce the loss function).

[0004] To determine the weight updates, techniques such as backpropagation can be used. Such techniques often involve backpropagating the loss function through the learning network. Based on the backpropagation of the loss function through the learning network and the input signals (activations) to the weight layers, weight updates for the weight layers can be determined and applied to the weights in the weight layers. This process of inference, loss function determination, and weight update is repeated for multiple iterations until the loss function is sufficiently small and the output signal is sufficiently correlated with the target output.

[0005] While training can produce learning networks capable of solving challenging problems, problems exist. Training can be time-consuming. It is desirable to reduce the latency or time required for iterations used to perform inference, loss function determination, and weight updates. Furthermore, learning networks store information during training. For example, at least temporarily, the loss function and activations, target output, and other information are stored. This information storage consumes memory. Particularly for edge devices using learning networks, this information storage can consume undesirably large amounts of memory. In some cases, it may even exhaust available memory. Therefore, improved techniques for training learning networks are desired. Attached Figure Description

[0006] Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

[0007] Figure 1 An embodiment of the data flow in an implementation of a learning network is described.

[0008] Figure 2 This is a flowchart depicting an embodiment of a method for training a learning network.

[0009] Figures 3-9 Examples of different latency and memory usage for training the learning network are described.

[0010] Figure 10 This is a table depicting examples of memory requirements for various learning network layers.

[0011] Figure 11 This is a schematic diagram depicting an embodiment of a system with an efficient architecture that can be used in AI accelerators.

[0012] Figure 12 An example of a system with an efficient architecture that can be used in AI accelerators is described.

[0013] Figure 13 An example of a system with an efficient architecture that can be used in AI accelerators is described.

[0014] Figure 14 An embodiment of a computing engine that can be used in an AI accelerator is depicted.

[0015] Figure 15 An embodiment of a computing engine that can be used in an AI accelerator and is capable of performing local updates is described.

[0016] Figure 16 An embodiment of a portion of an in-memory computing module that can be used in an AI accelerator is depicted.

[0017] Figure 17An embodiment of a portion of an in-memory computing module that can be used in an AI accelerator is depicted. Detailed Implementation

[0018] The invention can be implemented in various ways, including as a process; an apparatus; a system; a component of matter; a computer program product contained on a computer-readable storage medium; and / or a processor (such as a processor configured to execute instructions stored in and / or provided by memory coupled to the processor). In this specification, these implementations or any other form of the invention may be referred to as technology. Generally, the order of steps of the disclosed process can be varied within the scope of the invention. Unless otherwise stated, components such as processors or memory described as being configured to perform tasks can be implemented as general-purpose components temporarily configured to perform tasks at a given time or as specific components manufactured to perform tasks. As used herein, the term "processor" refers to one or more means, circuits, and / or processing cores configured to process data such as computer program instructions.

[0019] The appendix below, together with the description of the principles of the invention, Figure 1 This document provides a detailed description of one or more embodiments of the invention. The invention is described in relation to such embodiments, but is not limited to any particular embodiment. The scope of the invention is defined only by the claims, and the invention includes numerous alternatives, modifications, and equivalents. Numerous specific details are set forth in the following description to provide a thorough understanding of the invention. These details are provided for illustrative purposes, and the invention may be practiced according to the claims without requiring some or all of these specific details. For clarity, technical materials known in the art related to the invention have not been described in detail so as not to unnecessarily obscure the invention.

[0020] Learning networks (such as deep neural networks) consist of weight layers scattered with activation layers. Weights can be stored digital values ​​or analog resistors multiplied by an input signal (“activation”) such as a current, voltage, or bit signal. The output signal of the weight layer can be fed into the next activation layer (if any). The activation layer applies one or more activation functions (such as Tanh, ReLU, or SoftMax) and provides an output signal corresponding to the state of the neuron. The output signal from the activation layer can be provided as an activation to the next weight layer (if any). The combination of parameters such as the values ​​of the weights and the learning network (e.g., the number of layers and the connectivity between layers, the dimension of the layers, the type of activation function applied) is called a model.

[0021] The learning network is trained for a specific task, such as image recognition or as a large language model. Training optimizes the values ​​of the weights in each weight layer. Training thus provides the parameters of the model. Training may include iteratively performing inference, determining the loss function used for inference based on the target output, and performing weight updates based on the loss function. Therefore, the weights in one or more weight layers are adjusted to improve the correlation between the output signal of the learning network and the target output.

[0022] While training prepares a learning network and model for use, improvements are expected. For example, a learning network might be expected to be used in an edge device (e.g., a smartphone). Training a learning network consumes both time and memory. In some cases, the latency (time spent executing iterations of training) and / or memory consumed during training may be greater than expected. For instance, training might consume all the available memory on the edge device or might require more memory than is available at the edge device. Therefore, improvements in training are desired.

[0023] A method is described. The method includes profiling a learning network having multiple layers and associated memory. Layers include weight layers and activation layers. Multiple weight layers include weights. The method also includes determining a training technique for the model on the learning network based on the profiling. Determining the training technique includes optimizing the latency for at least one training iteration with respect to the capacity of the associated memory. One or more training iterations include at least one update to the weights of the weight layers.

[0024] In some embodiments, profiling further includes performing training iterations for the model using specific training techniques. Specific training techniques may include at least one of backpropagation, activation recalculation, partial activation recalculation, or feedback alignment. Performing training iterations may further include performing inference using training activation inputs, determining a loss function based on the inference, determining weight updates using the loss function, and updating the weights. In some embodiments, updating weights using the loss function includes backpropagating the loss function through layers.

[0025] The associated memory capacity may be less than the memory required for the training iteration. In this case, performing a training iteration may include loading the activations of the weight layers from remote memory. Performing a training iteration may also include storing the outputs of the weight layers in remote memory. Loading and storing can be repeated for each subsequent weight layer.

[0026] In some embodiments, profiling the model further includes determining a descriptor for the model and selecting a training technique from a library of multiple training techniques based on multiple descriptors. In some cases, multiple training techniques may be selected. These training techniques may be combined or used for specific situations. In some embodiments, the descriptor includes at least one of batch size, number of channels, width of input activations, height of input activations, stride, padding size, kernel size, pooling size, or dilation rate.

[0027] A system comprising memory and a processor coupled to the memory is described. The processor is configured to profile a model of a learning network. The learning network has layers and associated memory. Layers include weight layers and activation layers. Weight layers include weights. The processor is also configured to determine a training technique for the model on the learning network based on the profile. To determine the training technique, the processor is further configured to optimize the latency for at least one training iteration with respect to the capacity of the associated memory. One or more training iterations include updates to one or more weights of the weight layers.

[0028] In some embodiments, to profil the model, the processor is further configured to manage the execution of training iterations for the model using specific training techniques. Specific training techniques may include at least one of backpropagation, activation recalculation, partial activation recalculation, or feedback alignment. In some embodiments, executing training iterations further includes learning the network to perform inference using training activation inputs, determining a loss function based on the inference, determining weight updates using the loss function, and updating the weights. Updating the weights may include backpropagating the loss function through multiple layers.

[0029] In some cases, the associated memory capacity is less than the memory required for training iterations. In such situations, the learning network can be managed to perform training iterations by loading the activations of multiple weight layers from remote memory and storing the outputs of those weight layers in remote memory. This process of loading activations and storing outputs can be repeated for each subsequent weight layer.

[0030] In some embodiments, the learning network includes at least one tile. Each tile may include a computation engine and a general-purpose (GP) processor coupled to the computation engine. Each computation engine in the computation engine may include an in-memory computation (CIM) hardware module. The CIM hardware module stores at least a portion of the weights corresponding to a matrix and is configured to perform vector-matrix multiplication (VMM) on the matrix. The computation engine corresponds to a weight layer. The GP processor is coupled to the computation engine and configured to provide control instructions and data to the computation engine. The general-purpose processor may also apply activation functions. Therefore, the GP processor may correspond to an activation layer. In some embodiments, the GP processor is coupled to the computation engine via a streaming port and a control port. The control port is configured to provide control instructions to the computation engine. The streaming port is configured to exchange data between the GP processor and the computation engine. The CIM hardware module may include multiple storage units for storing multiple weights.

[0031] A computer program product contained in a non-transitory computer-readable medium is described. The computer program product includes computer instructions for profiling a model of a learning network. The learning network includes layers and associated memory. Layers include weight layers and activation layers. Weight layers include weights. The computer program product also includes computer instructions for determining a training technique for the model on the learning network based on the profiling. Determining the training technique includes optimizing the latency for at least one training iteration with respect to the capacity of the associated memory. One or more training iterations include at least one update to the weights of the weight layers.

[0032] Figure 1An embodiment of the data flow in a learning network 100 is depicted. The learning network 100 includes weight layers 110-1, 110-2, 110-3, and 110-4 (collectively or generally referred to as 110) and activation layers 120-1, 120-2, 120-3, and 120-4 (collectively or generally referred to as 120). Therefore, the learning network 100 includes four layers. An additional number of layers may be present. Although the weight layer 110 is shown interleaved with the activation layer 120, in some embodiments, additional layers (not shown) may be provided (e.g., between layers 110 and 120) and / or one or more weight layers 110 or activation layers 120 may be omitted. In some embodiments, each weight layer 110 performs a vector-matrix multiplication (VMM) of the input vector or input activation (A0) with a matrix of weights stored in the weight layer 110. In some embodiments, the weight layer 110 performs the VMM in hardware. For example, a hardware accelerator can be used to perform VMM operations on the input activation and stored weight matrix in parallel. Activation layer 120 can be used to apply one or more activation functions to the weighted output signal of weight layer 110. For example, activation layer 120 can perform ReLU, SoftMax, and / or Tanh operations. Although depicted as a layer consistent with weight layer 110, activation layer 120 can be implemented using one or more processors. For example, the weighted output signal from weight layer 110 can be provided to the processor for the application of activation functions. Although described in the context of a single iteration for training and use, the learning network 100 can operate in batch mode (i.e., on batches of data). Furthermore, memory is typically present for the learning network 100, but in Figure 2 It is not explicitly described in the text.

[0033] During the use of the learning network 100, an input vector or input activation (A0) is provided to the weight layer 110-1. For example, the input activation may be RGB data of an image or other data that is expected to be manipulated by the learning network 100. A first weighted output is provided from the weight layer 110-1 to the activation layer 120-1. The activation layer 120-1 applies the first activation function to the first weighted output and provides the first activation output (i.e., first activation - A1) to the weight layer 120-2. A second weighted output is provided from the weight layer 110-2 to the activation layer 120-2. The activation layer 120-2 applies the second activation function to the second weighted output and provides the second activation output (i.e., second activation - A2) to the weight layer 110-3. A third weighted output is provided from the weight layer 110-3 to the activation layer 120-3. The activation layer 120-3 applies the third activation function to the third weighted output and provides the third activation (A3) to the weight layer 110-4. The fourth weighted output is provided from the weight layer 110-4 to the activation layer 120-4. The activation layer 120-4 applies the fourth activation function to the fourth weighted output and provides the output signal (A4) from the learning network 100. This process of propagating the input activation through the training network 100 can be considered as inference performed by the learning network 100.

[0034] Before use, it is expected that the learning network 100 will be trained. For training, a loss function calculator 130 and a weight update block 140 are used. Although shown as blocks, the weight update block 140 and / or the loss function calculator 130 can be provided in hardware and / or software. During training of the learning network 100, the output signal (A4) of the learning network 100 is provided to the loss calculator 130. The loss calculator 130 determines the differences between the target output signals. Using one or more weight update techniques 140, updates to the weights in the weight layer 110 are determined, and the weights in the weight layer 110 are updated. The weight update block 140 may utilize techniques including, but not limited to, backpropagation, balance propagation, activation recalculation, partial activation recalculation, feedback alignment, and / or some other techniques (or combinations thereof). This process can continue iteratively until the desired accuracy is achieved.

[0035] Figures 3-9 This is a flowchart depicting an embodiment of a method 200 for training a learning network (such as learning network 100). However, method 200 can be used for other learning networks. Although a particular process is shown in sequence, the process can be executed in another order, including parallel processing. Furthermore, the process may have sub-steps. Method 200 can be used for a specific task and / or for a specific learning network. Therefore, method 200 can be considered as a method for determining the parameters of a model.

[0036] At 202, the model of the learning network is profiled. In some embodiments, 202 includes determining the latency or time spent on one or more iterations of one or more training techniques. In other words, 202 can be used to benchmark the performance of the model's learning network. In some embodiments, 202 includes performing one or more training iterations on the model using one or more specific training techniques. For example, the model may have specific input activations (one or more) and target outputs (one or more) for the task for which the learning network is being trained (e.g., image recognition or text generation). The profiling at 202 may include performing one or more inferences on the learning network for the specific input activations (one or more). Loss functions (one or more) for the output signal may be computed based on the model's target output. Updates to the weights are computed and applied in some embodiments using one or more training techniques. Thus, a training iteration may include operations for both inference (forward path) and weight updates (backward path). In some embodiments, a single iteration is used to profile a specific training technique. In some embodiments, multiple iterations may be performed (e.g., until the training network is partially or fully trained). In some embodiments, a single batch may be used. In some embodiments, multiple batches may be used. Specific training techniques may include one or more of backpropagation, activation recalculation, partial activation recalculation, balance propagation, feedback alignment, another technique, and / or a combination of techniques. For example, in some embodiments, backpropagation may be used. In such embodiments, the loss function is backpropagated through the layers of the network, and the weight updates can be determined using the activation and propagation loss function for each layer. The time spent on this iteration and the memory used in storing the activation and propagation loss function for each weight layer can be determined. In some embodiments, other techniques are also used, and memory and latency are determined.

[0037] In some embodiments, the memory associated with the learning network may be insufficient for the training techniques used in 202. In such embodiments, a remote storage device, such as off-chip dynamic random access memory (DRAM), can be used to supplement the memory associated with the learning network. In this case, performing training iterations may include loading the activations of weight layers from the remote memory. Performing training iterations may also include storing the outputs of the weight layers in the remote memory. Loading and storing can be repeated for each subsequent weight layer.

[0038] In some embodiments, training iterations may not be performed as part of profiling the model at point 202. For example, for the same learning network, method 200 may have been performed sufficiently many times, and for similar learning networks, for the same task, and / or for similar tasks for which the model's descriptor can be used to profil the model, method 200 may have been performed sufficiently. In other words, profiling the model at point 202 may include determining the model's descriptor. In some embodiments, the descriptor includes at least one of batch size, number of channels, width of input activations, height of input activations, stride, padding size, kernel size, pooling size, or dilation rate. For example, for image analysis, training data may include 256-pixel × 156-pixel images. In this example, the number of channels may be three (for red, green, and blue), the width of input activations is 256, and the height of input activations is 156. Based on these and / or other characteristics of the model (in some embodiments, including the task for which the model / learning network is being trained), the model can be benchmarked. Therefore, the latency of training iterations can be determined.

[0039] The method also includes determining one or more training techniques for the model on the learning network at 204. The determination of the training techniques used is based on the profiling performed at 202. In some embodiments, 204 includes optimizing the latency of one or more training iterations for a given capacity of associated memory of the learning network. At 202, the latency of the training iterations can be determined. In some embodiments, the capacity used at 204 is the total memory present and / or available to the learning network. In some embodiments, the capacity of associated memory used at 204 can be different. For example, some additional amount of associated memory can be reserved for other uses. In this case, the capacity used at 204 is less than the available associated memory. In some embodiments, other memory that can be accessed by the learning network can be included in the capacity. In this case, the capacity used at 204 is greater than the available associated memory. In some embodiments, optimization is performed, for example, on a processor associated with or a portion of the learning network.

[0040] In some embodiments, optimization is not required as part of 204. For example, for the same learning network, method 200 may have been performed sufficiently many times, and for similar learning networks, for the same task, and / or for similar tasks for which the descriptor of the model determined in 202 can be used to select a training technique at 204, method 200 may have been performed sufficiently. In some embodiments, 202 may include selecting one or more training techniques from a library of multiple training techniques that can be used to learn the network based on the descriptor. In some embodiments, 204 includes selecting multiple training techniques and combining the training techniques into one or more hybrid techniques. In some embodiments, the selected one or more training techniques may include one or more of backpropagation, activation recalculation, partial activation recalculation, balanced propagation, feedback alignment, another technique, and / or a combination of techniques.

[0041] Once one or more training techniques have been selected, the learning network can be trained at position 206. Therefore, the model's parameters, such as the optimized weights, can be determined. The learning network can then be used for the desired function.

[0042] For example, method 200 can be used in relation to learning network 100. At 202, the model of learning network 100 is profiled. For example, at 202, training inference can be performed for learning network 100 using the target output and input activation (A0). Using loss calculator 130, a loss function can be determined based on the target output and output signal (A4). The loss function can be provided to weight update block 140 to determine weight updates. In this example, it is assumed that the training technique used for 202 is backpropagation. To determine weight updates, weight update block 140 uses the backpropagation loss and activations A0, A1, A2, and A3. In some embodiments, the weights of layer 110 are also updated as part of 202. Therefore, the latency of training iterations can be determined. In some embodiments, the latency is determined as a function of the memory used to store activations (A0, A1, A2, and A3), target output, loss function, and backpropagation loss function. In some embodiments, 202 includes repeating inference, loss calculation, weight update determination, and weight updates for other training techniques. Latencies and memory used for such techniques can also be determined. 202 can be considered as profiling the model used to learn network 100 because the latency and / or memory used to train the learning network (i.e., train to optimize the weights / parameters of the task) for a task indicated by input activation and target output can be determined.

[0043] At 204, one or more training techniques (or one or more) to be used when training the learning network 100 are identified or selected. For example, for various training techniques, the latency determined at 202 can be optimized for a given capacity of memory associated with the learning network 100. In some embodiments, multiple techniques can be selected (e.g., for combining and / or for performing training of the learning network 100). At 206, the selected techniques can be used to train the learning network 100.

[0044] Using method 200, the training of the learning network can be improved. Due to the profiling and determination of the training techniques at 204, training can be performed in a more efficient manner. For example, a combination of reduced training time and appropriate amounts of memory used during training can be achieved. In the example of a learning network 100 with only four layers, method 200 may not be necessary because memory-intensive training techniques may not exhaust the available capacity of the associated memory. However, for edge devices using learning networks with a large number of layers, determining the characteristics of the model (e.g., dimensionality, latency, number of layers, and memory used) and identifying one or more training techniques based on the model can significantly improve training.

[0045] For example, Figures 3-9 This describes an embodiment of how model profiling can instruct different latency and memory usage during the training of the learning network. Therefore, method 200 can be used by… Figures 3-6 The learning network described. Figures 7-9 An example of how the model's profiling indicates the memory usage and latency of backpropagation in a four-layer learning network, as well as the form of activation recalculation, is described. Figure 3 An example is described that illustrates how the profiling of the model instructs the learning network's feedback alignment in terms of memory usage and latency, as well as the form of activation recalculation.

[0046] Figure 3 An embodiment of the data flow in a learning network 300 is depicted. The learning network 300 includes layers 310-1, 310-2, 310-3, and 310-4 (collectively or typically referred to as 310) and a loss calculator 330. Each layer 310 can be considered to include a weight layer and a corresponding activation layer. For example, layer 310-1 can be analogous to weight layer 110-1 combined with activation layer 120-1. Therefore, the learning network 300 includes four layers. An additional number of layers may exist. Furthermore, memory is typically provided for the learning network 300, but... Figures 4-6 It is not explicitly described in the text. However, in Figure 3 The associated memory is shown in the diagram. Figure 4 The instruction uses backpropagation. Therefore, activation (A) i(where i = 1, 2, 3, or 4) is shown as the output from a specific layer 310 in the forward direction (i.e., from input activation to output). Input activation A0 is provided to the first layer. The loss function for backpropagation ( Where 1, 2, 3, or 4) are shown as propagating in the reverse direction (i.e., from loss calculator 330 to first layer 310-1). The weight update of the corresponding layer is the outer product of the activation and the transpose of the loss function of the backpropagation. .

[0047] Figure 4 This is a graph 400 indicating the latency and memory usage of training iterations when backpropagation is used as a training technique for learning network 300. Graph 400 includes layers 410 and memory 420. Layers 410-1, 410-2, 410-3, and 410-4 (collectively or commonly referred to as 410) correspond to layers 310-1, 310-2, 310-3, and 310-4. Layer 430 corresponds to the loss calculator 330. Each cell 412 of layer 410 (only one of them is labeled) indicates whether the corresponding layer was used for a time interval. Memory 420 is used with the learning network 300. Memory 420 includes storage cells 422 corresponding to one or more storage locations where data is stored. Additionally, Figure 4 This indicates that two batches are being used for the training iteration. For layer 410: Instructions for the j-th th Batch passed the i-th th Forward propagation of layers (used for inference); Instructions for the j-th th Batch passed the i-th th Backpropagation of layers (used for backpropagation / weight update); Instructions for the j-th th Batch i th Update the weights in the layer; and Instructions for the j-th th Batch loss function calculation. Within memory 420, Instructions for the i-th th Layer and j th Batch activation storage; Instructions for the i-th th Layer and j th Storage of backpropagation loss in batches; Instructions for the j-th th Storage of the target output for the batch.

[0048] In the first time step, only the first layer 410-1 processes the data in the first batch of forward / inference paths. Through layer 410-1... This is illustrated. Input activation A0 is the only activation stored in cell 422 of memory 420 at this time step. In the second time step, the second layer 420-1 processes the first batch of data. Alternatively, a pipeline can be used in the learning network 300. Therefore, the first layer 410-1 also processes the data in the second batch of the forward / inference path. This is achieved through cell 412 of layer 410-2. and unit 412 in layer 410-1 To illustrate this. Additionally, both input activation A0 and activation A1 from the first layer 410-1 are stored in cell 422 of memory 420. This process continues through four layers 410 and the first four time steps. In the fifth time step, the fourth layer 410-4 processes the data in the second batch of the forward / inference path. Additionally, the loss calculator 430 determines the loss function for the first batch. This is achieved through the fifth time step in layer 410. and E 1 This is to indicate the purpose. To determine the loss function, the target output of the first batch is used. Therefore, memory 420 indicates that two batches are stored and used. All activations and the first batch of target outputs (T) 1 In the next time step, at E 2 The loss function for the second batch is determined here. For this purpose, the target output (T) of the second batch is stored and used. 2 The backpropagation loss of the first batch determined in the previous step. It is also stored. Backpropagation can begin. Therefore, layer 410-4 determines the weight update of layer 410-4 and backpropagates the loss of the first batch. Through layer 410... and In the corresponding time step The storage is used to indicate this. Additionally, the target output has been removed from storage to optimize memory usage. In subsequent time steps, the loss function of the first and second batches is backpropagated through layer 410 of the network, and the layer weights are updated. (As from...) Figure 5 As can be seen, the amount of memory used (e.g., in the ninth time step) is large. However, the latency is relatively low.

[0049] Figure 5This is a graph 500 indicating the latency and memory usage of backpropagation training iterations using a combination of activation recalculations and training techniques used to learn network 300. Graph 500 includes layers 510 and memory 520. Layers 510-1, 510-2, 510-3, and 510-4 (collectively or commonly referred to as layer 510) correspond to layers 310-1, 310-2, 310-3, and 310-4. Layer 530 corresponds to the loss calculator 330. Each cell 512 of layer 510 (only one of which is labeled) indicates whether the corresponding layer was used for a time interval. Memory 520 is used with the learning network 300. Memory 520 includes storage cells 522 corresponding to one or more storage locations where data is stored. Additionally, Figure 4 This indicates that two batches are being used for the training iteration. For Chart 500: and T j Pointing to Figure 6 Similar operations and storage amounts are used in the training process. For activation recalculation, only the input activation (A0) is stored throughout the entire training iteration. The remaining activations are computed once during the forward pass and recalculated as needed during backpropagation.

[0050] The first and second time steps of Chart 500 are similar to those of Chart 400. However, in the third time step, elements not currently in use are removed. Store only , and Only store activations that are currently in use or will be used in the next step. The process of discarding unused activations continues. In the fifth time step, at E... 1 Error Calculator 530 uses the first batch of activations The loss function is determined by the stored target output T1. Additionally, to prepare for the first batch of backpropagation, the first batch of input activations is repropagated through the first layer 510-1. At the first layer 510-1... To indicate this. Also... The second batch is propagated through the fourth layer. In the sixth time step, the loss function for the second batch is determined, and the second batch begins to be repropagated in the forward path to provide data for the backpropagation of the second batch. The process of forward repropagation to obtain appropriate activations for the batch and discarding unwanted activations continues. Additionally, the weights are updated. The comparison between Figures 400 and 500 indicates that the training technique used in Figure 500 (backpropagation combined with activation recalculation) requires less memory but has a longer latency than backpropagation alone.

[0051] Figure 6This is a graph 600 indicating the latency and memory usage of backpropagation training iterations using partial activation recalculation combinations of training techniques used to learn network 300. Graph 600 includes layers 610 and memory 620. Layers 610-1, 610-2, 610-3, and 610-4 (collectively or commonly referred to as layer 610) correspond to layers 310-1, 310-2, 310-3, and 310-4. Layer 630 corresponds to the loss calculator 330. Each cell 612 of layer 610 (only one of which is labeled) indicates whether the corresponding layer was used for a time interval. Memory 620 is used with the learning network 300. Memory 620 includes storage cells 622 corresponding to one or more storage locations where data is stored. Additionally, Figure 4 This indicates that two batches are being used for the training iteration. For chart 600: and Pointing to Figure 4 and 5 Similar operations and storage amounts are used. For partial activation recalculation, only the input activation (A0) and selected other activations are stored throughout the training iteration. In the illustrated embodiment, storage... and Continue this process until they are no longer needed. Calculate the remaining activations once during forward propagation, and recalculate them as needed during backpropagation.

[0052] The first and second time steps of Chart 600 are similar to those of Chart 400. However, in the third time step, elements not currently in use are removed. Store only , and .Apart from , and In addition, the process of discarding unused activations continues. In the fifth time step, at E... 1 Error Calculator 630 uses the first batch of activations The loss function is determined by the stored target output T1. The second batch is propagated through the fourth layer. In the sixth time step, the loss function is determined for the second batch. In the seventh time step... , stored Being positively repropagated to determine The loss function for the first batch is also backpropagated through layer 610-4. Similarly, in the eighth time step, the stored... Being positively repropagated to determine The loss function for the second batch is also backpropagated through layer 610-4. Forward propagation is then performed to obtain the appropriate activation and storage for the batch. and The process continues until these activations are no longer needed for iteration, and the unnecessary activations are discarded. The weights are also updated. A comparison of Figures 500 and 600 indicates that the training technique used in Figure 600 (backpropagation combined with partial activation recalculation) requires more memory than activation recalculation but has shorter latency. A comparison of Figures 400 and 600 indicates that the technique used in Figure 600 has longer latency and uses less memory compared to backpropagation alone.

[0053] Therefore, graphs 400, 500, and 600 indicate trade-offs between latency, computational intensity, and memory usage. In method 200, training techniques for graphs 400, 500, and 600 can be analyzed. This may include substantially determining graphs 400, 500, and 600. A training technique can then be selected from backpropagation, backpropagation with activation recomputation, and backpropagation with partial recomputation. For example, the latency of the three techniques can be optimized under the constraint that the memory used must not exceed a given capacity. For example, if the memory constraint is that no more than twelve blocks 422, 522, and 622 of one of graphs 400, 500, and 600 can be used at a given time, then backpropagation (graph 400) can be employed. In another example, if the memory constraint is that no more than eleven blocks 422, 522, and 622 of one of graphs 400, 500, and 600 can be used at a given time, then backpropagation with partial recomputation (graph 600) can be employed. If the memory constraint is that no more than eight blocks 422, 522, and 622 of one of graphs 400, 500, and 600 can be used at a given time, then backpropagation with full activation recalculation (graph 500) can be employed. Furthermore, the desired amount of activations stored for partial activation recalculation can be part of latency and memory optimization. Therefore, a trade-off between reducing latency and using more memory can be made. In this way, the training of learning network 300 can be improved.

[0054] Mathematically, the relationship between latency and memory for graphs 400, 500, and 600 can be described as follows. The training iterations for backpropagation (i.e., ...) can be given by the following formula. Figure 5 (As shown) the required time or delay: .

[0055] In the equation above, It is the total delay. It is a time delay error. It is the time delay of the forward path of layer i. It is the delay of the reverse path of layer i, and Let N be the latency used to determine and execute weight updates for layer i, and N be the number of layers. The required memory can be given by the following formula: .

[0056] In the equation above, Mem is the required memory, M A0 M is the memory required to store the input to activate A0. T M is the memory required to store the target. Ai M is the memory used to store the activation of layer i. e It is memory used for errors. This is the memory used to store the backpropagation loss, and N is the number of layers. However, if the activations are not stored, they are recalculated using activation recalculation. In this case (e.g.) Figure 7 As shown in the figure, the delay can be given by the following formula: .

[0057] The variables in the equation above are similar to those in backpropagation. The additional summation in the first term illustrates the recalculation of activations. In this case, the required memory is: .

[0058] The variables in the equation above are similar to those in backpropagation. Similarly, the delay for partial activation recalculation can be given by the following equation: .

[0059] The variables in the equation above are similar to those in backpropagation. In this case, the required memory can be given by the following equation: The variables in the equation above are similar to those in backpropagation. Additionally, S... i This represents a binary vector of activations stored. Using method 200 (specifically, procedure 204 of method 200), latency can be mathematically optimized for a given memory constraint. Therefore, training can be made more efficient.

[0060] Figure 7 An embodiment of the data flow in a learning network 700 is depicted. The learning network 700 includes layers 710-1, 710-2, 710-3, and 710-4 (collectively or typically referred to as 710) and a loss calculator 730. Each layer 710 can be considered to include a weight layer and a corresponding activation layer. For example, layer 710-1 can be analogous to weight layer 110-1 combined with activation layer 120-1. Therefore, the learning network 700 includes four layers. An additional number of layers may exist. Furthermore, memory is typically provided for the learning network 700, but... Figures 8-9 It is not explicitly described in the text. However, inFigure 7 The associated memory is shown in the diagram. Figure 8 The indication uses feedback alignment. Therefore, in addition to layer 710, transposed layers 720-1, 720-2, 720-3, and 720-4 (collectively or commonly referred to as 720) are also shown. Transposed layer 720 is used to determine weight updates by means of feedback alignment, using the transpose of the weight matrix of the corresponding layer 710. Therefore, activation (A i (where i = 1, 2, 7, or 4) is shown as the output from a specific layer 710 in the forward direction (i.e., from input activation to output). The loss function for backpropagation (d... i , where i = 1, 2, 7 or 4) is shown as propagating in the reverse direction (i.e., from loss calculator 730 to first layer 720-1).

[0061] Figure 8 This is a graph 800 indicating the latency and memory usage of training iterations for a training technique that aligns feedback for learning network 700. Graph 800 includes layers 810 and memory 820. Layers 810-1, 810-2, 810-3, and 810-4 (collectively or commonly referred to as layer 810) correspond to layers 710-1, 710-2, 710-3, and 710-4. Layer 830 corresponds to the loss calculator 730. Each cell 812 of layer 810 (only one is labeled) indicates whether the corresponding layer was used for a time interval. Memory 820 is used with the learning network 700. Memory 820 includes storage cells 822 corresponding to one or more storage locations where data is stored. Additionally, Figure 9 This indicates that two batches are being used for the training iteration. For layer 810: Instructions for the j-th th Batch passed the i-th th Forward propagation of layers (used for inference); Instructions for the j-th th Batch passed the i-th th Backpropagation of layers (used for backpropagation / weight update); Instructions for the j-th th Batch i th Update the weights in the layer; and Instructions for the j-th th Batch loss function calculation. Within memory 820, Instructions for the i-th th Layer and j th Batch activation storage; Instructions for the i-th th Layer and j th Storage of backpropagation loss in batches; Instructions for the j-th th Storage of the target output for the batch.

[0062] Figure 9 This is a graph 900 indicating the latency and memory usage of training iterations aligned with the feedback of the activation recalculation combination used in the training technique for learning network 700. Graph 900 includes layers 910 and memory 920. Layers 910-1, 910-2, 910-3, and 910-4 (collectively or commonly referred to as layer 910) correspond to layers 710-1, 710-2, 710-3, and 710-4. Layer 930 corresponds to the loss calculator 730. Each cell 912 of layer 910 (only one of which is labeled) indicates whether the corresponding layer was used for a time interval. Memory 920 is used with the learning network 300. Memory 920 includes storage cells 922 corresponding to one or more storage locations where data is stored. Additionally, Figure 8 This indicates that two batches are being used for the training iteration. For chart 900: and T j Pointing to Figure 10 Similar operations and storage amounts are used in the training process. For activation recalculation, only the input activation (A0) is stored throughout the entire training iteration. The remaining activations are computed once during the forward pass and recalculated as needed during backpropagation.

[0063] The comparison of graphs 800 and 900 indicates that feedback alignment with activation recalculation consumes less memory compared to feedback alignment alone. However, feedback alignment with activation recalculation is computationally more intensive. Graphs 400, 500, 600, 800, and 900 indicate the trade-offs between latency, computational intensity, and memory usage for various training techniques. In method 200, the training techniques of graphs 400, 500, 600, 800, and 900 can be profiled. This may include substantially determining graphs 400, 500, 600, 800, and 900 at 202. A training technique can then be selected from backpropagation, backpropagation with activation recalculation, backpropagation with partial recalculation, feedback alignment, and feedback alignment with activation recalculation. For example, the latency of the five techniques (optionally together with other techniques) can be optimized under the constraint that the memory used does not exceed a given capacity. In some embodiments, measurements of computational intensity or power consumption (e.g., peak and / or average power) can also be optimized. Therefore, trade-offs can be made between reducing training latency, memory consumption during transactions, and / or computational intensity during training. In this way, training of networks 100, 300, and / or 700 can be improved.

[0064] Therefore, method 200 can be used to improve the training of the learning network. In some embodiments, the learning network can be implemented using a hardware accelerator. In addition to layer-by-layer profiling of the model used for the learning network, other techniques can be used in combination with method 200. For example, layer 110 can be divided into smaller blocks and / or recomputed per convolutional kernel activation. Similarly, other training techniques, other types of learning networks and / or models can be used, and method 200 can be used to optimize other training techniques, other types of learning networks and / or models. As indicated above, descriptors of the model and / or learning network can be used to determine latency, memory usage, and / or the desired training techniques. For example, Figures 11-17 Indicates the memory required for various types of layers. The memory used in relation to the latency of the layers can be used to optimize the training of the learning network using method 200.

[0065] Hardware accelerators can be used to implement learning networks such as learning networks 100, 300, and 700. For example, Figures 11-17 The computational blocks and computational engines available when providing learning networks of 100, 300, and / or 700 are described. These computational blocks and / or computational engines can be part of a system-on-a-chip and / or a network-on-a-chip. Therefore, they can be used with... Figure 11 The hardware accelerators described herein are used in connection with method 200 and diagrams 400, 500, 600, 800 and / or 900.

[0066] Figure 11 This is a schematic diagram depicting an embodiment of system 1100 that can be used in a learning network. System 1100 is a computational block and can be considered an artificial intelligence (AI) accelerator with an efficient architecture. The computational block (or simply "block") 1100 can be implemented as a single integrated circuit. The computational block 1100 includes a general-purpose (GP) processor 1110 and computational engines 1120-0 to 1120-5 (collectively or generally referred to as computational engines 1120). Although five computational engines 1120 are shown, in other embodiments, a different number may be included. The GP processor 1110 is shown coupled to the computational engines 1120 via a computational bus (or other connector) 1140 and a bus 1150. In other embodiments, the GP processor 1110 may be connected to the computational engines 1120 in another manner. In some embodiments, the computational block 1100 may include on-block memory 1130. In other embodiments, memory 1130 may be omitted. In some embodiments, other components may be present in computation block 1100, such as cache or additional memory, one or more modules for applying activation functions, modules for moving data, and / or other modules.

[0067] GP processor 1110 is a Reduced Instruction Set Computer (RISC) processor. For example, GP processor 1110 may be a RISC-V processor or an ARM processor. In other embodiments, one or more different and / or additional general-purpose processors may be used. GP processor 1110 provides control instructions and data to computing engine 1120. GP processor 1110 implements one or more instruction sets used in controlling computing engine 1120. GP processor 1110 provides commands to computing engine 1120 and controls data movement to and / or from computing engine 1120. GP processor 1110 may therefore act as part of the control plane of computing engine 1120 and block 1100 (i.e., providing commands and being part of the data path).

[0068] In some embodiments, data is moved from memory 1130 or another source to one or more compute engines 1120 via GP processor 1110. Data can be sent from memory 1130 to the internal memory of GP processor 1110 and then via buses 1140 and 1150 to the appropriate compute engine 1120. For example, data from memory 1130 can be provided to a vector register file (not shown) of GP processor 1110 and then from GP processor 1110 to the appropriate compute engine 1120. Once the compute engines 1120 have performed their functions, the output is provided to GP processor 1110. Similarly, data can be moved from compute engine 1120 to memory 1130 or another destination via GP processor 1110. Therefore, GP processor 1110 can be part of both the control plane and the data plane of compute block 1100.

[0069] The GP processor 1110 can also perform other functions. The GP processor 1110 can apply one or more activation functions to data. For example, activation functions (e.g., ReLU, Tanh, and / or SoftMax) can be applied to the output of one or more computation engines 1120. Therefore, the GP processor 1110 can perform non-linear operations. The GP processor 1110 can also perform linear functions and / or other operations. However, compared to, for example, the graphics processing unit (GPU) or central processing unit (CPU) of a computer system with which the block 1100 can be used, the GP processor 1110 is still expected to have reduced functionality.

[0070] Computation engine 1120 is configured to efficiently and in parallel execute tasks that may be part of using (e.g., performing inference) and / or training (e.g., performing inference and / or updating weights) a model. Computation engine 1120 is coupled to GP processor 1110, and computation engine 1120 receives commands and, in at least some embodiments, data from GP processor 1110. Computation engine 1120 is a module that performs vector-matrix multiplication (VMM) in parallel. Therefore, computation engine 1120 can perform linear operations. Each computation engine 1120 includes an in-memory computation (CIM) hardware module (not included in...). Figure 11 (Specifically shown in the diagram). The CIM hardware module stores the weights corresponding to the matrix and is configured to perform VMM on the matrix in parallel. The computation engine 1120 may also include one or more Local Update (LU) modules (not shown in the diagram). Figure 11 (See in detail). Such LU modules (one or more) allow the computation engine 1120 to update the weights stored in the CIM.

[0071] The CIM module is a hardware module that stores data and performs operations. In some embodiments, the CIM module stores the weights of the model. In this way, the CIM module determines the maximum size of the model (i.e., the maximum number of weights or parameters) that can be handled by the computation block 1100. The CIM module stores the weights (or other data) in fully addressable cells. The CIM module also uses the weights to perform operations. More specifically, the CIM module performs a VMM, where the vector can be an input vector (e.g., activations) provided using the GP processor 1110, and the matrix can be the weights (i.e., data / parameters) stored by the CIM module. The CIM module can be considered to include (e.g., memory for storing weights) and (e.g., computational hardware for performing vector-matrix multiplication of the stored weights). In some embodiments, the vector can be a matrix. The CIM module may include an analog SRAM having multiple SRAM cells and configured to provide one or more outputs (e.g., one or more voltages) corresponding to the data (weights / parameters) stored in each cell of the SRAM multiplied by the corresponding element of the input vector. In some embodiments, the CIM module may include a digital SRAM having multiple SRAM cells and configured to provide one or more outputs corresponding to the elements of the input vector multiplied by data (weights / parameters) stored in each cell of the digital SRAM. Other configurations of the CIM module are possible. Each CIM module thus stores weights corresponding to the matrix in its cells and is configured to perform vector-matrix multiplication of the matrix with the input vector. In some embodiments, if the utilization of the computing engine is below a certain threshold (e.g., 170%-80%), the CIM module of the computing engine 1120 may be reused as memory. For example, in such an embodiment, the CIM may store repeated weights or vectors (e.g., activations).

[0072] To facilitate on-chip learning, a Local Update (LU) module (not shown) may also be provided in the computing engine 1120. The LU module is coupled to a corresponding CIM module. The LU module is used to update weights (or other data) stored in the CIM module. The LU module is considered local because it is located near the CIM module. For example, one or more LU modules of a particular computing engine 1120 may reside in the same integrated circuit as one or more CIM modules of the computing engine 1120. In some embodiments, the LU module is considered local because it is fabricated on the same substrate (e.g., the same silicon wafer) as the corresponding CIM module. In some embodiments, the LU module is also used when determining weight updates. In other embodiments, individual components may compute weight updates. For example, weight updates may be determined by the GP processor 1110, in software by one or more other processors that are not part of the computing block 1100, by other hardware that is part of the computing block 1100, by other hardware outside the computing block 1100, and / or some combination thereof, in addition to or in place of the LU module.

[0073] Memory 1130 may be or include static random access memory (SRAM) and / or some other type of memory. Memory 1130 is shown coupled to GP processor 1110. In other words, data movement between memory 1130 and computing engine 1120 can be performed via GP processor 1120. In some embodiments, memory 1130 may be coupled to computing bus 1140 (i.e., coupled to computing engine 1120). Memory 1130 may store activations (e.g., the result of an input vector provided to computing block 1100 and an activation function applied to the output of computing engine 1120). Memory 1130 may also store weights. For example, if it is desired to change the weights stored in computing engine 1120, memory 1130 may contain backup copies of the weights or different weights. In some embodiments, memory 1130 is organized into a cell library (e.g., an SRAM cell library). In such embodiments, a particular library of memory 1130 may serve a particular computing engine(s) within computing engine(s). In other embodiments, the library of memory 1130 can serve any computing engine 1120.

[0074] In operation, the GP processor 1110 provides an input vector to one or more computing engines 1120. The input vector is expected to be multiplied by weights that may have been previously stored in the computing engines(s) 1120. If the weight matrix and / or the input vector has too many elements for a single computing engine, the input vector can be provided to multiple computing engines 1120. In some such embodiments, a portion of the input vector is provided to each of the multiple computing engines 1120 (each storing a portion of the weights). In some embodiments, the input vector is provided from memory 1130 to the GP processor 1110 and from the GP processor 1110 to the computing engines(s) 1120. The GP processor 1110 also instructs the computing engines(s) 1120 to perform a Virtual Model (VMM). The computing engines(s) 1120 perform the VMM between the input vector and the weight matrix to provide an output. The VMM is performed element-wise on the input vector. The output of the computing engines(s) 1120 can be considered as an output vector. The output is provided by the computing engines(s) 1120 to the GP processor 1110. For example, the output can be stored in a vector register file of GP processor 1110. GP processor 1110 can also store the output (e.g., in memory 1130) and / or provide the output to another off-block component. GP processor 1110 can apply a function (e.g., an activation function) to the output. The result of the activation function applied to the output of computation engine 1120 can be stored in GP processor 1110 (e.g., in a buffer or vector register file). GP processor 1110 can also store the result in memory 1130 or off-block. GP processor 1110 can provide the result as an input vector to one or more other computation engines 1120 to apply different sets of weights to the result, where another set of weights is stored in one or more other computation engines 1120. Thus, one or more inferences with one or more different sets of weights can be performed. In some embodiments, training can also be performed by block 1100. In some such embodiments, GP processor 1110 or another component (such as a host) can determine the desired update of the weights. In some embodiments, the LU module (not shown) of the computing engine 1120 may be used to determine weights and apply updates to the weights.

[0075] As in Figure 12Remote memory 1190 is also shown. For example, remote memory 1190 may include or be DRAM memory. Remote memory 1190 can be used for long-term storage. For example, input activations for training, target outputs for training, and / or other information can be stored in DRAM 1190. This information can be loaded into computation block 1100 as needed. For example, if computation block 1190 includes insufficient memory to perform training iterations as part of method 200, activations and / or other data can be temporarily stored and loaded from DRAM 1190 during training iterations of method 200.

[0076] Therefore, computation block 1100 comprises two computation blocks: a GP processor 1110 and a computation engine 1120 working together. The GP processor 1110 can perform non-linear operations (e.g., activation functions), and the computation engine 1120 can perform linear operations (e.g., VMM). The GP processor 1110 resides in the control and data plane of the computation engine 1120. Thus, the GP processor 1110 and the computation engine 1120 are tightly coupled. Therefore, data can be moved more efficiently within block 1100. Operations such as VMM and applying activation functions to the output of the computation engine 1120 can be performed more efficiently. Furthermore, there is no need to design and fabricate a dedicated controller for computation block 1100. Instead, the GP processor 1110 is used. As a result, computation block 1100 can be designed and fabricated more flexibly and easily. For example, the activations applied by the GP processor 1110 can be updated by updating the GP processor 1110. No new dedicated controller is required. Therefore, machine learning functions can be performed more efficiently and easily. Additionally, computation block 1100 includes on-block memory 1130. Using on-block memory, for example as note-taking memory, considers that computation block 1100 is highly independent of other components (e.g., other blocks). Therefore, multiple blocks 1100 can more easily work in parallel. This can improve learning efficiency.

[0077] Figure 12This is a schematic diagram depicting an embodiment of a computation block 1200 that can be used in a learning network. Computation block 1200 can be an AI accelerator with an efficient architecture. Computation block 1200 is similar to computation block 1100. Computation block 1200 therefore includes GP processor 1210 and computation engines 1220-0 to 1220-5 (collectively or generally referred to as computation engine 1220), which are respectively similar to GP processor 1110 and computation engines 1110-0 to 1110-5. Although five computation engines 1210 are shown, in other embodiments, a different number may be included. GP processor 1210 is shown coupled to computation engine 1220 via computation bus (or other connector) 1240 and bus 1250. In other embodiments, GP processor 1210 may be connected to computation engine 1220 in another manner. Computation block 1200 may include on-block memory 1230, similar to on-block memory 1130. Memory 1230 may therefore be or include SRAM. Data movement between memory 1230 and compute engine 1220 can be performed via GP processor 1220. In some embodiments, memory 1230 can be coupled to compute bus 1240 (i.e., coupled to compute engine 1220). In the illustrated embodiment, compute block 1200 also includes bus 1260, direct memory access (DMA) module 1270, and grid stop 1280.

[0078] GP processor 1210 is similar to GP processor 1110. Therefore, GP processor 1210 can be a RISC-V processor or an ARM processor. In other embodiments, different and / or additional general-purpose processors(s) may be used. GP processor 1210 provides control instructions to computing engine 1220 and manages data flow. Data sent to or from computing engine 1220 may also pass through GP processor 1210. Therefore, GP processor 1210 may be part of the control plane and data plane of computing block 1200. GP processor 1210 may also perform other functions, including nonlinear functions. For example, GP processor 1210 may apply one or more activation functions to data. In some embodiments, GP processor 1210 may include vector processing units (not shown) that perform nonlinear operations, such as applying activation functions to data. Also explicitly shown as part of GP processor 1210 are local memories 1212 and 1214. In some embodiments, local memory 1212 stores instructions, while local memory 1214 stores data.

[0079] Computation engine 1220 is similar to computation engine 1120. Computation engine 1220 is configured to efficiently and in parallel execute tasks that may be part of using and / or training a model. Computation engine 1220 is coupled to GP processor 1210, and computation engine 1220 receives commands and, in at least some embodiments, data from GP processor 1210. Computation engine 1220 performs linear operations such as VMM in parallel. Each computation engine 1220 includes a CIM hardware module (not described in the text) similar to the one described for computation engine 1120. Figure 12 (Specifically shown in the diagram). The CIM hardware module stores the weights corresponding to the matrix and is configured to perform a VMM on the matrix. The computation engine 1220 may also include one or more LU modules (not shown in the diagram). Figure 12 (See details in the text).

[0080] Bus 1250 couples the GP processor 1210 to the computing bus 1240, and thus to the computing engine 1220. The computer bus 1250 includes a control bus 1252, a streaming bus 1254, and a status bus 1256. The control bus 1252, streaming bus 1254, and status bus 1256 are coupled to the control port (unlabeled), streaming port (unlabeled), and status port (unlabeled) of the GP processor 1210, respectively. The control bus 1252 receives instructions from the GP processor 1210 for the computing engine 1220. The computing engine 1220 executes operations based on these instructions. For example, the instructions may include load instructions for loading data from the GP processor 1210 to one or more identified compute engines 1220, store instructions for storing data from one or more identified compute engines 1220 to the GP processor 1210, and support instructions for identifying the address of one or more identified compute engines 1220 to which data is to be loaded and from which data will be read. The streaming bus 1254 may be a high-speed, high-bandwidth bus. In some embodiments, the streaming bus 1254 is 512 bits wide. Other bus widths are possible. The streaming bus 1254 is used to rapidly move data between the GP processor 1210 and the compute engines 1220. The status bus may be read from or written to the status register of the compute engine 1220. Thus, the GP processor 1210 may notify a particular compute engine 1220 that a task, such as a VMM, has been completed.

[0081] Compute block 1200 also includes DMA 1270 and grid stop 1280. DMA 1270 initiates data movement within compute block 1200. DMA 1270 can be used to move data from outside the block to the block and vice versa. Therefore, DMA 1270 can be used with the host (not shown) and / or other blocks (not shown in the table). Figure 12(As shown in the diagram) communication. For example, the DMA 1270 can be used to transfer an input vector (activation) from the host or another block (not shown in the diagram). Figure 12 Data is moved to memory 1230 (as shown in the diagram). If memory 1230 is also directly connected to compute engine 1220 (e.g., via compute bus 1240), DMA 1270 can be used to move data between memory 1230 and compute engine 1220. Grid stop 1280 provides an interface between compute block 1200 and the construction of a grid network including compute block 1200. Therefore, grid stop 1280 can be used to communicate with other compute blocks (not shown) that can be used with compute block 1200. Grid stop 1280 can also be used to communicate with remote DRAM 1290. Data can also be moved via bus 1260. Therefore, in some embodiments, data can be moved to and / or from memory 1230 and to block 1200 and / or from block 1200 via buses such as buses 1240, 1250, and / or 1260.

[0082] Computation block 1200 operates in a manner similar to computation block 1100. For example, data can be transferred on the block from the host or other blocks via DMA 1270 and / or grid stop 1280. Such data can be stored in memory 1230. Therefore, memory 1230 can store weights and input vectors. Weights can be loaded into one or more computation engines 1220 for use. For example, weights can be moved from memory 1230 to one or more CIM hardware modules of computation engine 1220(one or more) via GP processor 1210. For inference, input vectors are provided to one or more of computation engines 1220 by GP processor 1210. To do this, input vectors / activations can be moved from memory 1230 to GP processor 1210 and from GP processor 1210 to one or more computation engines 1220(one or more) via streaming bus 1254. One or more computation engines 1220 execute the VMM (Virtual Machine) in parallel, processing the elements of the input vector and the weight matrix (or multiple matrices) stored in the one or more computation engines 1220. The outputs of the one or more computation engines 1220 can be stored from the one or more computation engines 1220 to the GP processor 1210 via a streaming bus 1254. The GP processor 1210 can apply a function (e.g., an activation function) to the output. The result of the activation function applied to the output of the computation engines 1220 can be stored in the GP processor 1210 (e.g., a buffer, which is not in...). Figure 13(This is explicitly shown in the image). The GP processor 1210 can also store the results in memory 1230. The GP processor 1210 can provide the results to another block or host via grid stop 1280 or DMA 1270. The GP processor can provide the results as input vectors to one or more other computing engines 1220 to apply different sets of weights to the results, where another set of weights is stored in one or more other computing engines. Thus, one or more inferences with one or more different sets of weights can be performed. In some embodiments, training can also be performed by block 1200. In some such embodiments, the GP processor 1210 or another component (such as the host) can determine the desired updates to the weights. In some embodiments, the LU module (not shown) of computing engine 1220 can be used to determine the weights and apply updates to the weights.

[0083] Computation block 1200 can share the benefits of computation block 1100. The GP processor 1210 and computation engine 1220 are closely cooperating computation blocks. For example, the data and control plane of computation block 1200 may include memory 1230, GP processor 1210, buses 1240 and 1250, and computation engine 1220. Therefore, data can be moved more efficiently within block 1200, and operations such as VMM and activation functions can be performed more efficiently. Furthermore, there is no need to design and fabricate a dedicated controller for computation block 1200. As a result, computation block 1200 can be designed and fabricated more flexibly and easily. Therefore, machine learning functions can be performed more efficiently and easily. Additionally, the memory 1230 on the block considers computation block 1200 to be highly independent of other components (e.g., other blocks). Therefore, multiple blocks 1200 can work in parallel more easily and efficiency can be improved.

[0084] Figure 13This is a schematic diagram depicting an embodiment of a computation block 1300 that can be used in a learning network. Computation block 1300 can be an AI accelerator with an efficient architecture. Computation block 1300 is similar to computation blocks 1100 and 1200. Computation block 1300 thus includes a GP processor 1310, computation engines 1120 / 220, memory 1130 / 230, computation buses 1140 / 240, buses 1150 / 250, bus 1260, DMA 1270, and grid stop 1280, similar to GP processors 1110 / 210, computation engines 1120-0 to 1320-5 (collectively or generally referred to as computation engine 1320), memory 1330, computation buses 1340, bus 1350, bus 1360, DMA 1370, and grid stop 1380. Although five computation engines 1310 are shown, in other embodiments, another number may be included. GP processor 1310 is shown coupled to computing engine 1320 via compute bus (or other connector) 1340 and bus 1350. In other embodiments, GP processor 1310 may be connected to computing engine 1320 in an alternative manner. GP processor 1310 also includes memory 1312 and 1314, which are respectively similar to local memory 1212 and 1214. Data movement between memory 1330 and computing engine 1320 may be performed via GP processor 1320. For example, bus 1350 includes control bus 1352, streaming bus 1354, and status bus 1356, which are respectively similar to control bus 1252, streaming bus 1254, and status bus 1256. In some embodiments, memory 1330 may be coupled to compute bus 1340 (i.e., coupled to computing engine 1320). Remote memory 1390, similar to remote memory 1190 and / or 1290, is also shown.

[0085] GP processor 1310 is similar to GP processors 1110 and / or 1210. Therefore, GP processor 1310 can be a RISC-V processor or an ARM processor. In other embodiments, different and / or additional general-purpose processors(s) may be used. GP processor 1310 provides control instructions to computing engine 1320 and manages data flow. Data sent to or from computing engine 1320 may also pass through GP processor 1310. Therefore, GP processor 1310 may be part of both the control plane and the data plane of computing block 1300. GP processor 1310 may also perform other functions, including nonlinear functions. For example, GP processor 1310 may apply one or more activation functions to data. In some embodiments, GP processor 1310 may include vector processing units (not shown) that perform nonlinear operations, such as applying activation functions to data.

[0086] Additionally, the GP processor includes an additional fixed-function compute block (FFCB) 1316. In some embodiments, the FFCB 1316 is a single-instruction multiple-data arithmetic logic unit (SIMD ALU). In some embodiments, the FFCB 1316 can be configured in another manner. The FFCB 1316 can be a tightly coupled fixed-function unit for on-device inference and training of a learning network. In some embodiments, the FFCB 1316 performs nonlinear operations, digital format conversion, and / or dynamic scaling. In some embodiments, other and / or additional operations can be performed by the FFCB 1316. The FFCB 1316 can be coupled to a data path for a vector processing unit of the GP processor 1310.

[0087] Computation engine 1320 is similar to computation engines 1120 and / or 1220. Computation engine 1320 is configured to efficiently and in parallel execute tasks that may be part of using and / or training a model. Computation engine 1320 is coupled to GP processor 1310, and computation engine 1320 receives commands and, in at least some embodiments, data from GP processor 1310. Computation engine 1320 performs linear operations such as VMMs in parallel. Each computation engine 1320 includes a CIM hardware module (not described in the text) similar to the one described for computation engine 1120. Figure 13 (Specifically shown in the diagram). The CIM hardware module stores the weights corresponding to the matrix and is configured to perform a VMM on the matrix. The computation engine 1320 may also include one or more LU modules (not shown in the diagram). Figure 14 (See in detail). Additionally, the memory 1330 on the block considers computation of block 1300 to be highly independent of other components (e.g., other blocks). Therefore, multiple blocks 1300 can more easily work in parallel.

[0088] Figure 15 A computational engine 1400 that can be used in an AI accelerator is depicted. The computational engine 1400 can be part of an AI accelerator that can be deployed to use a model (not explicitly depicted) and to consider on-chip training (or, in other words, on-chip learning) of the model. The computational engine 1400 can therefore be used as one or more computational engines 1120, 1220, and / or 1320. The computational engine 1400 includes a CIM module 1430 and a LU module 1440. Although one CIM module 1430 and one LU module 1440 are shown, the computational engine can include another number of CIM modules 1430 and / or another number of LU modules 1440. For example, the computational engine can include three CIM modules 1430 and one LU module 1440, one CIM module 1430 and two LU modules 1440, or two CIM modules 1430 and two LU modules 1440.

[0089] CIM module 1430 is a hardware module for storing data and performing operations. In some embodiments, CIM module 1430 stores the weights of the model. CIM module 1430 also uses the weights to perform operations. More specifically, CIM module 1430 performs vector-matrix multiplication, where the vector can be an input vector provided by processor 1110 and the matrix can be the weights (i.e., data / parameters) stored by CIM module 1430. Thus, CIM module 1430 can be considered to include memory (e.g., storing weights) and computational hardware (e.g., performing vector-matrix multiplication of the stored weights). In some embodiments, the vector can be a matrix (i.e., an n×m vector where n>1 and m>1). For example, CIM module 1430 may include an analog static random access memory (SRAM) having a plurality of SRAM cells and configured to provide one or more outputs (e.g., one or more voltages) corresponding to the data (weights / parameters) stored in each cell of the SRAM multiplied by the corresponding element of the input vector. In some embodiments, CIM module 1430 may include a digital static SRAM having a plurality of SRAM cells and configured to provide one or more outputs corresponding to the corresponding element of an input vector multiplied by data (weights / parameters) stored in each cell of the digital SRAM. In some embodiments, CIM module 1430 may include an analog resistive random access memory (RAM) configured to provide an output (e.g., one or more voltages) corresponding to the corresponding element of an input vector multiplied by the impedance of each cell. Other configurations of CIM module 1430 are possible. Each CIM module 1430 thus stores weights corresponding to a matrix in its cells and is configured to perform vector-matrix multiplication of the matrix with the input vector.

[0090] To facilitate on-chip learning, a LU module 1440 may be provided. The LU module 1440 is coupled to a corresponding CIM module 1430. The LU module 1440 is used to update weights (or other data) stored in the CIM module 1430. The LU module 1440 is considered local because it is close to the CIM module 1430. For example, the LU module 1440 may reside on the same integrated circuit as the CIM module 1430. In some embodiments, the LU module 1440 for a particular computing engine resides on the same integrated circuit as the CIM module 1430. In some embodiments, the LU module 1440 is considered local because it is fabricated on the same substrate (e.g., the same silicon wafer) as the corresponding CIM module 1430. In some embodiments, the LU module 1440 is also used when determining weight updates. In other embodiments, separate components may compute weight updates. For example, in addition to or instead of LU module 1440, weight updates may be determined by the GP processor, determined in software by one or more other processors that are not part of computing engine 1400 and / or the corresponding AI accelerator (e.g., computing blocks 1100, 1200, or 1300), determined by other hardware that is part of computing engine 1400 and / or the corresponding AI accelerator (e.g., computing blocks 1100, 1200, or 1300), determined by other hardware outside of computing engine 1400 and / or the corresponding AI accelerator (e.g., computing blocks 1100, 1200, or 1300), and / or some combination thereof.

[0091] Using computation engine 1400 in the context of computation blocks 1100, 1200, or 1300 and / or similar systems improves the efficiency and performance of the learning network. The use of CIM module 1430 can significantly reduce the time required to perform vector-matrix multiplications that provide weighted signals. Therefore, performing inference(s) using computation engine 1400 may require less time and power. This can improve the efficiency of model training and use. LU module 1440 considers local updates to the weights in CIM module 1430. This can reduce the data movement that might otherwise be required for weight updates. Therefore, the time spent on training can be significantly reduced. In some embodiments, the time spent on weight updates using LU module 1440 can be orders of magnitude smaller (i.e., one-tenth the time) compared to not performing updates locally. This can improve the efficiency and performance of the learning network provided by system 1100.

[0092] Figure 16An embodiment of a computing engine 1500, which can be used in an AI accelerator and is capable of performing local updates, is depicted. The computing engine 1500 may be a hardware computing engine similar to computing engine 1400. The computing engine 1500 thus includes a CIM module 1530 and a LU module 1540, respectively similar to CIM module 1430 and LU module 1440. The computing engine 1500 also includes analog bit mixers 1504-1 to 1504-n (generally or collectively referred to as 1504), one or more analog-to-digital converters (ADCs) 1506-1 to 1506-n (generally or collectively referred to as 1506), an input cache 1550, an output cache 1560, and an address decoder 1570. Although specific numbers of components 1502, 1504, 1506, 1530, 1540, 1542, 1544, 1546, 1360 and 1570 are shown, another number or more of components 1502, 1504, 1506, 1530, 1540, 1542, 1544, 1546, 1360 and 1570 may be present.

[0093] CIM module 1530 is a hardware module that stores data corresponding to weights and performs vector-matrix multiplication. The vector is an input vector provided to CIM module 1530 (e.g., via input cache 1550), and the matrix includes the weights stored by CIM module 1530. In some embodiments, the vector may be a matrix. Figure 16 and 17 The document describes an example of an embodiment of the CIM module that can be used in the CIM module 1530.

[0094] Figure 16 An embodiment of a cell in an SRAM CIM module that can be used in CIM module 1530 is depicted. A DAC 1502 of a computing engine 1500 is also shown. For clarity, only one SRAM cell 1610 is shown. However, multiple SRAM cells 1610 can exist. For example, multiple SRAM cells 1610 can be arranged in a rectangular array. SRAM cells 1610 can store weights or portions of weights. The illustrated CIM module includes lines 1602, 1604, and 1618, transistors 1606, 1608, 1612, 1614, and 1616, and capacitor 1620 (C...). S ) and 1622 (C L ).exist Figure 16In the illustrated embodiment, DAC 1502 converts the digital input voltage into differential voltages V1 and V2 with a zero reference. These voltages are coupled to each cell within a row. DAC 1502 is thus used for differential time encoding. Lines 1602 and 1604 carry voltages V1 and V2 from DAC 1502, respectively. Line 1618 is connected to address decoder 1570 (not in... Figure 16 (as shown in the figure) coupling, and line 1618 is used to select cell 1610 (and in the illustrated embodiment, to select the entire row including cell 1610) via transistors 1606 and 1608.

[0095] During operation, the voltages of capacitors 1620 and 1622 are set to zero, for example, by means of a reset supplied to transistor 1616. DAC 1502 provides differential voltages on lines 1602 and 1604, and the address decoder (not in...) Figure 16 (As shown in the diagram) The row of cell 1610 is selected via line 1618. If SRAM cell 1610 stores logic 1, transistor 1612 delivers input voltage V1, and if SRAM cell 1610 stores 0, transistor 1614 delivers input voltage V2. Therefore, based on the contents of SRAM cell 1610, an appropriate voltage is provided to capacitor 1620. Capacitor 1620 is connected in series with capacitor 1622. Therefore, capacitors 1620 and 1622 act as a capacitive voltage divider. Each row in the column of SRAM cell 1610 corresponds to the voltage passed through and the capacitance C of capacitor 1620. S The capacitance C of capacitor 1622 L The corresponding total voltage contributes. Each row contributes its corresponding voltage to capacitor 1622. The output voltage across capacitor 1622 is measured. In some embodiments, this voltage is passed to the corresponding aBit mixer 1504 of the column. In some embodiments, transistors can be used instead of capacitors 1620 and 1622 to act as resistors, thus creating a resistive voltage divider instead of a capacitive voltage divider. Therefore, using Figure 17 As depicted in the configuration, the CIM module 1530 can use data stored in the SRAM cell 1610 to perform vector-matrix multiplication.

[0096] Figure 17 An embodiment of a cell in one embodiment of a digital SRAM module that can be used in a CIM module 1530 is depicted. For clarity, only one digital SRAM cell 1710 is labeled. However, multiple cells 1710 exist, and multiple cells 1710 can be arranged in a rectangular array. The corresponding transistors 1706 and 1708, line 1718, logic gate 1720, adder tree 1722, and digital mixer 1724 for each cell are also labeled. Because Figure 15The SRAM module shown is digital, so it can be accessed from... Figure 17 The computing engine 1500 depicted in the figure omits DAC 1502, aBit mixer 1504, and ADC 1506.

[0097] During operation, the address decoder 1570 (not in) Figure 17 (As shown in the diagram) Line 1718 is used to enable the row including digital SRAM cell 1710. Transistors 1706 and 1708 are enabled, allowing data stored in digital SRAM cell 1710 to be provided to logic gate 1720. Logic gate 1720 combines the data stored in digital SRAM cell 1710 with the input vector. Therefore, the binary weights stored in digital SRAM cell 1710 are combined with the binary input. The output of logic gate 1720 is accumulated in adder tree 1722 and combined by digital mixer 1724. Therefore, using... Figure 15 As depicted in the configuration, the CIM module 1530 can use data stored in the digital SRAM cell 1710 to perform vector-matrix multiplication.

[0098] Return to reference Figure 16 The CIM module 1530 therefore stores the weights corresponding to the matrix in its cells and is configured to perform vector-matrix multiplication of the matrix with the input vector. In some embodiments, the computation engine 1500 stores positive weights in the CIM module 1530. However, for some models and / or some applications, it may be desirable to use both positive and negative weights. In this case, bipolar weights (e.g., having a range of -S to +S) are mapped to a positive range (e.g., 0 to S). For example, the bipolar weight matrix W can be mapped to the positive weight matrix W. P , so that: Where J is a matrix of all ones of the same size as W, and S is the maximum value of the weights (e.g., for N-bit weights, ...). For simplicity, CIM module 1530 is typically similar to... Figure 15 The computation engine 1500 is discussed in the context of the simulated SRAM CIM module described in the text.

[0099] Input cache 1550 receives the input vector for which vector-matrix multiplication is desired. In some embodiments, the input vector is provided to the input cache by a GP processor (such as GP processor 1110). The input vector can be read from memory, from a cache or register in the processor, or otherwise obtained. Digital-to-analog converter (DAC) 1502 converts the digital input vector into analog for operation on the vector by CIM module 1530. Although shown as connected to only some parts of CIM module 1530, DAC 1502 can be connected to all cells of CIM module 1530. Alternatively, multiple DACs 1502 can be used to connect to all cells of CIM module 1530. Address decoder 1570 includes address circuitry configured to selectively couple vector adder 1544 and write circuitry 1542 to each cell of CIM module 1530. Address decoder 1570 selects cells in CIM module 1530. For example, address decoder 1570 can select individual cells, rows, or columns to be updated, perform vector-matrix multiplication, or output results. In some embodiments, the aBit mixer 1504 combines the results from the CIM module 1530. Using the aBit mixer 1504 can save the ADC 1506 and allow access to the analog output voltage.

[0100] One or more ADCs 1506 convert the analog results of vector-matrix multiplication into digital form. The output cache 1560 receives the results of vector-matrix multiplication and outputs the results from the computation engine 1500. Therefore, vector-matrix multiplication can be performed using the CIM module 1530.

[0101] LU module 1540 includes write circuitry 1542 and vector adder 1544. In some embodiments, LU module 1540 includes a weight update calculator 1546. In other embodiments, the weight update calculator 1546 may be a separate component and / or may not reside within computation engine 1500. The weight update calculator 1546 is used to determine how to update the weights stored in CIM module 1530. In some embodiments, updates are determined sequentially based on the target output of a learning system in which computation engine 1500 is a part. In some embodiments, the provided weight updates may be sign-based (e.g., increasing the positive sign in the gradient of the loss function and decreasing the negative sign in the gradient of the loss function). In some embodiments, weight updates may be ternary (e.g., increasing the positive sign in the gradient of the loss function, decreasing the negative sign in the gradient of the loss function, and keeping the weights unchanged for the zero gradient of the loss function). Other types of weight updates may be possible. In some embodiments, weight update calculator 1546 provides an update signal indicating how to update each weight. Weights stored in the cells of CIM module 1530 are sensed and increased, decreased, or remained unchanged based on update signals. Specifically, weight updates can be provided to vector adder 1544, which also reads the weights of the cells in CIM module 1530. More specifically, adder 1544 is configured to selectively couple to each cell of the CIM module via address decoder 1570. Vector adder 1544 receives weight updates and adds the weight updates to the weights of each cell. Thus, the sum of the weight updates and the weights is determined. The resulting sum (i.e., the updated weights) is provided to write circuitry 1542. Write circuitry 1542 is coupled to vector adder 1544 and the cells of CIM module 1530. Write circuitry 1542 writes the weights and the sum of the weight updates to each cell. In some embodiments, LU module 1540 further includes a local batch weight update calculator (not in the local batch weight update calculator) coupled to vector adder 1544. Figure 15 (As shown in the image). This batch weight update calculator is configured to determine weight updates.

[0102] The computing engine 1500 may also include a control unit 1540. The control unit 1540 generates control signals depending on the operating mode of the computing engine 1500. The control unit 1540 is configured to provide control signals to the CIM hardware module 1530 and the LU module 1549. Some of the control signals correspond to inference mode. Some of the control signals correspond to training or weight update mode. In some embodiments, the control signals are generated by a control processor (not in the instruction set architecture) based on the instruction set architecture (ISA). Figure 16 As shown in the diagram, but similar to processor 1110, it controls the mode.

[0103] In inference mode, the input data is multiplied by stored weights, and the output is obtained after processing the ADC 1506. This mode may include many steps. For example, if capacitors arranged in a voltage divider are used to provide the output (e.g., in...), ​ If the capacitor (or other storage element) is reset, it may be set to zero or a precharge value, depending on its functionality. The capacitor divider operation is enabled to provide the output of the vector-matrix multiplication. The aBit mixer 1504 is enabled. One or more ADCs 1506 are also enabled. Data is stored in the output cache 1560 for transmission to the computation engine or other desired location. This process can be repeated for the entire vector multiplication. In weight update mode, weight update signals can be generated sequentially by the weight update calculator 1546. In parallel, cells in the rows of the CIM module 1530 are read row by row and passed to the adder 1544 for the corresponding weight updates.

[0104] Using the computation engine 1500 improves the efficiency and performance of the learning network. The CIM module 1530 significantly reduces the time required to perform vector-matrix multiplications. Therefore, performing one or more inference operations using the computation engine 1500 may require less time and power. This improves the efficiency of model training and use. The LU module 1540 performs local updates to the weights stored in the units of the CIM module 1530 using components 1542, 1544, and 1546. This reduces the data movement that might otherwise be required for weight updates. Therefore, the time spent on training can be significantly reduced. The efficiency and performance of the learning network using the computation engine 1500 can be improved.

[0105] Therefore, method 200 can be used in relation to the learning network provided by using one or more computer blocks 1100, 1200 and / or 1300 and computing engines 1400 and / or 1500. Thus, in addition to improved training, the learning network may also have improved performance.

[0106] While the foregoing embodiments have been described in considerable detail for purposes of clarity, the invention is not limited to the details provided. Many alternative embodiments of the invention exist. The disclosed embodiments are illustrative and not restrictive.

Claims

1. A method comprising: The model of a learning network with multiple layers and associated memory is dissected, wherein the multiple layers include multiple weight layers and multiple activation layers, and the multiple weight layers include multiple weights; as well as Based on the profiling, a training technique for the model on the learning network is determined, the determination including optimizing the latency of at least one training iteration for the capacity of the associated memory, the at least one training iteration including at least one update of the plurality of weights of the plurality of weight layers.

2. The method of claim 1, wherein the profiling further comprises: Specific training techniques are used to perform training iterations for the model.

3. The method of claim 2, wherein the specific training technique includes at least one of backpropagation, activation recalculation, partial activation recalculation, or feedback alignment.

4. The method of claim 3, wherein performing the training iteration further comprises: Use training to activate inputs to perform inference; The loss function is determined based on the reasoning described above; The loss function is used to determine multiple updates to the multiple weights; as well as Update the multiple weights.

5. The method of claim 4, wherein using the loss function to determine the plurality of updates further comprises: The loss function is backpropagated through the multiple layers.

6. The method of claim 2, wherein the capacity of the associated memory is less than the memory required for the training iteration, and performing the training iteration further comprises: Load the activation of the weight layer from the plurality of weight layers from remote memory; The output of the weighted layer is stored in the remote memory; as well as The loading and storing process is repeated for each subsequent weight layer among the plurality of weight layers.

7. The method of claim 1, wherein the profiling further comprises: Determine multiple descriptors for the model; as well as Based on the multiple descriptors, at least a portion of the training techniques are selected from a library that includes various training techniques.

8. The method of claim 7, wherein the plurality of descriptors includes at least one of batch size, number of channels, width of input activation, height of input activation, stride, padding size, kernel size, pooling size, or dilation rate.

9. A system comprising: Memory; as well as A processor, which is coupled to the memory and configured to: The model of a learning network with multiple layers and associated memory is dissected, wherein the multiple layers include multiple weight layers and multiple activation layers, and the multiple weight layers include multiple weights; as well as Based on the profiling, a training technique for the model on the learning network is determined to determine that the processor is further configured to optimize the latency of at least one training iteration for the capacity of the associated memory, the at least one training iteration including at least one update of the plurality of weights of the plurality of weight layers.

10. The system of claim 9, wherein the processor is configured to further include the processor being configured to: The administrator performs training iterations on the model using specific training techniques.

11. The system of claim 10, wherein the specific training technique includes at least one of backpropagation, activation recalculation, partial activation recalculation, or feedback alignment.

12. The system of claim 11, wherein performing the training iteration further includes the learning network: Use training to activate inputs to perform inference; The loss function is determined based on the reasoning described above; The loss function is used to determine multiple updates to the plurality of weights; and Update the multiple weights.

13. The system of claim 12, wherein using the loss function to determine the plurality of updates further comprises: The loss function is backpropagated through the multiple layers.

14. The system of claim 10, wherein the capacity of the associated memory is less than the memory required for the training iteration, and performing the training iteration further comprises: Load the activation of the weight layer from the plurality of weight layers from remote memory; The output of the weighted layer is stored in the remote memory; as well as The loading and storing process is repeated for each subsequent weight layer among the plurality of weight layers.

15. The system of claim 9, wherein the processor is configured to profile further comprises the processor being configured to: Determine multiple descriptors for the model; and Based on the multiple descriptors, at least a portion of the training techniques are selected from a library that includes various training techniques.

16. The system of claim 15, wherein the plurality of descriptors includes at least one of batch size, number of channels, width of input activation, height of input activation, stride, padding size, kernel size, pooling size, or dilation rate.

17. The system of claim 9, wherein the learning network comprises at least one block, each of the at least one block comprising: Multiple computing engines, each of which includes an in-memory computing (CIM) hardware module, the CIM hardware module storing a portion of the multiple weights corresponding to the matrix and configured to perform vector-matrix multiplication (VMM) on the matrix, the multiple computing engines corresponding to the multiple weight layers; as well as A general-purpose (GP) processor, coupled to the plurality of computing engines and configured to provide control instructions and data to the plurality of computing engines, the general-purpose processor corresponding to the plurality of activation layers.

18. The system of claim 17, wherein the GP processor is coupled to the plurality of computing engines via a streaming port and a control port, the control port being configured to provide the control instructions to the plurality of computing engines, and the streaming port being configured to exchange data between the GP processor and the plurality of computing engines.

19. The system of claim 17, wherein the CIM hardware module includes a plurality of storage units for storing the plurality of weights.

20. A computer program product contained in a non-transitory computer-readable medium and comprising computer instructions for performing the following operations: A model of a learning network with multiple layers and associated memory is analyzed, wherein the multiple layers include multiple weight layers and multiple activation layers, and the multiple weight layers include multiple weights; and Based on the profiling, a training technique for the model on the learning network is determined, the determination including optimizing the latency of at least one training iteration for the capacity of the associated memory, the at least one training iteration including at least one update of the plurality of weights of the plurality of weight layers.

Images