Executing large artificial intelligence models on memory constrained devices

Abstract

The methods, systems, devices, and computer program products described herein support execution of large AI models on memory-constrained target devices that are communicatively connected to a parameter server that stores a master copy of the AI model. The AI model can be decomposed into smaller parts (e.g., layers or sub-layers), and each part can be executed on the target device as efficiently as possible. After execution of a part of the AI model is complete, another part of the AI model can be downloaded and executed at the target device. To improve efficiency, input samples can be divided into micro-batches, and multiple micro-batches that are executed in sequence can form a mini-batch. The size of a group of micro-batches or mini-batches can be adjusted manually or automatically to reduce communication overhead.

Description

Background Technology

[0001] Artificial intelligence has already had a profound impact on many aspects of modern society. Machine learning (a subset of artificial intelligence that uses mathematical algorithms to process large datasets) is gaining popularity in commercial applications and is increasingly appearing in consumer products. Deep learning, a branch of machine learning, is based on algorithms used to model high-level abstractions in data. Many applications of artificial intelligence are driven by deep learning, such as natural language processing, speech recognition, and image analysis.

[0002] However, many challenges hinder the widespread adoption of deep learning. These challenges include the complexity of managing large datasets and the significant time and resources required to train deep learning networks. For example, speech recognition programs may require data from multiple dialects and demographic data, potentially including terabytes of data for a single language. The complexity of a deep neural network (DNN) can be represented by the number of parameters, making it more complex the more parameters it contains. Furthermore, optimizing hyperparameters (parameters defined before the learning process of an artificial intelligence (AI) model begins) can drastically impact the performance of the AI ​​model. Moreover, substantial computing power is required to process the massive amounts of data used to train such AI models.

[0003] In deep learning, certain types of AI models may require the processing power of GPUs (Graphics Processing Units) with high memory capacity. To improve throughput, multiple GPUs can run in data parallelism, which typically requires synchronizing hundreds of millions to billions of parameters stored separately on different GPUs. This approach may be limited by the memory capacity of the GPUs and may not achieve the GPUs' maximum computational efficiency. Summary of the Invention

[0004] This summary is provided to introduce, in a simplified form, the selection of concepts further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0005] This paper describes methods, systems, apparatuses, and computer program products that enable the execution of arbitrarily large AI models on memory-constrained target devices communicatively connected to a parameter server.

[0006] Specifically, this paper describes a method, system, apparatus, and computer program product comprising a target device communicatively connected to a parameter server, the target device being configured to: download a portion of an AI model from the parameter server to the target device's memory; store a set of micro-batches of the dataset in the target device's memory; execute the micro-batches on a first sub-part of the downloaded portion of the AI ​​model to generate activations; and download the weights of a second sub-part of the downloaded portion of the AI ​​model from the parameter server to the target device's memory. The target device is also configured to execute the micro-batches on a third sub-part of the downloaded portion of the AI ​​model to generate gradients; download the weights and activations of a fourth sub-part of the downloaded portion of the AI ​​model; and simultaneously, execute the micro-batches on the fourth sub-part using the downloaded weights and activations, download the weights and activations of a fifth sub-part of the downloaded portion of the AI ​​model from the parameter server, and send the gradients of the third sub-part to the parameter server.

[0007] This document also describes methods, systems, apparatus, and computer program products including a parameter server communicatively connected to a target device, the parameter server being configured to: store a master copy of an AI model; determine a micro-batch size suitable for the target device; and transmit a portion of the AI ​​model to the target device.

[0008] Other features and advantages, as well as the structure and operation of various examples, are described in detail below with reference to the accompanying drawings. Note that the concepts and techniques are not limited to the specific examples described herein. Such examples are presented herein for illustrative purposes only. Additional examples will be readily apparent to those skilled in the art based on the teachings contained herein. Attached Figure Description

[0009] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments of the present application and, together with the description, further serve to explain the principles of the embodiments and enable those skilled in the art to make and use the embodiments.

[0010] Figure 1 This is a block diagram of a system that supports the execution of arbitrarily large AI models on memory-constrained target devices, according to an example embodiment.

[0011] Figure 2 A flowchart illustrating a process for running an AI model on a memory-constrained device during forward propagation, according to an example embodiment, is shown.

[0012] Figure 3 A flowchart is shown illustrating a process for running an AI model on a memory-constrained device during a reverse pass, according to an example embodiment.

[0013] Figure 4A table representing a forward pass through a machine learning model, according to an example embodiment, is shown.

[0014] Figure 5 A table representing a machine learning model is shown, according to an example embodiment.

[0015] Figure 6 A flowchart illustrating the process of running an AI model on a memory-constrained device at a parameter server according to an example embodiment is shown.

[0016] Figure 7 A flowchart illustrating the process of generating activation during forward propagation at the parameter server according to an example embodiment is shown.

[0017] Figure 8 A flowchart illustrating the process of updating an AI model at a parameter server according to an example embodiment is shown.

[0018] Figure 9 A block diagram illustrating parameter reduction in a multi-stage parallel system according to an example embodiment is shown.

[0019] Figure 10 A timing diagram of parameter reduction in multi-stage parallelism according to an example embodiment is shown.

[0020] Figure 11 A flowchart illustrating the parallel parameter reduction process in a provision system according to an example embodiment is shown.

[0021] Figure 12 A flowchart illustrating the process of providing mixed-precision training of an AI model according to an example embodiment is shown.

[0022] Figure 13 A flowchart illustrating a process for training an AI model using multiple target devices, according to an example embodiment, is shown.

[0023] Figure 14 A flowchart illustrating a process for dynamically executing AI modeling, according to an example embodiment, is shown.

[0024] Figure 15 A flowchart is shown illustrating a process for determining the computational accuracy of dynamic execution used for AI modeling, according to an example embodiment.

[0025] Figure 16 A flowchart is shown illustrating the process of determining whether to stop or continue the execution of an AI model based on the accuracy of the AI ​​model, according to an example embodiment.

[0026] Figure 17 This is a block diagram of an example computer system in which embodiments can be implemented.

[0027] The features and advantages of the embodiments will become more apparent from the specific embodiments described below in conjunction with the accompanying drawings, in which similar reference numerals identify corresponding elements throughout. In the drawings, similar reference numerals generally denote identical, functionally similar, and / or structurally similar elements. The leftmost (or plurality of) numerals of the corresponding reference numerals indicate the first appearance of the element in the drawing. Detailed Implementation

[0028] I. Introduction

[0029] The following detailed description discloses many embodiments. The scope of this patent application is not limited to the disclosed embodiments, but also includes combinations of the disclosed embodiments and modifications to the disclosed embodiments.

[0030] References to "an embodiment," "an embodiment," "an exemplary embodiment," etc., in the specification indicate that the described embodiment may include specific features, structures, or characteristics, but each embodiment may not necessarily include specific features, structures, or characteristics. Furthermore, such phrases do not necessarily refer to the same embodiment. Additionally, when a feature, structure, or characteristic is described in connection with an embodiment, it is believed that combining it with other embodiments (whether explicitly described or not) to achieve such a feature, structure, or characteristic is within the knowledge of those skilled in the art.

[0031] In this discussion, unless otherwise stated, adjectives such as “substantially,” “approximately,” and “about” that modify one or more features of embodiments of this disclosure are understood to mean that the condition or feature is defined within an acceptable tolerance for operation of the embodiments for their intended application.

[0032] Many exemplary embodiments are described below. Note that any section / subsection headings provided herein are not intended to be limiting. Embodiments are described throughout this document, and any type of embodiment may be included under any section / subsection. Furthermore, embodiments disclosed in any section / subsection may be combined in any way with any other embodiments described in the same or different sections / subsections.

[0033] II. Example Implementation

[0034] The exemplary embodiments described herein are provided for illustrative purposes and are not limiting. The examples described herein can be applied to any type of target crawling system. Further structural and operational embodiments (including modifications / changes) will become apparent to those skilled in the art from the teachings herein.

[0035] Deep learning has many applications, including natural language processing, speech recognition, image analysis, machine translation, object classification and detection in photographs, automatic handwriting generation, automated games, and generative chatbots. Deep learning models are widely used in various tasks because of their ability to mimic the human brain.

[0036] In deep learning, large AI models (e.g., those trained for natural language processing or image analysis) may require multiple GPUs with high memory capacity to perform their training. To improve speed, these GPUs can have high-speed interfaces, such as high-bandwidth memory (HBM) interfaces. However, even with high-quality hardware, there are ways to improve the inference and training process of large AI models. For example, there are two methods to parallelize the training of AI models to improve throughput: model parallelism and data parallelism.

[0037] Model parallelism involves dividing the learning model into multiple parts and placing these parts on different computing nodes (e.g., placing the first half of a layer on a first GPU and the second half on a second GPU, or splitting intermediate layers and allocating them to separate GPUs). For example, a typical large AI model with 24 layers can be run on a GPU in the following way: Forward passes are executed layer by layer on the same mini-batch, such as by starting with a mini-batch on layer 1, then layer 2, and so on up to layer 24. After each layer, the activations of that layer (also referred to here as hidden activations, hidden states, or intermediate results) can be saved (e.g., on-chip or off-chip) for backward passes, which can be executed in a similar layer-by-layer manner (in reverse order) on the same mini-batch. For example, a mini-batch could be executed on layer 24, then layer 23, and so on up to layer 1, after which the AI ​​model is updated. Sometimes, as a trade-off between computational cost and efficient memory usage, hidden activations can be recomputed in backward passes. In some types of AI models (e.g., natural language processing), there may be many parameters, but the mini-batch size may be small (e.g., a few gigabytes). In other types of models (e.g., dense networks or computer vision models), the number of parameters may be relatively small, but the number of hidden activations may be large. Typically, these types of models may not be able to run on devices without global memory, such as application-specific integrated circuit (ASIC) devices. Therefore, the available technique used is model parallelism, where the model is split across multiple devices. However, model parallelism is inefficient due to long periods of dormant memory and computation time.

[0038] Furthermore, GPUs can have certain data structures mapped to their global memory, which is off-chip and connected to a high-speed memory interface (e.g., HBM). For example, input and output activations may reside on-chip, and sometimes gradients as well, while master copies of weights and hidden activations may be stored off-chip. Several residency issues arise with these data structures. Weights may be loaded before actual use, thus consuming valuable memory. Hidden activations may be generated after the forward pass is complete, but may not be needed before the backward pass. Additionally, global memory data is moved in and out of the chip via loading and storing, leading to memory access amplification even with limited time buffering through caches and registers. Therefore, when running large AI models (e.g., using model parallelism or unified memory addressing techniques) on such GPUs or GPU clusters, the AI ​​model size depends on the number of devices, and the performance penalty of communication overhead cannot be scaled or hidden.

[0039] Data parallelism involves partitioning the input data across computing devices, with each device holding a complete copy of the learned model, known as a complex or worker. Each complex computes the gradient of its portion of data, and these gradients are combined to update the model parameters. In asynchronous distributed stochastic gradient descent (SGD), each complex accesses a shared memory space where global parameters are stored. After copying the parameters in its local memory, a complex can compute the gradient with respect to its current weights and the updated weights, then apply the updated weights to the global parameters in the shared memory space. The advantage of this configuration is that complexes can work at their own pace without waiting for other complexes to finish computed their gradients. However, there is no way to ensure that when one complex computes its gradient with respect to a set of parameters, another complex has not updated its global parameters, resulting in the global parameters being updated with stale gradients. In synchronous distributed SGD, each GPU may run mini-batches of the input data (or, samples), then stop execution to synchronize all model parameters by exchanging gradients that are tuned by the backpropagation loss of the AI ​​model. This method is highly limited by the memory capacity of the GPUs. When an AI model requires more memory than a single GPU, model compilation on that GPU may fail due to an out-of-memory error. Data parallelism typically requires synchronizing hundreds of millions to billions of parameters stored separately on different GPUs. Therefore, this approach may not achieve the maximum computational efficiency of GPUs because they require long pauses in computation to complete synchronization.

[0040] The embodiments described herein overcome the difficulty of enabling the execution of AI models on devices with large on-chip memory but no global memory. The embodiments described herein can execute AI models of any size in a fast and efficient manner on memory-constrained devices such as GPUs, ASICs, or FPGAs (Field-Programmable Gate Arrays). In example embodiments, any AI model of any size can be executed on an ASIC that lacks global memory but can execute AI models faster than a GPU. Therefore, the embodiments described herein support the execution of large AI models on memory-constrained devices.

[0041] Example implementations can be implemented in a system with at least one parameter server and one target device. A master copy of the AI ​​model can reside in the parameter server. The AI ​​model can be decomposed into smaller parts or blocks (e.g., individual layers), and each part or layer can be executed as efficiently as possible on the target device. After one layer is completed, the next layer is executed. To improve balance and efficiency, the technique iterates on the same layer with a large number of input samples until (a) the next layer is loaded onto the target device, thus completely hiding its latency, or (b) the next layer is loaded after the current layer is completed, exposing its latency, but minimizing overhead through the long computation cycle of the current layer. To lengthen the current computation cycle, the input samples can be divided into micro-batches. A set of micro-batches forms a mini-batch, which is a term for a number of samples used for each update (for training) or a number of samples provided (for inference) in each inference cycle. By using a set of micro-batches and / or mini-batches as a knob that can be adjusted manually or automatically (e.g., using software using an AI framework), communication overhead can be minimized or even reduced to zero.

[0042] If an AI model can be optimized with a large batch size, such as in the case of natural language processing models, vision models, or models with a high weight / activation ratio, the embodiments described herein will allow these models to run at peak performance on one or more memory-constrained devices. Therefore, according to example embodiments, large AI models can be executed on target devices with less memory than is required to efficiently run a large AI model. In other words, AI models can be executed with the smallest possible device batch size, at which point peak efficiency (i.e., efficient TFLOPs) can be achieved. For example, the performance of an AI model may depend solely on the efficiency of the computational throughput of the library running on the target device, i.e., TFLOPs (teraFLOPS). Floating-point operations per second (FLOPS) is a metric for computer performance, for example, measuring the ability of an algorithm or computer hardware to perform one trillion floating-point operations per second. In other example embodiments, large AI models can be executed on multiple target devices whose combined memory (e.g., global memory) may be less than the memory required to efficiently run a large AI model.

[0043] A. Executing large-scale artificial intelligence models on memory-constrained target devices

[0044] Supporting the execution of large AI models on memory-constrained devices can be achieved in several ways. For example, Figure 1 This is a block diagram of a system, according to an example embodiment, that supports the execution of arbitrarily large AI models on memory-constrained target devices. Figure 1 As shown, system 100 includes parameter server 102 and target devices 134a-134k. Although Figure 1 Only one parameter server 102 is shown, but system 100 may include multiple parameter servers. Similarly, although Figure 1 The target devices 134a-134k are depicted, but system 100 may include fewer or more target devices. Based on information regarding... Figure 1 The following discussion of the system 100 described herein will reveal other structural and operational embodiments to those skilled in the art.

[0045] Parameter server 102 may include any type of computing device (mobile or stationary). Parameter server 102 can provide functionality to other programs or devices, such as sharing data or resources or performing computations. Parameter server 102 may include memory 104 configured to store data (e.g., datasets, software programs, AI models) and processor 132 configured to perform programming functions. Parameter server 102 may include commercial off-the-shelf components and / or custom components, and may be a standalone device or part of another computing device. Parameter server 102 may include... Figure 1Other components not shown include peripheral interfaces, communication interfaces, integrated devices, multiprocessors, and different types of memory. In embodiments, parameter server 102 can be implemented as one or more of a CPU, FPGA, or ASIC. For example, as a CPU, parameter server 102 may include an electronic circuitry system within a computing device that executes instructions of a computer program by performing operations (e.g., mathematical, logical, control, or input / output) based on instructions. As an FPGA, parameter server 102 may include an array of programmable logic blocks configured to perform complex combinational functions or other operations. As an ASIC or system-on-a-chip, parameter server 102 may include a custom integrated circuit configured to perform operations based on computer program instructions.

[0046] Parameter server 102 can be configured to store AI model 106 in memory 104. AI model 106 may include weights 108, and activations 112 and gradients 110 may be stored in memory 104 during the execution of AI model 106. Parameter server 102 may also store dataset 114, which may be a training or testing dataset. Parameter server 102 may also include computer program logic (e.g., computer program code or instructions) for performing operations. For example, parameter server 102 may include AI model manager 116, which is configured to manage AI model 106 during inference or training. AI model manager 116 includes computer program logic for managing AI model 106, such as data manager 118, batch manager 120, transmitter 122, and output data manager 124. Output data manager 124 is configured to receive and manage output data from target devices 134a-134k, as well as other data, for managing AI model 106. The output data manager 124 includes a weight updater 126 configured to update the weights 108 of the AI ​​model 106, a precision formatter 128 configured to manage precision (e.g., mixed precision training, precision conversion, etc.) formats, and a model evaluator 132 configured to evaluate the AI ​​model 106 and manage the execution of the AI ​​model 106 accordingly. In an example embodiment, the AI ​​model manager 116 may include a weight updater 126. Figure 1 The AI ​​model manager 116 may have fewer or more components as shown. In other embodiments, the functions of the components may overlap.

[0047] Parameter server 102 can serve one or more target devices 134a-134k. Parameter server 102 can be communicatively connected to target devices 134a-134k via a suitable interface (such as Peripheral Component Interconnect (PCI) or PCI Express (PCIe)) and / or a network (e.g., for cloud computing or edge computing). In an example embodiment, parameter server 102 and one or more target devices 134a-134k can reside on the same chip or can reside on different chips or different devices. In an example embodiment, parameter server 102 and target devices 134a-134k can include software for communicating with each other. For example, parameter server 102 can include driver software specifically designed for communication, such as sending commands (e.g., initiating function calls to target devices 134a-134k) and receiving responses.

[0048] The target devices 134a-134k can each include Figure 1 An example of the features shown for target device 134a is provided. Target device 134a includes a data interface 136, a processor 140, and a memory 142. Target devices 134a-134k may each include multiple processors and different types of interfaces and memory, even if... Figure 1A single data interface is described for target device 134a. In example embodiments, target devices 134a-134k have the same hardware specifications, such as the same memory size. In other example embodiments, target devices 134a-134k may have different hardware specifications. Target devices 134a-134k may be specifically designed to perform computationally intensive operations in an accelerated manner. For example, target devices may include high computational density, fixed-function processors (e.g., processor 140), and other general-purpose capabilities. Target devices 134a-134k may be managed by parameter server 102 to run specific computations. For example, parameter server 102 may execute a main program that prepares input data for processing at target devices 134a-134k, invokes parallel routines (e.g., cores) at target devices 134a-134k, and receives results after the routines terminate. The parameter server 102 may also utilize a high-level computing language, computing platform, or framework that includes a comprehensive library of acceleration algorithms and data structures to make it easier to accelerate computations on target devices 134a-134k. In an example embodiment, target devices 134a-134k may be implemented as a GPU (e.g., a dedicated or general-purpose GPU), ASIC, FPGA, or edge device (e.g., a standalone device or microcontroller residing at the end or edge of a network connection and potentially having a small memory footprint). In an embodiment, target devices 134a-134k may be memory-constrained devices, although this is not necessarily the case in all cases. For example, target devices 134a-134k may be memory-constrained individually or in one or more groups, making them unable to efficiently run large AI models due to insufficient memory. For illustrative purposes, Figure 1 The characteristics of target device 134a are described below, as representatives of each target device in target devices 134a-134k.

[0049] Data interface 136 can be configured to interface target device 134a with parameter server 102 and other devices, including other target devices. For example, data interface 136 may include PCI, PCIe, and / or HBM. Processor 140 is configured to perform operations requested by parameter server 102 and operations specifically for target device 134a. Memory 142 is configured to store data and computer program logic; for example, memory 142 includes accelerator 144, which is configured to perform functions and / or accelerate certain operations, for example, as indicated by parameter server 102. Accelerator 144 includes a data downloader 146 configured to download data (e.g., a model and / or its data, such as weights, activations, and datasets), a data manager 148 configured to store or otherwise manage the downloaded data, a layer executor 150 configured to execute an AI model or a portion thereof (i.e., execute a dataset on an AI model or a portion thereof), and an output manager 152 configured to manage output data (e.g., gradients and activations) generated from model execution, for example, by saving, sending, or restoring output data. In an example embodiment, accelerator 144 may include a... Figure 1 The accelerator 144 may have fewer or more components as shown. In other embodiments, the functionality of the components of the accelerator 144 may overlap.

[0050] Combination Figure 2 Further operational aspects of parameter server 102 and target device 134a are described below. Figure 2 A flowchart 200 illustrating a process for running an AI model on a memory-constrained device during forward propagation, according to an example embodiment, is shown. Although referenced... Figure 1 The system 100 is described, but Figure 2 The process is not limited to this system. Based on Figure 2 Flowchart 200 and Figure 1 The following discussion of System 100 will provide other structural and operational embodiments that will be of interest to those skilled in the art.

[0051] Flowchart 200 begins at step 202. In step 202, a portion of the artificial intelligence (AI) model is downloaded from a parameter server to the target device's memory; the parameter server stores a master copy of the AI ​​model. For example, Figure 1Target device 134a, specifically data downloader 146, can be configured to download a portion of AI model 106 from parameter server 102 to memory 142 of target device 134a. In an example embodiment, one or more target devices 134a-134k can form a group configured to run instances of the AI ​​model. For example, one group may include target device 134a, while another group may include target devices 134b-134k. The memory required by the AI ​​model has both invariant and variable requirements. For example, the size of weights or parameters may be invariant for a particular precision type (e.g., 32-bit), and variable requirements may depend on the batch size of the target devices. In an example embodiment, target device 134a (or a group of target devices 134a-134k) can be configured to run a large AI model, and the memory of target device 134a (or the combined or global memory of the group of target devices 134a-134k) may be smaller than the memory required for efficiently running the AI ​​model. Efficient AI operation means that the AI ​​model runs at the minimum device batch size, thereby achieving peak efficiency (i.e., effective TLOP). In other words, the AI ​​model does not benefit from a larger device batch size (there is no more efficient TFLOP). For example, target device 134a may have memory smaller than the overall size of the AI ​​model for a particular optimal batch size. The optimal batch size for the target device is the batch size that achieves the desired accuracy (e.g., 85%) at the highest possible throughput (e.g., the rate at which the dataset is processed). The optimal group batch size is the global batch size (e.g., the batch size of a group of target devices) divided by the number of groups of target devices communicatively connected to parameter server 102.

[0052] AI model 106 can include any type of machine learning model, which can have a variety of applications in many fields, such as natural language processing, autonomous vehicles, image processing, deep learning robots, automatic machine translation, and automatic handwriting generation. AI model 106 can have any type of deep learning architecture, such as deep neural networks, recurrent neural networks, and convolutional neural networks.

[0053] A simple neural network may consist of several layers, one for receiving input signals and another for sending output signals. One or more hidden layers or processing layers may be located between the input and output layers. In a DNN constructed to generate one or more inferences, there may be many hidden layers composed of artificial neurons. Such neurons may include an activation function, constant inputs, other inputs, and an output. The neuron can produce an output by performing an activation function on a weighted version of the input. The inputs to the activation function are weighted according to their respective weights. For example, the inputs may include normalized data. The activation function may be configured to accept a single digit (e.g., a linear combination of weighted inputs) based on all inputs and perform a fixed operation, such as sigmoid, tanh, or rectified linear unit options. Constant inputs may be constant values.

[0054] A single neuron may not do much work on its own, and useful AI models typically involve the combined computational work of a large number of neurons working together. For example, a DNN can include multiple neurons assembled in layers and connected in a cascaded manner. These layers can include an input layer, an output layer, and some hidden layers in between. The output of each neuron in each layer can be weighted according to certain weights and then used as input to neurons in the next layer. Other interconnection strategies known in the art can be employed. The neurons in the input layer can be configured to accept normalized or otherwise feature-engineered or processed data corresponding to user data. The output of each neuron in the input or hidden layer can be weighted according to the weights of its corresponding output edge and then used as input to each neuron in the next layer. The output(s) of the output layer include the output of the DNN or AI model. In the context of inference, such output can be (multiple) inferences or (multiple) predictions. Building such a DNN is only the beginning of generating useful machine learning or AI models. The accuracy of the inference generated by such an AI model requires the selection of a suitable activation function, and then each weight of the entire model is tuned to provide accurate output. The process of tuning such weights is called "training". Training a DNN or other type of network requires a set of training data with known properties. For example, if a DNN aims to predict the probability that an input image of an animal is a cat, the training data will include images of many different cats, and typically not only cat images but also images of other similar animals. Training requires preprocessing the image data corresponding to each image according to normalization and / or feature extraction techniques known in the art to generate input features for the DNN, which are then provided as input to the network, for example, as input to neurons in the input layer.

[0055] Subsequently, each neuron in a layer performs its corresponding activation operation, the output of which is weighted and fed forward in the forward pass to the next layer until the output layer generates the DNN(s). The DNN(s) output(s) can be compared with known or expected values ​​of the output, and the differences can be fed back in the backward pass through the DNN to adjust the weights contained therein according to backpropagation algorithms known in the art. Image features can be input into the AI ​​model again with updated weights, and new outputs are generated. Training involves iterating the AI ​​model on the training dataset and updating the weights at each iteration. The AI ​​model is said to be trained once it reaches sufficient accuracy, or its output has converged and changes in the weights have little effect. The trained model can then be used to evaluate arbitrary input data whose nature is unknown beforehand and which the model has not previously considered (e.g., a new picture of an animal) and output the expected inference (e.g., the probability that the image is a picture of a cat).

[0056] Gradient descent is an algorithm frequently used in training AI models. Gradient descent involves an objective function (e.g., a loss function or cost function, which can be many), and the goal is to minimize this function. The objective function is used to monitor the error in the AI ​​model's predictions. Therefore, by minimizing this function, the lowest error value can be found, thus improving the accuracy of the AI ​​model. Stochastic gradient descent (SGD) is a variant of the gradient descent algorithm that computes the error and updates the model for each sample in the training dataset. SGD updates frequently and learns faster, but it is computationally expensive and may require longer training times on large datasets. Batch SGD is another variant that computes the error for each sample in the training dataset, but updates the AI ​​model only after processing the entire dataset (i.e., at the end of the training phase). Batch SGD updates less frequently and is more computationally efficient than SGD. The separation of prediction error computation and model update in batch SGD makes the algorithm suitable for parallel processing-based implementations, but the update at the end of the training epoch requires additional complexity to accumulate the prediction error across the entire dataset and is typically implemented in a way that requires the entire training dataset in memory and is available for the algorithm. Mini-batch SGD is another variant of SGD that splits the training dataset into mini-batches, which are used to compute model errors and update parameters. Implementations can sum gradients over mini-batches, further reducing gradient variance. Therefore, mini-batch SGD strikes a balance between SGD and batch SGD. Mini-batch SGD requires an additional "mini-batch size" hyperparameter configured for the learning algorithm. Error information can accumulate in the training examples on mini-batches. The mini-batch size can be configured to be a power of 2 suitable for the memory requirements of the target device or accelerator hardware, such as 32, 64, 128, 256, etc. The batch size can act as a tuning factor for the learning process, where smaller values ​​result in a fast convergence at the cost of noise during training, while larger values ​​result in a slow convergence with accurate estimates of error gradients.

[0057] Return to reference Figure 2In step 202, the downloaded portion of AI model 106 may include any part of AI model 106. AI model 106 may be decomposed into layers, composite layers, or composite fractional layers (e.g., subdivided into 1.5x layers). The AI ​​model is neatly divided at the layers, so these models can be partitioned as a whole. In one example embodiment, the downloaded portion may include one or more layers of AI model 106. However, in another example embodiment, fractional partitioning is possible depending on the AI ​​model and other factors. One reason that fractional partitioning may be necessary is that such fractional parts may fit within the target device, while entire layers may not. This would support an implementation where any number of layers can be run without memory outage issues, rather than encountering memory outage errors after a certain number of layers. In an example embodiment, the portion of AI model 106 downloaded to target device 134a may include any part of AI model 106 up to the entire AI model 106. Target device 134a may download portions of AI model 106 in various ways. For example, target device 134a can download the next part of AI model 106 to one or more memory buffers while executing the current part of AI model 106. This method can use slightly more memory and special libraries, but can produce a higher-performance AI model 106. In another example, target device 134a can execute the current sub-part, synchronize, and then download the next sub-part. This method may be slightly slower, but does not require buffering. Flowchart 200 continues to step 204.

[0058] In step 204, the micro-batch set of the dataset is stored in the memory of the target device. For example, as Figure 1As shown, a set of micro-batches of dataset 114 can be downloaded from parameter server 102 via data downloader 146. Data manager 148 can then store the set of micro-batches in memory 142, buffer, or any other known memory structure of target device 134a. Dataset 114 can be user input data, such as a training dataset for training, a test dataset for testing purposes, or arbitrary input data for inference. The set of micro-batches includes multiple micro-batches configured to be executed sequentially at target device 134a. The set of micro-batches forms mini-batches, which include a number of samples updated for training AI model 106 or a number of samples provided for inference in each inference cycle. Each micro-batch in the set of micro-batches can have a micro-batch size that can be configured automatically or manually. In an embodiment, the micro-batch size can be selected based on the execution rate of the multiple micro-batches and the communication rate between target device 134a and parameter server 102. For example, the micro-batch size can be initially selected for target device 134a based on its hardware specifications, and then adjusted as needed during iterations to adequately hide communication latency. In an embodiment, the optimal micro-batch size can be a trade-off between the required memory and the percentage of communication overhead that can be hidden. Because more communication overhead is hidden, computation may require more memory. Therefore, the micro-batch size may be large enough to fully utilize the execution of layers in the target device, but small enough to fit within the target device's memory.

[0059] Flowchart 200 continues to step 206, which executes a set of micro-batches on a first sub-part of the downloaded portion of the AI ​​model to generate activations. For example, the set of micro-batches can be executed at target device 134a by layer executor 150 on a first sub-part of the portion of the AI ​​model 106 downloaded by data downloader 146. In an example embodiment where the downloaded portion of AI model 106 includes one or more layers, the set of micro-batches can be executed one layer at a time on one or more downloaded layers of AI model 106 to generate activations. Activations can be values ​​as intermediate results, such as the output of each micro-batch execution. Activations can be internal data required in backpropagation to determine how the weights 108 of AI model 106 should be adjusted. After each micro-batch is executed for a sub-part (e.g., a layer) of AI model 106, the activations can be saved on target device 134a, sent to parameter server 102 to save memory, or discarded to save memory and then recalculated. For example, if AI model 106 has 12 layers and each mini-batch has 8 micro-batches, activations can be stored 96 times during forward pass and restored 96 times during backward pass. If not all activations are saved during forward pass, they can be recalculated during backward pass. In the example embodiment, the storage of micro-batch activations can occur (whether at target device 134a or at parameter 102) while target device 134a is executing different micro-batches. In the example embodiment, the restoration or recalculation of activations can occur before or during the execution of a sub-part as needed; for example, the restoration / recalculation of activations for the next micro-batch can occur in parallel with the execution of the current micro-batch.

[0060] Flowchart 200 ends at step 208. In step 208, the weights of the second sub-part of the downloaded portion of the AI ​​model are downloaded from the parameter server to the memory of the target device. For example, if the downloaded portion of the AI ​​model includes multiple layers, the weights of the second layer can be downloaded to the memory 142 of the target device 134a via the data downloader 146. In an example embodiment, downloading the weights of the next layer can occur while the current layer is being executed. For example, the target device 134a can be configured to simultaneously execute a micro-batch set of the dataset on the second sub-part using the downloaded weights of the second sub-part, and download the weights of the third sub-part of the downloaded portion of the AI ​​model 106 from the parameter server 102 to the memory 142 of the target device 134a. For example, the layer executor 150 can execute a micro-batch set on a layer using the weights already downloaded for that layer, while the data downloader 146 downloads the weights 106 of the next layer of the AI ​​model. Alternatively, target device 134a can be configured to serially execute micro-batch sets on the second sub-part using the download weights of the second sub-part, and download the weights of the third sub-part of the download portion of AI model 106 from parameter server 102 to memory 142 of target device 134a. For example, layer executor 150 can execute micro-batch sets on a layer using weights already downloaded for that layer, and after the execution of that layer, data downloader 146 can download the weights for the next layer of AI model 106.

[0061] Therefore, the execution of AI model 106 continues at the target device 134a one sub-part at a time, as described above, while other sub-parts of AI model 106 can also be executed at other target devices. For example, in a forward pass, a set of micro-batches or mini-batches are executed at the first layer, then the second layer, and so on, until the last layer.

[0062] Once the forward pass of AI model 106 is complete, the reverse pass can be performed. For example, Figure 3 A flowchart 300 illustrating a process for running an AI model on a memory-constrained device during a reverse pass, according to an example embodiment, is shown. Although referenced... Figure 1 The system 100 is described, but Figure 3 The process is not limited to this system. Based on Figure 3 Flowchart 300 and Figure 1 The following discussion of System 100 will make other structural and operational embodiments readily apparent to those skilled in the art.

[0063] Flowchart 300 begins at step 302, where a micro-batch set is executed on the third sub-section of the download portion of the AI ​​model to generate gradients. For example, the micro-batch set can be executed by layer executor 150 on the third sub-section of the download portion of the AI ​​model 106 to generate gradients for that third sub-section. If the AI ​​model 106 has 24 layers, a micro-batch set can be executed on the 24th layer to generate gradients for that layer to begin backpropagation.

[0064] Flowchart 300 continues to step 304. In step 304, the weights and activations of the fourth sub-part of the downloaded portion of the AI ​​model are downloaded. For example, the weights and activations of the fourth sub-part of the downloaded portion of the AI ​​model 106 can be downloaded from parameter server 102 to target device 134a by data downloader 146. For example, if AI model 106 has 24 layers, the weights and activations from layer 23 can be downloaded from parameter server 102 to target device 134a.

[0065] In step 306, simultaneously, a micro-batch set is executed on the fourth sub-part using the downloaded weights and output activations, the weights and output activations of the fifth sub-part of the downloaded portion of the AI ​​model are downloaded from the parameter server, and the gradients of the third sub-part are sent to the parameters. For example, in parallel, simultaneously or substantially simultaneously, the layer executor 150 may use the downloaded weights and output activations of the sub-part to execute a micro-batch set on the fourth sub-part, the data downloader 146 may download the weights and output activations of the fifth sub-part of the downloaded portion of the AI ​​model 106 from the parameter server 102, and the output manager 152 may send the gradients of the third sub-part of the AI ​​model 106 to the parameter server 102. In an example embodiment where model 106 has 24 layers, the target device 134a may be configured to execute multiple steps in parallel or simultaneously. In this embodiment, the target device 134a can be configured to simultaneously execute layer 23 using the downloaded weights and output activations of layer 23, download the weights and output activations of layer 22 from parameter 102, and send the gradient 110 generated for layer 24 to parameter server 102.

[0066] The target device 134a is configured to continue the steps above in flowchart 300 so as to execute the entire dataset 114 in micro-batch on AI model 106 one sub-part (e.g., layer) at a time for the reverse pass (i.e., layer 24, layer 23, ... and layer 1).

[0067] like Figure 2 and Figure 3 The forward and backward propagation described in the text can be as follows: Figure 4 and Figure 5 Visualized as depicted in [the text]. For example, Figure 4Table 400, representing a forward pass through a machine learning model with 24 layers, is shown according to an example embodiment. Table 400 is for a target device that can be implemented as... Figure 1 The target device is 134a. Table 400 has three rows; row 410 shows the execution of the micro-batch set sequentially on each layer of the AI ​​model, where 10 of them form a mini-batch. Row 412 shows a set of actions of the target device (e.g., receiving weights from the parameter server), and row 414 shows another set of actions that the target device can take (e.g., sending activations to the parameter server). Data exchange at the target device can be accomplished via an interface, such as... Figure 4 The PCI is shown in Table 400. Although the AI ​​model has 24 layers, only layers 1, 2, and 24 are shown in detail in Table 400 because the execution of the AI ​​model at each layer is similar. For example, column 402 of Table 400 depicts the execution of a set of 10 micro-batches at layer 1, with the 10 micro-batches forming mini-batches. During this execution, the target device receives weights from layer 2 (the next layer to be executed). Since each micro-batch is executed at layer 1, the activation of that micro-batch can be saved if memory and / or other resources allow (e.g., at the target device or parameter server). Then, as shown in column 406 of Table 400, the same set of 10 micro-batches is executed at layer 2, while receiving weights from layer 3 and saving the activation of each micro-batch. This process continues for all layers of the AI ​​model until the last layer, namely layer 24, which may be referred to as the “decoding layer” (DL), “embedding layer”, or “output layer”. When a micro-batch set is executed at the last layer (layer 24), its weights and activations are determined at the target device and sent to the parameter server, as shown in column 408 of Table 400.

[0068] Figure 5 Table 500, illustrating a representation of the backpropagation via a machine learning model with 24 layers according to an example embodiment, is shown. Table 500 is for a target device that can be implemented as... Figure 1 The target device is 134a. Table 500 has four rows; row 510 shows the execution of the micro-batch set sequentially on each layer of the AI ​​model, where 16 of them form a mini-batch. Row 512 shows one set of actions on the target device (e.g., loading weights and activations from the parameter server), row 514 shows another set of actions (e.g., sending gradients to the parameter server), and row 516 shows yet another set of actions the target device can take (e.g., parameter reduction). Data exchange at the target device can be accomplished via an interface, such as... Figure 5The PCI is shown. Although the AI ​​model has 24 layers, only layers 24, 23, 22, and 1 are shown in detail in Table 500 because the execution of the AI ​​model on each layer is similar. For example, column 502 of Table 500 depicts a set of 16 micro-batches executed on layer 24, forming mini-batches. During this execution, the target device loads the weights and activations of layer 23 (the next layer to be executed in the backpropagation). Then, in column 504, the set of 16 micro-batches is executed on layer 23 using the loaded weights and activations. In parallel or simultaneously (or substantially simultaneously), the target device is configured to load the weights and activations of layer 22 (the next layer to be executed); send the gradients of the most recently executed layer 24 to the parameter server; and reduce the parameters of the AI ​​model. In column 506, the same set of 16 micro-batches is executed on layer 22 using the loaded weights and activations of that layer. During the execution of layer 22, the weights and activations of layer 21 are loaded, the gradients of layer 23 are sent to the parameter server, and the parameters are reduced at the target device. In column 508, the same set of 16 micro-batches is executed for layer 1. Simultaneously with this execution, the gradients of layer 2 are sent to the parameter server, and the parameters are reduced at the target device.

[0069] By running many micro-batches on top of the same layer, there is sufficient time to hide or cover the latency of preparing the next layer. Therefore, the total memory complexity of the target device can be two layers plus one layer of hidden activation and one layer of output activation.

[0070] For example, combining Figures 2-5 In the description, the target device is used as a supporting component for executing large AI models on memory-constrained devices. (See below for more details.) Figure 6-8 In the description, the parameter server can be used as a supporting component for executing large AI models on memory-constrained devices. For example, Figure 6 A flowchart 600 illustrating a process for running an AI model on a memory-constrained device at a parameter server according to an example embodiment is shown. Although referenced... Figure 1 The system 100 is described, but Figure 6 The process is not limited to this system. Based on Figure 6 Flowchart 600 and Figure 1 The following discussion of System 100 will make other structural and operational embodiments readily apparent to those skilled in the art.

[0071] Flowchart 600 begins at step 602. In step 602, a master copy of the artificial intelligence model is stored at a parameter server, which is communicatively connected to the target device. For example, as... Figure 1As shown, a primary copy of AI model 106 can be stored in memory 104 of parameter server 102 by data manager 118. Parameter server 102 can communicate with target devices 134a-134k via suitable means (such as PCI and PCIe interfaces or other network interfaces). In the example embodiment, parameter server 102 stores a complete copy of AI model 106, while target devices 134a-134k can store a portion of AI model 106 instead of the entire copy.

[0072] In step 604, a suitable micro-batch size for the target device is determined. For example, such as... Figure 1 As shown, target device 134a may be a memory-constrained device with a memory size smaller than the total size of AI model 106 stored at parameter server 102 for a specific optimal batch size. Batch manager 120 is configured to determine a suitable micro-batch size for target device 134a, for example, by taking into account the memory size and / or other hardware specifications of target device 134a. In one embodiment, batch manager 120 configures the micro-batch size for load balancing, such that the ratio of execution time to communication time is maximized. For example, the micro-batch size may depend on the computation time of the target device (C), the size of the sub-parts (S) to be transmitted, and the communication bandwidth (B) between the target device and the parameter server system. In this example, the micro-batch size may be determined as Minimum_numMicroBatches = S / B / C. This equation may be static, but in some cases (e.g., neural architecture search), the micro-batch size may be determined dynamically. Thus, when the ratio of execution time to communication time can be manipulated, the parameter server has more time to perform complex data parallelism or background tasks. In the example embodiment, the micro-batch size may be dynamically configured at certain times or boundary points during the training or inference process. For example, the micro-batch size can be dynamically configured at the end of the iteration, but it must be constant for mini-batch iterations in both forward and backward propagation.

[0073] return Figure 6Flowchart 600 ends at step 606. In step 606, a portion of the AI ​​model is transmitted to the target device. For example, transmitter 122 may transmit a portion of AI model 106 from parameter server 102 to target device 134a. AI model 106 may be divided into different parts in any number of ways. For example, a part may be a layer, a combination of layers, or a combination of fractional layers of AI model 106. The size of the part may be determined based on the memory available on target device 134a, such that the part has an optimal size for target device 134a. For example, AI model manager 116 may take into account the hardware specifications of target device 134a when determining the size of the part to be sent to target device 134a. Transmitter 122 may transmit a portion of AI model 106 to target device 134a while target device 134a is executing another part, thereby requiring target device 134a to buffer the part. Alternatively, transmitter 122 may transmit a portion of AI model 106 to target device 134a after target device 134a has completed the current part to avoid the need to buffer the part. In this alternative example, the target device 134a can perform synchronization after the execution of the current part, before receiving the part.

[0074] The parameter server 102, or more specifically, the AI ​​model manager 116, can perform additional steps to improve the throughput of distributed training and inference of AI models on memory-constrained devices. For example, Figure 7 A flowchart 700 is shown illustrating a process for generating activation during forward propagation at a parameter server according to an example embodiment. Although referenced... Figure 1 The system 100 is described, but Figure 7 The process is not limited to this system. Based on Figure 7 Flowchart 700 and Figure 1 The following discussion of System 100 will make other structural and operational embodiments readily apparent to those skilled in the art.

[0075] Figure 7The process begins at step 702, where activations are received from the target device after each micro-batch is executed. For example, after each micro-batch is executed on the target device, the output data manager 124 may receive activations from target devices 134a-134k. Activations may include hidden activations, intermediate results of micro-batch execution, or outputs of micro-batch execution at target device 134a. In an example embodiment, activations are received from target device 134a after each micro-batch. In this embodiment, saving and / or storing activations after each micro-batch can provide optimal efficiency for executing AI model 106. Therefore, when target device 134a is executing mini-batches comprising multiple micro-batches, activations for each micro-batch within the multiple micro-batches can be saved after each micro-batch execution within the multiple micro-batches. In another example embodiment, activations are saved at target device 134a. In yet another example embodiment, not all activations are saved in the forward pass; only the data required to recalculate activations in the backward pass is saved. Such data may include the input state of a specific sub-section of the AI ​​model, for example, because input states may require less memory space than output states. Therefore, the memory space of the target device 134a can be saved by not saving all activations during the forward pass.

[0076] Flowchart 700 ends at step 704, where an output activation is generated based on the received activations for a sub-part of the downloaded portion of the AI ​​model. For example, weight updater 126 may generate an output activation based on activations received from target devices 134a-134k for a sub-part of the downloaded portion of the AI ​​model 106. In an example embodiment, the generated output activation may be stored as activation 112 in memory 104 of parameter server 102. In an example where the sub-part includes a layer, the output activation of that layer may be generated by weight updater 126 based on a hidden activation received after each micro-batch is executed at target devices 134a-134k.

[0077] Parameter server 102 can perform additional steps to improve the throughput of distributed training and inference of AI models on memory-constrained devices. For example, Figure 8 A flowchart 800 illustrating a process for updating an AI model at a parameter server according to an example embodiment is shown. (Although references are not included.) Figure 1 The system 100 is described, but Figure 8 The process is not limited to this system. Based on Figure 8 Flowchart 800 and Figure 1 The following discussion of System 100 will make other structural and operational embodiments readily apparent to those skilled in the art.

[0078] Flowchart 800 begins at step 802, where gradients are received from the target device. For example, Figure 1 The output data manager 124 can receive gradients from the target devices 134a-134k. Gradients are adjustments calculated by backpropagating the prediction error of the AI ​​model. Therefore, a gradient is a value representing the difference between where the model weights are located and where they should be located. Gradients can be placed in a data structure, such as a matrix. In an example embodiment, gradients can be received after the execution of each microbatch, and the output data manager 124 and / or weight updater 126 are configured to accumulate received gradients until a certain number of microbatches have been executed, and then use the received gradients to perform additional calculations and / or update the AI ​​model 106. In another example embodiment, gradients can be accumulated at the target devices 134a-134k for each microbatch, and then sent to the parameter server 102 after each microbatch is completed.

[0079] In step 804, the weights of the AI ​​model are updated based on the received gradients. For example, Figure 1 The weight updater 126 can use gradients received from target devices 134a-134k to update the weights 108 of the AI ​​model 106. The received gradients can be further processed (e.g., averaged) before being used to update the AI ​​model 106 with the processed gradients. In an example embodiment, the output data manager 124 receives gradients after each microbatch and accumulates gradients on the mini-batch before the weight updater 126 updates the AI ​​model 106 by updating the weights 108. In another embodiment, the output data manager 124 receives gradients after each microbatch, and the weight updater 126 updates the AI ​​model 106 at this time. For example, for an image analysis model, the mini-batch size can be set to 512 images, so that after execution of 512 images, the gradients of the mini-batch can be provided to the parameter server to update the model. However, if the target devices can only accommodate micro-batches of 16 images each, gradients can accumulate after each microbatch is executed at the target device, and the gradients are only applied to the model after 32 microbatches. Therefore, the microbatch method is mathematically equivalent to the minibatch method. In other words, executing 512 images in a mini-batch and then applying the gradients of that mini-batch to the model is mathematically the same as executing 16 images in a micro-batch to accumulate the gradients for each micro-batch, until 512 images have been executed in 32 micro-batches or one mini-batch, and then the accumulated gradients are applied to the model. B. Data Parallelism in Distributed Training of Artificial Intelligence Models

[0080] One challenge in deep learning when training distributed deep learning models in large-scale environments is communication between the target devices. For example, exchanging gradients across all target devices (e.g., in implementations without a parameter server) is a time-consuming process. Typically, in synchronous data-parallel distributed deep learning, the main computational steps involve computing gradients on mini-batches using GPUs, calculating the mean of the gradients via inter-GPU communication, and then updating the model. To compute the mean of the gradients, a communication operation (e.g., AllReduce) can be used to reduce the target array across all GPUs into a single array and return the single array to all GPUs. Even in schemes that use a parameter server, GPUs may be needed to cache all layers of the AI ​​model.

[0081] In example embodiments, executing the dataset in micro-batches on a sub-part of the AI ​​model at a time offers several advantages, particularly for training such AI models in a distributed manner in a data-parallel fashion. For instance, this technique enables one or more parameter servers to reduce (e.g., optimize, average, and update) all parameters of the AI ​​model while reducing parameters occurring on the target device. Therefore, parameter reduction can occur simultaneously at different levels (e.g., the target device level and the parameter server level). A benefit of this technique is zero or near-zero communication overhead in large-scale data parallelism.

[0082] For example, Figure 9 A diagram illustrating multi-level parallel parameter reduction in a system 900 according to an example embodiment is shown. System 900 includes a parameter server 902 and target devices 906a-906n, and the system can be implemented as a system 100 having a parameter server 102 and target devices 134a-134k. Figure 9 As shown, while target devices 906a-906n are performing parameter reduction on a specific sub-part of the AI ​​model (e.g., the current layer) at target device level 908, parameter server 902 can also perform parameter reduction on another sub-part of the AI ​​model (e.g., the previous layer) at parameter server level 904. Therefore, parameter server 902 can be responsible for reducing parameters, such as averaging gradients and / or otherwise optimizing them, and then performing subsequent weight updates of the AI ​​model outside the target device in parallel with the computation at the target device, thereby accelerating the overall computation. More parameter servers can be added to system 900, and this multi-level parallel parameter reduction technique scales well with the addition of parameter servers to reduce communication overhead even at commodity network speeds. For example, parameter servers can perform parameter server-level parameter reduction in parallel with the target devices performing target device-level parameter reduction.

[0083] Figure 10A timing diagram 1000 illustrating parameter reduction in a multi-stage parallel system according to an example embodiment is shown. For example, Figure 1000 depicts a system (such as...) Figure 9 The system 900 shown illustrates multi-level parallel parameter reduction of the AI ​​model executed in the system. Figure 1000 shows a timeline 1004 with different time periods 1014, 10416, and 1018. During each time period, parallel execution can be performed with the target device (e.g., Figure 9 Target devices 906a-906n) and parameter servers (e.g., Figure 9 The AI ​​model training-related task 1002 is executed at the parameter server 902 to improve computation speed.

[0084] For example, during the first time period 1014, the target device can execute task 1008, which is the computation of the current layer N. Simultaneously, it also executes task 1010, which is a full reduction operation between the target devices in the previous layer N+1. The result 1024 of the full reduction operation of the previous layer N+1 is sent to the parameter server. Furthermore, during the first time period 1014, the parameter server executes tasks 1006 and 1012. Task 1006 is the preparation for the next layer N-1, and task 1012 is the parameter reduction for the layer before that N+2. The preparation for the next layer N-1 includes sending necessary data 1020 (e.g., the weights and activations of the AI ​​model) to the target device.

[0085] During the second time period 1016, the target device can execute task 1008 based on the received data 1020. Task 1008 is the calculation of layer N-1, and simultaneously executes task 1010, which is a full reduction operation between target devices in layer N. The result 1026 of the full reduction operation of layer N is sent to the parameter server. Furthermore, during the second time period 1014, the parameter server executes tasks 1006 and 1012. Task 1006 is the preparation for layer N-2, and task 1012 is the parameter reduction for layer N+1. The preparation for layer N-2 includes sending the necessary data 1022 to the target device.

[0086] During the second time period 1016, the target device can execute task 1008 based on the received data 1020. Task 1008 is the calculation of layer N-1. Simultaneously, it also executes task 1010, which is a full reduction operation between target devices in layer N. The result 1026 of the full reduction operation in layer N is sent to the parameter server. Furthermore, during the second time period 1016, the parameter server executes tasks 1006 and 1012. Task 1006 is the preparation for layer N-2, and task 1012 is the parameter reduction for layer N+1. The preparation for layer N-2 includes sending the necessary data 1022 to the target device.

[0087] The multi-stage reduction process continues in a similar manner at both the parameter server and the target device for each time period until the AI ​​model training is complete. For example, during the third time period 1018, the target device can execute task 1008 based on the received data 1022. Task 1008 is the computation of layer N-2, and simultaneously execute task 1010, which is a full reduction operation between the target device and layer N-1. Furthermore, during the third time period 1018, the parameter server executes tasks 1006 and 1012. Task 1006 is the preparation for layer N-3, and task 1012 is the parameter reduction for layer N.

[0088] Multi-stage parallel parameter reduction processes can be implemented in various ways. For example, Figures 11-13 This process is demonstrated for distributed training of AI models. More specifically, Figure 11 A flowchart 1100 illustrates a parallel parameter reduction process in a system according to an example embodiment. For example, parallel parameter reduction can be performed by... Figure 9 The system 900 shown includes parameters for server 902 and target devices 906a-906n and / or Figure 1 The parameters of system 100 shown are executed by server 102 and target devices 134a-134k.

[0089] Flowchart 1100 begins at step 1102, where a master copy of the AI ​​model is stored. For example, as... Figure 1 As shown, when AI model 106 is trained, for example using dataset 114, the data manager 118 can store a master copy of AI model 106 together with its associated weights 108, activations 112 and gradients 110 at parameter server 102.

[0090] In step 1104, a portion of the AI ​​model is transmitted to the target device. For example, such as... Figure 1 As shown, transmitter 122 transmits a portion of AI model 106 from parameter server 102 to target device 134a, which may be a memory-constrained device. Therefore, in this example embodiment, target device 134a may not have sufficient memory to efficiently execute AI model 106. In another example embodiment, target device 134a may have large enough memory to store the entire AI model 106, but downloading and storing only a portion of AI model 106 as needed, rather than storing the entire instance of AI model 106, may be more efficient.

[0091] In step 1106, the appropriate micro-batch size for the target device is determined. (Refer to the above...) Figure 2 and Figure 6As described, the micro-batch size can be automatically or manually configured by the batch manager 120 at discrete points during the training of the AI ​​model 106, based on the communication rate between the target device 134a and the parameter server 102. In one embodiment, the batch manager 116 may initially select the micro-batch size based on the hardware specifications of the target device 134a and then iteratively adjust it to the optimal micro-batch size, for example, based on the computation time of the target device 134a, the size of the sub-parts of the AI ​​model 106 to be transmitted, and / or the communication bandwidth of the system 100.

[0092] Flowchart 1100 ends at step 1108. In step 1108, simultaneously, a micro-batch set of the training dataset is executed on the target device at the first sub-part of the transmission portion of the AI ​​model to generate gradients, parameter reduction is performed for the second sub-part of the transmission portion of the AI ​​model, and the weights of the third sub-part of the transmission portion of the AI ​​model are sent to the target device. For example, when the target device 134a executes a micro-batch set of dataset 114 on the first sub-part (e.g., the current layer) of AI model 106, weight updater 126 can perform parameter reduction for the second sub-part (e.g., the previous layer) of AI model 106, and simultaneously (or substantially simultaneously), transmitter 112 can send the weights of the third sub-part (e.g., the next layer) of AI model 106 to the target device 134a. For example, parameter server 102 can be configured to... Figure 10 The figure 1000 shown is used to perform these tasks.

[0093] In an example embodiment, weight updater 126 is configured to perform parameter reduction using gradients received from target device 134a, which are generated by target device 134a performing micro-batch sets of dataset 114 on a second sub-part (e.g., the previous layer) of AI model 106 at target device 134a. Weight updater 126 is also configured to generate an average of the received gradients in any manner known in the art. For example, weight updater 126 can generate an average of the received gradients by using operations and libraries provided in the AI ​​framework. Weight updater 126 can also perform other operations on and / or otherwise optimize the received gradients. Weight updater 126 is also configured to update AI model 106 using the average of the received gradients by updating weights 108.

[0094] In the example embodiment, target devices 134a-134k are configured to perform parameter reduction on gradients generated by target devices 134a-134k in a manner similar to parameter server 102. For example, output manager 154 may generate an average value of the gradients generated by target device 134a. Output manager 154 may also perform other operations on the gradients and / or optimize them in other ways.

[0095] In addition to the processes depicted in flowchart 1100, parameter server 102 can perform additional processes. Training an AI model requires computational and memory resources, and larger AI models require even more. Deep learning systems can use single-precision (i.e., 32-bit) format (a common floating-point format), double-precision (i.e., 64-bit) format, or half-precision (i.e., 16-bit) format for computational workloads, such as storing and updating data like weights, activations, and gradients. Mixed-precision methods combine the use of different numerical formats within a single computational workload. By using mixed-precision training, memory bandwidth requirements can be reduced because fewer bits can be used to store the same number of values. Computation time on the processor can also be improved, allowing the processor to provide higher throughput for reduced-precision mathematics. Furthermore, some devices and AI frameworks can include automatic support for mixed-precision methods. For example, Figure 12 A flowchart 1200 illustrates a process for providing mixed-precision training of an AI model according to an example embodiment. Based on Figure 12 Flowchart 1200 and Figure 1 The following discussion of System 100 will make other structural and operational embodiments readily apparent to those skilled in the art.

[0096] Flowchart 1200 begins at step 1202, where the weights of the fourth sub-part of the AI ​​model's transmission portion are converted to a first-precision format before being sent to the target device. For example, as... Figure 1 As shown, before the transmitter 122 sends the converted weights to the target device 134a, the precision formatter 128 can convert the weights 108 of the AI ​​model 106 into a first precision (e.g., half precision) format. For example, using a lower precision format can reduce computation time. In the example embodiment, any precision format can be used as needed to optimize the performance of the AI ​​model 106.

[0097] In step 1204, the gradient received from the target device is converted to a second-precision format. For example, as... Figure 1 As shown, precision formatter 128 is configured to convert gradients received from target device 134a into a second-precision (e.g., single-precision) format. In an example embodiment, the gradient-to-second-precision format conversion may be performed before or after certain operations on the received gradient (e.g., summation, averaging, etc.). In other embodiments, the received gradient may be simply converted to a second-precision format before being stored as gradient 110 in memory 104.

[0098] In step 1206, the transformed gradient is used to update the weights. For example, as... Figure 1As shown, the weights 108 of the AI ​​model 106 can be updated by the weight updater 126 using the transformed gradient.

[0099] In embodiments, flowchart 1200 can be performed with fewer or more steps, or with different steps than those shown. For example, different mixed-precision methods can be utilized with different levels of precision. For instance, for training iterations of a sub-part (e.g., layer) of AI model 106, weights 108 can be converted to half-precision format for forward propagation, and the generated activations can also be maintained in half-precision format. In backward propagation, weights 108 can be maintained in half-precision format along with the generated gradients. Once the average gradient has been computed, it can be converted to single-precision format before updating the weights 108 of AI model 106. Many other operational embodiments can be implemented using system 100 for a variety of reasons. For example, weight updates (e.g., weight gradient multiplied by the learning rate) may become too small to be represented in half-precision to maintain model accuracy. Single-precision or double-precision formats may result in more computation time and / or resources for training the model.

[0100] Parameter server 102 can execute additional processes to manage target devices 134a-134k. For example, Figure 13 A flowchart 1300 illustrates a process for training an AI model using multiple target devices according to an example embodiment. Based on... Figure 13 Flowchart 1300 and Figure 1 The following discussion of System 100 will make other structural and operational embodiments readily apparent to those skilled in the art.

[0101] Flowchart 1300 begins at step 1302, where another portion of the AI ​​model is transmitted to another target device. For example, transmitter 122 could transmit to another target device (such as...) Figure 9 The target device 906n shown transmits another part of the AI ​​model 106. In the example embodiment, multiple target devices can be used to accelerate the training time of the AI ​​model. System 900 may include any number of target devices, from one to multiple, each of which is communicatively connected to parameter server 902 via one or more suitable interfaces (e.g., PCI or PCIe).

[0102] In step 1304, gradients are received from another target device to perform parameter reduction for another part of the AI ​​model. Continuing the example of step 1302, the target device 906n can send gradients to the output data manager 124 at the parameter server 902 for use in a portion of the AI ​​model 106 received and executed on the target device 906n.

[0103] C. Dynamic multi-layer execution of artificial intelligence modeling

[0104] Another significant advantage of the execution paradigm above (i.e., executing the dataset in micro-batches on a sub-part of the AI ​​model at a time) is that it only requires statically defining the sub-part or a portion thereof (e.g., a layer or sublayer), rather than the entire model computation graph as conventionally required. Therefore, the number of layers within the AI ​​model can be dynamically modified based on any number of factors, such as performance, alternative datasets, or other statistical observations.

[0105] Novel models based on Neural Architecture Search (NAS) and its probabilistic counterparts are emerging, and dynamically executed frictionless methods offer improved modeling techniques, which are currently very challenging to develop. NAS is a technique or algorithm that searches for the optimal neural network architecture based on a set of defined building blocks that can be used to build the neural network. These building blocks can be sampled and pieced together to construct a network similar to other known networks in the field, but can include different combinations and configurations of building blocks. NAS-built networks can be trained and tested, and the building blocks can be tuned based on the test results. NAS-built networks can be improved through operations such as adding layers, removing layers, or otherwise modifying layers.

[0106] Therefore, the technology that allows the number of layers within an AI model to be dynamically modified based on any number of factors is highly beneficial in NAS and other application areas. For example, Figure 14 A flowchart 1400 illustrating a process for dynamically executing AI modeling, according to an example embodiment, is shown. (Although references are provided...) Figure 1 The system 100 is described, but Figure 14 The method is not limited to this system. Based on... Figure 1 The following discussion of system 100 will reveal other structural and operational embodiments to those skilled in the art. Flowchart 1400 may include, for example, those already referenced above. Figure 1 , Figure 2 and Figure 6 The steps described herein will not be described in detail below for the sake of brevity.

[0107] Flowchart 1400 begins at step 1402, where a master copy of the AI ​​model is stored in the parameter server. For example, as... Figure 1 As shown, the data manager is configured to store a primary copy of the AI ​​model 106 in the memory 104 of the parameter server 102.

[0108] In step 1404, a micro-batch size suitable for the target device is determined. For example, batch manager 120 may be configured to determine a micro-batch size suitable for target device 134a. In an example embodiment, target device 134a may be a memory-constrained device, such that the memory of target device 134a may be insufficient to efficiently execute AI model 106. In an alternative embodiment, target device 134a may be able to accommodate the entire AI model 106. However, in this embodiment, it may be more efficient or preferable to download and store only a portion of AI model 106 at a given time, rather than the entire instance of AI model 106.

[0109] In step 1406, a portion of the AI ​​model is transmitted to the target device. For example, transmitter 122 may be configured to transmit a portion of AI model 106 to target device 120b.

[0110] In step 1408, output data can be received from the target device. This output data is generated by executing a micro-batch set of the dataset on the target device at a sub-part of the AI ​​model's transmission portion. For example, the output data manager 124 can be configured to receive data from... Figure 1 The target device 134a receives the output. The output data can be generated by executing a set of micro-batches of a dataset (e.g., dataset 114) on a sub-part (e.g., layer or sub-layer) of the transport portion (e.g., layer or sub-layer) of the AI ​​model 106 at the target device 134a. The output data can be, for example, activations and gradients generated on the forward and backward passes for inference or for training the AI ​​model 106. In an example embodiment, the target device 134a can send a signal instead of output data indicating that a set of micro-batches has been executed at the target device 134a. The parameter server 102 can then be configured to perform actions (e.g., execute subsequent steps) based on this signal rather than on the output data.

[0111] Flowchart 1400 ends at step 1410. In step 1410, the AI ​​model is evaluated based on one or more metrics to determine whether any changes to the execution of the AI ​​model are needed. For example, model evaluator 130 may be configured to evaluate AI model 106 based on one or more metrics to determine whether any changes to the execution of AI model 106 are needed, such as dynamically increasing or decreasing the number of layers to be executed.

[0112] One or more metrics can be based on any number of factors, such as current performance, alternative datasets, or other statistical observations. In an example embodiment, one or more metrics include the accuracy statistics of gradients and weights of sub-parts (e.g., layers or sub-layers) of the transport portion of the AI ​​model 106. For example, Figure 15A flowchart illustrating a process for determining the computational accuracy of dynamic execution used for AI modeling, according to an example embodiment, is shown. (Although references are provided...) Figure 1 The system 100 is described, but Figure 15 The method is not limited to this system. Based on... Figure 1 The following discussion of System 100 will make other structural and operational embodiments clear to those skilled in the art.

[0113] Flowchart 1500 includes step 1502. In step 1502, the computational precision to be applied to the execution of the next microbatch set is determined. For example, precision formatter 128 may be configured to determine the computational precision to be applied to the execution of the next microbatch set. The computational precision may be a fixed-point representation of gradients and weights (e.g., 16-bit, 32-bit, 64-bit). For example, to improve energy efficiency or data bit storage or for any other reason, precision formatter 128 may be configured to determine an appropriate precision to apply to the execution of the next microbatch set at target device 134a. In an example embodiment, precision formatter 128 may be configured to detect vanishing gradients and adjust the computational precision accordingly without requiring the implementation of any specific method.

[0114] In another embodiment, one or more metrics include measurements of the accuracy of the AI ​​model. For example, Figure 16 A flowchart 1600 illustrates the process of determining whether to stop or continue the execution of an AI model based on its accuracy. (Although referenced...) Figure 1 The system 100 is described, but Figure 16 The method is not limited to this system. Based on... Figure 1 The following discussion of System 100 will make other structural and operational embodiments readily apparent to those skilled in the art.

[0115] Flowchart 1600 includes step 1602. In step 1602, the execution of the AI ​​model stops when the accuracy of the AI ​​model exceeds a predetermined threshold, or continues when the accuracy of the AI ​​model does not exceed the predetermined threshold. For example, Figure 1The model accessor 130 is configured to determine whether to stop or continue executing the AI ​​model based on its accuracy. Specifically, the model accessor 130 can stop the execution of the AI ​​model when its accuracy exceeds a predetermined threshold, or it can continue the execution of the AI ​​model when its accuracy does not exceed the predetermined threshold. For example, the accuracy measurement of the AI ​​model 106 could be classification accuracy, which is the ratio of the number of correctly predicted data to the total number of input samples. In an example embodiment, the execution of the AI ​​model 106 can be paused when its accuracy measurement exceeds a predetermined threshold, which can be any predefined value, such as 95%. According to this example embodiment, the execution of the AI ​​model 106 can continue when its accuracy measurement does not exceed the predetermined threshold of 95%, for example, when its accuracy is 80%. In an example embodiment, the continuation of the AI ​​model 106's execution can be dynamically determined, for example, for some number of layers or until the next evaluation of the AI ​​model 106. In an example embodiment, AI model 106 may be executed one sub-part (e.g., layer or sub-layer) at a time, and AI model 106 may be evaluated after each sub-part is executed.

[0116] Other metrics can be used to evaluate AI model 106, such as log loss, metrics derived from the confusion matrix, area under the curve, Fl score, mean absolute error, and mean squared error. When using other metrics, appropriate thresholds for each metric can be determined and applied to the evaluation of AI model 106. Other factors, such as new datasets being used, can lead to changes in the evaluation and / or performance of AI model 106.

[0117] In the foregoing discussion of flowcharts 200, 300, 600-800, and 1100-1600, it should be understood that sometimes these steps may be performed in a different order or even simultaneously with other steps. Other operational embodiments will be readily apparent to those skilled in the art. It should also be noted that the foregoing general description of the operation of systems 100 and 900 is for illustrative purposes only, and embodiments of systems 100 and 900 may include different hardware and / or software and may operate in a manner different from that described above.

[0118] III. Example Computer System Implementation

[0119] Each of the parameter server 102, target devices 134a-134k, parameter server 904, target devices 906a-906n, and flowcharts 200, 300, 600-800, and / or 1100-1600 can be implemented in hardware or in combination with software and / or firmware. For example, the parameter server 102, target devices 134a-134k, parameter server 904, target devices 906a-906n, and flowcharts 200, 300, 600-800, and / or 1100-1600 can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, parameter server 102, target devices 134a-134k, parameter server 904, and target devices 906a-906n, as well as flowcharts 200, 300, 600-800, and / or 1100-1600, can be implemented as hardware logic / circuit systems.

[0120] For example, in one embodiment, parameter server 102, target devices 134a-134k, parameter server 904, and target devices 906a-906n, as well as one or more of flowcharts 200, 300, 600-800, and / or 1100-1600, can be implemented together in any combination within the SoC. The SoC may include an integrated circuit chip comprising one or more of the following: a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or other circuitry, and may optionally execute received program code and / or include embedded firmware to perform functions.

[0121] Figure 17 Exemplary implementations of computing device 1700 in which embodiments can be implemented are depicted. For example, parameter server 102, target devices 134a-134k, parameter server 904, and target devices 906a-906n can each be implemented in one or more computing devices similar to computing device 1700 in fixed or mobile computer embodiments, including one or more features and / or alternative features of computing device 1700. The description of computing device 1700 provided herein is provided for illustrative purposes and is not intended to be limiting. Embodiments can be implemented in other types of computer systems, as known to those skilled in the art.

[0122] like Figure 17As shown, computing device 1700 includes one or more processors (referred to as processor circuitry 1702), system memory 1704, and a bus 1706 coupling various system components, including system memory 1704, to processor circuitry 1702. Processor circuitry 1702 is an electrical and / or optical circuit implemented as a central processing unit (CPU), microcontroller, microprocessor, and / or other physical hardware processor circuitry in one or more physical hardware circuitry device elements and / or integrated circuit devices (semiconductor chip or die). Processor circuitry 1702 can execute program code stored in a computer-readable medium, such as operating system 1730, application program 1732, other program 1734, etc. Bus 1706 represents one or more types of bus architectures, including memory bus or memory controller, peripheral bus, accelerated graphics port, and processor or local bus using any of the various bus architectures. System memory 1704 includes read-only memory (ROM) 1708 and random access memory (RAM) 1710. The Basic Input / Output System 1712 (BIOS) is stored in ROM 1708.

[0123] The computing device 1700 also includes one or more of the following drives: a hard disk drive 1714 for reading from and writing to a hard disk, a disk drive 1716 for reading from or writing to a removable disk 1718, and an optical disc drive 1720 for reading from or writing to a removable optical disc 1722, such as a CD-ROM, DVD-ROM, or other optical media. The hard disk drive 1714, disk drive 1716, and optical disc drive 1720 are connected to the bus 1706 via a hard disk drive interface 1724, a disk drive interface 1726, and an optical disc drive interface 1728, respectively. The drives and their associated computer-readable media provide the computer with non-volatile storage of computer-readable instructions, data structures, program modules, and other data. Although hard disks, removable disks, and removable optical discs have been described, other types of hardware-based computer-readable storage media, such as flash memory cards, digital video disks, RAM, ROM, and other hardware storage media, can be used to store data.

[0124] Multiple program modules may be stored on a hard disk, magnetic disk, optical disk, ROM, or RAM. These programs include an operating system 1730, one or more application programs 1732, other programs 1734, and program data 1736. Application programs 1732 or other programs 1734 may include, for example, computer program logic (e.g., computer program code or instructions) for implementing parameter server 102, target devices 134a-134k, parameter server 904, and target devices 906a-906n, as well as flowcharts 200, 300, 600-800, and / or 1100-1600 (including any suitable steps of flowcharts 200, 300, 600, and / or 1100-1600), and / or other embodiments described herein.

[0125] Users can input commands and information into computing device 1700 using input devices such as keyboard 1738 and pointing device 1740. Other input devices (not shown) may include microphone, joystick, game controller, satellite antenna, scanner, touchscreen and / or touchpad, voice recognition system for receiving voice input, gesture recognition system for receiving gesture input, etc. These and other input devices are typically connected to processor circuitry 1702 via serial port interface 1742, which is coupled to bus 1706, but may also be connected via other interfaces such as parallel port, game port, or Universal Serial Bus (USB).

[0126] Display screen 1744 is also connected to bus 1706 via an interface (such as video adapter 1746). Display screen 1744 may be external to computing device 1700 or incorporated into computing device 1700. Display screen 1744 may display information and a user interface for receiving user commands and / or other information (e.g., via touch, finger gestures, virtual keyboard, etc.). In addition to display screen 1744, computing device 1700 may include other peripheral output devices (not shown), such as speakers and printers.

[0127] Computing device 1700 is connected to network 1748 (e.g., the Internet) via an adapter or network interface 1750, modem 1752, or other means for establishing communication on the network. Modem 1752 (which may be internal or external) may be connected to bus 1706 via serial port interface 1742, such as... Figure 17 As shown in the diagram, another interface type (including the parallel interface) can be used to connect to bus 1706.

[0128] As used herein, the terms “computer program medium,” “computer-readable medium,” and “computer-readable storage medium” are used to refer to physical hardware media, such as a hard disk associated with hard disk drive 1714, removable disk 1718, removable optical disk 1722, other physical hardware media such as RAM, ROM, flash memory cards, digital video disks, zip disks, MEM, nanotechnology-based storage devices, and other types of physical / tangible hardware storage media. Such computer-readable storage media are distinct from and do not overlap with communication media (excluding communication media). Communication media contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves. The term “modulated data signal” refers to a signal whose characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, communication media include wireless media such as acoustic, RF, infrared, and other wireless media, as well as wired media. Embodiments also relate to such communication media that are separate from and do not overlap with embodiments relating to computer-readable storage media.

[0129] As described above, computer programs and modules (including application program 1732 and other programs 1734) can be stored on a hard disk, magnetic disk, optical disk, ROM, RAM, or other hardware storage media. Such computer programs can also be received via network interface 1750, serial port interface 1742, or any other interface type. When executed or loaded by an application, such computer programs enable computing device 1700 to implement the features of the embodiments described herein. Therefore, such computer programs represent a controller for computing device 1700.

[0130] The embodiments also relate to computer program products that include computer code or instructions stored on any computer-readable medium. Such computer program products include hard disk drives, optical disk drives, storage device packages, portable memory sticks, memory cards, and other types of physical storage hardware.

[0131] IV. Additional Example Implementations

[0132] This document describes a system. In one embodiment, the system includes: a set of target devices, the set of target devices comprising one or more target devices, each of the target devices being communicatively connected to a parameter server that stores a master copy of an artificial intelligence (AI) model, the set of target devices being configured to run instances of the AI ​​model, each target device including: a downloader configured to download a portion of the AI ​​model from the parameter server; a data manager configured to store a collection of micro-batches of a dataset; and an executor configured to execute the collection of micro-batches on a first sub-part of the downloaded portion of the AI ​​model to generate activations; and the downloader is also configured to download weights of a second sub-part of the downloaded portion of the AI ​​from the parameter server.

[0133] In one embodiment of the aforementioned system, the executor is further configured to execute a micro-batch set of the dataset on the second sub-part using the download weights of the second sub-part; the downloader is further configured to download the weights of the third sub-part of the download portion of the AI ​​model from the parameter server to the memory of the target device; wherein the execution of the micro-batch set of the dataset on the second sub-part and the download of the weights of the third sub-part are performed simultaneously.

[0134] In another embodiment of the aforementioned system, the executor is further configured to execute a micro-batch set on the second sub-part using the download weights of the second sub-part; and the downloader is further configured to download the weights of the third sub-part of the download portion of the AI ​​model from the parameter server; wherein the execution of the micro-batch set of the dataset on the second sub-part and the download of the weights of the third sub-part are executed serially.

[0135] In an additional embodiment of the aforementioned system, the micro-batch set includes multiple micro-batches configured to be executed sequentially, and the micro-batch set forms a mini-batch, which includes a number of samples for training updated each time or a number of samples for inference provided in each inference cycle.

[0136] In another embodiment of the aforementioned system, each of the target devices further includes an output manager configured to: send the activation of a first sub-part to a parameter server or store the activation on the target device for forward propagation during the training of the AI ​​model; and restore the activation for reverse propagation during the training of the AI ​​model.

[0137] In another embodiment of the aforementioned system, each microbatch in the microbatch set has a microbatch size, which is selected based on the execution rate of multiple microbatch sets and the communication rate between each target device and the parameter server.

[0138] In another embodiment of the aforementioned system, the executor is further configured to perform a micro-batch set on a third sub-part of the downloaded portion of the AI ​​model to generate gradients; the downloader is further configured to download the weights and output activations of a fourth sub-part of the downloaded portion of the AI ​​model; and simultaneously, the executor is further configured to perform a micro-batch set on the fourth sub-part using the downloaded weights and output activations, the downloader is configured to download the weights and output activations of a fifth sub-part of the downloaded portion of the AI ​​model from the parameter server, and the output manager is further configured to send the gradients of the third sub-part to the parameter server.

[0139] In another embodiment of the aforementioned system, the target device includes at least one of the following: an application-specific integrated circuit, a graphics processing unit, or an edge device.

[0140] This paper describes a method implemented in a target device. The method includes: downloading a portion of an artificial intelligence (AI) model from a parameter server; storing a set of micro-batches of the dataset; executing the micro-batches on a first sub-part of the downloaded portion of the AI ​​model to generate activations; and downloading the weights of a second sub-part of the downloaded portion of the AI ​​model from the parameter server.

[0141] An embodiment of the aforementioned method further includes simultaneously performing a micro-batch set of the dataset on the second sub-part using the download weights of the second sub-part; and the weights of the third sub-part of the download portion of the AI ​​model downloaded from the parameter server.

[0142] Another embodiment of the aforementioned method further includes sequentially performing micro-batch sets on the second sub-part using the download weights of the second sub-part; and the weights of the third sub-part of the download portion of the AI ​​model downloaded from the parameter server.

[0143] In another embodiment of the aforementioned method, the micro-batch set includes multiple micro-batches configured to be executed sequentially, and the micro-batch set forms a mini-batch, which includes several samples updated each time for training or several samples saved for inference in each inference cycle.

[0144] Another embodiment of the aforementioned method further includes: sending the activation of the first sub-part to the parameter server during the training of the AI ​​model or storing the activation on the target device for forward propagation; and restoring the activation during the training of the AI ​​model for reverse propagation.

[0145] In an additional embodiment of the aforementioned method, each microbatch in the microbatch set has a microbatch size that is configurable based on the execution rate of multiple microbatch sets and the communication rate between the target device and the parameter server.

[0146] Another embodiment of the aforementioned method further includes: performing a micro-batch set on a third sub-part of the downloaded portion of the AI ​​model to generate gradients; downloading the weights and output activations of a fourth sub-part of the downloaded portion of the AI ​​model; and simultaneously, using the downloaded weights and output activations, performing a micro-batch set on the fourth sub-part to download the weights and output activations of a fifth sub-part of the downloaded portion of the AI ​​model from a parameter server, and sending the gradients of the third sub-part to the parameter server.

[0147] This document describes a system. In one embodiment, the system includes: a parameter server communicatively connected to a set of target devices, the set of target devices including one or more target devices configured to run instances of an artificial intelligence (AI) model; the parameter server including: a data manager configured to store a master copy of the AI ​​model; a batch manager configured to determine a micro-batch size suitable for each target device in the set of target devices; and a transmitter configured to transmit a portion of the AI ​​model to the set of target devices.

[0148] In one embodiment of the aforementioned system, the micro-batch size is based on the execution rate of the micro-batch set at each target device and the communication rate between the target device and the parameter server.

[0149] In another embodiment of the aforementioned system, the parameter server further includes an output data manager configured to: receive activations from each target device after each mini-batch is executed; and generate output activations for sub-parts of the downloaded portion of the AI ​​model based on the received activations.

[0150] In another embodiment of the aforementioned system, the parameter server further includes a weight updater configured to update the weights of the AI ​​model based on gradients received from each target device.

[0151] In another embodiment of the aforementioned system, the parameter server includes a central processing unit, a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC).

[0152] V. Conclusion

[0153] While various embodiments of the disclosed subject matter have been described above, it should be understood that they are presented as examples only and not as limitations. Those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of the embodiments as defined in the appended claims. Therefore, the breadth and scope of the disclosed subject matter should not be limited to any of the exemplary embodiments described above, but should be defined solely by the appended claims and their equivalents.

Claims

1. A system comprising: A set of target devices, comprising one or more target devices, each target device being communicatively connected to a parameter server storing a master copy of an artificial intelligence (AI) model, the set of target devices being configured to run instances of the AI ​​model, each target device including an integrated circuit with on-chip memory and storing a portion of the AI ​​model, at least one target device including an integrated circuit chip having an on-chip memory size smaller than the overall size of the AI ​​model, each target device comprising: The downloader is configured to download the weights of the first layer for the portion of the AI ​​model from the parameter server during a first time period. A data manager is configured to store sets of micro-batches of a dataset, wherein input samples are split into said sets of micro-batches; An actuator, configured by the integrated circuit chip to execute the micro-batch set on the first layer of the portion of the AI ​​model during a second time period to generate activations; and The downloader is also configured to download the weights of the second layer for the portion of the AI ​​from the parameter server during the second time period, wherein the computation cycle generated by the executor executing the micro-batch set hides the delay caused by the downloader downloading the weights of the second layer.

2. The system according to claim 1, wherein The executor is also configured to execute the micro-batch set of the dataset on the second layer using the weights downloaded for the second layer; The downloader is also configured to download the weights of the third layer for the portion of the AI ​​model from the parameter server to the memory of the target device; The micro-batch set of the dataset is executed on the second layer, and the weights for the third layer are downloaded simultaneously.

3. The system according to claim 1, wherein... The executor is also configured to execute the micro-batch set on the second layer using the weights downloaded for the second layer; and The downloader is also configured to download the weights of the third layer for the portion of the AI ​​model from the parameter server; The micro-batch set of the dataset is executed on the second layer, and the weights for the third layer are downloaded sequentially.

4. The system of claim 1, wherein the micro-batch set comprises a plurality of micro-batches configured to be executed sequentially, the micro-batch set forming a mini-batch, the mini-batch comprising a plurality of samples for training updated each time or a plurality of samples for inference provided in each inference cycle.

5. The system of claim 4, wherein each of the target devices further comprises an output manager, the output manager being configured to: Send the activation for the first layer to the parameter server or save the activation on the target device for positive propagation during the training of the AI ​​model; And to restore the activation for backpropagation during the training of the AI ​​model.

6. The system of claim 4, wherein each micro-batch in the micro-batch set has a micro-batch size, the micro-batch size being selected based on the execution rate of the plurality of micro-batches and the communication rate between each target device and the parameter server.

7. The system of claim 1, wherein each target device comprises at least one of the following: an application-specific integrated circuit, a graphics processing unit, or an edge device.

8. A system comprising: A set of target devices, comprising one or more target devices, each target device being communicatively connected to a parameter server storing a master copy of an artificial intelligence (AI) model, the set of target devices being configured to run instances of the AI ​​model, each target device including an integrated circuit with on-chip memory and storing a portion of the AI ​​model, each target device comprising: The downloader is configured to download the weights of the first layer for the portion of the AI ​​model from the parameter server during a first time period. A data manager is configured to store sets of micro-batches of a dataset, wherein input samples are split into said micro-batches; and An executor is configured to execute the micro-batch set on the first layer of the portion of the AI ​​model during a second time period to generate activations; and The downloader is also configured to download the weights of the second layer for the portion of the AI ​​from the parameter server during the second time period, wherein the computation cycle generated by the executor executing the micro-batch set hides the latency caused by the downloader downloading the weights of the second layer. The actuator is also configured to execute the micro-batch set on the third layer of the portion of the AI ​​model to generate gradients; The downloader is also configured to download the weights and output activations of the fourth layer for said portion of the AI ​​model; and The actuator is also configured to activate the execution of the micro-batch set on the fourth layer using the downloaded weights and the output. The downloader is configured to download the weights and output activations of the fifth layer for that part of the AI ​​model from the parameter server, and The output manager is also configured to send the gradient for the third layer to the parameter server. The micro-batch set is executed on the fourth layer, the download of the weights and output activations of the fifth layer for the portion of the AI ​​model and the sending of the gradients for the third layer are performed simultaneously.

9. The system of claim 8, wherein each target device comprises at least one of the following: an application-specific integrated circuit, a graphics processing unit, or an edge device.

10. A method implemented in a target device, comprising: In the first time period, weights for a portion of the first layer of an artificial intelligence (AI) model are downloaded from a parameter server. The target device includes an integrated circuit chip and stores a portion of the AI ​​model. The integrated circuit chip has an on-chip memory size smaller than the overall size of the AI ​​model, and the size of the portion is determined at least based on the available memory on the target device and the size of one or more layers of the AI ​​model. The target device is configured to run an instance of the AI ​​model. A set of micro-batches of a stored dataset, wherein input samples are split into said set of micro-batches; The integrated circuit chip executes the micro-batch set on the first layer of the portion of the AI ​​model stored in the on-chip memory during a second time period on the target device to generate activation; as well as During the second time period, the weights of the second layer for the portion of the AI ​​model are downloaded from the parameter server, wherein the computation cycle of executing the micro-batch set hides the delay caused by downloading the weights of the second layer.

11. The method of claim 10, further comprising: At the same time The micro-batch set of the dataset is executed on the second layer using the weights downloaded for the second layer; as well as Download the weights of the third layer for that part of the AI ​​model from the parameter server.

12. The method of claim 10, further comprising: Continuous The micro-batch set is executed on the second layer using the weights downloaded for the second layer; as well as Download the weights of the third layer for that part of the AI ​​model from the parameter server.

13. The method of claim 10, wherein the micro-batch set comprises a plurality of micro-batches configured to be executed sequentially, the micro-batch set forming a mini-batch, the mini-batch comprising a plurality of samples for training updated each time or a plurality of samples for inference provided in each inference cycle.

14. The method of claim 13, further comprising: Send the activation for the first layer to the parameter server or save the activation on the target device for positive propagation during the training of the AI ​​model; And to restore the activation for backpropagation during the training of the AI ​​model.

15. The method of claim 13, wherein each microbatch in the microbatch set has a microbatch size, the microbatch size being configurable based on the execution rate of the plurality of microbatches and the communication rate between the target device and the parameter server.

16. A method implemented in a target device, comprising: In the first time period, download the weights of the first layer for the artificial intelligence (AI) model from the parameter server. A set of micro-batches of a stored dataset, wherein input samples are split into said set of micro-batches; In the second time period, the micro-batch set is executed on the first layer of the portion of the AI ​​model to generate activations; During the second time period, the weights of the second layer for the portion of the AI ​​model are downloaded from the parameter server, wherein the computation cycle of generating the micro-batch set hides the delay caused by downloading the weights of the second layer; The micro-batch set is executed on the third layer of the portion of the AI ​​model to generate gradients; Download the weights and output activations of the fourth layer for the aforementioned portion of the AI ​​model; as well as At the same time The downloaded weights and outputs are used to activate the execution of the micro-batch set on the fourth layer. Download the weights and output activations of the fifth layer for that part of the AI ​​model from the parameter server, and The gradient for the third layer is sent to the parameter server.

17. A system comprising: A parameter server is communicatively connected to a set of target devices, the set of target devices comprising one or more target devices configured to run instances of an artificial intelligence (AI) model, each target device including an integrated circuit with on-chip memory storing portions of the AI ​​model, at least one target device including an integrated circuit chip having an on-chip memory size smaller than the overall size of the AI ​​model, the parameter server comprising: The data manager is configured to store a primary copy of the AI ​​model; A batch manager is configured to determine the micro-batch size applicable to each target device in the set of target devices, wherein the input samples are divided into the micro-batch set, and The transmitter is configured to transmit weights of a portion of the first layer of the AI ​​model to the group of target devices during a first time period. Simultaneously, during a second time period, the integrated circuit chip performs micro-batch processing of the training dataset on the first layer of the portion of the AI ​​model stored in the on-chip memory at the at least one target device to generate gradients. The weight updater is configured to perform parameter reduction for the second layer of the AI ​​model, and The transmitter is also configured to send the third-layer weights for the portion of the AI ​​model to the target device, wherein the micro-batch size determined by the batch manager is such that the computation cycle generated by the integrated circuit chip executing the micro-batch set hides the delay caused by the transmitter sending the third-layer weights.

18. The system of claim 17, wherein the micro-batch size is selected based on the execution rate of the micro-batch set at each target device and the communication rate between the target device and the parameter server.

19. The system of claim 17, wherein the parameter server further comprises an output data manager, the output data manager being configured to: Activation is received from each target device after each mini-batch is executed; and Based on the received activation, output activation is generated for the layer of the portion of the AI ​​model.

20. The system of claim 17, wherein the weight updater is further configured to: The weights of the AI ​​model are updated based on the gradients received from each target device.

21. The system of claim 17, wherein the parameter server comprises a central processing unit, a field-programmable gate array, or an application-specific integrated circuit.

Images