Dynamic multi-layer execution for artificial intelligence modeling

Abstract

The methods, systems, devices, and computer program products described herein support execution of large AI models on memory-constrained target devices that are communicatively connected to a parameter server that stores a master copy of the AI model. The AI model can be decomposed into smaller parts (e.g., layers or sub-layers), and each part can be executed on the target device as efficiently as possible. After execution of a part of the AI model is complete, another part of the AI model can be downloaded and executed at the target device. This paradigm of executing a part of the AI model at a time allows for dynamic execution of large AI models.

Description

BACKGROUND

[0001] Artificial intelligence has had a tremendous impact on many aspects of modern society. Machine learning, a subset of artificial intelligence that employs mathematical algorithms to process large data sets, is increasingly popular in business applications and is appearing more and more in consumer products. Deep learning, a branch of machine learning, is based on algorithms for modeling high-level abstractions in data. Many applications of artificial intelligence are driven by deep learning, such as natural language processing, speech recognition, and image analysis.

[0002] However, there are many challenges that impede the widespread adoption of deep learning. These challenges include the complexity of managing large data sets and the large amount of time and resources required to train deep learning networks. For example, a speech recognition program can require data from multiple dialects and demographic data, which can include terabytes of data for a single language. The complexity of a deep neural network (DNN) can be represented by the number of parameters, such that the more parameters that exist, the more complex the DNN. Additionally, optimizing hyperparameters, parameters defined before the learning process of an artificial intelligence (AI) model begins, can greatly affect the performance of the AI model. Further, a large amount of computing power is required to process the large amount of data used to train such AI models.

[0003] In deep learning, certain kinds of AI models can require the processing power of GPUs (graphics processing units) with high memory capacity. To improve throughput, multiple GPUs can be run in a data-parallel manner, which typically requires synchronizing hundreds of millions to billions of parameters stored in different GPUs, respectively. This approach can be limited by the memory capacity of the GPUs and can not achieve the maximum computing efficiency of the GPUs. SUMMARY

[0004] This summary is provided to introduce a selection of concepts that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0005] Methods, systems, apparatuses, and computer program products are described herein that support the execution of arbitrarily large AI models on memory-constrained target devices that are communicatively connected to a parameter server. In particular, example embodiments provide a process for dynamic execution of AI modeling.

[0006] In particular, described herein are methods, systems, apparatuses, and computer program products that include a parameter server communicatively connected to a target device, the parameter server including: a data manager configured to store a master copy of an AI model; a batch manager configured to determine a mini-batch size suitable for the target device; a transmitter configured to transmit a portion of the AI model to the target device; an output data manager configured to receive output data from the target device, the output data generated by performing a set of mini-batches of a dataset on a sub-portion of the transmitted portion of the AI model at the target device; and a model evaluator configured to evaluate the AI model based on one or more metrics to determine whether any changes need to be performed to the AI model.

[0007] Other features and advantages, and the structure and operation of the various examples are described in detail below with respect to the accompanying drawings. Note that the concepts and technologies are not limited to the specific examples described herein. Such examples are presented herein for illustrative purposes only. Additional examples will readily occur to those skilled in the relevant arts based on the teachings hereinafter discussed. BRIEF DESCRIPTION OF DRAWINGS

[0008] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments of the present application and, together with the description, further serve to explain the principles of the embodiments and to enable a person skilled in the relevant art to make and use the embodiments.

[0009] Figure 1 is a block diagram of a system that supports execution of arbitrarily large AI models on memory-constrained target devices, in accordance with example embodiments.

[0010] Figure 2 is a flow diagram illustrating a process that provides running an AI model on a memory-constrained device during forward pass, in accordance with example embodiments.

[0011] Figure 3 is a flow diagram illustrating a process that provides running an AI model on a memory-constrained device during backward pass, in accordance with example embodiments.

[0012] Figure 4 is a table representing a forward pass through a machine learning model, in accordance with example embodiments.

[0013] Figure 5 is a table representing a backward pass through a machine learning model, in accordance with example embodiments.

[0014] Figure 6 is a flow diagram illustrating a process that provides running an AI model on a memory-constrained device at a parameter server, in accordance with example embodiments.

[0015] Figure 7A flow diagram illustrating a process to provide generation of activations at a parameter server during forward pass, according to an example embodiment, is shown.

[0016] Figure 8 A flow diagram illustrating a process to provide updating of an AI model at a parameter server, according to an example embodiment, is shown.

[0017] Figure 9 A block diagram illustrating parameter reduction with multi-stage parallelism in a system, according to an example embodiment, is shown.

[0018] Figure 10 A timing diagram illustrating parameter reduction with multi-stage parallelism, according to an example embodiment, is shown.

[0019] Figure 11 A flow diagram illustrating a process to provide parallel parameter reduction in a system, according to an example embodiment, is shown.

[0020] Figure 12 A flow diagram illustrating a process to provide mixed precision training of an AI model, according to an example embodiment, is shown.

[0021] Figure 13 A flow diagram illustrating a process to provide training of an AI model using multiple target devices, according to an example embodiment, is shown.

[0022] Figure 14 A flow diagram illustrating a process to provide dynamic execution of AI modeling, according to an example embodiment, is shown.

[0023] Figure 15 A flow diagram illustrating a process to provide determining a computational precision of dynamic execution of AI modeling, according to an example embodiment, is shown.

[0024] Figure 16 A flow diagram illustrating a process to provide determining whether to stop or continue execution of an AI model based on accuracy of the AI model, according to an example embodiment, is shown.

[0025] Figure 17 is a block diagram of an example computer system in which embodiments can be implemented.

[0026] The features and advantages of the embodiments will become more apparent from the detailed description set forth below when taken in conjunction with the drawings, in which like reference characters identify corresponding elements throughout. In the drawings, like reference numbers generally indicate identical, functionally similar, and / or structurally similar elements. The drawing in which an element first appears is indicated by the left-most digit(s) in the corresponding reference number. DETAILED DESCRIPTION

[0027] I. INTRODUCTION

[0028] The following DETAILED DESCRIPTION discloses many embodiments. The scope of the patent application is not limited to the disclosed embodiments, but includes combinations of the disclosed embodiments, and modifications to the disclosed embodiments.

[0029] Reference in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” or the like, means that a described embodiment can include a particular feature, structure, or characteristic, but every embodiment can not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Furthermore, when a

[0030] In the discussion, unless otherwise stated, adjectives such as “substantially”, “approximately”, and “about” modifying a condition or relationship are understood to mean that the condition or relationship is defined to be within acceptable limits for the operation of the embodiment for which it is being defined.

[0031] Many example embodiments are described below. Note that any headings provided herein are not intended to be limiting. Embodiments are described throughout this document, and any type of embodiment can be included under any heading. Moreover, embodiments disclosed in any heading can be combined with any other embodiment described in the same heading and / or a different heading in any manner.

[0032] II. Example Embodiments

[0033] The example embodiments described herein are provided for illustrative purposes, and are not limiting. The examples described herein can be applicable to any type of target crawling system. Additional structural and operational embodiments, including modifications / variations, will become apparent to those skilled in the art with the teachings herein.

[0034] Deep learning has many applications, including natural language processing, speech recognition, image analysis, machine translation, object classification and detection in photos, automatic handwriting generation, automatic game playing, generative model chatbots. Deep learning models are widely used in various tasks due to their ability to mimic the human brain.

[0035] In deep learning, large AI models (e.g., trained for natural language processing or image analysis) can require multiple GPUs with high memory capacity to perform their training. To improve speed, these GPUs can have high-speed interfaces, such as high-bandwidth memory (HBM) interfaces. However, even with high-quality hardware, there are ways to improve the inference and training processes for large AI models. For example, there are two ways to parallelize the training of AI models to improve throughput: model parallelism and data parallelism.

[0036] Model parallelism includes dividing a learning model into multiple parts and placing those parts on different compute nodes (e.g., placing the first half of a layer on a first GPU and the second half of the layer on a second GPU, or splitting intermediate layers and assigning them to separate GPUs). For example, a typical large AI model with 24 layers can be run on GPUs by performing forward passes on the same mini-batch layer by layer, such as by starting with layer 1, then layer 2, and so on, up to layer 24. After each layer, the activations for that layer (also referred to herein as hidden activations, hidden states, or intermediate results) can be saved (e.g., on-chip or off-chip) for backward pass, which can be performed on the same mini-batch in a similar layer-by-layer fashion (in reverse order). For example, the mini-batch can be executed on layer 24, then layer 23, and so on, up to layer 1, after which the AI model is updated. Sometimes, as a tradeoff between computational cost and efficient memory usage, hidden activations can be recomputed in the backward pass. In certain types of AI models (e.g., natural language processing), there can be many parameters, but the mini-batch size can be small (e.g., a few kilobytes). In other types of models (such as dense networks or computer vision models), the number of parameters can be relatively small, but the number of hidden activations can be large. Generally, these types of models can not be able to run on devices without global memory, such as application-specific integrated circuit (ASIC) devices. Thus, the available technique used is model parallelism, where the model is split across multiple devices. However, model parallelism is inefficient due to long sleep memory and computation time.

[0037] Further, a GPU can have certain data structures mapped to its global memory, which is off-chip and connected with a high-speed memory interface (e.g., HBM). For example, input and output activations can reside on-chip, sometimes so do the gradients, while the main copies of weights and hidden activations can be stored off-chip. There are several residency issues with these data structures. Weights can be loaded before they are actually used, taking up valuable memory. Hidden activations can be generated after a forward pass is completed, but can not be needed until a backward pass. Further, global memory data is moved in and out of the chip via loads and stores, causing memory access amplification even with limited time buffering through caches and registers. Thus, when running large AI models in such a GPU or a cluster of GPUs (e.g., using model parallelism or unified memory addressing techniques), the AI model size depends on the number of devices, and the performance loss from communication overhead cannot be accommodated or hidden.

[0038] Data parallelism is that input data is split across computing devices, and each device holds a complete copy of the learning model, called a replica or worker. Each replica computes the gradient of its partial data, and the gradients are combined to update the model parameters. In asynchronous distributed stochastic gradient descent (SGD), each replica has access to a shared memory space where global parameters are stored. After copying the parameters in its local memory, a replica can compute the gradient and updated weights with respect to its current weights, and then apply the updated weights to the global parameters in the shared memory space. The advantage of this configuration is that replicas can work at their own pace without waiting for other replicas to finish computing their gradients. However, there is no way to ensure that when one replica computes the gradient with respect to a set of parameters, another replica does not update the global parameters, causing the global parameters to be updated with stale gradients. In synchronous distributed SGD, each GPU can run a mini-batch of input data (or, samples), and then stop execution to synchronize all model parameters by exchanging gradients, which are adjustments computed by backpropagating loss through the AI model. This method is highly limited by the GPU memory capacity. In cases where the AI model requires more memory than a single GPU, model compilation on that GPU can fail with a memory out-of-error. Data parallelism typically requires synchronizing hundreds of millions to billions of parameters stored separately in different GPUs. Thus, this method can not achieve the maximum computing efficiency of GPUs because they need to pause for a long time in computation to complete synchronization.

[0039] Embodiments described herein overcome such difficulties to support the running of AI models on devices with large on-chip memory but without global memory. Embodiments described herein can execute any arbitrary size AI model in a memory-constrained device such as a GPU, ASIC, or FPGA (field programmable gate array) in a fast and efficient manner. In example embodiments, any arbitrary size AI model can be executed on an ASIC that has no global memory but can execute the AI model faster than a GPU. Thus, embodiments described herein support the execution of large AI models on memory-constrained devices.

[0040] Example embodiments can be implemented in a system with at least one parameter server and one target device. A master copy of an AI model can reside in the parameter server. The AI model can be broken into smaller parts or chunks (e.g., individual layers), and each part or layer can be executed on the target device as efficiently as possible. After a layer is completed, the next layer is executed. To improve balance and efficiency, the technique iterates over the same layer for a large number of input samples until (a) the next layer is loaded onto the target device, thus completely hiding its latency, or (b) the next layer is loaded after the current layer is completed, exposing its latency, but minimizing the overhead through the long computation period of the current layer. To make the current computation period long, the input samples can be divided into micro-batches. A group of micro-batches forms a mini-batch, which is a term for a number of samples used for each update (for training) or provided in each inference period (for inference). By using the size of a group of micro-batches and / or mini-batches as a knob that can be adjusted manually or automatically (e.g., using software of an AI framework), the communication overhead can be minimized or even reduced to zero.

[0041] If an AI model can be optimized with larger batch sizes, such as in the case of natural language processing models, vision models, or models with high weight / activation ratios, the embodiments described herein will allow these models to run at peak performance on one or more memory-constrained devices. Thus, according to example embodiments, large AI models can be executed on target devices whose memory is less than the memory required to efficiently run the large AI models. In other words, the AI models can be executed at a minimum device batch size at which peak efficiency of speed (i.e., efficient TFLOP) can be achieved. For example, the performance of the AI model depends only on the efficiency of the computational throughput of the library running on the target device, i.e., TFLOP (teraFLOPS). Flops per second (FLOPS) is a measure of a computer's performance, e.g., to measure the ability of an algorithm or computer hardware to perform one trillion floating-point operations per second. In other example embodiments, large AI models can be executed on multiple target devices whose combined memory (e.g., global memory) can be less than the memory required to efficiently run the large AI models.

[0042] A. Executing large artificial intelligence models on memory-constrained target devices

[0043] Supporting execution of large AI models on memory-constrained devices can be implemented in a variety of ways. For example, Figure 1 is a block diagram of a system that supports execution of arbitrary large AI models on memory-constrained target devices according to example embodiments. As Figure 1 illustrated in FIG. 1, system 100 includes parameter server 102 and target devices 134a-134k. While Figure 1 only one parameter server 102 is illustrated in FIG. 1, system 100 can include multiple parameter servers. Similarly, while Figure 1 target devices 134a-134k are depicted in FIG. 1, system 100 can include fewer or more numbers of target devices. Based on the following discussion of system 100 as Figure 1 depicted in FIG. 1, other structural and operational embodiments will be apparent to those skilled in the relevant art.

[0044] Parameter server 102 can include any type of computing device (mobile or stationary). Parameter server 102 can provide functionality for other programs or devices, such as sharing data or resources or performing computations. Parameter server 102 can include memory 104 configured to store data (e.g., data sets, software programs, AI models) and processor 132 configured to perform programmed functions. Parameter server 102 can include commercial off-the-shelf components and / or custom components, and can be a standalone device or part of another computing device. Parameter server 102 can include Figure 1other components not shown, such as peripheral interfaces, communication interfaces, integrated devices, multi-processors, and different types of memory. In embodiments, the parameter server 102 can be implemented as one or more of a CPU, FPGA, or ASIC. For example, as a CPU, the parameter server 102 can include electronic circuitry within a computing device that executes instructions of a computer program by performing operations (e.g., mathematical, logical, control, or input / output) based on the instructions. As a FPGA, the parameter server 102 can include an array of programmable logic blocks configured to perform complex combinational functions or other operations. As an ASIC or system-on-a-chip, the parameter server 102 can include a custom integrated circuit configured to perform operations based on computer program instructions.

[0045] The parameter server 102 can be configured to store the AI model 106 in the memory 104. The AI model 106 can include weights 108, and during execution of the AI model 106, activations 112 and gradients 110 can be stored in the memory 104. The parameter server 102 can also store a dataset 114, which can be a training or testing dataset. The parameter server 102 can also include computer program logic (e.g., computer program code or instructions) for performing operations. For example, the parameter server 102 can include an AI model manager 116 configured to manage the AI model 106 during inference or training of the AI model 106. The AI model manager 116 includes computer program logic for managing the AI model 106, such as a data manager 118, a batch manager 120, a transmitter 122, and an output data manager 124. The output data manager 124 is configured to receive and manage output data from the target devices 134a-134k, as well as other data, for managing the AI model 106. The output data manager 124 includes a weight updater 126 configured to update the weights 108 of the AI model 106, a precision formatter 128 configured to manage precision (e.g., mixed precision training, precision conversion, etc.) formats, and a model evaluator 132 configured to evaluate the AI model 106 and manage execution of the AI model 106 accordingly. In example embodiments, the AI model manager 116 can include fewer or more components than shown in FIG. 1. Figure 1 In other embodiments, the functions of the components of the AI model manager 116 can overlap.

[0046] The parameter server 102 can serve one or more target devices 134a-134k. The parameter server 102 can be communicatively connected to the target devices 134a-134k via a suitable interface, such as a peripheral component interconnect (PCI) or PCI express (PCIe), and / or a network (e.g., for cloud computing or edge computing). In example embodiments, the parameter server 102 and the one or more target devices 134a-134k can reside on the same chip or can reside on different chips or different devices. In example embodiments, the parameter server 102 and the target devices 134a-134k can include software for communicating with each other. For example, the parameter server 102 can include driver software specifically designed for communication, such as sending commands (e.g., initiating function calls to the target devices 134a-134k) and receiving responses.

[0047] The target devices 134a-134k can each include Figure 1 The target device 134a includes the data interface 136, the processor 140, and the memory 142. The target devices 134a-134k can each include multiple processors and different types of interfaces and memories, even though Figure 1A single data interface is depicted for target device 134a. In example embodiments, target devices 134a-134k have the same hardware specifications, such as the same memory size. In other example embodiments, target devices 134a-134k can have different hardware specifications. Target devices 134a-134k can be specifically designed to perform compute-intensive operations in an accelerated manner. For example, target devices can include high compute density, fixed-function processors (e.g., processors 140), and other general-purpose capabilities. Target devices 134a-134k can be managed by parameter server 102 to run specific computations. For example, parameter server 102 can execute a main program that prepares input data for processing at target devices 134a-134k, invokes parallel routines (e.g., kernels) at target devices 134a-134k, and receives results after the routines terminate. Parameter server 102 can also utilize a high-level computing language, computing platform, or framework that includes a comprehensive library of accelerated algorithms and data structures to make it easier to accelerate computations on target devices 134a-134k. In example embodiments, target devices 134a-134k can be implemented as GPUs (e.g., specialized or general-purpose GPUs), ASICs, FPGAs, or edge devices (e.g., standalone devices or microcontrollers that reside at the end or edge of a network connection and can have a small memory footprint). In embodiments, target devices 134a-134k can be memory-constrained devices, although this is not necessarily the case in all situations. For example, target devices 134a-134k individually or in one or more groups can be memory-constrained such that they are unable to efficiently run large AI models due to a lack of memory. For illustrative purposes, Figure 1 Features of target device 134a are described below as representative of each of target devices 134a-134k.

[0048] Data interface 136 can be configured to interface target device 134a with parameter server 102 and other devices, including other target devices. For example, data interface 136 may include PCI, PCIe, and / or HBM. Processor 140 is configured to perform operations requested by parameter server 102 and operations specifically for target device 134a. Memory 142 is configured to store data and computer program logic; for example, memory 142 includes accelerator 144, which is configured to perform functions and / or accelerate certain operations, for example, as indicated by parameter server 102. Accelerator 144 includes a data downloader 146 configured to download data (e.g., a model and / or its data, such as weights, activations, and datasets), a data manager 148 configured to store or otherwise manage the downloaded data, a layer executor 150 configured to execute an AI model or a portion thereof (i.e., execute a dataset on an AI model or a portion thereof), and an output manager 152 configured to manage output data (e.g., gradients and activations) generated from model execution, for example, by saving, sending, or restoring output data. In an example embodiment, accelerator 144 may include a... Figure 1 The accelerator 144 may have fewer or more components as shown. In other embodiments, the functionality of the components of the accelerator 144 may overlap.

[0049] Combination Figure 2 Further operational aspects of parameter server 102 and target device 134a are described below. Figure 2 A flowchart 200 illustrating a process for running an AI model on a memory-constrained device during forward propagation, according to an example embodiment, is shown. Although referenced... Figure 1 The system 100 is described, but Figure 2 The process is not limited to this system. Based on Figure 2 Flowchart 200 and Figure 1 The following discussion of System 100 will provide other structural and operational embodiments that will be of interest to those skilled in the art.

[0050] Flowchart 200 begins at step 202. In step 202, a portion of the artificial intelligence (AI) model is downloaded from a parameter server to the target device's memory; the parameter server stores a master copy of the AI ​​model. For example, Figure 1The target device 134a, specifically the data downloader 146, can be configured to download a portion of the AI model 106 from the parameter server 102 into the memory 142 of the target device 134a. In example embodiments, one or more target devices 134a-134k can form a group that is configured to run an instance of the AI model. For example, one group can include the target device 134a, while another group can include the target devices 134b-134k. The memory required by the AI model has both invariable requirements and variable requirements. For example, the size of the weights or parameters can be invariable for a particular precision type (e.g., 32 bits), and the variable requirements can depend on the target device batch size. In example embodiments, the target device 134a (or, a group of target devices 134a-134k) can be configured to run a large AI model, and the memory of the target device 134a (or, the combined memory of the group of target devices 134a-134k or a global memory) can be smaller than the memory required to run the AI model efficiently. Running the AI model efficiently means that the AI model is run with a minimum device batch size, achieving peak efficiency in speed (i.e., effective TLOP). In other words, the AI model does not benefit from a larger device batch size (without more effective TFLOP). For example, the target device 134a can have a memory that is smaller than the overall size of the AI model that is smaller than a particular optimal batch size. The optimal batch size of a target device is a batch size that achieves a desired accuracy (e.g., 85%) at the highest possible throughput (e.g., rate of processing a dataset). The optimal group batch size is the global batch size (e.g., batch size of a group of target devices) divided by the number of groups of target devices that are communicatively connected to the parameter server 102.

[0051] The AI model 106 can include any type of machine learning model that can have a variety of applications in many fields, such as natural language processing, autonomous vehicles, image processing, deep learning robots, automatic machine translation, automatic handwriting generation. The AI model 106 can have any type of deep learning architecture, such as a deep neural network, a recurrent neural network, and a convolutional neural network.

[0052] A simple neural network can include several layers, one to receive input signals and another to send output signals. One or more hidden or processing layers can be located between the input and output layers. In a DNN built to generate one or more inferences, there can be many hidden layers composed of artificial neurons. Such neurons can include an activation function, a constant input, other inputs, and an output. The neuron can produce an output by performing an activation function on a weighted version of the inputs. The inputs to the activation function are weighted according to their respective weights. For example, the inputs can include normalized data. The activation function can be configured to take a single number (e.g., a linear combination of the weighted inputs) based on all of the inputs and perform a fixed operation, such as a sigmoid, tanh, or rectified linear unit option. The constant input can be a constant value.

[0053] A single neuron by itself can not accomplish much work, and useful AI models typically include the combined computational work of a large number of neurons working in concert. For example, a DNN can include multiple neurons assembled in layers and connected in a cascading fashion. These layers can include an input layer, an output layer, and some hidden layers in between. The output of each layer of neurons can be weighted according to certain weights and then used as input to the neurons in the next layer. Other interconnection strategies known in the art can be employed. The neurons of the input layer can be configured to accept normalized or otherwise feature engineered or processed data corresponding to user data. The output of each neuron of the input or hidden layers can be weighted according to its corresponding output edge weight and then applied as input at each neuron of the next layer. The output(s) of the output layer include the output of the DNN or AI model. In the context of inferences, such outputs can be inference(s) or prediction(s). Building such a DNN is only the beginning of generating a useful machine learning or AI model. The accuracy of the inferences generated by such an AI model requires the selection of a suitable activation function, and then each weight of the entire model is adjusted to provide accurate outputs. The process of adjusting such weights is referred to as “training.” Training a DNN or other type of network requires a set of training data with known characteristics. For example, if the DNN is intended to predict the probability that an input image of an animal is a cat, the training data will include many images of cats, and typically not only images of cats, but also images of other similar animals. Training requires pre-processing of the image data corresponding to each image according to normalization and / or feature extraction techniques known in the art to produce input features for the DNN, which are then provided as input to the network, e.g., as input to the neurons of the input layer.

[0054] Thereafter, each neuron of the layer performs its respective activation operation, the outputs of the activation operations are weighted and fed forward in a forward pass to the next layer until the output(s) of the DNN are generated by the output layer. The output(s) of the DNN can be compared to known or expected values of the output and the differences can be fed back in a backward pass through the DNN to adjust the weights contained therein according to backpropagation algorithms known in the art. With the AI model including the updated weights, image features can be input to the model again and new outputs generated. Training includes iterating the AI model over a training dataset and updating the weights at each iteration. Once the AI model reaches a sufficient accuracy, or its outputs have converged and the weights change have little effect, the AI model is said to have been trained. The trained model can then be used to evaluate arbitrary input data, the nature of which is not known beforehand, that the model has not previously considered (e.g., a new picture of an animal), and output the desired inferences (e.g., the probability that the image is of a cat).

[0055] Gradient descent is an algorithm often used in training AI models. Gradient descent involves an objective function (e.g., a loss function or cost function) (of which there can be many) and the goal is to minimize the function. The objective function is used to monitor errors in the predictions of the AI model. Thus, by minimizing the function, the lowest error value can be found, thereby improving the accuracy of the AI model. Stochastic gradient descent (SGD) is a variant of the gradient descent algorithm that computes errors and updates the model for each sample in the training dataset. SGD updates frequently and has a faster learning speed, but is computationally expensive and can take longer to train on large datasets. Batch SGD is another variant that computes errors for each sample in the training dataset, but updates the AI model only after performing the entire dataset (i.e., at the end of the training phase). Batch SGD updates less frequently and is more computationally efficient than SGD. The separation of prediction error computation and model update for batch SGD makes the algorithm suitable for parallel processing-based implementations, but the update at the end of the training epoch requires additional complexity to accumulate prediction errors for the entire dataset and is typically implemented in a way that requires the entire training dataset in memory and available to the algorithm. Mini-batch SGD is yet another variant of SGD that splits the training dataset into small batches that are used to compute model errors and update parameters. Implementations can sum gradients over the mini-batch, thereby further reducing the variance of the gradient. Thus, mini-batch SGD strikes a balance between SGD and batch SGD. Mini-batch SGD requires an additional “mini-batch size” hyperparameter to be configured for the learning algorithm. Error information can be accumulated over the training examples of the mini-batch. The mini-batch size can be configured to be a function of the computing architecture on which the AI model is being executed, for example, a power of two, such as 32, 64, 128, 256, etc., that fits the memory requirements of the target device or accelerator hardware. The batch size can be tuned as a regulation to the learning process, where smaller values give a learning process that converges quickly at the cost of noise in the training process, while larger values give a learning process that converges slowly with accurate estimates of the error gradient.

[0056] Referring back to Figure 2At step 202, the downloaded portion of the AI model 106 can include any portion of the AI model 106. The AI model 106 can be broken down into layers or complex layers or complex fractional layers (e.g., into 1.5x layers). AI models are neatly segmented at layers, so these models can be divided holistically. In one example embodiment, the downloaded portion can include one or more layers of the AI model 106. However, in another example embodiment, fractional division is possible depending on the AI model and other factors. One reason fractional division can be needed is that such fractional portions can fit inside the target device, while the entire layer can not. This would support an implementation where any number of layers can be run without running into a memory insufficient problem, as opposed to running into a memory insufficient error after a particular number of layers. In an example embodiment, the portion of the AI model 106 downloaded to the target device 134a can include any portion of the AI model 106 up to the entire AI model 106. The target device 134a can download the portion of the AI model 106 in various ways. For example, the target device 134a can download the next portion of the AI model 106 into one or more memory buffers while executing the current portion of the AI model 106. This approach can use a little more memory and special libraries, but can result in a higher performing AI model 106. In another example, the target device 134a can execute the current sub-portion, synchronize, and then download the next sub-portion. This approach can be a little slower, but does not require buffering. The flowchart 200 continues with step 204.

[0057] At step 204, the set of mini-batches of the dataset is stored in the memory of the target device. For example, as described above with respect to FIG. 1, the target device 134a can store the set of mini-batches of the dataset in the memory 136a. Figure 1As shown in FIG. 1, a micro-batch set of a data set 114 can be downloaded from the parameter server 102 via the data downloader 146. The data manager 148 can then store the micro-batch set in the memory 142, a buffer, or any other known memory structure of the target device 134a. The data set 114 can be user input data, such as a training data set for training, a test data set for testing purposes, or any input data for inference. The micro-batch set includes a plurality of micro-batches configured to be executed in order at the target device 134a. The micro-batch set forms a mini-batch that includes several samples for training of the AI model 106 per update or several samples for inference provided in each inference cycle. Each micro-batch in the micro-batch set can have a micro-batch size that can be configured automatically or manually. In embodiments, the micro-batch size can be selected based on the execution rate of the plurality of micro-batches and the communication rate between the target device 134a and the parameter server 102. For example, the micro-batch size can be initially selected for the target device 134a based on its hardware specifications, and then the micro-batch size can be adjusted as needed during the iteration process to sufficiently hide the communication latency. In embodiments, the optimal micro-batch size can be a trade-off between the memory required and the percentage of communication overhead that can be hidden. As more communication overhead is hidden, more memory can be required for computation. Thus, the micro-batch size can be large enough to fully utilize the execution of the layers in the target device, but small enough to fit in the memory of the target device.

[0058] The flowchart 200 continues with step 206, which performs a set of microbatches on a first sub-portion of the downloaded portion of the AI model to generate activations. For example, the set of microbatches can be performed on the first sub-portion of the portion of the AI model 106 downloaded by the data downloader 146 at the target device 134a by the layer executor 150. In example embodiments in which the downloaded portion of the AI model 106 includes one or more layers, the set of microbatches can be performed on the one or more downloaded layers of the AI model 106 one layer at a time to generate activations. The activations can be values that are intermediate results, e.g., the output of each microbatch execution. The activations can be internal data needed in backpropagation to determine how the weights 108 of the AI model 106 should be adjusted. After each microbatch is executed for a sub-portion (e.g., a layer) of the AI model 106, the activations can be saved on the target device 134a, sent to the parameter server 102 to save memory, or discarded to save memory and then recomputed. For example, if the AI model 106 has 12 layers and each mini-batch has 8 microbatches, the activations can be stored 96 times during forward propagation and recovered 96 times during backpropagation. If not all activations are saved during forward propagation, the activations can be recomputed during backpropagation. In example embodiments, the storing of the activations of a microbatch can occur while the target device 134a is executing a different microbatch, whether at the target device 134a or at the parameter 102. In example embodiments, the recovery of activations or the recomputation of activations can occur before execution of a sub-portion or as needed during execution of a sub-portion, e.g., the recovery / recomputation of the activations of the next microbatch can occur in parallel with the execution of the current microbatch.

[0059] The flowchart 200 ends with step 208. In step 208, the weights of the second sub-portion of the download portion of the AI model are downloaded from the parameter server into the memory of the target device. For example, if the download portion of the AI model includes multiple layers, the weights of the second layer can be downloaded into the memory 142 of the target device 134a via the data downloader 146. In example embodiments, downloading the weights of the next layer can occur while the current layer is being executed. For example, the target device 134a can be configured to concurrently execute a set of minibatches of a dataset on the second sub-portion using the downloaded weights of the second sub-portion and download the weights of a third sub-portion of the download portion of the AI model 106 from the parameter server 102 into the memory 142 of the target device 134a. For example, the layer executor 150 can execute a set of minibatches on a layer using the weights that have been downloaded for that layer while the data downloader 146 downloads the weights of the next layer of the AI model 106. Alternatively, the target device 134a can be configured to serially execute a set of minibatches on the second sub-portion using the downloaded weights of the second sub-portion and download the weights of a third sub-portion of the download portion of the AI model 106 from the parameter server 102 into the memory 142 of the target device 134a. For example, the layer executor 150 can execute a set of minibatches on a layer using the weights that have been downloaded for that layer and, after execution of that layer, the data downloader 146 can download the weights of the next layer of the AI model 106.

[0060] Accordingly, execution of the AI model 106 continues at the target device 134a one sub-portion at a time as described above, while other sub-portions of the AI model 106 can also be executed at other target devices. For example, in a forward pass, a set of minibatches or mini-batches are executed at a first layer, followed by a second layer, and so on until the last layer.

[0061] Once the forward pass of the AI model 106 is complete, a backward pass can be executed. For example, Figure 3 A flowchart 300 illustrating a process of providing running an AI model on a memory constrained device during a backward pass is shown, in accordance with an example embodiment. Although described with reference to the system 100 of Figure 1 The process of Figure 3 is not limited to this system. Based on the flowchart 300 of Figure 3 and the following discussion of the system 100 of Figure 1 other structural and operational embodiments will be apparent to persons skilled in the relevant art.

[0062] The flowchart 300 begins at step 302 with the execution of a micro batch set on the third subpart of the download portion of the AI model to generate a gradient. For example, the micro batch set can be executed by the layer executor 150 on the third subpart of the download portion of the AI model 106 to generate a gradient for the third subpart. If the AI model 106 has 24 layers, the micro batch set can be executed on the 24th layer to generate a gradient for that layer to begin the backward pass.

[0063] The flowchart 300 continues at step 304. In step 304, the weights and activations of the fourth subpart of the download portion of the AI model are downloaded. For example, the weights and activations from the fourth subpart of the download portion of the AI model 106 can be downloaded from the parameter server 102 to the target device 134a by the data downloader 146. For example, if the AI model 106 has 24 layers, the weights and activations from layer 23 can be downloaded from the parameter server 102 to the target device 134a.

[0064] In step 306, concurrently, a micro batch set is executed on the fourth subpart using the downloaded weights and output activations, the weights and output activations of the fifth subpart of the download portion of the AI model are downloaded from the parameter server, and the gradient of the third subpart is sent to the parameter. For example, in a parallel manner, concurrently or substantially concurrently, a micro batch set can be executed on the fourth subpart by the layer executor 150 using the downloaded weights and output activations of the subpart, the weights and output activations of the fifth subpart of the download portion of the AI model 106 can be downloaded from the parameter server 102 by the data downloader 146, and the gradient of the third subpart of the AI model 106 can be sent to the parameter server 102 by the output manager 152. In an example embodiment in which the model 106 has 24 layers, the target device 134a can be configured to execute multiple steps in parallel or concurrently. In this embodiment, the target device 134a can be configured to concurrently execute layer 23 using the downloaded weights and output activations of layer 23, download the weights and output activations of layer 22 from the parameter 102, and send the gradient 110 generated for layer 24 to the parameter server 102.

[0065] The target device 134a is configured to continue the above steps of the flowchart 300 in order to complete execution of the entire dataset 114 in micro batches on the AI model 106 one subpart (e.g., layer) at a time for a backward pass in reverse order (i.e., layer 24, layer 23,... and layer 1).

[0066] As described in Figure 2 and Figure 3 forward and backward passes can be visualized as depicted in Figure 4 and Figure 5 For example, Figure 4Table 400 is shown representing forward pass through a machine learning model having 24 layers, according to an example embodiment. Table 400 is for a target device, which can be implemented as a target device 134a of Figure 1 Table 400 has three rows, row 410 shows execution of a set of microbatches in turn on each of the layers of the AI model, 10 of which form a minibatch here. Row 412 shows a set of actions of the target device (e.g., receiving weights from the parameter server), and row 414 shows another set of actions the target device can take (e.g., sending activations to the parameter server). Data exchanges at the target device can be done via an interface, such as a PCI shown in Figure 4 Although the AI model has 24 layers, only the 1st layer, the 2nd layer, and the 24th layer are shown in detail in Table 400, as the execution of the AI model on each layer is similar. For example, column 402 of Table 400 depicts execution of a set of 10 microbatches on layer 1, 10 of which form a minibatch. During this execution, the target device receives weights for the 2nd layer (the next layer to be executed). As each microbatch is executed on the 1st layer, the activations of that microbatch can be saved (e.g., at the target device or the parameter server) if memory and / or other resources permit. Then, as shown in column 406 of Table 400, the same set of 10 microbatches is executed on the 2nd layer, while receiving weights for the 3rd layer and saving the activations of each microbatch. This process continues for all layers of the AI model, until the last layer, i.e., layer 24, which can be referred to as a “decoding layer” (DL) or an “embedding layer” or an “output layer.” When the set of microbatches is executed on the last layer (the 24th layer), its weights and activations are determined at the target device and sent to the parameter server, as shown in column 408 of Table 400.

[0067] Figure 5 Table 500 is shown representing backward pass through a machine learning model having 24 layers, according to an example embodiment. Table 500 is for a target device, which can be implemented as a target device 134a of Figure 1 Table 500 has four rows, row 510 shows execution of a set of microbatches in turn on each of the layers of the AI model, 16 of which form a minibatch here. Row 512 shows a set of actions of the target device (e.g., loading weights and activations from the parameter server), row 514 shows another set of actions (e.g., sending gradients to the parameter server), and row 516 shows another set of actions the target device can take (e.g., parameter reduction). Data exchanges at the target device can be done via an interface, such as a PCI shown in Figure 5The PCI is shown. Although the AI ​​model has 24 layers, only layers 24, 23, 22, and 1 are shown in detail in Table 500 because the execution of the AI ​​model on each layer is similar. For example, column 502 of Table 500 depicts a set of 16 micro-batches executed on layer 24, forming mini-batches. During this execution, the target device loads the weights and activations of layer 23 (the next layer to be executed in the backpropagation). Then, in column 504, the set of 16 micro-batches is executed on layer 23 using the loaded weights and activations. In parallel or simultaneously (or substantially simultaneously), the target device is configured to load the weights and activations of layer 22 (the next layer to be executed); send the gradients of the most recently executed layer 24 to the parameter server; and reduce the parameters of the AI ​​model. In column 506, the same set of 16 micro-batches is executed on layer 22 using the loaded weights and activations of that layer. During the execution of layer 22, the weights and activations of layer 21 are loaded, the gradients of layer 23 are sent to the parameter server, and the parameters are reduced at the target device. In column 508, the same set of 16 micro-batches is executed for layer 1. Simultaneously with this execution, the gradients of layer 2 are sent to the parameter server, and the parameters are reduced at the target device.

[0068] By running many micro-batches on top of the same layer, there is sufficient time to hide or cover the latency of preparing the next layer. Therefore, the total memory complexity of the target device can be two layers plus one layer of hidden activation and one layer of output activation.

[0069] For example, combining Figures 2-5 In the description, the target device is used as a supporting component for executing large AI models on memory-constrained devices. (See below for more details.) Figures 6-8 In the description, the parameter server can be used as a supporting component for executing large AI models on memory-constrained devices. For example, Figure 1 A flowchart 600 illustrating a process for running an AI model on a memory-constrained device at a parameter server according to an example embodiment is shown. Although referenced... Figure 6 The system 100 is described, but Figure 6 The process is not limited to this system. Based on Figure 1 Flowchart 600 and Figure 1 The following discussion of System 100 will make other structural and operational embodiments readily apparent to those skilled in the art.

[0070] Flowchart 600 begins at step 602. In step 602, a master copy of the artificial intelligence model is stored at a parameter server, which is communicatively connected to the target device. For example, as... Figure 1As shown, a primary copy of AI model 106 can be stored in memory 104 of parameter server 102 by data manager 118. Parameter server 102 can communicate with target devices 134a-134k via suitable means (such as PCI and PCIe interfaces or other network interfaces). In the example embodiment, parameter server 102 stores a complete copy of AI model 106, while target devices 134a-134k can store a portion of AI model 106 instead of the entire copy.

[0071] In step 604, a suitable micro-batch size for the target device is determined. For example, such as... Figure 6 As shown, target device 134a may be a memory-constrained device with a memory size smaller than the total size of AI model 106 stored at parameter server 102 for a specific optimal batch size. Batch manager 120 is configured to determine a suitable micro-batch size for target device 134a, for example, by taking into account the memory size and / or other hardware specifications of target device 134a. In one embodiment, batch manager 120 configures the micro-batch size for load balancing, such that the ratio of execution time to communication time is maximized. For example, the micro-batch size may depend on the computation time of the target device (C), the size of the sub-parts (S) to be transmitted, and the communication bandwidth (B) between the target device and the parameter server system. In this example, the micro-batch size may be determined as Minimum_numMicroBatches = S / B / C. This equation may be static, but in some cases (e.g., neural architecture search), the micro-batch size may be determined dynamically. Thus, when the ratio of execution time to communication time can be manipulated, the parameter server has more time to perform complex data parallelism or background tasks. In the example embodiment, the micro-batch size may be dynamically configured at certain times or boundary points during the training or inference process. For example, the micro-batch size can be dynamically configured at the end of the iteration, but it must be constant for mini-batch iterations in both forward and backward propagation.

[0072] return Figure 7The flowchart 600 ends at step 606. At step 606, a portion of the AI model is transmitted to the target device. For example, the transmitter 122 can transmit a portion of the AI model 106 from the parameter server 102 to the target device 134a. The AI model 106 can be divided into different portions in any number of ways. For example, the portion can be a layer, a combination of layers, or a combination of a small number of layers of the AI model 106. The portion size can be determined based on the memory available on the target device 134a such that the portion has an optimal size for the target device 134a. For example, the AI model manager 116 can consider the hardware specifications of the target device 134a when determining the size of the portion to send to the target device 134a. The transmitter 122 can transmit a portion of the AI model 106 to the target device 134a while the target device 134a is executing another portion, requiring the target device 134a to buffer the portion. Alternatively, the transmitter 122 can transmit a portion of the AI model 106 to the target device 134a after the target device 134a completes a current portion to avoid the need to buffer the portion. In this alternative example, the target device 134a can perform a synchronization after execution of the current portion before receiving the portion.

[0073] The parameter server 102, or specifically the AI model manager 116, can perform additional steps to improve the throughput of distributed training and inference of AI models on memory-constrained devices. For example, Figure 1 A flowchart 700 showing a process to provide activations at a parameter server for generating during forward pass is shown in accordance with an example embodiment. Although described with reference to the system 100 of Figure 7 , the process of Figure 7 is not limited to this system. Based on the flowchart 700 of Figure 1 and the following discussion of the system 100 of Figure 7 , other structural and operational embodiments will be apparent to those of ordinary skill in the art.

[0074] Figure 8begins with step 702, where activations are received from the target devices after each micro-batch is executed. For example, after each micro-batch is executed by a target device, the output data manager 124 can receive activations from the target devices 134a-134k. For example, the activations can include hidden activations, or intermediate results of executing the micro-batches, or outputs of executing the micro-batches at the target device 134a. In an example embodiment, the activations are received from the target device 134a after each micro-batch. In this embodiment, saving and / or storing the activations after each micro-batch can provide optimal efficiency in executing the AI model 106. Thus, when the target device 134a is executing a mini-batch that includes multiple micro-batches, the activations for each of the multiple micro-batches can be saved after each of the multiple micro-batches is executed. In another example embodiment, the activations are saved at the target device 134a. In yet another example embodiment, not all activations are saved in the forward pass, only the data needed to recompute the activations in the backward pass. Such data can include the input states of a particular subpart of the AI model, for example, because the input states can require less memory space than the output states. Thus, the memory space of the target device 134a can be conserved by not saving all of the activations during the forward pass.

[0075] The flowchart 700 ends with step 704, where output activations are generated for a subpart of a downloaded portion of the AI model based on the received activations. For example, the weight updater 126 can generate output activations for a subpart of a downloaded portion of the AI model 106 based on the activations received from the target devices 134a-134k. In an example embodiment, the generated output activations can be saved in the memory 104 of the parameter server 102 as the activations 112. In an example where the subpart includes a layer, the output activations for the layer can be generated by the weight updater 126 from the hidden activations received after each micro-batch is executed at the target devices 134a-134k.

[0076] The parameter server 102 can perform additional steps to improve the throughput of distributed training and inference of AI models on memory-constrained devices. For example, Figure 1 A flowchart 800 is shown that provides a process at a parameter server for updating an AI model, according to an example embodiment. Although described with reference to the system 100 of Figure 8 The process of Figure 8 is not limited to this system. Based on Figure 1 The following discussion of the flowchart 800 and Figure 1 The system 100 will be apparent to those skilled in the relevant art.

[0077] The flowchart 800 begins with step 802, where gradients are received from the target devices. For example,Figure 1 The output data manager 124 can receive the gradients from the target devices 134a-134k. The gradients are adjustments calculated by backpropagating the prediction error through the AI model. Thus, the gradients are values that represent the difference between where the model weights are and where the weights should be. The gradients can be placed in a data structure, such as a matrix. In an example embodiment, the gradients can be received after the execution of each mini-batch, and the output data manager 124 and / or the weight updater 126 are configured to accumulate the received gradients until a particular number of mini-batches have been executed, and then use the received gradients to perform additional calculations and / or update the AI model 106. In another example embodiment, the gradients can be accumulated at the target devices 134a-134k for each mini-batch, and then sent to the parameter server 102 after each mini-batch is completed.

[0078] In step 804, the weights of the AI model are updated based on the received gradients. For example, Figure 9 The weight updater 126 can use the gradients received from the target devices 134a-134k to update the weights 108 of the AI model 106. The received gradients can be further processed (e.g., averaged) before the AI model 106 is updated with the processed gradients. In an example embodiment, the output data manager 124 receives the gradients after each mini-batch and accumulates the gradients over the mini-batches before the weight updater 126 updates the AI model 106 by updating the weights 108. In another embodiment, the output data manager 124 receives the gradients after each mini-batch and the weight updater 126 updates the AI model 106 at that time. For example, for an image analysis model, the size of the mini-batch can be set to 512 images, thus, after the execution of 512 images, the gradients of the mini-batch can be provided to the parameter server to update the model. However, if the target devices can each only accommodate a mini-batch of 16 images, then the gradients can be accumulated after each mini-batch is executed at the target devices, and the gradients are only applied to the model after 32 mini-batches. Thus, the mini-batch method is mathematically equivalent to the mini-batch method. That is, executing 512 images in one mini-batch and then applying the gradients of that mini-batch to the model is mathematically the same as executing 16 images in a mini-batch to accumulate the gradients of each mini-batch until 512 images are executed in 32 mini-batches or one mini-batch, and then applying the accumulated gradients to the model.

[0079] B. Data parallelism in distributed training of artificial intelligence models

[0080] One challenge in deep learning when training distributed deep learning models in large-scale environments is communication between the target devices. For example, exchanging gradients across all target devices (e.g., in implementations without a parameter server) is a time-consuming process. Typically, in synchronous data-parallel distributed deep learning, the main computational steps involve computing gradients on mini-batches using GPUs, calculating the mean of the gradients via inter-GPU communication, and then updating the model. To compute the mean of the gradients, a communication operation (e.g., AllReduce) can be used to reduce the target array across all GPUs into a single array and return the single array to all GPUs. Even in schemes that use a parameter server, GPUs may be needed to cache all layers of the AI ​​model.

[0081] In example embodiments, executing the dataset in micro-batches on a sub-part of the AI ​​model at a time offers several advantages, particularly for training such AI models in a distributed manner in a data-parallel fashion. For instance, this technique enables one or more parameter servers to reduce (e.g., optimize, average, and update) all parameters of the AI ​​model while reducing parameters occurring on the target device. Therefore, parameter reduction can occur simultaneously at different levels (e.g., the target device level and the parameter server level). A benefit of this technique is zero or near-zero communication overhead in large-scale data parallelism.

[0082] For example, Figure 9 A diagram illustrating multi-level parallel parameter reduction in a system 900 according to an example embodiment is shown. System 900 includes a parameter server 902 and target devices 906a-906n, and the system can be implemented as a system 100 having a parameter server 102 and target devices 134a-134k. Figure 10 As shown, while target devices 906a-906n are performing parameter reduction on a specific sub-part of the AI ​​model (e.g., the current layer) at target device level 908, parameter server 902 can also perform parameter reduction on another sub-part of the AI ​​model (e.g., the previous layer) at parameter server level 904. Therefore, parameter server 902 can be responsible for reducing parameters, such as averaging gradients and / or otherwise optimizing them, and then performing subsequent weight updates of the AI ​​model outside the target device in parallel with the computation at the target device, thereby accelerating the overall computation. More parameter servers can be added to system 900, and this multi-level parallel parameter reduction technique scales well with the addition of parameter servers to reduce communication overhead even at commodity network speeds. For example, parameter servers can perform parameter server-level parameter reduction in parallel with the target devices performing target device-level parameter reduction.

[0083] Figure 9A timing diagram 1000 illustrating parameter reduction in a multi-stage parallel system according to an example embodiment is shown. For example, Figure 1000 depicts a system (such as...) Figure 9 The system 900 shown illustrates multi-level parallel parameter reduction of the AI ​​model executed in the system. Figure 1000 shows a timeline 1004 with different time periods 1014, 10416, and 1018. During each time period, parallel execution can be performed with the target device (e.g., Figure 9 Target devices 906a-906n) and parameter servers (e.g., Figures 11-13 The AI ​​model training-related task 1002 is executed at the parameter server 902 to improve computation speed.

[0084] For example, during the first time period 1014, the target device can execute task 1008, which is the computation of the current layer N. Simultaneously, it also executes task 1010, which is a full reduction operation between the target devices in the previous layer N+1. The result 1024 of the full reduction operation of the previous layer N+1 is sent to the parameter server. Furthermore, during the first time period 1014, the parameter server executes tasks 1006 and 1012. Task 1006 is the preparation for the next layer N-1, and task 1012 is the parameter reduction for the layer before that N+2. The preparation for the next layer N-1 includes sending necessary data 1020 (e.g., the weights and activations of the AI ​​model) to the target device.

[0085] During the second time period 1016, the target device can execute task 1008 based on the received data 1020. Task 1008 is the calculation of layer N-1, and simultaneously executes task 1010, which is a full reduction operation between target devices in layer N. The result 1026 of the full reduction operation of layer N is sent to the parameter server. Furthermore, during the second time period 1014, the parameter server executes tasks 1006 and 1012. Task 1006 is the preparation for layer N-2, and task 1012 is the parameter reduction for layer N+1. The preparation for layer N-2 includes sending the necessary data 1022 to the target device.

[0086] During the second time period 1016, the target device can execute task 1008 based on the received data 1020. Task 1008 is the calculation of layer N-1. Simultaneously, it also executes task 1010, which is a full reduction operation between target devices in layer N. The result 1026 of the full reduction operation in layer N is sent to the parameter server. Furthermore, during the second time period 1016, the parameter server executes tasks 1006 and 1012. Task 1006 is the preparation for layer N-2, and task 1012 is the parameter reduction for layer N+1. The preparation for layer N-2 includes sending the necessary data 1022 to the target device.

[0087] The multi-stage reduction process continues in a similar manner at both the parameter server and the target device for each time period until the AI ​​model training is complete. For example, during the third time period 1018, the target device can execute task 1008 based on the received data 1022. Task 1008 is the computation of layer N-2, and simultaneously execute task 1010, which is a full reduction operation between the target device and layer N-1. Furthermore, during the third time period 1018, the parameter server executes tasks 1006 and 1012. Task 1006 is the preparation for layer N-3, and task 1012 is the parameter reduction for layer N.

[0088] Multi-stage parallel parameter reduction processes can be implemented in various ways. For example, Figure 11 This process is demonstrated for distributed training of AI models. More specifically, Figure 9 A flowchart 1100 illustrates a parallel parameter reduction process in a system according to an example embodiment. For example, parallel parameter reduction can be performed by... Figure 1 The system 900 shown includes parameters for server 902 and target devices 906a-906n and / or Figure 1 The parameters of system 100 shown are executed by server 102 and target devices 134a-134k.

[0089] Flowchart 1100 begins at step 1102, where a master copy of the AI ​​model is stored. For example, as... Figure 1 As shown, when AI model 106 is trained, for example using dataset 114, the data manager 118 can store the master copy of AI model 106 together with its associated weights 108, activations 112 and gradients 110 at parameter server 102.

[0090] In step 1104, a portion of the AI ​​model is transmitted to the target device. For example, such as... Figure 2 As shown, transmitter 122 transmits a portion of AI model 106 from parameter server 102 to target device 134a, which may be a memory-constrained device. Therefore, in this example embodiment, target device 134a may not have sufficient memory to efficiently execute AI model 106. In another example embodiment, target device 134a may have large enough memory to store the entire AI model 106, but downloading and storing only a portion of AI model 106 as needed, rather than storing the entire instance of AI model 106, may be more efficient.

[0091] In step 1106, the appropriate micro-batch size for the target device is determined. (Refer to the above...) Figure 6 and Figure 10As described, the micro-batch size can be automatically or manually configured by the batch manager 120 at discrete points during the training of the AI ​​model 106, based on the communication rate between the target device 134a and the parameter server 102. In one embodiment, the batch manager 116 may initially select the micro-batch size based on the hardware specifications of the target device 134a and then iteratively adjust it to the optimal micro-batch size, for example, based on the computation time of the target device 134a, the size of the sub-parts of the AI ​​model 106 to be transmitted, and / or the communication bandwidth of the system 100.

[0092] Flowchart 1100 ends at step 1108. In step 1108, simultaneously, a micro-batch set of the training dataset is executed on the target device at the first sub-part of the transmission portion of the AI ​​model to generate gradients, parameter reduction is performed for the second sub-part of the transmission portion of the AI ​​model, and the weights of the third sub-part of the transmission portion of the AI ​​model are sent to the target device. For example, when the target device 134a executes a micro-batch set of dataset 114 on the first sub-part (e.g., the current layer) of AI model 106, weight updater 126 can perform parameter reduction for the second sub-part (e.g., the previous layer) of AI model 106, and simultaneously (or substantially simultaneously), transmitter 112 can send the weights of the third sub-part (e.g., the next layer) of AI model 106 to the target device 134a. For example, parameter server 102 can be configured to... Figure 12 The figure 1000 shown is used to perform these tasks.

[0093] In an example embodiment, weight updater 126 is configured to perform parameter reduction using gradients received from target device 134a, which are generated by target device 134a performing micro-batch sets of dataset 114 on a second sub-part (e.g., the previous layer) of AI model 106 at target device 134a. Weight updater 126 is also configured to generate an average of the received gradients in any manner known in the art. For example, weight updater 126 can generate an average of the received gradients by using operations and libraries provided in the AI ​​framework. Weight updater 126 can also perform other operations on and / or otherwise optimize the received gradients. Weight updater 126 is also configured to update AI model 106 using the average of the received gradients by updating weights 108.

[0094] In the example embodiment, target devices 134a-134k are configured to perform parameter reduction on gradients generated by target devices 134a-134k in a manner similar to parameter server 102. For example, output manager 154 may generate an average value of the gradients generated by target device 134a. Output manager 154 may also perform other operations on the gradients and / or optimize them in other ways.

[0095] In addition to the processes depicted in flowchart 1100, parameter server 102 can perform additional processes. Training an AI model requires computational and memory resources, and larger AI models require even more. Deep learning systems can use single-precision (i.e., 32-bit) format (a common floating-point format), double-precision (i.e., 64-bit) format, or half-precision (i.e., 16-bit) format for computational workloads, such as storing and updating data like weights, activations, and gradients. Mixed-precision methods combine the use of different numerical formats within a single computational workload. By using mixed-precision training, memory bandwidth requirements can be reduced because fewer bits can be used to store the same number of values. Computation time on the processor can also be improved, allowing the processor to provide higher throughput for reduced-precision mathematics. Furthermore, some devices and AI frameworks can include automatic support for mixed-precision methods. For example, Figure 12 A flowchart 1200 illustrates a process for providing mixed-precision training of an AI model according to an example embodiment. Based on Figure 1 Flowchart 1200 and Figure 1 The following discussion of System 100 will make other structural and operational embodiments readily apparent to those skilled in the art.

[0096] Flowchart 1200 begins at step 1202, where the weights of the fourth sub-part of the AI ​​model's transmission portion are converted to a first-precision format before being sent to the target device. For example, as... Figure 1 As shown, before the transmitter 122 sends the converted weights to the target device 134a, the precision formatter 128 can convert the weights 108 of the AI ​​model 106 into a first-precision (e.g., half-precision) format. For example, using a lower-precision format can reduce computation time. In the example embodiment, any precision format can be used as needed to optimize the performance of the AI ​​model 106.

[0097] In step 1204, the gradient received from the target device is converted to a second-precision format. For example, as... Figure 1 As shown, precision formatter 128 is configured to convert gradients received from target device 134a into a second-precision (e.g., single-precision) format. In an example embodiment, the gradient-to-second-precision format conversion may be performed before or after certain operations on the received gradient (e.g., summation, averaging, etc.). In other embodiments, the received gradient may be simply converted to a second-precision format before being stored as gradient 110 in memory 104.

[0098] In step 1206, the transformed gradient is used to update the weights. For example, as... Figure 13As shown, the weights 108 of the AI ​​model 106 can be updated by the weight updater 126 using the transformed gradient.

[0099] In embodiments, flowchart 1200 can be performed with fewer or more steps, or with different steps than those shown. For example, different mixed-precision methods can be utilized with different levels of precision. For instance, for training iterations of a sub-part (e.g., layer) of AI model 106, weights 108 can be converted to half-precision format for forward propagation, and the generated activations can also be maintained in half-precision format. In backward propagation, weights 108 can be maintained in half-precision format along with the generated gradients. Once the average gradient has been computed, it can be converted to single-precision format before updating the weights 108 of AI model 106. Many other operational embodiments can be implemented using system 100 for a variety of reasons. For example, weight updates (e.g., weight gradient multiplied by the learning rate) may become too small to be represented in half-precision to maintain model accuracy. Single-precision or double-precision formats may result in more computation time and / or resources for training the model.

[0100] Parameter server 102 can execute additional processes to manage target devices 134a-134k. For example, Figure 13 A flowchart 1300 illustrates a process for training an AI model using multiple target devices according to an example embodiment. Based on... Figure 1 Flowchart 1300 and Figure 9 The following discussion of System 100 will make other structural and operational embodiments readily apparent to those skilled in the art.

[0101] Flowchart 1300 begins at step 1302, where another portion of the AI ​​model is transmitted to another target device. For example, transmitter 122 could transmit to another target device (such as...) Figure 14 The target device 906n shown transmits another part of the AI ​​model 106. In the example embodiment, multiple target devices can be used to accelerate the training time of the AI ​​model. System 900 may include any number of target devices, from one to multiple, each of which is communicatively connected to parameter server 902 via one or more suitable interfaces (e.g., PCI or PCIe).

[0102] In step 1304, gradients are received from another target device to perform parameter reduction for another part of the AI ​​model. Continuing the example of step 1302, the target device 906n can send gradients to the output data manager 124 at the parameter server 902 for use in a portion of the AI ​​model 106 received and executed on the target device 906n.

[0103] C. Dynamic multi-layer execution of artificial intelligence modeling

[0104] Another significant advantage of the execution paradigm above (i.e., executing the dataset in micro-batches on a sub-part of the AI ​​model at a time) is that it only requires statically defining the sub-part or a portion thereof (e.g., a layer or sublayer), rather than the entire model computation graph as conventionally required. Therefore, the number of layers within the AI ​​model can be dynamically modified based on any number of factors, such as performance, alternative datasets, or other statistical observations.

[0105] Novel models based on Neural Architecture Search (NAS) and its probabilistic counterparts are emerging, and dynamically executed frictionless methods offer improved modeling techniques, which are currently very challenging to develop. NAS is a technique or algorithm that searches for the optimal neural network architecture based on a set of defined building blocks that can be used to build the neural network. These building blocks can be sampled and pieced together to construct a network similar to other known networks in the field, but can include different combinations and configurations of building blocks. NAS-built networks can be trained and tested, and the building blocks can be tuned based on the test results. NAS-built networks can be improved through operations such as adding layers, removing layers, or otherwise modifying layers.

[0106] Therefore, the technology that allows the number of layers within an AI model to be dynamically modified based on any number of factors is highly beneficial in NAS and other application areas. For example, Figure 1 A flowchart 1400 illustrating a process for dynamically executing AI modeling, according to an example embodiment, is shown. (Although references are provided...) Figure 14 The system 100 is described, but Figure 1 The method is not limited to this system. Based on... Figure 1 The following discussion of system 100 will reveal other structural and operational embodiments to those skilled in the art. Flowchart 1400 may include, for example, those already referenced above. Figure 2 , Figure 6 and Figure 1 The steps described herein will not be described in detail below for the sake of brevity.

[0107] Flowchart 1400 begins at step 1402, where a master copy of the AI ​​model is stored in the parameter server. For example, as... Figure 1 As shown, the data manager is configured to store a primary copy of the AI ​​model 106 in the memory 104 of the parameter server 102.

[0108] In step 1404, a micro-batch size suitable for the target device is determined. For example, batch manager 120 may be configured to determine a micro-batch size suitable for target device 134a. In an example embodiment, target device 134a may be a memory-constrained device, such that the memory of target device 134a may be insufficient to efficiently execute AI model 106. In an alternative embodiment, target device 134a may be able to accommodate the entire AI model 106. However, in this embodiment, it may be more efficient or preferable to download and store only a portion of AI model 106 at a given time, rather than the entire instance of AI model 106.

[0109] In step 1406, a portion of the AI ​​model is transmitted to the target device. For example, transmitter 122 may be configured to transmit a portion of AI model 106 to target device 120b.

[0110] In step 1408, output data can be received from the target device. This output data is generated by executing a micro-batch set of the dataset on the target device at a sub-part of the AI ​​model's transmission portion. For example, the output data manager 124 can be configured to receive data from... Figure 15 The target device 134a receives the output. The output data can be generated by executing a set of micro-batches of a dataset (e.g., dataset 114) on a sub-part (e.g., layer or sub-layer) of the transport portion (e.g., layer or sub-layer) of the AI ​​model 106 at the target device 134a. The output data can be, for example, activations and gradients generated on the forward and backward passes for inference or for training the AI ​​model 106. In an example embodiment, the target device 134a can send a signal instead of output data indicating that a set of micro-batches has been executed at the target device 134a. The parameter server 102 can then be configured to perform actions (e.g., execute subsequent steps) based on this signal rather than on the output data.

[0111] Flowchart 1400 ends at step 1410. In step 1410, the AI ​​model is evaluated based on one or more metrics to determine whether any changes to the execution of the AI ​​model are needed. For example, model evaluator 130 may be configured to evaluate AI model 106 based on one or more metrics to determine whether any changes to the execution of AI model 106 are needed, such as dynamically increasing or decreasing the number of layers to be executed.

[0112] One or more metrics can be based on any number of factors, such as current performance, alternative datasets, or other statistical observations. In an example embodiment, one or more metrics include the accuracy statistics of gradients and weights of sub-parts (e.g., layers or sub-layers) of the transport portion of the AI ​​model 106. For example, Figure 1A flowchart illustrating a process for determining the computational accuracy of dynamic execution used for AI modeling, according to an example embodiment, is shown. (Although references are provided...) Figure 15 The system 100 is described, but Figure 1 The method is not limited to this system. Based on... Figure 16 The following discussion of System 100 will make other structural and operational embodiments clear to those skilled in the art.

[0113] Flowchart 1500 includes step 1502. In step 1502, the computational precision to be applied to the execution of the next microbatch set is determined. For example, precision formatter 128 may be configured to determine the computational precision to be applied to the execution of the next microbatch set. The computational precision may be a fixed-point representation of gradients and weights (e.g., 16-bit, 32-bit, 64-bit). For example, to improve energy efficiency or data bit storage or for any other reason, precision formatter 128 may be configured to determine an appropriate precision to apply to the execution of the next microbatch set at target device 134a. In an example embodiment, precision formatter 128 may be configured to detect vanishing gradients and adjust the computational precision accordingly without requiring the implementation of any specific method.

[0114] In another embodiment, one or more metrics include an accuracy measurement of the AI ​​model. For example, Figure 1 A flowchart 1600 illustrates the process of determining whether to stop or continue the execution of an AI model based on its accuracy. (Although referenced...) Figure 16 The system 100 is described, but Figure 1 The method is not limited to this system. Based on... Figure 1 The following discussion of System 100 will make other structural and operational embodiments readily apparent to those skilled in the art.

[0115] Flowchart 1600 includes step 1602. In step 1602, the execution of the AI ​​model stops when the accuracy of the AI ​​model exceeds a predetermined threshold, or continues when the accuracy of the AI ​​model does not exceed the predetermined threshold. For example, Figure 17The model accessor 130 is configured to determine whether to stop or continue executing the AI ​​model based on its accuracy. Specifically, the model accessor 130 can stop the execution of the AI ​​model when its accuracy exceeds a predetermined threshold, or it can continue the execution of the AI ​​model when its accuracy does not exceed the predetermined threshold. For example, the accuracy measurement of the AI ​​model 106 could be classification accuracy, which is the ratio of the number of correctly predicted data to the total number of input samples. In an example embodiment, the execution of the AI ​​model 106 can be paused when its accuracy measurement exceeds a predetermined threshold, which can be any predefined value, such as 95%. According to this example embodiment, the execution of the AI ​​model 106 can continue when its accuracy measurement does not exceed the predetermined threshold of 95%, for example, when its accuracy is 80%. In an example embodiment, the continuation of the AI ​​model 106's execution can be dynamically determined, for example, for some number of layers or until the next evaluation of the AI ​​model 106. In an example embodiment, AI model 106 may be executed one sub-part (e.g., layer or sub-layer) at a time, and AI model 106 may be evaluated after each sub-part is executed.

[0116] Other metrics can be used to evaluate AI model 106, such as log loss, metrics derived from the confusion matrix, area under the curve, Fl score, mean absolute error, and mean squared error. When using other metrics, appropriate thresholds for each metric can be determined and applied to the evaluation of AI model 106. Other factors, such as new datasets being used, can lead to changes in the evaluation and / or performance of AI model 106.

[0117] In the foregoing discussion of flowcharts 200, 300, 600-800, and 1100-1600, it should be understood that sometimes these steps may be performed in a different order or even simultaneously with other steps. Other operational embodiments will be readily apparent to those skilled in the art. It should also be noted that the foregoing general description of the operation of systems 100 and 900 is for illustrative purposes only, and embodiments of systems 100 and 900 may include different hardware and / or software and may operate in a manner different from that described above.

[0118] III. Example Computer System Implementation

[0119] Each of the parameter server 102, target devices 134a-134k, parameter server 904, target devices 906a-906n, and flowcharts 200, 300, 600-800, and / or 1100-1600 can be implemented in hardware or in combination with software and / or firmware. For example, the parameter server 102, target devices 134a-134k, parameter server 904, target devices 906a-906n, and flowcharts 200, 300, 600-800, and / or 1100-1600 can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, parameter server 102, target devices 134a-134k, parameter server 904, and target devices 906a-906n, as well as flowcharts 200, 300, 600-800, and / or 1100-1600, can be implemented as hardware logic / circuit systems.

[0120] For example, in one embodiment, parameter server 102, target devices 134a-134k, parameter server 904, and target devices 906a-906n, as well as one or more of flowcharts 200, 300, 600-800, and / or 1100-1600, can be implemented together in any combination within the SoC. The SoC may include an integrated circuit chip comprising one or more of the following: a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or other circuitry, and may optionally execute received program code and / or include embedded firmware to perform functions.

[0121] Figure 17 Exemplary implementations of computing device 1700 in which embodiments can be implemented are depicted. For example, parameter server 102, target devices 134a-134k, parameter server 904, and target devices 906a-906n can each be implemented in one or more computing devices similar to computing device 1700 in fixed or mobile computer embodiments, including one or more features and / or alternative features of computing device 1700. The description of computing device 1700 provided herein is provided for illustrative purposes and is not intended to be limiting. Embodiments can be implemented in other types of computer systems, as known to those skilled in the art.

[0122] like Figure 17As shown, computing device 1700 includes one or more processors (referred to as processor circuitry 1702), system memory 1704, and a bus 1706 coupling various system components, including system memory 1704, to processor circuitry 1702. Processor circuitry 1702 is an electrical and / or optical circuit implemented as a central processing unit (CPU), microcontroller, microprocessor, and / or other physical hardware processor circuitry in one or more physical hardware circuitry device elements and / or integrated circuit devices (semiconductor chip or die). Processor circuitry 1702 can execute program code stored in a computer-readable medium, such as operating system 1730, application program 1732, other program 1734, etc. Bus 1706 represents one or more types of bus architectures, including memory bus or memory controller, peripheral bus, accelerated graphics port, and processor or local bus using any of the various bus architectures. System memory 1704 includes read-only memory (ROM) 1708 and random access memory (RAM) 1710. The Basic Input / Output System 1712 (BIOS) is stored in ROM 1708.

[0123] The computing device 1700 also includes one or more of the following drives: a hard disk drive 1714 for reading from and writing to a hard disk, a disk drive 1716 for reading from or writing to a removable disk 1718, and an optical disc drive 1720 for reading from or writing to a removable optical disc 1722, such as a CD-ROM, DVD-ROM, or other optical media. The hard disk drive 1714, disk drive 1716, and optical disc drive 1720 are connected to the bus 1706 via a hard disk drive interface 1724, a disk drive interface 1726, and an optical disc drive interface 1728, respectively. The drives and their associated computer-readable media provide the computer with non-volatile storage of computer-readable instructions, data structures, program modules, and other data. Although hard disks, removable disks, and removable optical discs have been described, other types of hardware-based computer-readable storage media, such as flash memory cards, digital video disks, RAM, ROM, and other hardware storage media, can be used to store data.

[0124] Multiple program modules may be stored on a hard disk, magnetic disk, optical disk, ROM, or RAM. These programs include an operating system 1730, one or more application programs 1732, other programs 1734, and program data 1736. Application programs 1732 or other programs 1734 may include, for example, computer program logic (e.g., computer program code or instructions) for implementing parameter server 102, target devices 134a-134k, parameter server 904, and target devices 906a-906n, as well as flowcharts 200, 300, 600-800, and / or 1100-1600 (including any suitable steps of flowcharts 200, 300, 600, and / or 1100-1600), and / or other embodiments described herein.

[0125] Users can input commands and information into computing device 1700 using input devices such as keyboard 1738 and pointing device 1740. Other input devices (not shown) may include microphone, joystick, game controller, satellite antenna, scanner, touchscreen and / or touchpad, voice recognition system for receiving voice input, gesture recognition system for receiving gesture input, etc. These and other input devices are typically connected to processor circuitry 1702 via serial port interface 1742, which is coupled to bus 1706, but may also be connected via other interfaces such as parallel port, game port, or Universal Serial Bus (USB).

[0126] Display screen 1744 is also connected to bus 1706 via an interface (such as video adapter 1746). Display screen 1744 may be external to computing device 1700 or incorporated into computing device 1700. Display screen 1744 may display information and a user interface for receiving user commands and / or other information (e.g., via touch, finger gestures, virtual keyboard, etc.). In addition to display screen 1744, computing device 1700 may include other peripheral output devices (not shown), such as speakers and printers.

[0127] Computing device 1700 is connected to network 1748 (e.g., the Internet) via an adapter or network interface 1750, modem 1752, or other means for establishing communication on the network. Modem 1752 (which may be internal or external) may be connected to bus 1706 via serial port interface 1742, such as... ​ As shown in the diagram, another interface type (including the parallel interface) can be used to connect to bus 1706.

[0128] As used herein, the terms “computer program medium,” “computer-readable medium,” and “computer-readable storage medium” are used to refer to physical hardware media, such as a hard disk associated with hard disk drive 1714, removable disk 1718, removable optical disk 1722, other physical hardware media such as RAM, ROM, flash memory cards, digital video disks, zip disks, MEM, nanotechnology-based storage devices, and other types of physical / tangible hardware storage media. Such computer-readable storage media are distinct from and do not overlap with communication media (excluding communication media). Communication media contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves. The term “modulated data signal” refers to a signal whose characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, communication media include wireless media such as acoustic, RF, infrared, and other wireless media, as well as wired media. Embodiments also relate to such communication media that are separate from and do not overlap with embodiments relating to computer-readable storage media.

[0129] As described above, computer programs and modules (including application program 1732 and other programs 1734) can be stored on a hard disk, magnetic disk, optical disk, ROM, RAM, or other hardware storage media. Such computer programs can also be received via network interface 1750, serial port interface 1742, or any other interface type. When executed or loaded by an application, such computer programs enable computing device 1700 to implement the features of the embodiments described herein. Therefore, such computer programs represent a controller for computing device 1700.

[0130] The embodiments also relate to computer program products that include computer code or instructions stored on any computer-readable medium. Such computer program products include hard disk drives, optical disk drives, storage device packages, portable memory sticks, memory cards, and other types of physical storage hardware.

[0131] IV. Additional Example Implementations

[0132] This document describes a system. In one embodiment, the system includes: a parameter server communicatively connected to a target device, the parameter server including: a data manager configured to store a master copy of an artificial intelligence (AI) model; a batch manager configured to determine micro-batch sizes suitable for the target device; a transmitter configured to transmit a portion of the AI ​​model to the target device; an output data manager configured to receive output data from the target device, the output data being generated by executing a collection of micro-batches of a dataset on the target device at a sub-part of the transmission portion of the AI ​​model; and a model evaluator configured to evaluate the AI ​​model based on one or more metrics to determine whether any changes need to be performed on the AI ​​model.

[0133] In the aforementioned system embodiments, the execution of micro-batch sets is used for inference or for training AI models.

[0134] In another embodiment of the aforementioned system, the micro-batch set includes multiple micro-batches configured to be executed sequentially, and the micro-batch set forms a mini-batch, which includes a number of samples for training updated each time or a number of samples for inference provided in each inference cycle.

[0135] In another embodiment of the aforementioned system, one or more metrics include the precision statistics of gradients and weights of sub-parts of the transmission part of the AI ​​model.

[0136] In another embodiment of the aforementioned system, the parameter server further includes a precision formatter configured to determine the computational precision to be applied to the execution of the next micro-batch set.

[0137] In an additional embodiment of the aforementioned system, one or more metrics include an accuracy measurement of the AI ​​model.

[0138] In another embodiment of the aforementioned system, the parameter server further includes a model evaluator configured to: stop the execution of the AI ​​model when the accuracy measurement of the AI ​​model exceeds a predetermined threshold; or continue the execution of the AI ​​model when the accuracy measurement of the AI ​​model does not exceed the predetermined threshold.

[0139] This paper describes a method implemented in a parameter server. The method includes: storing a master copy of an artificial intelligence (AI) model in the parameter server; determining a micro-batch size suitable for a target device; transmitting a portion of the AI ​​model to the target device; receiving output data from the target device, which is generated by executing a collection of micro-batches of a dataset on the target device at a sub-part of the transmitted portion of the AI ​​model; and evaluating the AI ​​model based on one or more metrics to determine if any changes need to be performed on the AI ​​model.

[0140] In one embodiment of the aforementioned method, the micro-batch set includes multiple micro-batches configured sequentially, and the micro-batch set includes mini-batches, which form a set of samples for training updated each time or samples for inference provided in each inference cycle.

[0141] In another embodiment of the aforementioned method, one or more metrics include the precision statistics of gradients and weights of sub-parts of the transmission part of the AI ​​model.

[0142] One embodiment of the aforementioned method also includes determining the computational precision to be applied to the execution of the next microbatch set.

[0143] In another embodiment of the aforementioned method, one or more metrics include an accuracy measurement of the AI ​​model.

[0144] In another embodiment of the aforementioned method, the micro-batch size is configurable based on the execution rate of the micro-batch set at the target device and the communication rate between the target device and the parameter server.

[0145] Another embodiment of the aforementioned method further includes: stopping the execution of the AI ​​model when the accuracy measurement of the AI ​​model exceeds a predetermined threshold; or continuing the execution of the AI ​​model when the accuracy measurement of the AI ​​model does not exceed the predetermined threshold.

[0146] This document also describes a computer program product. The computer program product includes a computer-readable storage device on which computer program logic is recorded, the computer program logic causing the processor-based computer system, when executed, to perform a method comprising: storing a master copy of an artificial intelligence (AI) model in a parameter server; determining a micro-batch size suitable for a target device; transferring a portion of the AI ​​model from the parameter server to the target device; receiving output data from the target device, the output data being generated by executing a set of micro-batches of a dataset on the target device at a sub-part of the transfer portion of the AI ​​model; and evaluating the AI ​​model based on a metric to determine whether to continue executing the AI ​​model.

[0147] In the aforementioned embodiments of the computer program product, micro-batch sets are executed for inference or for training AI models.

[0148] In one embodiment of the aforementioned computer program product, the micro-batch set includes multiple micro-batches configured to be executed sequentially, and the micro-batch set forms a mini-batch, which includes several samples for training updated each time or several samples for inference provided in each inference cycle.

[0149] In an additional embodiment of the aforementioned computer program product, one or more metrics include precision statistics of gradients and weights of sub-parts of the transmission portion of the AI ​​model.

[0150] In another embodiment of the aforementioned computer program product, the method further includes determining the computational precision to be applied to the execution of the next micro-batch set.

[0151] In an additional embodiment of the aforementioned computer program product, one or more metrics include an accuracy measurement of the AI ​​model; the method further includes: stopping the execution of the AI ​​model when the accuracy measurement of the AI ​​model exceeds a predetermined threshold; or continuing the execution of the AI ​​model when the accuracy measurement of the AI ​​model does not exceed the predetermined threshold.

[0152] V. Conclusion

[0153] While various embodiments of the disclosed subject matter have been described above, it should be understood that they are presented as examples only and not as limitations. Those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of the embodiments as defined in the appended claims. Therefore, the breadth and scope of the disclosed subject matter should not be limited to any of the exemplary embodiments described above, but should be defined solely by the appended claims and their equivalents.

Claims

1. A system comprising: A parameter server, communicatively connected to the target device, includes: The data manager is configured to store a primary copy of the artificial intelligence (AI) model; A transmitter is configured to transmit a portion of the AI ​​model to the target device; An output data manager is configured to receive output data from the target device, the output data being generated by performing a collection of micro-batches of a dataset on a sub-part of the transmitted portion of the AI ​​model on the target device; A model evaluator is configured to evaluate the AI ​​model based on one or more metrics corresponding to the output data to determine changes to the AI ​​model. The output data manager is also configured to modify the AI ​​model to incorporate the determined changes; and The transmitter is further configured to send an update to the target device simultaneously with the execution of a second micro-batch set of the dataset on a second sub-part of the transmitted portion of the AI ​​model at the target device, the update indicating that the modified AI model should be applied during the execution of a third micro-batch set of the dataset.

2. The system of claim 1, wherein the micro-batch set is executed for inference or for training the AI ​​model.

3. The system of claim 1, wherein the micro-batch set comprises a plurality of micro-batches configured to be executed sequentially, the micro-batch set forming a mini-batch, the mini-batch comprising a plurality of samples for training updated each time or a plurality of samples for inference provided in each inference cycle.

4. The system of claim 1, wherein the one or more metrics include precision statistics of gradients and weights for the sub-parts of the transmitted portion of the AI ​​model.

5. The system of claim 4, wherein the parameter server further includes a precision formatter configured to determine the computational precision to be applied to the execution of the next micro-batch set.

6. The system of claim 1, wherein the one or more metrics include an accuracy measurement of the AI ​​model.

7. The system of claim 6, wherein the parameter server further comprises a model evaluator configured to: When the accuracy metric of the AI ​​model exceeds a predetermined threshold, the execution of the AI ​​model is stopped; or If the accuracy metric of the AI ​​model does not exceed the predetermined threshold, the execution of the AI ​​model continues.

8. A method implemented in a parameter server, comprising: Store the master copy of the artificial intelligence (AI) model on the parameter server; Transmit a portion of the AI ​​model to the target device; Output data is received from the target device, the output data being generated by performing a set of micro-batches of a dataset on a sub-part of the transmitted portion of the AI ​​model on the target device; The AI ​​model is evaluated based on one or more metrics corresponding to the output data to determine changes to the AI ​​model; Modify the AI ​​model to incorporate the determined changes; as well as While the second micro-batch set of the dataset is being executed on the target device at the second sub-part of the portion of the transmitted AI model, an update is sent to the target device, the update indicating that the modified AI model should be applied when executing the third micro-batch set of the dataset.

9. The method of claim 8, wherein the micro-batch set is executed for inference or for training the AI ​​model.

10. The method of claim 8, wherein the micro-batch set comprises a plurality of micro-batches configured to be executed sequentially, the micro-batch set forming a mini-batch, the mini-batch comprising a plurality of samples for training updated each time or a plurality of samples for inference provided in each inference cycle.

11. The method of claim 8, wherein the one or more metrics include precision statistics of gradients and weights for the sub-parts of the transmitted portion of the AI ​​model.

12. The method of claim 11, further comprising: Determine the computational precision to be applied to execute the next microbatch set.

13. The method of claim 12, wherein the one or more metrics include an accuracy measurement of the AI ​​model.

14. The method of claim 13, further comprising: When the accuracy metric of the AI ​​model exceeds a predetermined threshold, the execution of the AI ​​model is stopped. or If the accuracy metric of the AI ​​model does not exceed the predetermined threshold, the execution of the AI ​​model continues.

15. A computer program product including a computer-readable storage device having computer program logic recorded thereon, the computer program logic causing the processor-based computer system to perform a method when executed by a processor-based computer system, the method comprising: Store the master copy of the artificial intelligence (AI) model on the parameter server; A portion of the AI ​​model is transmitted from the parameter server to the target device; Output data is received from the target device, the output data being generated by performing a set of micro-batches of a dataset on a sub-part of the transmitted portion of the AI ​​model on the target device; The AI ​​model is evaluated based on one or more metrics corresponding to the output data to determine changes to the AI ​​model; Modify the AI ​​model to incorporate the determined changes; as well as While the second micro-batch set of the dataset is being executed on the target device at the second sub-part of the portion of the transmitted AI model, an update is sent to the target device, the update indicating that the modified AI model should be applied when executing the third micro-batch set of the dataset.

16. The computer program product of claim 15, wherein the execution of micro-batch sets is used for inference or for training the AI ​​model.

17. The computer program product of claim 15, wherein the micro-batch set comprises a plurality of micro-batches configured to be executed sequentially, the micro-batch set forming a mini-batch, the mini-batch comprising a plurality of samples for training updated each time or a plurality of samples for inference provided in each inference cycle.

18. The computer program product of claim 15, wherein the one or more metrics include precision statistics of gradients and weights for the sub-parts of the transmitted portion of the AI ​​model.

19. The computer program product of claim 18, wherein the method further comprises: Determine the computational precision to be applied to execute the next microbatch set.

20. The computer program product of claim 19, wherein the one or more metrics include an accuracy measurement of the AI ​​model; The method further includes: When the accuracy metric of the AI ​​model exceeds a predetermined threshold, the execution of the AI ​​model is stopped. or If the accuracy metric of the AI ​​model does not exceed the predetermined threshold, the execution of the AI ​​model continues.

Images