Accelerating deep learning with inter-iteration scheduling

Abstract

An apparatus relates to accelerating deep learning with inter-iteration scheduling based on operational classification associated with deep learning. The device comprises an interface circuit module (1120) and a programmable circuit module. And instructions to cause the programmable circuit module to: classify an operational group of the distributed deep learning workload based on a resource utilization of the operational group; selecting at least two operations in the operation group for overlapping execution based on the classification and dependency analysis of the at least two operations in the operation group; and performing distributed training of the distributed deep learning workload based on an execution schedule including overlapping execution of the selected at least two operations.

Description

Technical Field

[0001] The present disclosure relates generally to machine learning, and more particularly, to methods and apparatus for accelerating deep learning using inter-iteration scheduling based on classification of operations associated with deep learning. Background Art

[0002] Machine learning is a subfield of artificial intelligence. In machine learning, instead of providing explicit instructions, the programmer supplies data to the model. The model generates predictions and, in some examples, is trained to improve prediction accuracy. The programmer can also adjust model parameters to further improve prediction accuracy. A deep neural network (DNN) model is a machine learning model based on an artificial neural network. DNNs can be trained across multiple computing units in distributed training. In distributed training, the workload is split between multiple computing units: CPUs, GPUs, TPUs, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 is an example illustration of a system for accelerating deep learning.

[0004] Figure 2 is a block diagram of an example deep learning accelerator circuit module.

[0005] Figure 3 is an example execution schedule for training deep learning workloads.

[0006] Figure 4 After the operation classification Figure 3 The example performs scheduling.

[0007] Figure 5 It follows the segment that identifies the execution schedule for overlapping execution. Figure 4 The example performs scheduling.

[0008] Figure 6 is an example processor circuit module that can be executed to implement Figure 2 A flowchart of example machine-readable instructions and / or example operations of a deep learning accelerator circuit module.

[0009] Figure 7 is a flowchart representative of example machine readable instructions and / or example operations that may be executed by an example processor circuit module to classify operations for a distributed deep learning workload.

[0010] Figure 8 is a flow diagram representative of example machine readable instructions and / or example operations executable by an example processor circuit module to identify overlapping operations in a distributed deep learning workload.

[0011] Fig. 9is a flow diagram representative of example machine readable instructions and / or example operations executable by example processor circuit modules to implement inter / intra-iteration overlap.

[0012] Fig.10 is a flow diagram representative of example machine-readable instructions and / or example operations executable by example processor circuit modules to perform staleness-aware optimization of distributed deep learning workloads.

[0013] Fig.11 is a block diagram of an example processing platform including a processor circuit module configured to perform Figure 6-10 Example machine readable instructions and / or example operations to implement Figure 2 Deep learning accelerator circuit module.

[0014] Fig.12 yes Fig.11 A block diagram of an example implementation of a processor circuit module.

[0015] Fig.13 yes Fig.11 A block diagram of another example implementation of a processor circuit module.

[0016] Fig.14 is a block diagram of an example software distribution platform (e.g., one or more servers) for distributing software (e.g., Figure 6-10 The software corresponding to example machine-readable instructions of the present invention is distributed to client devices associated with end users and / or consumers (e.g., for licensing, sale, and / or use), retailers (e.g., for sale, resale, licensing, and / or sublicensing), and / or original equipment manufacturers (OEMs) (e.g., for inclusion in products to be distributed to, for example, retailers and / or to other end users, such as direct purchase customers).

[0017] In general, throughout the (one or more) drawings and the accompanying written description, the same reference numerals will be used to refer to the same or similar parts. The drawings are not to scale. Alternatively, the thickness of a layer or region may be exaggerated in the drawings. Although the drawings show layers and regions with clear lines and boundaries, some or all of these lines and / or boundaries may be idealized. In practice, boundaries and / or lines may be unobservable, mixed and / or irregular.

[0018] As used herein, unless otherwise specified, the term "above" describes the relationship of two parts relative to the ground. A first part is above a second part if the second part has at least one portion between the ground and the first part. Likewise, as used herein, a first part is "below" a second part when the first part is closer to the ground than the second part. As described above, a first part can be above or below a second part when one or more of the following are present: there are other parts between them, there are no other parts between them, wherein the first part and the second part are in contact, or the first part and the second part are not in direct contact with each other.

[0019] Notwithstanding the foregoing, in the context of semiconductor devices, "above" is not referenced to the ground but instead refers to a bulk region of a base semiconductor substrate (e.g., a semiconductor wafer) on which components of the integrated circuit are formed. Specifically, as used herein, a first component of an integrated circuit is "above" a second component when the first component is further away from the bulk region of the semiconductor substrate than the second component.

[0020] As used in this patent, stating that any part (e.g., a layer, film, portion, region, or plate) is in any way on another part (e.g., positioned on another part, located on another part, disposed on another part, or formed on another part, etc.) indicates that either the referenced part is in contact with the other part, or the referenced part is above the other part, with one or more intermediate parts located between the referenced part and the other part.

[0021] As used herein, unless otherwise indicated, connection references (e.g., attachment, coupling, connection, and engagement) may include intermediate members between elements referenced by the connection reference and / or relative movement between those elements. Thus, a connection reference does not necessarily imply that two elements are directly connected and / or are in a fixed relationship to each other. As used herein, stating that any part is "in contact" with another part is defined to mean that there is no intermediate part between the two parts.

[0022] Unless otherwise specifically stated, descriptors such as "first," "second," "third," etc., used herein are not intended to attribute or otherwise indicate any meaning of priority, physical order, arrangement in a list, and / or ordering in any manner, but are merely used as labels and / or arbitrary names to distinguish elements to facilitate understanding of the disclosed examples. In some examples, the descriptor "first" may be used to refer to an element in the detailed description, while a different descriptor such as "second" or "third" may be used to refer to the same element in the claims. In such instances, it should be understood that such descriptors are only used to clearly identify those elements that may otherwise share the same name, for example.

[0023] As used herein, "approximately" and "about" modify their subject matter / values ​​to recognize the potential for variations that occur in real-world applications. For example, as will be understood by one of ordinary skill in the art, "approximately" and "about" may modify dimensions that may not be exact due to manufacturing tolerances and / or other real-world imperfections. For example, "approximately" and "approximately" may indicate that such a dimension may be within a tolerance range of + / - 10%, unless otherwise specified in the description below. As used herein, "substantially real time" means occurring in a near instantaneous manner, recognizing that there may be real-world delays in computing time, transmission, etc. Therefore, unless otherwise specified, "substantially real time" means real time + / - 1 second.

[0024] As used herein, the phrase "in communication" (including variations thereof) encompasses direct communication and / or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or constant communication, but instead also includes selective communication at periodic intervals, scheduled intervals, non-periodic intervals and / or one-time events.

[0025] As used herein, a "processor circuit module" is defined to include: (i) one or more dedicated circuits configured to perform (one or more) specific operations and including one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors); and / or (ii) one or more general-purpose semiconductor-based circuits, programmable with instructions for performing specific operations and including one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors). Examples of processor circuit modules include programmable microprocessors, field programmable gate arrays (FPGAs) that can instantiate instructions, central processor units (CPUs), graphics processor units (GPUs), digital signal processors (DSPs), XPUs or microcontrollers, and integrated circuits such as application-specific integrated circuits (ASICs). For example, an XPU may be implemented by a heterogeneous computing system that includes multiple types of processor circuit modules (e.g., one or more FPGAs, one or more CPUs, one or more GPUs, one or more DSPs, etc. and / or combinations thereof) and (one or more) application programming interfaces (APIs) that can assign (one or more) computing tasks to whichever processing circuit module(s) of the multiple types of processor circuit modules is most suitable for performing the (one or more) computing tasks. DETAILED DESCRIPTION

[0026] Artificial intelligence (AI), including machine learning (ML), deep learning (DL), and / or other artificial machine-driven logic, enables a machine (e.g., a computer, logic circuit, etc.) to process input data using a model to generate outputs based on patterns and / or associations previously learned by the model via a training process. For example, a model can be trained with data to recognize patterns and / or associations, and follow such patterns and / or associations when processing input data so that (one or more) other inputs result in (one or more) outputs consistent with the recognized patterns and / or associations.

[0027] In general, implementing an ML / AI system involves two phases, a learning / training phase and an inference phase. In the learning / training phase, a training algorithm is used to train the model to operate according to patterns and / or associations based on, for example, training data. In general, the model includes internal parameters that guide how input data is transformed into output data, such as through a series of nodes and connections within the model. In addition, hyperparameters are used as part of the training process to control how learning is performed (e.g., learning rate, number of layers to be used in the machine learning model, etc.). Hyperparameters are defined as training parameters that are determined before the training process is started.

[0028] Different types of training can be performed based on the type and / or expected output of the ML / AI model. For example, supervised training uses inputs and corresponding expected (e.g., labeled) outputs to select parameters for the ML / AI model that reduce model error (e.g., by iteratively selecting combinations of parameters). As used herein, labeling refers to the expected output of the machine learning model (e.g., classification, expected output value, etc.). Alternatively, unsupervised training (e.g., for deep learning, subsets of machine learning, etc.) involves inferring patterns from inputs to select parameters for the ML / AI model (e.g., without the benefit of expected (e.g., labeled) outputs).

[0029] Deep learning is an ML method based on learning data representations, rather than task-specific processes. Deep learning models attempt to define the relationship between input data and associated output data. Deep learning is computationally intensive and requires a lot of processing power and resources. In recent years, deep neural network (DNN) workloads have increased in scope and complexity. Therefore, it is challenging to train large DNN models on a single chip. Similarly, existing solutions for training large DNN models on multiple chips have shortcomings and fail to address the various technical challenges associated with such training (e.g., scalability and hardware resource utilization efficiency). As DNN workloads and models require growing computing power, training DNNs on a single chip becomes increasingly challenging.

[0030] DNN workloads can be understood in terms of execution graphs. An execution graph is a directed acyclic graph (DAG) where nodes represent computations and edges between nodes represent execution dependencies. Training a neural network is more computationally intensive than inference for a given neural network because the execution graph used for training includes forward propagation operations (e.g., forward pass) for computing losses and backward propagation operations (e.g., backward pass) for computing gradients. The operations of the computation graph can be executed based on a topological sort, but such execution scheduling may not take advantage of parallel execution opportunities.

[0031] Multi-chip DNN training can alleviate the problems faced by single-chip architectures, but the distribution of DNN training introduces additional computational overhead. Scheduling training operations in a way that provides high hardware utilization efficiency can also be difficult.

[0032] The examples disclosed herein schedule distributed deep learning operations according to the computational characteristics of the distributed deep learning operations. In some examples, the operations are divided into one of four categories: compute-constrained, memory-constrained, I / O-constrained, and network-constrained. Next, inter-iteration overlapping execution can be performed based on inter-iteration dependency analysis. Finally, a stale-aware distributed optimizer generates an execution schedule based on inter-iteration overlapping execution and identified communication operations. As described herein, overlapping execution can refer to full overlap and / or partial overlap of multiple (e.g., two or more) operations.

[0033] Some examples disclosed herein provide overlapping operation scheduling between iterations and improve distributed hardware resource utilization by assigning priorities to different operation types. Some examples disclosed herein improve DNN execution in heterogeneous computing environments. For example, a graphics processing unit (GPU) can perform compute-limited operations, while a dataflow accelerator can perform memory-limited operations.

[0034] Go to the attached picture, Figure 1 is a diagram of an example distributed computing system 100. The distributed computing system 100 includes an example deep learning accelerator circuit module 102, an example neural network 104, example first training data 106a, example second training data 106b, example third training data 106c, an example first workstation 108, an example second workstation 110, an example third workstation 112, an example fourth workstation 114, and an example network 116.

[0035] It is impractical (e.g., requires too much execution time, insufficient available memory, etc.) to train the neural network 104 using only the first workstation 108 with the example training data 106a-106c. In order to efficiently train the neural network 104, the training workload is distributed to the second workstation 110, the third workstation 112, and the fourth workstation 114. Thus, the example system 100 causes the training workload (e.g., the training data 106a-c) to be distributed among the second workstation 110, the third workstation 112, and the fourth workstation 114.

[0036] Before and / or during workload execution, neural network 104 is transmitted to one or more of second workstation 110, third workstation 112, and fourth workstation 114. By transmitting neural network 104 to each of the workstations, the workstations can each partially train neural network 104. The results of each partial training can be combined by example first workstation 108 to produce a final training model that integrates the training performed on each copy of neural network 104.

[0037] Each of the first workstation 108, the second workstation 110, the third workstation 112, and the fourth workstation 114 executes an instance of the deep learning accelerator circuit module 102. The deep learning accelerator circuit module 102 generates an execution schedule that takes into account overlapping operations between iterations and is resource contention aware. Figure 2 The structure and function of the deep learning accelerator circuit module 102 are described in association.

[0038] First workstation 108, second workstation 110, third workstation 112, and fourth workstation 114 are connected via network 116. In some examples, neural network 104 may be trained on a single machine having multiple processing elements that each handle a portion of the machine learning workload.

[0039] exist Figure 1 In the example of FIG. 1 , a separate instance of the deep learning accelerator circuit module 102 is included in each of the first workstation 108, the second workstation 110, the third workstation 112, and / or the fourth workstation 114. However, in some examples, the deep learning accelerator circuit module 102 may not be included in one or more of the first workstation 108, the second workstation 110, the third workstation 112, and / or the fourth workstation 114.

[0040] Figure 2 is a block diagram of a deep learning accelerator circuit module 102 for accelerating deep learning operations. Figure 2The deep learning accelerator circuit module 102 can be instantiated (e.g., created, generated for any length of time, concretized, implemented, etc.) by a processor circuit module such as a central processing unit that executes instructions. Additionally or alternatively, Figure 2 The deep learning accelerator circuit module 102 can be instantiated (e.g., created, generated for any length of time, materialized, implemented, etc.) by an ASIC or FPGA configured to perform operations corresponding to the instructions. It should be understood that Figure 2 Some or all of the circuit modules of can thus be instantiated at the same or different times. For example, some or all of the circuit modules can be instantiated in one or more threads that execute concurrently on the hardware and / or serially on the hardware. In addition, in some examples, Figure 2 Some or all of the circuit modules may be implemented by a microprocessor circuit module that executes instructions to implement one or more virtual machines and / or containers.

[0041] The example deep learning accelerator circuit module 102 includes an example operation classification circuit module 202. The operation classification circuit module 202 classifies operations including deep learning training execution scheduling. Example operations classified by the operation classification circuit module 202 may include data loaders (e.g., for reading training samples, customizing data loading order, batches, etc.), linear layer operations, convolutional layer operations, optimizer operations (e.g., stochastic gradient descent operations), etc.

[0042] In some examples, the operation classification circuit module 202 can classify operations as compute-bound, memory-bound, I / O-bound, and / or network-bound. A compute-bound operation is an operation for which the time to complete the operation is primarily determined by the processor circuit module. A memory-bound operation is an operation in which the time to complete the operation is primarily determined by memory speed and / or availability. An I / O-bound operation is an operation in which the time to complete the operation is primarily determined by input / output overhead. A network-bound operation is an operation in which the time to complete the operation is primarily determined by communication overhead.

[0043] The operation classification circuit module 202 can classify multiple operations of the distributed deep learning workload as network-limited, computationally limited, memory-limited, or input / output-limited based on the resource utilization of one of the two or more operations. Multiple operations classified by the example operation classification circuit module 202 can be assigned scheduling and / or execution priorities. In some examples, input / output-limited operations are assigned a higher scheduling priority than computationally limited operations. Based on testing and / or analysis of resource usage during execution of the operation, the operation can be associated with a category. Such information can be saved and used for future classification of the same operation.

[0044] The various classified operations may also be transmitted to and / or executed on different computing units. For example, compute-limited instructions may be executed on a graphics processing unit, while memory-limited instructions may be executed on a data flow accelerator unit. In some examples, the operation classification circuit module 202 is instantiated by a processor circuit module that executes operation classification instructions and / or is configured to perform operations such as those performed by Figure 6-10 The flowcharts represent the operations of those operations.

[0045] In some examples, the deep learning accelerator circuit module 102 includes a component for classifying the operation of the device. For example, the component for classification can be implemented by the operation classification circuit module 202. In some examples, the operation classification circuit module 202 can be implemented by a processor circuit module (such as Fig.11 For example, the operation classification circuit module 202 may be instantiated by a processor circuit module 1112 that executes machine executable instructions. Fig.12 The example microprocessor 1200 is instantiated by the machine executable instructions such as those provided by at least Figure 7 In some examples, the operation classification circuit module 202 may be instantiated by a hardware logic circuit module, which may be an ASIC, XPU, or other processor configured to perform operations corresponding to machine-readable instructions. Fig.13 The FPGA circuit module 1300 is implemented. Additionally or alternatively, the operation classification circuit module 202 can be instantiated by any other combination of hardware, software and / or firmware. For example, the operation classification circuit module 202 can be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, ASICs, XPUs, comparators, operational amplifiers (op-amps), logic circuits, etc.), and the at least one or more hardware circuits are configured to execute some or all of the machine-readable instructions and / or execute some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are equally suitable.

[0046] The example deep learning accelerator circuit module 102 includes an example inter-iteration scheduling circuit module 204. The example inter-iteration scheduling circuit module 204 generates an inter-iteration DAG. To perform inter-iteration analysis, two DAGs each representing a single directed graph can be combined by connecting the final operation of a forward propagation (e.g., forward pass) to the first operation for a subsequent backward propagation (e.g., backward pass). Thus, the final node and / or operation of the forward propagation can be connected to the first node and / or operation of the subsequent backward propagation layer. In examples where reasoning is performed, two DAGs corresponding to separate (e.g., independent) reasoning iterations can be connected (e.g., connected by virtual nodes).

[0047] The example inter-iteration scheduling circuit module 204 can identify operations for partially overlapping execution based on the resource type of the operation and any dependencies between the operations. For example, if a first operation is restricted by a first operation type (e.g., I / O restricted), a second operation is restricted by a second operation type (e.g., memory restricted), and there is no data dependency between the first and second operations, then the first and second operations can be classified for overlapping execution (e.g., at least partially overlapping execution). In some examples, network-restricted communication operations are prioritized for overlapping (e.g., at least partially overlapping) execution with other types of operations. In some execution DAGs, execution paths between parent nodes and child nodes that do not include network-restricted communication can be overlapped with network-restricted communication to accelerate both distributed training and heterogeneous computing.

[0048] The example inter-iteration scheduling circuit module 204 may perform some or all of the operations shown in Tables 1 and 2 below:

[0049] Table 1

[0050] Table 1 shows an example algorithm for identifying candidate operations. In Table 1, a list of communication operations is generated. For each communication operation that is not the last all-reduce operation in the DDP wrapper (e.g., a synchronization process across machines), each node in the operation DAG is analyzed to see if it falls between the parent node and child node of the communication operation. Therefore, the method of Table 1 maintains a list of candidate overlapping operations for the communication operation.

[0051] Table 2

[0052] Table 2 shows an example scheduling algorithm that can be utilized by the inter-iteration scheduling circuit module 204. The inter-iteration scheduling circuit module 204 identifies overlaps in distributed training operations for communication operations. The method of Table 2 also prunes the candidate list and reorders the operations in the candidate list according to priority (e.g., I / O-limited, compute-limited, and memory-limited). The inter-iteration scheduling circuit module 204 can also identify non-communication operations in the candidate list or the second candidate list.

[0053] In some examples, the deep learning accelerator circuit module 102 includes components for inter-iteration / intra-iteration scheduling. For example, the components for inter-iteration / intra-iteration scheduling can be implemented by the example inter-iteration scheduling circuit module 204. In some examples, the example inter-iteration scheduling circuit module 204 can be implemented by a processor circuit module (such as Fig.11 For example, the example inter-iteration scheduling circuit module 204 may be instantiated by an example processor circuit module 1112 that executes machine executable instructions. Fig.12 The example microprocessor 1200 is instantiated by the machine executable instructions such as at least Figure 6 Block 602 and / or Figure 7 In some examples, the example inter-iteration scheduling circuit module 204 may be instantiated by a hardware logic circuit module, which may be implemented by an ASIC, XPU, or other processor configured to perform operations corresponding to machine-readable instructions. Fig.13 FPGA circuit module 1300. Additionally or alternatively, the example inter-iteration scheduling circuit module 204 can be instantiated by any other combination of hardware, software and / or firmware. For example, the example inter-iteration scheduling circuit module 204 can be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, ASICs, XPUs, comparators, operational amplifiers (op-amps), logic circuits, etc.), and the at least one or more hardware circuits are configured to execute some or all of the machine-readable instructions and / or execute some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are equally suitable.

[0054] The example deep learning accelerator circuit module 102 includes an example staleness-aware distributed optimization circuit module 206. The staleness-aware distributed optimization circuit module 206 is a staleness-aware distributed optimizer based on synchronous stochastic gradient descent (S-SGD). S-SGD distributes training operations to multiple workers to speed up training. However, S-SGD also introduces communication overhead for exchanging model parameters and / or gradients in each iteration.

[0055] Synchronous S-SGD uses data parallelism to train models with multiple workers (e.g. Figure 1 The deep learning model (e.g., Figure 1 Each worker gets a portion of the data (e.g., a mini-batch), where gradient updates are performed in parallel by the workers. In some examples, the model is updated using the average gradients from the various workers.

[0056] The staleness-aware distributed optimization circuit module 206 identifies additional communication operations in some models (e.g., models with a large first layer). In models with a large first layer (e.g., a layer close to the input data), the staleness-aware distributed optimization circuit module 206 identifies additional communication overhead that does not overlap with computation.

[0057] In some examples, the deep learning accelerator circuit module 102 includes a component for performing staleness-aware distributed optimization. For example, the component for performing staleness-aware distributed optimization can be implemented by the staleness-aware distributed optimization circuit module 206. In some examples, the staleness-aware distributed optimization circuit module 206 can be implemented by a processor circuit module (such as Fig.11 For example, the staleness-aware distributed optimization circuit module 206 may be instantiated by a processor circuit module 1112 that executes machine executable instructions. Fig.12 The example microprocessor 1200 is instantiated by the machine executable instructions such as at least Figure 6 Block 604 and / or Figure 8 In some examples, the stale-aware distributed optimization circuit module 206 may be instantiated by a hardware logic circuit module, which may be implemented by an ASIC, XPU, or other processor configured to perform operations corresponding to machine-readable instructions. Fig.13 The FPGA circuit module 1300 is implemented. Additionally or alternatively, the stale-aware distributed optimization circuit module 206 can be instantiated by any other combination of hardware, software and / or firmware. For example, the stale-aware distributed optimization circuit module 206 can be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, ASICs, XPUs, comparators, operational amplifiers (op-amps), logic circuits, etc.), and the at least one or more hardware circuits are configured to execute some or all of the machine-readable instructions and / or execute some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are equally suitable.

[0058] The example deep learning accelerator circuit module 102 includes an example neural network circuit module 208. The neural network circuit module 208 implements a convolutional neural network (e.g., a deep neural network) that includes various convolutional layers, maximum pooling layers, fixed embedding layers, global average layers, etc. In some examples, the example neural network circuit module 208 may include additional and / or alternative machine learning models for predicting the class label of a given example input data. For example, the neural network circuit module 208 may interoperate with any other classification algorithm (e.g., logistic regression, naive Bayes, k-nearest neighbor, decision tree, support vector machine) to provide improved classification results.

[0059] The example neural network circuit module 208 includes a neural network training circuit module. In some examples, the neural network circuit module 208 can be initialized with random weights. The neural network circuit module 208 can then retrieve training data (e.g., labeled test data) and adjust the weights to produce results consistent with the labeled test data (e.g., minimize the loss function). The weights of the neural network circuit module 208 are adjusted based on gradient descent. However, the neural network circuit module 208 can be adjusted based on any other suitable optimization algorithm.

[0060] The example neural network circuit module 208 can retrieve training data from the example data storage device 212 and use the retrieved data to train the example neural network circuit module 208. In some examples, the neural network circuit module 208 can perform pre-processing on the training data. In some examples, the neural network circuit module 208 can deduplicate elements of the training set before training.

[0061] In some examples, the deep learning accelerator circuit module 102 includes components for implementing a neural network. For example, the components for implementing a neural network may be implemented by a neural network circuit module 208. In some examples, the neural network circuit module 208 may be implemented by a processor circuit module (such as Fig.11 For example, the neural network circuit module 208 may be instantiated by a processor circuit module 1112 that executes machine executable instructions. Fig.12 The example microprocessor 1200 is instantiated by the machine executable instructions such as at least Figure 6 In some examples, the neural network circuit module 208 may be instantiated by a hardware logic circuit module, which may be implemented by an ASIC, XPU, or other processor configured to perform operations corresponding to machine-readable instructions. Fig.13 Additionally or alternatively, the neural network circuit module 208 may be instantiated by any other combination of hardware, software, and / or firmware. For example, the neural network circuit module 208 may be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, ASICs, XPUs, comparators, operational amplifiers (op-amps), logic circuits, etc.), which are configured to execute some or all of the machine-readable instructions and / or execute some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are equally suitable.

[0062] The example deep learning accelerator circuit module 102 includes an example communication circuit module 210. The example communication circuit module 210 transmits and / or receives information associated with the example deep learning accelerator circuit module 102. For example, multiple workstations (e.g., Figure 1 The first workstation 108, the second workstation 110, the third workstation 112, and the fourth workstation 114) can communicate with the server to transmit / receive training data, classification results, trained models (e.g., neural network 104), etc. In some examples, the example communication circuit module 210 can transmit the model to a cloud server (e.g., a cloud server including an instance of the deep learning accelerator circuit module 102).

[0063] The example communication circuit module 210 may also coordinate communications between the operation classification circuit module 202, the inter-iteration scheduling circuit module 204, the example staleness-aware distributed optimization circuit module 206, the neural network circuit module 208, the training circuit module 210, and the data storage device 212. For example, such communications may occur via the communication bus 214.

[0064] In some examples, the deep learning accelerator circuit module 102 includes components for facilitating communication. For example, the components for facilitating communication may be implemented by the example communication circuit module 210. In some examples, the example communication circuit module 210 may be implemented by a processor circuit module (such as Fig.11 For example, the example communication circuit module 210 may be instantiated by executing machine executable instructions. Fig.12 The example microprocessor 1200 is instantiated by the machine executable instructions such as at least Figure 6 In some examples, the communication circuit module 210 may be instantiated by a hardware logic circuit module, which may be implemented by an ASIC, XPU, or other processor configured to perform operations corresponding to machine-readable instructions. Fig.13 The example communication circuit module 210 is implemented by the FPGA circuit module 1300. Additionally or alternatively, the example communication circuit module 210 can be instantiated by any other combination of hardware, software and / or firmware. For example, the example communication circuit module 210 can be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, ASICs, XPUs, comparators, operational amplifiers (op-amps), logic circuits, etc.), and the at least one or more hardware circuits are configured to execute some or all of the machine-readable instructions and / or execute some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are equally suitable.

[0065] The example deep learning accelerator circuit module 102 includes an example data storage device 212. The example data storage device 212 stores training data used to train the example neural network circuit module 208. The example data storage device 212 may also store results of classifications performed by the example neural network circuit module 208, classifications generated by the operation classification circuit module 202, schedules generated by the inter-iteration scheduling circuit module 204, information related to stale gradients, etc.

[0066] In some examples, the deep learning accelerator circuit module 102 includes a component for storing data generated by the deep learning accelerator circuit module 102. For example, the component for storing data can be implemented by the example data storage device 212. In some examples, the example data storage device 212 can be implemented by a processor circuit module (such as Fig.11 For example, the example data storage device 212 may be implemented by executing machine executable instructions. Fig.12 The example microprocessor 1200 is instantiated by the machine executable instructions such as those provided by at least Figure 6 In some examples, the example data storage device 212 may be instantiated by a hardware logic circuit module, which may be an ASIC, XPU, or other processor configured to perform operations corresponding to machine-readable instructions. Fig.13 FPGA circuit module 1300. Additionally or alternatively, the example data storage device 212 may be instantiated by any other combination of hardware, software, and / or firmware. For example, the example data storage device 212 may be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, ASICs, XPUs, comparators, operational amplifiers (op-amps), logic circuits, etc.), which are configured to execute some or all of the machine-readable instructions and / or execute some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are equally suitable.

[0067] Although in Figure 2 The implementation is shown in the figure Figure 1 An example of a deep learning accelerator circuit module 102, but in Figure 2 One or more of the elements, processes and / or devices illustrated in the figure may be combined, divided, rearranged, omitted, eliminated and / or implemented in any other manner. In addition, Figure 1The example operation classification circuit module 202, the example inter-iteration scheduling circuit module 204, the example staleness-aware distributed optimization circuit module 206, the example neural network circuit module 208, the example communication circuit module 210, the example data storage device 212 and / or more generally the example deep learning accelerator circuit module 102 can be implemented by hardware alone or by hardware combined with software and / or firmware. Thus, for example, Figure 1 Any of the example operation classification circuit module 202, the example inter-iteration scheduling circuit module 204, the example stale-aware distributed optimization circuit module 206, the example neural network circuit module 208, the example communication circuit module 210, the example data storage device 212, and / or more generally the example deep learning accelerator circuit module 102 can be implemented by a processor circuit module, (one or more) analog circuits, (one or more) digital circuits, (one or more) logic circuits, (one or more) programmable processors, (one or more) programmable microcontrollers, (one or more) graphics processing units ((one or more) GPUs), (one or more) digital signal processors ((one or more) DSPs), (one or more) application-specific integrated circuits ((one or more) ASICs), (one or more) programmable logic devices ((one or more) PLDs) and / or (one or more) field programmable logic devices ((one or more) FPLDs) (such as field programmable gate arrays (FPGAs)). Further, Figure 1 An example deep learning accelerator circuit module 102 may include in addition to Figure 2 In addition to or in lieu of those shown in Figure 2 One or more elements, processes and / or devices of those illustrated, and / or may include more than one of any or all of the illustrated elements, processes and devices.

[0068] Figure 3 300 (e.g., execution graph, computation graph, etc.) for training a deep learning workload. Specifically, the execution schedule is an execution schedule of an example deep learning recommendation model (DLRM). DLRM inputs include both dense (e.g., represented as floating point values) inputs and sparse (e.g., represented as indices into an embedding table) inputs. Figure 4-6 Picture shows Figure 2 How the deep learning accelerator circuit module 102 can optimize the execution schedule 300. However, Figure 3-6 Only one example of the deep learning accelerator circuit module classifying operations and generating an execution schedule is illustrated, and the deep learning accelerator circuit module 102 can optimize any type of deep learning workload.

[0069] The example execution schedule 300 is represented as a directed acyclic graph (DAG). Each node in the graph (e.g., data loader operation 302) represents an operation. Each edge in the graph (e.g., first edge 350) represents the flow of data from one operation (e.g., first operation) to another operation (e.g., second operation). The execution schedule 300 is an execution schedule for one training iteration. Therefore, the example execution schedule 300 only shows the intra-iteration optimization (e.g., forward pass, backward pass, and weight update) for training mode execution.

[0070] The execution schedule 300 starts at a data loader operation 302. The data loader operation 302 is a data loading operation that can iterate over a data set. The data loader operation 302 (e.g., loading training data) is a parent node of a sparse embedding operation 304 (e.g., an operation on a sparse tensor), a bottom multi-layer perceptron (bot mlp) operation 306 (e.g., an operation on a multi-layer perceptron), and a dense embedding operation 308 (e.g., an operation on a dense tensor). Therefore, a first edge 350 connects the data loader operation 302, the sparse embedding operation 304, the bottom multi-layer perceptron operation 306, and the dense embedding operation 308. The catall-to-all operation 310 (e.g., a connect operation) depends on the sparse embedding operation 304. Therefore, the interaction operation 312 depends on: the catall-to-all operation 310 (e.g., connect), the bottom multi-layer perceptron (bot mlp) operation 306, and the dense embedding operation 308. The remaining operations 314-332 exhibit dependencies according to the same principle.

[0071] Figure 4 It is by Figure 2 After the operation classification circuit module 202 classifies the operation Figure 3 Example execution scheduling. Operation classification circuit module 202 classifies operations into one of four categories: network-constrained, memory-constrained, compute-constrained, and I / O-constrained. The four categories of classification are illustrated in legend 402. A first legend entry 404 illustrates a first pattern (e.g., a dashed pattern) for identifying network-constrained operations. A second legend entry 406 illustrates a second pattern (e.g., a white background) for identifying memory-constrained operations. A third legend entry 408 illustrates a third pattern (e.g., a gray background) for identifying compute-constrained operations. A fourth legend entry 410 illustrates a fourth pattern (e.g., a black background) for identifying I / O-constrained operations. Example operation classification circuit module 202 identifies Figure 3 The execution schedule 300 generates Figure 4 The classified execution schedule 400.

[0072] Each operation of the execution schedule 400 has been classified into one of the four categories presented in the legend 402. For example, the data loader operation 302 is I / O limited. The first all-to-all-connect operation 310, the second all-to-all-connect operation 326, the first all-reduce operation 324, the second all-reduce operation 330, and the third all-reduce operation 332 are network limited. The bottom multi-layer perceptron operation 306, the interactive operation 312, the top multi-layer perceptron (top mlp) operation 314, the top multi-layer perceptron back propagation (topmlp_bwd) operation 320, the interactive back propagation (bwd) operation 322, and the bottom multi-layer perceptron back propagation (bot mlp_bwd) operation are classified as memory limited. The inter-iteration scheduling circuit module 204 and the stale-aware distributed optimization circuit module 206 can generate a schedule based on the classified execution schedule 400.

[0073] Figure 5 is after the segment of the third execution schedule 500 for overlapping execution is identified by the inter-iteration scheduling circuit module 204 and / or the stale-aware distributed optimization circuit module 206 Figure 4 The example third execution schedule 500 includes a first operation 502 for overlapping execution and a second operation 504 for overlapping execution. The first operation 502 for overlapping execution is a group of operations identified by the inter-iteration scheduling circuit module 204 as being available for overlapping execution. For example, the all-to-fully connected operation 310 is a network-restricted operation, and thus one or more of the bottom multi-layer perceptron operation 306, the dense embedding operation 308, and / or the zero gradient operation 340 may at least partially overlap with the execution of the all-to-fully connected operation 310.

[0074] Figure 6-10 The diagram shows a method that can be executed to configure the processor circuit module to implement Figure 2 Flowchart of example machine readable instructions for the deep learning accelerator circuit module 102 of FIG. The machine readable instructions may be for use by a processor circuit module (such as the following in combination with Fig.11 The processor circuit module 1112 shown in the example processor platform 1100 discussed below and / or in combination Fig.12 and / or Fig.13The program may be embodied in software, which is stored on one or more non-transitory computer-readable storage media such as a compact disk (CD), a floppy disk, a hard disk drive (HDD), a solid-state drive (SSD), a digital versatile disk (DVD), a Blu-ray disk, a volatile memory (e.g., any type of random access memory (RAM), etc.) or a non-volatile memory (e.g., an electrically erasable programmable read-only memory (EEPROM), FLASH memory, HDD, SSD, etc.) associated with a processor circuit module located in one or more hardware devices, but the entire program and / or its portion may alternatively be executed by one or more hardware devices other than the processor circuit module and / or embodied in firmware or dedicated hardware. The machine-readable instructions may be distributed across multiple hardware devices and / or executed by two or more hardware devices (e.g., a server and a client hardware device). For example, the client hardware device may be implemented by an endpoint client hardware device (e.g., a hardware device associated with a user) or an intermediate client hardware device (e.g., a radio access network (RAN) gateway that can facilitate communication between a server and an endpoint client hardware device). Similarly, a non-transitory computer-readable storage medium may include one or more media located in one or more hardware devices. In addition, although reference is made to Figure 6-10 The flowchart illustrated in FIG. 1 describes an example procedure, but may alternatively be implemented using Figure 2 Many other methods of the example deep learning accelerator circuit module 102. For example, the execution order of the boxes may be changed, and / or some of the boxes described in the boxes may be changed, eliminated, or combined. Additionally or alternatively, any or all of the boxes in the boxes may be implemented by one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) that are configured to perform corresponding operations without executing software or firmware. The processor circuit modules may be distributed in different network locations and / or located locally in one or more hardware devices (e.g., single-core processors (e.g., single-core central processing units (CPUs)), multi-core processors in a single machine (e.g., multi-core CPUs, XPUs, etc.), multiple processors distributed among multiple servers in a server rack, multiple processors distributed among one or more server racks, CPUs and / or FPGAs located in the same package (e.g., the same integrated circuit (IC) package or in two or more separate housings, etc.).

[0075] The machine-readable instructions described herein may be stored in one or more of a compressed format, an encrypted format, a segmented format, a compiled format, an executable format, a packaged format, etc. The machine-readable instructions as described herein may be stored as data or a data structure (e.g., as a portion of an instruction, a code, a representation of a code, etc.) that may be used to create, manufacture, and / or generate machine-executable instructions. For example, the machine-readable instructions may be segmented and stored on one or more storage devices and / or computing devices (e.g., servers) located in the same or different locations of a network or a collection of networks (e.g., in the cloud, in an edge device, etc.). The machine-readable instructions may require one or more of installation, modification, adaptation, updating, combination, supplementation, configuration, decryption, decompression, unpacking, distribution, reassignment, compilation, etc., so that they are directly readable, interpretable, and / or executable by a computing device and / or other machine. For example, machine-readable instructions may be stored in multiple portions that are individually compressed, encrypted, and / or stored on separate computing devices, where the portions, when decrypted, decompressed, and / or combined, form a set of machine-executable instructions that implement one or more operations that may collectively form a program such as the one described herein.

[0076] In another example, the machine-readable instructions may be stored in a state in which they are readable by a processor circuit module, but require the addition of a library (e.g., a dynamic link library (DLL)), a software development kit (SDK), an application programming interface (API), etc., in order to execute the machine-readable instructions on a particular computing device or other device. In another example, the machine-readable instructions may need to be configured (e.g., settings are stored, data are entered, network addresses are recorded, etc.) before the machine-readable instructions and / or (one or more) corresponding programs can be executed in whole or in part. Therefore, a machine-readable medium as used herein may include machine-readable instructions and / or (one or more) programs, regardless of the specific format or state of the machine-readable instructions and / or (one or more) programs when stored or otherwise at rest or in transition.

[0077] The machine-readable instructions described herein may be represented by any past, present, or future instruction language, scripting language, programming language, etc. For example, the machine-readable instructions may be represented using any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, Hypertext Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0078] As mentioned above, Figure 6-10The example operations of can be implemented using executable instructions (e.g., computer and / or machine readable instructions) stored on one or more non-transitory computer and / or machine readable media, such as optical storage devices, magnetic storage devices, HDDs, flash memory, read-only memory (ROM), CDs, DVDs, caches, any type of RAM, registers, and / or any other storage devices or storage disks, wherein information can be stored for any duration (e.g., for an extended period of time, permanently, for brief instances, for temporary buffering, and / or for caching of information). As used herein, the terms non-transitory computer-readable medium, non-transitory computer-readable storage medium, non-transitory machine-readable medium, and non-transitory machine-readable storage medium are expressly defined to include any type of computer-readable storage device and / or storage disk, and do not include propagating signals and do not include transmission media. As used herein, the terms "computer-readable storage device" and "machine-readable storage device" are defined to include any physical (mechanical and / or electrical) structure for storing information, but do not include propagating signals and do not include transmission media. Examples of computer-readable storage devices and machine-readable storage devices include any type of random access memory, any type of read-only memory, solid-state memory, flash memory, optical disks, magnetic disks, disk drives, and / or redundant arrays of independent disks (RAID) systems. As used herein, the term "device" refers to a physical structure, such as a mechanical and / or electrical device, hardware, and / or circuit module, that may or may not be configured by computer-readable instructions, machine-readable instructions, etc. and / or manufactured to execute computer-readable instructions, machine-readable instructions, etc.

[0079] "Include" and "comprising" (and all forms and tenses thereof) are used herein as open-ended terms. Thus, whenever a claim employs any form of "includes" or "comprising" (e.g., comprises, includes, comprising, including, having, etc.) as a preamble or in any type of claim recitation, it is understood that additional elements, terms, etc. may be present without falling outside the scope of the corresponding claim or recitation. As used herein, when the phrase "at least" is used as a transitional term, such as in the preamble of a claim, it is open-ended in the same manner as the terms "includes" and "comprising" are open-ended. When used, for example, in a form such as A, B, and / or C, the term "and / or" refers to any combination or subset of A, B, and C, such as (1) A alone, (2) B alone, (3) C alone, (4) A and B, (5) A and C, (6) B and C, or (7) A and B and C. As used herein in the context of describing structures, components, articles, objects, and / or things, the phrase “at least one of A and B” is intended to refer to implementations that include any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing structures, components, articles, objects, and / or things, the phrase “at least one of A or B” is intended to refer to implementations that include any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. As used herein in the context of describing the execution or performance of processes, instructions, actions, activities, and / or steps, the phrase “at least one of A and B” is intended to refer to implementations that include any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing the execution or performance of a process, instruction, action, activity, and / or step, the phrase "at least one of A or B" is intended to refer to an implementation that includes any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0080] As used herein, singular references (e.g., "a," "an," "first," "second," etc.) do not exclude the plural. As used herein, the term "a" or "an" object refers to one or more of that object. The terms "a" (or "an"), "one or more," and "at least one" are used interchangeably herein. In addition, although listed separately, multiple components, elements, or method actions may be implemented by, for example, the same entity or object. In addition, although individual features may be included in different examples or claims, these may be combined, and the inclusion in different examples or claims does not imply that a combination of features is not feasible and / or advantageous.

[0081] Figure 6 is a flow diagram representative of example machine readable instructions and / or example operations 600 that may be executed and / or instantiated by a processor circuit module to accelerate deep learning. Figure 6 The machine readable instructions and / or operations 600 begin at block 602 where, Figure 2 The operation classification circuit module 202 classifies the operations of the distributed deep learning workload. Figure 7 The example operation 602 is described in more detail. Instructions continue at block 604 where, Figure 2 The inter-iteration scheduling circuit module 204 identifies overlapping operations in the distributed deep learning workload. Figure 8 Operation 604 is described in further detail.

[0082] At block 606, Figure 2 The example inter-iteration scheduling circuit module 206 enables inter-iteration / intra-iteration execution of selected overlapping operations. Fig. 9 The operation of block 606 is described in more detail. At block 608, Figure 2 The example stale-aware distributed optimization circuit module 206 performs stale-aware training of the neural network. Fig.10 The operations of block 608 are described in more detail. The instruction ends.

[0083] Figure 7 is a flow diagram representative of example machine readable instructions and / or example operations 602 that may be executed and / or instantiated by a processor circuit module to classify operations for a distributed deep learning workload. Figure 7 The machine readable instructions and / or operations 602 begin at block 702 where, Figure 2 The example operation classification circuit module 202 determines whether the operation is a communication operation. At block 704, if Figure 2If the example operation classification circuit module 202 determines that the operation is a communication operation, control moves to block 706, where the operation is classified as network limited. Figure 2 If the example operation classification circuit module 202 determines that the operation is not a communication operation, control moves to block 708 where Figure 2 An example operation of the classification circuit module 202 compares the computation time with the memory transfer time.

[0084] At block 710, if Figure 2 If the example operation classification circuit module 202 determines that the CPU / GPU calculation time is greater than the memory transfer time, control moves to block 712, where Figure 2 The operation classification circuit module 202 classifies the computation as computationally limited. If at block 710, Figure 2 If the example operation classification circuit module 202 determines that the operation does not have a CPU / GPU compute time greater than the memory transfer time, control moves to block 714 .

[0085] At block 714, Figure 2 The operation classification circuit module 202 compares the memory transfer time with the input / output transfer time. At block 716, if Figure 2 If the example operation classification circuit module 202 determines that the memory transfer time is greater than the I / O transfer time, control moves to block 718 where, Figure 2 The operation classification circuit module 202 classifies the operation as memory limited. If at block 716, if Figure 2 If the example operation classification circuit module 202 determines that the operation does not have a memory transfer time greater than the I / O transfer time, control moves to block 720 to classify the operation as I / O bound. Instructions return to block 604.

[0086] Figure 8 is a flow diagram representative of example machine readable instructions and / or example operations 604 that may be executed and / or instantiated by a processor circuit module to identify overlapping operations in an execution schedule. Figure 8 The machine readable instructions and / or operations 604 begin at block 802 where, Figure 2 The example inter-iteration scheduling circuit module 204 retrieves the inter-iteration directed graph. Figure 3-5 The graph described is a DAG having nodes representing operations and edges representing dependencies between operations, but the information represented by the graph may be stored in any suitable format (e.g., a table, a sparse matrix, etc.) that includes data associating operations and dependencies between operations.

[0087] At block 804, Figure 2The example inter-iteration scheduling circuit module 204 profiles the workload associated with the graph for execution time. In some examples, the workload profile can be pre-computed and loaded (e.g., the computational characteristics of the operation are known) rather than determined in real time. At block 806, Figure 2 The example inter-iteration scheduling circuit module 204 identifies communication operations in the execution graph that are not the last full reduce.

[0088] At block 808, Figure 2 The example inter-iteration scheduling circuit module 204 identifies the parent node and output node of each identified communication operation. At block 810, Figure 2 The example inter-iteration scheduling circuit module 204 generates a candidate list with the identified communication operations. For example, the candidate list of communication operations can be compiled to identify communication operations for overlapping execution. Finally, at block 812, Figure 2 The example inter-iteration scheduling circuit module 204 associates the entry in the candidate list with a node in the inter-iteration directed graph that is between the parent node and the child node of the candidate operation. Instructions return to block 606.

[0089] Fig. 9 is a flow diagram representative of example machine readable instructions and / or example operations 606 that may be executed and / or instantiated by a processor circuit module to enable inter / intra-iteration overlap.

[0090] Fig. 9 The machine readable instructions and / or operations 606 of begin at block 902 where, Figure 2 The example inter-iteration scheduling circuit module 204 retrieves the inter-iteration directed graph, the communication list, and the candidate overlapping operations. At block 904, Figure 2 The example inter-iteration scheduling circuit module 204 marks the candidate count of the non-communication operation. Then, at block 906, Figure 2 The example inter-iteration scheduling circuit module 204 determines the count of non-communication operations. The operations of blocks 904 and 906 (e.g., and more generally, operation 606) correspond to the algorithm in Table 2 above. For example, the candidate counts of non-communication operations can be associated with priorities (e.g., a higher count associated with a greater priority). At block 908, Figure 2 The example inter-iteration scheduling circuit module 204 also prioritizes the operations according to the priority of the following: I / O-limited, compute-limited, and memory-limited operations. For example, if there are more than one non-communication operations that may overlap with the communication operation, the priority of the non-communication operation may be determined by assigning the I / O-limited operation as the highest priority, the compute-limited operation as the second highest priority, and the memory-limited operation as the third highest priority. The instruction returns to block 608.

[0091] Fig.10 is a flow diagram representative of example machine readable instructions and / or example operations 608 that may be executed and / or instantiated by a processor circuit module to perform staleness-aware optimization of distributed deep learning workloads.

[0092] Fig.10 The machine readable instructions and / or operations 608 of begin at block 1002 where, Figure 2 The example stale-aware distributed optimization circuit module 206 identifies stale portions of a tensor. For example, some computations may be performed using stale weights until Figure 2 The staleness-aware distributed optimization circuit module 206 utilizes the gradient. At block 1004, Figure 2 The example staleness-aware distributed optimization circuit module 206 transmits the stale gradient to another computing device in a staleness-aware communication. Figure 2 The stale-aware distributed optimization circuit module 206 may update the weights with stale gradients (eg, identified based on stale-aware communications). At block 1006, Figure 2 The example stale-aware distributed optimization circuit module 206 updates the weights of the relevant neural network with the stale gradient. At block 1008, Figure 2 The example stale-aware distributed optimization circuit module 206 performs stale-aware training of the neural network. In some examples, the learning rate can be adjusted based on the identification of the number of stale communications. The instructions end.

[0093] Fig.11 is constructed to execute and / or instantiate Figure 6-10 Machine readable instructions and / or operations to implement Figure 2 1. The processor platform 1100 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cell phone, a smart phone, an iPad, etc.). TM tablet computer), personal digital assistant (PDA), Internet appliance, DVD player, CD player, digital video recorder, Blu-ray player, game console, personal video recorder, set-top box, headsets (e.g., augmented reality (AR) headsets, virtual reality (VR) headsets, etc.) or other wearable devices, or any other type of computing device.

[0094] The illustrated example processor platform 1100 includes a processor circuit module 1112. The illustrated example processor circuit module 1112 is hardware. For example, the processor circuit module 1112 can be implemented by one or more integrated circuits, logic circuits, FPGAs, microprocessors, CPUs, GPUs, DSPs, and / or microcontrollers from any desired series or manufacturer. The processor circuit module 1112 can be implemented by one or more semiconductor-based (e.g., silicon-based) devices. In this example, the processor circuit module 1112 implements the example operation classification circuit module 202, the example inter-iteration scheduling circuit module 204, the example stale-aware distributed optimization circuit module 206, the example neural network circuit module 208, the example communication circuit module 210, and the example data storage device 212.

[0095] The processor circuit module 1112 of the illustrated example includes a local memory 1113 (e.g., cache, registers, etc.). The processor circuit module 1112 of the illustrated example communicates with a main memory (including a volatile memory 1114 and a non-volatile memory 1116) via a bus 1118. The volatile memory 1114 may be a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory The non-volatile memory 1116 may be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 1114 , 1116 of the illustrated example is controlled by a memory controller 1117 .

[0096] The processor platform 1100 of the illustrated example also includes an interface circuit module 1120. The interface circuit module 1120 may be implemented by hardware according to any type of interface standard, such as an Ethernet interface, a universal serial bus (USB) interface, interface, a near field communication (NFC) interface, a peripheral component interconnect (PCI) interface, and / or a peripheral component interconnect express (PCIe) interface.

[0097] In the illustrated example, one or more input devices 1122 are connected to the interface circuit module 1120. The input device(s) 1122 permit a user to enter data and / or commands into the processor circuit module 1112. The input device(s) 1122 may be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a track pad, a trackball, an isopoint device, and / or a speech recognition system.

[0098] One or more output devices 1124 are also connected to the interface circuit module 1120 of the illustrated example. The output device(s) 1124 may be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube (CRT) display, an in-place switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Therefore, the interface circuit module 1120 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics processor circuit module (such as a GPU).

[0099] The interface circuit module 1120 of the illustrated example also includes a communication device (such as a transmitter, receiver, transceiver, modem, residential gateway, wireless access point and / or network interface) to facilitate the exchange of data with an external machine (e.g., any kind of computing device) through the network 1126. Communication can be carried out through, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-site wireless system, a cellular telephone system, an optical connection, etc.

[0100] The processor platform 1100 of the illustrated example also includes one or more mass storage devices 1128 for storing software and / or data. Examples of such mass storage devices 1128 include magnetic storage devices, optical storage devices, floppy disk drives, HDDs, CDs, Blu-ray disk drives, redundant arrays of independent disks (RAID) systems, solid-state storage devices (such as flash memory devices and / or SSDs), and DVD drives.

[0101] can be Figure 6-10 The machine-readable instructions 1132 implemented by the machine-readable instructions may be stored in the mass storage device 1128, in the volatile memory 1114, in the non-volatile memory 1116, and / or on a removable non-transitory computer-readable storage medium such as a CD or DVD.

[0102] Fig.12 yes Fig.11 1112 of the processor circuit module 1112. In this example, Fig.11 The processor circuit module 1112 is implemented by the microprocessor 1200. For example, the microprocessor 1200 may be a general-purpose microprocessor (eg, a general-purpose microprocessor circuit module). The microprocessor 1200 executes Figure 6-10Some or all of the machine-readable instructions of the flowchart of the deep learning accelerator circuit module 102 are effectively instantiated as a logic circuit for performing operations corresponding to those machine-readable instructions. In some such examples, the deep learning accelerator circuit module 102 is instantiated by the hardware circuit of the microprocessor 1200 in combination with the instructions. For example, the microprocessor 1200 can be implemented by a multi-core hardware circuit module such as a CPU, a DSP, a GPU, an XPU, etc. Although it can include any number of example cores 1202 (for example, 1 core), the microprocessor 1200 of this example is a multi-core semiconductor device including N cores. The cores 1202 of the microprocessor 1200 can operate independently or can cooperate to execute machine-readable instructions. For example, the machine code corresponding to the firmware program, the embedded software program, or the software program can be executed by one of the cores in the core 1202, or can be executed by multiple cores in the core 1202 at the same or different times. In some examples, the machine code corresponding to the firmware program, the embedded software program, or the software program is split into threads and executed in parallel by two or more cores in the core 1202. The software program may correspond to Figure 6-11 A flowchart may represent a portion or all of machine-readable instructions and / or operations.

[0103] The core 1202 can communicate via a first example bus 1204. In some examples, the first bus 1204 can be implemented by a communication bus to implement communications associated with one (or more) of the cores 1202. For example, the first bus 1204 can be implemented by at least one of an inter-integrated circuit (I2C) bus, a serial peripheral interface (SPI) bus, a PCI bus, or a PCIe bus. Additionally or alternatively, the first bus 1204 can be implemented by any other type of computing or electrical bus. The core 1202 can obtain data, instructions, and / or signals from one or more external devices via an example interface circuit module 1206. The core 1202 can output data, instructions, and / or signals to one or more external devices via the interface circuit module 1206. Although the cores 1202 of this example include example local memory 1220 (e.g., a level 1 (L1) cache that can be split into an L1 data cache and an L1 instruction cache), the microprocessor 1200 also includes example shared memory 1210 (e.g., a level 2 (L2 cache)) that can be shared by the cores for high-speed access to data and / or instructions. Data and / or instructions can be transferred (e.g., shared) by writing to and / or reading from the shared memory 1210. The local memory 1220 and the shared memory 1210 of each core in the cores 1202 can be part of a hierarchy of storage devices, including cache memory and main memory (e.g., Fig.11114, 1116). Typically, memories at higher levels in the hierarchy exhibit lower access times and have smaller storage capacities than memories at lower levels. Changes in various levels of the cache hierarchy are managed (e.g., coordinated) by a cache coherence policy.

[0104] Each core 1202 may be referred to as a CPU, DSP, GPU, etc., or any other type of hardware circuit module. Each core 1202 includes a control unit circuit module 1214, an arithmetic and logic (AL) circuit module (sometimes referred to as an ALU) 1216, a plurality of registers 1218, a local memory 1220, and a second example bus 1222. Other structures may exist. For example, each core 1202 may include a vector unit circuit module, a single instruction multiple data (SIMD) unit circuit module, a load / store unit (LSU) circuit module, a branch / jump unit circuit module, a floating point unit (FPU) circuit module, etc. The control unit circuit module 1214 includes a semiconductor-based circuit configured to control (e.g., coordinate) data movement within the corresponding core 1202. The AL circuit module 1216 includes a semiconductor-based circuit configured to perform one or more mathematical and / or logical operations on data within the corresponding core 1202. The AL circuit module 1216 of some examples performs integer-based operations. In other examples, the AL circuit module 1216 also performs floating-point operations. In still other examples, the AL circuit module 1216 may include a first AL circuit module that performs integer-based operations and a second AL circuit module that performs floating-point operations. In some examples, the AL circuit module 1216 may be referred to as an arithmetic logic unit (ALU). The register 1218 is a semiconductor-based structure for storing data and / or instructions such as the results of one or more of the operations performed by the AL circuit module 1216 of the corresponding core 1202. For example, the register 1218 may include (one or more) vector registers, (one or more) SIMD registers, (one or more) general registers, (one or more) flag registers, (one or more) segment registers, (one or more) machine-specific registers, (one or more) instruction pointer registers, (one or more) control registers, (one or more) debug registers, (one or more) memory management registers, (one or more) machine check registers, etc. The register 1218 may be arranged in a manner such as Fig.11 Alternatively, registers 1218 may be organized in any other arrangement, format, or structure including being distributed throughout core 1202 to reduce access time. Second bus 1222 may be implemented by at least one of an I2C bus, an SPI bus, a PCI bus, or a PCIe bus.

[0105] Each core 1202 and / or more generally the microprocessor 1200 may include additional and / or alternative structures to those shown and described above. For example, there may be one or more clock circuits, one or more power supply devices, one or more power gates, one or more cache home agents (CHA), one or more converged / common mesh stop points (converged / common mesh stop, CMS), one or more shifters (e.g., (one or more) barrel shifters) and / or other circuit modules. The microprocessor 1200 is a semiconductor device manufactured to include many transistors interconnected to implement the structure described above in one or more integrated circuits (ICs) contained in one or more packages. The processor circuit module may include one or more accelerators and / or collaborate with one or more accelerators. In some examples, the accelerator is implemented by a logic circuit module to perform certain tasks faster and / or more efficiently than can be completed by a general-purpose processor. Examples of accelerators include ASICs and FPGAs such as those discussed herein. A GPU or other programmable device may also be an accelerator. The accelerator may be on board the processor circuit module, in the same chip package as the processor circuit module, and / or in one or more packages separate from the processor circuit module.

[0106] Fig.13 yes Fig.11 1 is a block diagram of another example implementation of the processor circuit module 1112 of FIG. 1. In this example, the processor circuit module 1112 is implemented by an FPGA circuit module 1300. For example, the FPGA circuit module 1300 can be implemented by an FPGA. The FPGA circuit module 1300 can be used, for example, to execute a program that may otherwise be executed by executing corresponding machine-readable instructions. Fig.12 However, once configured, the FPGA circuit module 1300 instantiates the machine-readable instructions in hardware and can therefore typically perform the operations faster than the operations could be performed by a general-purpose microprocessor executing corresponding software.

[0107] More specifically, as described above Fig.12 The microprocessor 1200 (which can be programmed to execute the Figure 6-10 In contrast, a general purpose device that contains some or all of the machine-readable instructions represented by a flowchart, but whose interconnections and logic circuit modules are fixed once they are made, Fig.13 The example FPGA circuit module 1300 includes circuits that can be configured and / or interconnected in different ways after fabrication to instantiate, for example, Fig.13The flowcharts represent some or all of the interconnections and logic circuit modules of the machine-readable instructions. In particular, the FPGA circuit module 1300 can be viewed as an array of logic gates, interconnections, and switches. The switches can be programmed to change how the logic gates are interconnected by the interconnections, effectively forming one or more dedicated logic circuits (unless and until the FPGA circuit module 1300 is reprogrammed). The configured logic circuits enable the logic gates to cooperate in different ways to perform different operations on the data received by the input circuit module. Those operations may correspond to the operations performed by Figure 6-10 Thus, FPGA circuit module 1300 can be configured to effectively instantiate a dedicated logic circuit module. Figure 6-10 1300 can execute some or all of the machine-readable instructions of the flowchart to perform the operations corresponding to those software instructions in a dedicated manner similar to an ASIC. Therefore, the FPGA circuit module 1300 can execute the same operations faster than a general-purpose microprocessor can execute the same operations. Figure 6-10 Operations corresponding to some or all of the machine-readable instructions.

[0108] exist Fig.13 In the example of FPGA circuit module 1300, FPGA circuit module 1300 is configured to be programmed (and / or reprogrammed one or more times) by an end user via a hardware description language (HDL) such as Verilog. Fig.13 The FPGA circuit module 1300 includes an example input / output (I / O) circuit module 1302 for obtaining data from and / or outputting data to an example configuration circuit module 1304 and / or external hardware 1306. For example, the configuration circuit module 1304 may be implemented by an interface circuit module that may obtain machine-readable instructions to configure the FPGA circuit module 1300 or (one or more) portions thereof. In some such examples, the configuration circuit module 1304 may obtain machine-readable instructions from a user, a machine (e.g., a hardware circuit module (e.g., a programming or dedicated circuit module) that may implement an artificial intelligence / machine learning (AI / ML) model to generate instructions), or the like. In some examples, the external hardware 1306 may be implemented by an external hardware circuit module. For example, the external hardware 1306 may be implemented by Fig.12 FPGA circuit module 1300 also includes an example logic gate circuit module 1308, a plurality of example configurable interconnects 1310, and an array of example storage circuit modules 1312. Logic gate circuit module 1308 and configurable interconnects 1310 can be configured to instantiate a circuit that can correspond to Figure 6-10 One or more operations of at least some of the machine-readable instructions and / or other desired operations. Fig.13The logic gate circuit modules 1308 shown in are made in groups or blocks. Each block includes a semiconductor-based electrical structure that can be configured into a logic circuit. In some examples, the electrical structure includes a logic gate (e.g., an AND gate, an OR gate, a NAND gate, etc.) that provides a basic building block for the logic circuit. There is an electrically controllable switch (e.g., a transistor) in each of the logic gate circuit modules 1308 so that the configuration of the electrical structure and / or the logic gate can form a circuit for performing the desired operation. The logic gate circuit module 1308 may include other electrical structures, such as a lookup table (LUT), a register (e.g., a flip-flop or a latch), a multiplexer, etc.

[0109] The illustrated example configurable interconnect 1310 is a conductive path, trace, via, etc. that may include an electrically controllable switch (e.g., a transistor), the state of which may be changed by programming (e.g., using an HDL instruction language) to activate or deactivate one or more connections between one or more logic gate circuit modules in logic gate circuit module 1308 in order to program the desired logic circuit module.

[0110] The storage circuit module 1312 of the illustrated example is configured to store (one or more) results of one or more operations performed by the corresponding logic gates. The storage circuit module 1312 can be implemented by registers, etc. In the illustrated example, the storage circuit module 1312 is distributed among the logic gate circuit module 1308 to facilitate access and improve execution speed.

[0111] Fig.13 The example FPGA circuit module 1300 also includes an example dedicated operation circuit module 1314. In this example, the dedicated operation circuit module 1314 includes a dedicated circuit module 1316, which can be called to implement common functions, thereby avoiding the need to program these functions on site. Examples of such dedicated circuit modules 1316 include memory (e.g., DRAM) controller circuit modules, PCIe controller circuit modules, clock circuit modules, transceiver circuit modules, memory, and multiplier-accumulator circuit modules. There may be other types of dedicated circuit modules. In some examples, the FPGA circuit module 1300 may also include an example general programmable circuit module 1318, such as an example CPU 1320 and / or an example DSP 1322. Additionally or alternatively, there may be other general programmable circuit modules 1318 that can be programmed to perform other operations, such as GPUs, XPUs, etc.

[0112] Although Fig.12 and Fig.13 Picture shows Fig.11These are two example implementations of the processor circuit module 1112, but many other implementations are contemplated. For example, as mentioned above, a modern FPGA circuit module may include an onboard CPU, such as Fig.13 One or more of the example CPUs 1320. Thus, Fig.11 The processor circuit module 1112 can also be combined Fig.12 An example microprocessor 1200 and Fig.13 In some such hybrid examples, the Figure 6-10 The first portion of the machine-readable instructions represented by the flowchart may be represented by Fig.12 1202, executed by one or more of the cores 1202 Figure 6-10 The second portion of the machine-readable instructions represented by the flowchart may be represented by Fig.13 FPGA circuit module 1300 is executed, and / or by Figure 6-10 The third portion of the machine-readable instructions represented by the flowchart may be executed by the ASIC. It should be understood that Figure 2 Some or all of the circuit modules of can therefore be instantiated at the same or different times. For example, some or all of the circuit modules can be instantiated in one or more threads that are executed concurrently and / or serially. In addition, in some examples, Figure 2 Some or all of the circuit modules of the system may be implemented in one or more virtual machines and / or containers executed on a microprocessor.

[0113] In some examples, Fig.11 The processor circuit module 1112 may be in one or more packages. For example, Fig.12 The microprocessor 1200 and / or Fig.13 The FPGA circuit module 1300 may be in one or more packages. In some examples, the XPU may be comprised of a FPGA circuit module 1300 that may be in one or more packages. Fig.11 The XPU may be implemented by a processor circuit module 1112. For example, the XPU may include a CPU in one package, a DSP in another package, a GPU in yet another package, and an FPGA in yet another package.

[0114] The diagram is used to Fig.11 A block diagram of an example software distribution platform 1405 for distributing software to hardware devices owned and / or operated by a third party is provided in Fig.14 The example software distribution platform 1405 may be implemented by any computer server, data facility, cloud service, etc. that is capable of storing software and delivering software to other computing devices. A third party may be a customer of the entity that owns and / or operates the software distribution platform 1405. For example, the entity that owns and / or operates the software distribution platform 1405 may be a Fig.11 The third party may be a consumer, user, retailer, OEM, etc. who purchases the software and / or licenses the software for use and / or resale and / or sublicense. In the illustrated example, the software distribution platform 1405 includes one or more servers and one or more storage devices. As described above, the storage device stores the machine-readable instructions 1132, which may correspond to Figure 6-10 The example machine-readable instructions 600 of the example software distribution platform 1405 are provided. One or more servers of the example software distribution platform 1405 communicate with the example network 1410, which may correspond to any one or more of the Internet and / or the example network 116 described above. In some examples, as part of a business transaction, one or more servers respond to a request to transfer software to a requesting party. Payment for the delivery, sale, and / or license of the software may be handled by one or more servers of the software distribution platform and / or by a third-party payment entity. The server enables a purchaser and / or licensee to download the machine-readable instructions 1132 from the software distribution platform 1105. For example, the example network 1410 may correspond to Figure 6-10 The software of the example machine readable instructions 600 can be downloaded to the example processor platform 1100, which is to execute the machine readable instructions 1132 to implement the deep learning accelerator circuit module 102. In some examples, one or more servers of the software distribution platform 1405 periodically provide software, deliver software, and / or force updates to software (e.g., Fig.11 Example machine readable instructions 1132) to ensure that improvements, patches, updates, etc. are distributed and applied to the software at the end-user devices.

[0115] From the above, it will be appreciated that example systems, methods, devices, and articles of manufacture that accelerate deep learning have been disclosed. The disclosed systems, methods, devices, and articles of manufacture improve the efficiency of using computing devices by efficiently scheduling multi-chip DNN training-based computing features of operations including DNN training. Therefore, the disclosed systems, methods, devices, and articles of manufacture are directed to one or more improvements in the operation of a machine such as a computer or other electronic and / or mechanical device.

[0116] Example methods, devices, systems, and articles of manufacture for accelerating deep learning are disclosed herein. Further examples and combinations thereof include the following:

[0117] Example 1 includes a system comprising an interface circuit module, a programmable circuit module, and instructions for programming the programmable circuit module to classify operation groups of an operation group based on resource utilization of the operation groups of a distributed deep learning workload, select at least two operations in the operation group for overlapping execution based on the classification and a dependency analysis of the at least two operations of the operation group, and perform distributed training of the distributed deep learning workload based on an execution schedule that includes overlapping execution of the selected at least two operations.

[0118] Example 2 includes the system of any of the preceding examples, wherein the programmable circuit module is used to classify the operation of the distributed deep learning workload as one of network-constrained, compute-constrained, memory-constrained, or input / output-constrained.

[0119] Example 3 includes the system of any of the preceding examples, wherein the programmable circuit module is used to perform inter-iteration analysis of two operations in the operation group using a directed graph, wherein the edges of the directed graph connect forward operations and backward operations with the same weight.

[0120] Example 4 includes the system of any of the preceding examples, wherein dependency analysis of at least two operations in the group of operations indicates whether the at least two operations have different classifications and whether data dependencies exist between the at least two operations.

[0121] Example 5 includes the system of any of the preceding examples, wherein the at least two operations are selected for overlapping execution in response to the at least two operations having different classifications and having no data dependency between the at least two operations.

[0122] Example 6 includes the system of any of the preceding examples, wherein a first operation of the distributed deep learning workload is compute-constrained, a second operation of the distributed deep learning workload is memory-constrained, and wherein the system further comprises a graphics processing unit for performing the compute-constrained operation, and a dataflow accelerator for performing the memory-constrained operation. Example 7 includes the system of Example 1, wherein the programmable circuit module is used to assign scheduling priorities to at least two operations in the operation group, and wherein the input / output-constrained operation is assigned a higher scheduling priority than the compute-constrained operation.

[0123] Example 8 includes the system of any of the preceding examples, wherein the programmable circuit module is to identify communication operations for overlapping execution in the at least two operations in the group of operations.

[0124] Example 9 includes the system of any of the preceding examples, wherein, in response to a number of communication operations being greater than a number of non-communication operations identified for overlapping execution, the programmable circuit module is to identify operations in the communication operations for asynchronous execution.

[0125] Example 10 includes a computer-readable medium comprising instructions that, when executed by a processor circuit module, cause the processor circuit module to classify operation groups of a distributed deep learning workload based on resource utilization of the operation groups of the operation groups, select at least two operations in the operation group for overlapping execution based on the classification and a dependency analysis of the at least two operations of the operation group, and perform distributed training of the distributed deep learning workload based on an execution schedule that includes overlapping execution of the selected at least two operations.

[0126] Example 11 includes the computer-readable medium of any of the preceding examples, wherein the instructions, when executed, cause the processor circuit module to classify the operation of the distributed deep learning workload as one of network-constrained, compute-constrained, memory-constrained, or input / output-constrained.

[0127] Example 12 includes a computer-readable medium as described in any of the preceding examples, wherein the instructions, when executed, cause the processor circuit module to perform inter-iteration analysis of two operations in the operation group using a directed graph, wherein the edges of the directed graph connect forward operations and backward operations with the same weight.

[0128] Example 13 includes the computer-readable medium of any of the preceding examples, wherein a dependency analysis of at least two operations in the group of operations indicates whether the at least two operations have different classifications and whether a data dependency exists between the at least two operations.

[0129] Example 14 includes the computer-readable medium of any of the preceding examples, wherein, in response to the at least two operations having different classifications and having no data dependency between the at least two operations, the at least two operations are selected for overlapping execution.

[0130] Example 15 includes the computer-readable medium of any of the foregoing examples, wherein the first operation of the distributed deep learning workload is compute-constrained, the second operation of the distributed deep learning workload is memory-constrained, and wherein the system further comprises a graphics processing unit for performing the compute-constrained operation and a data flow accelerator for performing the memory-constrained operation. Example 16 includes the non-transitory computer-readable medium of Example 10, wherein the instructions, when executed, cause the processor circuit module to assign scheduling priorities to the at least two operations in the operation group, and wherein the input / output-constrained operation is assigned a higher scheduling priority than the compute-constrained operation.

[0131] Example 17 includes the computer readable medium of any of the preceding examples, wherein the instructions, when executed, cause the processor circuit module to identify communication operations for overlapping execution in the at least two operations in the group of operations.

[0132] Example 18 includes the computer-readable medium of any of the preceding examples, wherein, in response to a number of communication operations being greater than a number of non-communication operations identified for overlapping execution, the instructions, when executed, cause the processor circuit module to identify operations among the communication operations for asynchronous execution.

[0133] Example 19 includes a method comprising: executing instructions using a processor circuit module to classify operation groups of a distributed deep learning workload based on resource utilization of the operation groups, executing instructions using a processor circuit module to select at least two operations in the operation group for overlapping execution based on classification and dependency analysis of at least two operations in the operation group, and executing instructions using a processor circuit module to perform distributed training of the distributed deep learning workload based on an execution schedule that includes overlapping execution of the selected at least two operations.

[0134] Example 20 includes the method of any of the preceding examples, further comprising classifying operations of the distributed deep learning workload as one of network-constrained, compute-constrained, memory-constrained, or input / output-constrained.

[0135] Example 21 includes the method of any of the preceding examples, further comprising performing inter-iteration analysis of two operations in the operation group using a directed graph, wherein edges of the directed graph connect forward operations and backward operations with the same weight.

[0136] Example 22 includes the method of any of the preceding examples, wherein dependency analysis of at least two operations in the group of operations indicates whether the at least two operations have different classifications and whether data dependencies exist between the at least two operations.

[0137] Example 23 includes the method of any of the preceding examples, wherein, in response to the at least two operations having different classifications and having no data dependency between the at least two operations, the at least two operations are selected for overlapping execution.

[0138] Example 24 includes the method of any of the preceding examples, wherein a first operation of the distributed deep learning workload is compute-bound, a second operation of the distributed deep learning workload is memory-bound, and wherein the system further includes performing the compute-bound operation on a graphics processing unit and performing the memory-bound operation on a dataflow accelerator unit.

[0139] Example 25 includes the method of any of the preceding examples, further comprising assigning scheduling priorities to the at least two operations in the group of operations, and wherein input / output bound operations are assigned a higher scheduling priority than compute bound operations.

[0140] Example 26 includes the method of any of the preceding examples, further comprising identifying communication operations for overlapping execution in the at least two operations in the group of operations.

[0141] Example 27 includes the method of any of the preceding examples, further comprising: in response to the number of communication operations being greater than the number of non-communication operations identified for overlapping execution, identifying operations in the communication operations for asynchronous execution.

[0142] Example 28 includes a system comprising an interface circuit module, a programmable circuit module, and instructions for programming the programmable circuit module to classify first operations and second operations of a distributed deep learning workload based on a first resource utilization of the first operation and a second resource utilization of the second operation, perform dependency analysis on the first operation and the second operation, including identification of parent nodes and output nodes of the first operation and the second operation, and generate an execution schedule for inference including overlapping execution of the first operation and the second operation.

Claims

1. A system comprising: Interface circuit module; Programmable circuit module; as well as Instructions, the instructions are used to program the programmable circuit module to: classifying the operation groups based on resource utilization of the operation groups for distributed deep learning workloads; selecting the at least two operations in the group of operations for overlapping execution based on the classification and dependency analysis of the at least two operations in the group of operations; as well as Distributed training of the distributed deep learning workload is performed based on an execution schedule that includes overlapping execution of the selected at least two operations.

2. The system according to claim 1, wherein: The programmable circuit module is to classify the operation of the distributed deep learning workload as one of network-bound, compute-bound, memory-bound, or input / output-bound.

3. The system according to claim 1, wherein: The programmable circuit module is to perform inter-iteration analysis of two operations in the operation group using a directed graph, wherein an edge of the directed graph connects a forward operation and a backward operation with the same weight.

4. The system according to claim 1, wherein: The dependency analysis of the at least two operations in the group of operations indicates whether the at least two operations have different classifications and whether there is a data dependency between the at least two operations.

5. The system according to claim 4, wherein: In response to the at least two operations having different classifications and having no data dependency between the at least two operations, the at least two operations are selected for overlapping execution.

6. The system according to claim 1, wherein: The first operation of the distributed deep learning workload is compute bound, the second operation of the distributed deep learning workload is memory bound, and wherein the system further comprises: a graphics processing unit for performing said compute-limited operation; and A data flow accelerator is provided for performing the memory bound operation.

7. The system according to claim 1, wherein: The programmable circuit module is to assign scheduling priorities to the at least two operations in the group of operations, and wherein the input / output bound operation is assigned a higher scheduling priority than the compute bound operation.

8. The system according to claim 1, wherein: The programmable circuit module is to identify communication operations for overlapping execution in the at least two operations in the group of operations.

9. The system according to claim 8, wherein: In response to the number of communication operations being greater than the number of non-communication operations identified for overlapping execution, the programmable circuit module is to identify operations among the communication operations for asynchronous execution.

10. A non-transitory computer readable medium comprising instructions that, when executed by a processor circuit module, cause the processor circuit module to: classifying the operation groups based on resource utilization of the operation groups for distributed deep learning workloads; selecting the at least two operations in the group of operations for overlapping execution based on the classification and dependency analysis of the at least two operations in the group of operations; as well as Distributed training of the distributed deep learning workload is performed based on an execution schedule that includes overlapping execution of the selected at least two operations.

11. The non-transitory computer readable medium of claim 10, wherein: The instructions, when executed, cause the processor circuit module to classify the operation of the distributed deep learning workload as one of network-constrained, compute-constrained, memory-constrained, or input / output-constrained.

12. The non-transitory computer readable medium of claim 10, wherein: The instructions, when executed, cause the processor circuit module to perform inter-iteration analysis on two operations in the group of operations using a directed graph, wherein edges of the directed graph connect forward operations and backward operations with the same weight.

13. The non-transitory computer readable medium of claim 10, wherein: The dependency analysis of the at least two operations in the group of operations indicates whether the at least two operations have different classifications and whether there is a data dependency between the at least two operations.

14. The non-transitory computer readable medium of claim 13, wherein: In response to the at least two operations having different classifications and having no data dependency between the at least two operations, the at least two operations are selected for overlapping execution.

15. The non-transitory computer readable medium of claim 10, wherein: The first operation of the distributed deep learning workload is compute bound, the second operation of the distributed deep learning workload is memory bound, and wherein the system further comprises: a graphics processing unit for performing said compute-limited operation; and A data flow accelerator is provided for performing the memory bound operation.

16. The non-transitory computer readable medium of claim 10, wherein: The instructions, when executed, cause the processor circuit module to assign scheduling priorities to the at least two operations in the group of operations, and wherein the input / output bound operation is assigned a higher scheduling priority than the compute bound operation.

17. The non-transitory computer readable medium of claim 10, wherein: The instructions, when executed, cause the processor circuit module to identify communication operations for overlapping execution in the at least two operations in the group of operations.

18. The non-transitory computer readable medium of claim 17, wherein: In response to the number of communication operations being greater than the number of non-communication operations identified for overlapping execution, the instructions, when executed, cause the processor circuit module to identify operations of the communication operations for asynchronous execution.

19. A method comprising: classifying the operation groups of the distributed deep learning workload based on resource utilization of the operation groups by executing instructions using processor circuit modules; selecting, by executing instructions with the processor circuit module, the at least two operations in the group of operations for overlapping execution based on the classification and dependency analysis of the at least two operations in the group of operations; as well as Distributed training of the distributed deep learning workload is performed based on an execution schedule including overlapping execution of the selected at least two operations by executing instructions using the processor circuit module.

20. The method of claim 19, further comprising classifying the operation of the distributed deep learning workload as one of network-constrained, compute-constrained, memory-constrained, or input / output-constrained.

21. The method of claim 19, further comprising performing inter-iteration analysis of two operations in the group of operations using a directed graph, wherein edges of the directed graph connect forward operations and backward operations with the same weight.

22. The method according to claim 19, wherein: The dependency analysis of the at least two operations in the group of operations indicates whether the at least two operations have different classifications and whether there is a data dependency between the at least two operations.

23. The method according to claim 22, wherein: In response to the at least two operations having different classifications and having no data dependency between the at least two operations, the at least two operations are selected for overlapping execution.

24. The method according to claim 19, wherein: The first operation of the distributed deep learning workload is compute bound, the second operation of the distributed deep learning workload is memory bound, and wherein the system further comprises: performing compute-bound operations on a graphics processing unit; and Memory bound operations are performed on the dataflow accelerator unit.

25. The method of claim 19, further comprising assigning scheduling priorities to the at least two operations in the group of operations, and wherein input / output bound operations are assigned a higher scheduling priority than compute bound operations.

26. The method of claim 19, further comprising identifying communication operations for overlapping execution in the at least two operations in the group of operations.

27. The method according to claim 26, further comprising: In response to the number of communication operations being greater than the number of non-communication operations identified for overlapping execution, operations among the communication operations are identified for asynchronous execution.

28. A system comprising: Interface circuit module; Programmable circuit module; as well as Instructions, the instructions are used to program the programmable circuit module to: classifying the first operation and the second operation based on a first resource utilization of the first operation and a second resource utilization of the second operation of the distributed deep learning workload; performing dependency analysis on the first operation and the second operation, including identifying parent nodes and output nodes of the first operation and the second operation; as well as An execution schedule is generated for the speculation including overlapping execution of the first operation and the second operation.

Images