Method and system for accelerating AI training using advanced interconnect technology

Abstract

According to various embodiments, a method and system for accelerating artificial intelligence (AI) model training using advanced interconnect communication technology and performing system zero-value compression on a distributed training system is provided. During each iteration of performing a scatter-reduce process on a cluster of processors (117, 119, 121) arranged in a logical ring to train a neural network model (701), the processor receives a compressed data block from a previous processor (107) in the logical ring, performs operations on the received compressed data block and the compressed data block generated on the processor (117, 119, 121) to obtain a calculated data block, and sends the calculated data block to a subsequent processor (117, 119, 121) in the logical ring (702). The compressed data block calculated based on the corresponding data block from multiple processors (117, 119, 121) can be identified on each processor (117, 119, 121) and distributed to each of the other processors (117, 119, 121) and decompressed therein for use in AI model training (703).

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to machine learning. More specifically, embodiments of the present disclosure relate to neural network training. Background Art

[0002] In order to solve complex problems, neural networks are becoming increasingly complex. Complex neural networks require complex deep learning algorithms and higher data bandwidth during the training phase. As a result, training time, cost, and power consumption are increased. To accelerate training, advanced servers (e.g., faster servers or server clusters with more complex interfaces) have been used to improve computing and communication to reduce the cost of expensive hardware. However, existing solutions still face challenges in terms of performance and cost. Summary of the invention

[0003] According to a first aspect, some embodiments of the present disclosure provide a computer-implemented method for training an artificial intelligence (AI) model using a data processing (DP) accelerator, the method comprising: receiving a request from a central processing unit (CPU) for training an AI model based on a set of training data having multiple data blocks distributed from the CPU; and performing multiple DP iterations by multiple general-purpose processing units (GPUs) arranged in a logical ring to train the AI ​​model, comprising: for each of the DP iterations, during a first DP cycle, each of the GPUs respectively and in parallel performs a first predetermined DP operation on one of the data blocks to generate a corresponding DP result, and during a second DP cycle, each of the GPUs transmits the first corresponding DP result to a downstream GPU in the logical ring via an inter-processor link for further processing therein.

[0004] According to a second aspect, some embodiments of the present disclosure provide a data processing system comprising: at least one central processing unit (CPU); and multiple general-purpose processing units (GPUs) coupled to the CPU, wherein each of the GPUs is configured to perform an artificial intelligence (AI) data processing (DP) operation distributed from the CPU, the operation comprising: receiving a request from the central processing unit (CPU) for training an AI model based on a set of training data having multiple data blocks distributed from the CPU, and performing multiple DP iterations by the multiple general-purpose processing units (GPUs) arranged in a logical ring to train the AI ​​model, comprising: for each of the DP iterations, during a first DP cycle, each of the GPUs respectively and in parallel performs a first predetermined DP operation on one of the data blocks to generate a corresponding DP result, and during a second DP cycle, each of the GPUs transmits the first corresponding DP result to a downstream GPU in the logical ring via an inter-processor link for further processing therein.

[0005] According to a third aspect, some embodiments of the present disclosure provide a non-temporary machine-readable medium having instructions stored therein, which, when executed by a processor, causes the processor to perform operations for artificial intelligence (AI) training, the operations comprising: receiving a request from a central processing unit (CPU) for training an AI model based on a set of training data having multiple data blocks distributed from the CPU; and performing multiple DP iterations by multiple general-purpose processing units (GPUs) arranged in a logical ring to train the AI ​​model, comprising: for each of the DP iterations, during a first DP cycle, each of the GPUs respectively and in parallel performs a first predetermined DP operation on one of the data blocks to generate a corresponding DP result, and during a second DP cycle, each of the GPUs transmits the first corresponding DP result to a downstream GPU in the logical ring via an inter-processor link for further processing therein. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Embodiments of the present disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like references indicate similar elements.

[0007] Figure 1 An example of a system for AI model training according to an embodiment is shown.

[0008] FIG. 2A to FIG. 2F An exemplary process of data transfer in AI model training according to an implementation is shown.

[0009] Figure 3 Shown is the FIG. 2A to FIG. 2F An exemplary process for improving the process in .

[0010] Figure 4 An exemplary architecture of data compression, data manipulation, and interconnection buses according to an embodiment is shown.

[0011] Figure 5 A zero sum compression technique according to an embodiment is shown.

[0012] Figure 6 Exemplary operations on compressed data according to embodiments are shown.

[0013] Figure 7 An exemplary process of AI model training according to an implementation is shown. DETAILED DESCRIPTION

[0014] Various embodiments and aspects of the present disclosure will be described with reference to the details discussed below, and the accompanying drawings will illustrate various embodiments. The following description and the accompanying drawings are illustrative of the present disclosure and should not be construed as limiting the present disclosure. In order to provide a thorough understanding of the various embodiments of the present disclosure, many specific details are described. However, in some cases, in order to provide a concise discussion of the embodiments of the present disclosure, known or conventional details are not described.

[0015] Reference to "one embodiment" or "an embodiment" in the specification means that a particular feature, structure, or characteristic described in conjunction with the embodiment can be included in at least one embodiment of the present disclosure. The phrase "in one embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment.

[0016] According to various embodiments, methods and systems are provided herein for accelerating artificial intelligence (AI) training using advanced interconnect communication technologies. Using software and hardware components, the embodiments described in this disclosure can significantly reduce interconnect communication bandwidth requirements, power consumption, and reduce training time, thereby improving training performance without the cost of accuracy loss. AI model training is performed using system data compression and decompression on a distributed system, combined with an efficient All-Reduce algorithm.

[0017] According to one embodiment, a computer-implemented method for AI model training includes the following operations: performing multiple iterations in a scatter-reduce process on a processor cluster, each of which can be a graphics processing unit (GPU). The processors can be arranged in a logical ring to train a neural network model, each processor including multiple data blocks, each data block representing a set of parameters in the neural network model or a set of gradients for updating a set of parameters.

[0018] During each iteration, a processor receives a compressed data block from a previous processor in the logical ring, performs an operation on the received compressed data block and a compressed data block generated on the processor to obtain a calculated data block, and sends the calculated data block to a subsequent processor in the logical ring. At the end of the plurality of iterations, each data block on the plurality of processors will be compressed and operated on. The method also includes identifying an operation on each of the plurality of processors for a compressed data block that is calculated from corresponding data blocks of the plurality of processors.

[0019] In one embodiment, the identified compressed data block can be distributed to each of the other processors in the logical ring and can be decompressed on the processor for updating parameters in the neural network model. The processor can be attached to a central processing unit (CPU) in a different system of the distributed AI model training system. In one embodiment, each processor may include a hardware-based or software-based compression module that compresses and decompresses data blocks using zero-value compression techniques. The compressed data block can be represented by a data structure having a bit mask portion and a compressed data portion, and the bit mask portion includes bits indicating positions in the data block having non-zero values.

[0020] According to one embodiment, when a request for AI training is received from a central processing unit (CPU), each of the general processing units (GPUs) arranged in a logical ring is configured to iteratively perform data processing (DP) operations on data blocks distributed from the CPU in a pipelined manner. Relative to the CPU, each GPU operates as a DP accelerator. For each iteration, during the first DP cycle, each GPU performs a first predetermined DP operation (e.g., data compression) on one of the data blocks in parallel and generates a corresponding DP result. During the second DP cycle, each GPU transmits the corresponding DP result to the corresponding downstream GPU in the logical ring via the corresponding inter-processor link for further processing therein. For the purpose of illustration, a GPU is used as an example of a DP accelerator, however, other types of processors or processing logic may also be used as DP accelerators.

[0021] In one embodiment, during the second DP cycle, each GPU also receives processing results from its corresponding upstream GPU in the logical ring via the corresponding inter-processor link for further processing at the GPU. In one embodiment, during the third DP cycle, each of the GPUs simultaneously performs a second predetermined DP operation (e.g., a combination operation, such as summation) on a first data block processed by itself (e.g., processing results) and a second data block received from its upstream GPU (e.g., processing results of the upstream GPU). In one embodiment, during the fourth DP cycle, each of the GPUs performs further DP operations, such as data decompression operations.

[0022] Figure 1 An example of a system for AI model training according to an embodiment is shown. Figure 1As shown, the system includes a GPU cluster 101 dispersed in a plurality of servers (e.g., server A 103 and server B 105), wherein each server includes one or more CPUs, and each CPU is associated with one or more DP accelerators such as GPUs. The server may include a CPU 107 and a CPU 109 communicating with each other via an Ethernet connection 111. Figure 1 In the exemplary system shown, each CPU may have multiple GPUs connected to the CPU via a PCIe (Peripheral Component Interconnect Express) switch. For example, in server A 103, GPU 117, GPU 119, and GPU 121 are connected to CPU A 107 via PCIe switch A 113; in server B 105, GPU 123, GPU 125, and GPU 127 are connected to CPU B 109 via PCIe switch B 115.

[0023] CPU 107 and CPU 109 may communicate with each other via an inter-processor link such as Ethernet connection 111 to coordinate the task of training a neural network. For example, a job command may be distributed to each server via Ethernet connection 111. The job command may then be dispatched from the CPU in the server to the GPU connected to the CPU. Once the job command is dispatched, data between the GPUs in the system may be passed via corresponding inter-processor links or connections such as a Cache Coherent Interconnect for Accelerators (CCIX) compatible connection. Figure 1 As shown, the GPUs in the system are arranged in a bidirectional ring topology, although a unidirectional ring topology can be used.

[0024] CCIX is an open cache coherent interconnect architecture developed by the CCIX Consortium. CCIX is designed to simplify communication between central processing units (e.g., CPUs) and various accelerators (e.g., GPUs) in a system through cache coherence extensions to standard PCIe. CCIX is a high-performance chip-to-chip interconnect architecture that provides a cache coherence framework for heterogeneous system architectures. Cache coherence between the central processing unit and various other accelerators in the system is always automatically maintained. Each CCIX-enabled device includes at least one CCIX port, which is pin-compatible with any other CCIX-enabled device. CCIX supports a large number of topologies, such as chip-to-chip, chip-to-switch-to-chip, mesh, daisy chain, and ring.

[0025] In one embodiment, the GPU is configured to perform AI training operations on data blocks distributed from its corresponding CPU in a pipelined manner. Each of the GPUs also communicates with each other via an inter-processor link. The GPUs can be configured in a ring arrangement to receive processing results from an upstream GPU for further data processing. Each GPU may further send the processing results to its corresponding downstream GPU for further processing therein. Therefore, each of the GPUs performs its assigned DP operations in parallel and sends its DP results to the downstream GPU. At the same time, each GPU also receives processing results from its upstream GPU for further processing therein.

[0026] FIG. 2A to FIG. 2F An exemplary process of data transfer in AI model training according to an embodiment is shown. Although three GPUs 203, GPU 205, and GPU 207 are shown herein, the exemplary process may use as many GPUs as possible (e.g., thousands of GPUs) depending on multiple factors, such as the complexity of the neural network to be trained, the size of the training data, and the training speed desired by the user.

[0027] Examples of neural networks that can be trained on the example system include multilayer perceptron (MLP) neural networks, which include a collection of connected neurons. Neurons in an MLP neural network can be fully connected, as each neuron in one layer is connected to each neuron in a subsequent layer with parameters (e.g., weights and biases).

[0028] During the training of a neural network model, gradient descent (i.e., backpropagation) can be used to determine a set of parameters to minimize the difference between the expected value and the actual output of the neural network model. Gradient descent includes the following steps: calculating the gradient of the loss / error function, and updating the existing parameters in response to the gradient. This cycle can be repeated until a minimum value of the loss function is reached.

[0029] In one embodiment, the training data set for the neural network model can be divided into multiple subsets, each subset is used to train the neural network model on one of the GPUs, so that the training of the neural network can be performed in parallel by multiple GPUs. Each GPU can have a complete copy of the neural network model.

[0030] Each subset of the training data set can be logically divided into multiple equal-sized data blocks. In an exemplary process, the number of blocks can be equal to the number of GPUs. Parallelized training of the neural network model requires multiple iterations for gradient descent. For each iteration, each GPU runs the forward propagation of the neural network model on the data on the GPU, followed by error back propagation to calculate the gradient of the loss relative to the network parameters. The GPUs can then communicate with each other to calculate the statistics of the gradient (e.g., the average, maximum, or minimum value), and apply the statistics (e.g., the average gradient) to obtain updated parameters. The neural network model can have a large number of parameters (e.g., billions of parameters), and each parameter can be associated with a gradient value. In this way, for a neural network, the size of the gradient can be huge, and transferring the gradient between GPUs can take up a lot of bandwidth.

[0031] Return to reference FIG. 2A to FIG. 2F , an exemplary process illustrates an algorithm for reducing the bandwidth requirements for data transfer between GPUs. In one embodiment, bandwidth as used herein is the maximum data transfer rate on a given network connection. The algorithm may include two processes. The first process is a Scatter-Reduce process, and the second process is an Allgather process. During the Scatter-Reduce process, the GPUs may exchange data so that each GPU ends up with a large number of final result blocks. During the Allgather process, the GPUs may exchange those result blocks so that all GPUs end up with a complete final result.

[0032] Each GPU may include one or more applications configured to divide a subset of the training data set on that GPU into equal-sized data blocks; in an exemplary system, the number of data blocks on each GPU is the number of GPUs. During training of the neural network model, each data block may generate its own set of gradients.

[0033] In this example, as shown above, since there are three GPUs in the system, the number of data blocks on each GPU is three. From the subset of training data on GPU #0 203, three sets of gradients a0 215, b0 231, c0 237 can be generated; from the subset of training data on GPU #1 205, three sets of gradients a1 217, b1 223, c1 239 can also be generated. Similarly, from the subset of training data on GPU #2 207, three sets of gradients a2 219, b2 235, c2 241 are generated. In one embodiment, different gradient groups on each GPU can be stored in an array or another data structure.

[0034] As an illustration, the algorithm is designed to sum the gradients generated by each subset of the training dataset so that when the algorithm completes, each GPU will have the sum of the gradients generated from the training dataset.

[0035] The GPUs in the exemplary process may have N-1 iterations during the Scatter-Reduce process, where N is the total number of GPUs in the system. Thus, the GPUs in the exemplary system may have 2 iterations. In each iteration, each GPU may send a set of gradients on the GPU to its right neighbor, receive a set of gradients from its left neighbor, and add the two sets of gradients to generate a new set of gradients. For each iteration, the set of gradients sent and received by each GPU is different; the nth GPU starts by sending the nth set of gradients, and receives the (n-1)th set of gradients, and then processes backwards from there.

[0036] FIG. 2A to FIG. 2C The Scatter-Reduce process is shown. Figure 2A The data transfer in the first iteration of the Scatter-Reduce process is shown. After completing the first send and the first receive, each GPU will have an array element with a value representing the sum of two sets of gradients on two different GPUs. For example, the first element a1 on GPU 205 may include the sum of the gradient groups from the second GPU 205 and the first GPU 203. Figure 2B The data transfer in the second iteration of the Scatter-Reduce process is shown, and the intermediate sums after the first iteration of the Scatter-Reduce process is completed are also shown. In the second iteration, the Scatter-Reduce process continues, and at the end of the Scatter-Reduce process (i.e., after the second iteration in this example), each GPU will have an array element that contains the sum of all gradients in the corresponding array elements across all GPUs. Figure 2C The final state at the end of the Scatter-Reduce process is shown.

[0037] FIG. 2D to FIG. 2F The Allgather process is shown, which proceeds identically to Scatter-Reduce and also has N-1 iterations, except that the received gradients overwrite the gradients in the corresponding array elements on the receiving GPU instead of accumulating the gradients received by the GPU. Figure 2D Figure 2 shows the data transfer in the first iteration of the Allgather process. Figure 2E As shown in , after completing the first iteration, each GPU will have two array elements that each contain the sum of all gradients in the corresponding array elements across all GPUs. Figure 2EThe Allgather process is shown in the second iteration, which is the final iteration in the exemplary process. Figure 2F At the end of the Allgather process shown, the GPU will have fully accumulated gradients from the entire training data set. Since all data passes occur synchronously in discrete iterations, the exemplary process is bandwidth-optimal.

[0038] Figure 3 Shown is the FIG. 2A to FIG. 2F An exemplary process for improving the process in . In one embodiment, Figure 3 The exemplary process shown in can be used to pass gradients for updating neural network parameters during the training of a neural network model, where tens of megabytes of data may need to be passed between distributed servers and operated in coordination. This requires efficient hardware and software to improve performance and latency.

[0039] In one embodiment, the exemplary process utilizes the All-Reduce algorithm and software and hardware co-design to improve performance and latency. Software and hardware co-design refers to designing hardware and software simultaneously to achieve the desired functionality. The exemplary process uses hardware components such as the Cache Coherent Interconnect (CCIX) for accelerators, for connecting GPUs in a cluster, and software modules such as zero-value compression modules and other compression modules to enable hardware calculations based on compressed data. The exemplary process uses system data compression on a distributed system designed to perform an efficient All-Reduce process. Therefore, the gradients generated from different subsets of the training data set can be accumulated and distributed to each GPU more quickly, thereby accelerating AI model training.

[0040] exist Figure 3 The left part shows a typical All-Reduce process 302, which has been FIG. 2A to FIG. 2F The right part describes the improved All-Reduce process using system compression on a distributed system. Figure 3 Three GPUs are used as an example, and the three GPUs are arranged to form a logical ring.

[0041] In the typical All-Reduce process 302 and the improved All-Reduce process, the data blocks to be transferred between CPUs are stored in a data structure (e.g., an array), and the data blocks can be gradients generated from different blocks of a subset of a training data set used to train a neural network model. Each GPU can have a complete copy of the neural network model to be trained. The gradients are transferred between GPUs for updating the parameters of the neural network model.

[0042] In one embodiment, during the first iteration or first processing cycle of the Scatter-Reduce process, the data blocks on each GPU may be compressed by a compression module, which may be implemented in hardware or as a software module. For example, in operations 301, 315, and 329, data block a0 on GPU #0 203, data block b1 on GPU #1 205, and data block c2 in GPU #2 207 are compressed, respectively.

[0043] The compressed data block may be sent to the adjacent GPU in the next processing cycle. For example, in operation 303, the compressed data block on GPU#0 203 may be sent to GPU#1 205, in operation 317, the compressed data block on GPU#1 205 may be sent to GPU#2 207, and in operation 331, the compressed data block on GPU#2 207 may be sent to GPU#0 203.

[0044] In one embodiment, while compressed data blocks are being sent to neighboring GPUs, different data blocks on each GPU may be compressed and added to the compressed data received as described above. In this example embodiment, an addition operation is used as an example; other operations (e.g., multiplication, deduction, and mathematical average) may also be used.

[0045] For example, in operation 305, data block c0 on GPU#0 203 may be compressed and added to compressed data block c2 received from GPU#2 207. In operation 319, data block a1 on GPU#1 205 may be compressed and added to compressed data block a0 received from GPU#0 203. In operation 333, data block b2 on GPU#2 207 may be compressed and added to compressed data block b1 received from GPU#1 205.

[0046] The above process may be repeated for each remaining iteration of the Scatter-Reduce process. The number of iterations may be the number of GPUs minus 1. Thus, the Scatter-Reduce process 305 in the improved All-Reduce process may have 2 iterations. In each of the remaining iterations, instead of sending the original compressed data block on the GPU, each GPU may send the sum of the compressed data blocks from multiple GPUs to the next GPU.

[0047] For example, during the second iteration, in operation 307, GPU#0 203 may send the sum of compressed data block c0 and compressed data block c2 to GPU#1 205. In operation 321, GPU#1 205 may send the sum of compressed data block a0 and compressed data block a1 to GPU#2 207. In operation 335, GPU#2 207 may send the sum of compressed data block b1 and compressed data block b2 to GPU#0 203.

[0048] In one embodiment, while the sum of compressed data blocks is being sent to the neighboring GPUs, each GPU may compress the remaining data blocks on the GPU and add the compressed data blocks to the sum of compressed data blocks previously received from the previous GPU in the logical ring. For example, in operation 309, data block b0 on GPU #0 203 may be compressed and added to the sum of compressed data blocks b1 and b2. In operation 323, data block c1 on GPU #1 205 may be compressed and added to the sum of compressed data blocks c0 and c2. In operation 337, data block a2 on GPU #2 207 may be compressed and added to the sum of compressed data blocks a0 and a1.

[0049] Thus, at the end of the Scatter-Reduce process, in the exemplary process, each GPU will have the sum of the compressed data blocks from the corresponding position in the array across all GPUs.

[0050] During the Allgather process, each GPU may distribute the sum of the compressed data blocks from the corresponding positions in the array to each other's GPUs, so that at the end of the Allgather process, each GPU will have a copy of the sum of all compressed data blocks. Then, as shown in operations 313, 327, and 341, each GPU may decompress the compressed sum. The decompressed sum on each GPU may be used to update the parameters of the copy of the neural network model on that GPU.

[0051] Figure 4 An exemplary architecture of data compression, data manipulation, and interconnection buses according to an embodiment is shown.

[0052] Figure 4 The diagram in FIG. 4 shows a data flow of compressing original data blocks 405 and 407, transferring the compressed data blocks on interconnection buses 416 and 418, performing operations 413 and 419 on the compressed data, and decompressing the compressed data into original data.

[0053] like Figure 4As shown, a pair of compression modules and decompression modules can be used on each GPU. For example, on GPU A 401, compression module 411 and decompression module 409 are used, while on GPU B 403, compression module 417 and decompression module 415 are used.

[0054] For compression modules 411 and 417, any compression algorithm may be used. An exemplary compression algorithm is a zero-value compression algorithm / technique, which is described in detail in the following disclosure. With a zero-value ratio of 50%, using a zero-value compression algorithm can save nearly 50% of the bandwidth of data transfer; when combined with interconnection buses and various operations on compressed data, the bandwidth benefit can exceed 50%.

[0055] Figure 5 A zero sum compression technique according to an embodiment is shown. Figure 5 , matrix 513 is the original 4×4 data array used to train the neural network model. Data structure 510 shows a compressed form of matrix 513 using a zero value compression technique. Data structure 510 includes multiple fields, such as type field 501, length field 503, bit mask field 505, and compressed data field 507. Matrix 513 and data structure 510 can be converted to each other using compression 511 and decompression 509.

[0056] In one embodiment, the type field 501 indicates the data type of the values ​​in the matrix 513. Examples of data types may include floating point (FP) 32, FP16, and integer (INT) 8. The length indicates the total size of the bit mask field 505 and the compressed data field 507 in bytes; or indicates the size of the compressed data field 507 with a fixed size bit mask byte. The bit mask field 505 field is set to "1" to indicate a non-zero value in a specific position in the matrix 513, and is set to "0" to indicate a zero value. The compressed data field 507 only includes non-zero value data with the correct alignment / offset. The bit mask field can also be used by a decompression module (e.g., Figure 4 The decompression module 409 or 415 in is used to write the non-zero values ​​back to their original positions in the 4×4 data array 513.

[0057] Figure 6 FIG. 2 shows an exemplary operation on compressed data according to an embodiment. Figure 6 As shown, a sum operation is provided as an example to illustrate how to operate on two compressed data blocks.

[0058] In one embodiment, compressed data 617 is a data structure representing matrix A 613 in a compressed form thereof, and compressed data 619 is a data structure representing matrix B 615 in a compressed form thereof. Figure 5The compression technique depicted in generates these two structures, and a decompression module (eg, decompression module 409 or 415) may be used to decompress the two structures into matrix A 613 and matrix B 615, respectively.

[0059] In one embodiment, to sum the two compressed matrices 613 and 615 in their compressed form, a hardware compression module (e.g., Figure 4 The compression module 411 or 417 in may first pipeline the two compressed data structures 617 and 619, compare the bits in the bit mask field in one data structure with the bits in the bit mask field in the other data structure, and output a result 621 based on the comparison.

[0060] By passing data between GPUs in compressed form, the bandwidth requirements for data transfer can be reduced. In addition, compressed data blocks take up less memory than their uncompressed form because fewer bits are read from and written to memory during operation, thus also reducing the memory requirements for operating compressed data blocks.

[0061] For example, a sum operation may require two reads and one write. Since the data read from the memory and written to the memory is in compressed form, the memory requirements for the sum operation are reduced.

[0062] Figure 7 An exemplary process 700 of AI model training according to an embodiment is shown. Process 700 may be performed by processing logic that may include software, hardware, or a combination thereof.

[0063] Return to reference Figure 7 In operation 701, multiple iterations are performed on multiple processors arranged in a logical ring to train a neural network model, wherein each processor includes multiple data blocks. In operation 702, for each of the multiple iterations, a processor among the multiple processors receives a compressed data block from a previous processor in the logical ring, performs an operation on the received compressed data block and the compressed data block generated on the processor to obtain a calculated data block, and sends the calculated data block to a subsequent processor in the logical ring. In operation 703, a compressed data block calculated based on a corresponding data block from the multiple processors is identified on each of the multiple processors. The identified data block is distributed to each of the other processors and decompressed therein for use in AI model training, such as updating parameters of a neural network model.

[0064] Note that some or all of the components shown and described above may be implemented in software, hardware, or a combination thereof. For example, such a component may be implemented as software installed and stored in a permanent storage device, which may be loaded and executed by a processor (not shown) in a memory to implement the processes or operations described throughout this application. Alternatively, such a component may be implemented as executable code programmed or embedded in dedicated hardware such as an integrated circuit (e.g., a dedicated IC or ASIC), a digital signal processor (DSP), or a field programmable gate array (FPGA), which may be accessed from an application via a corresponding driver and / or operating system. In addition, such a component may be implemented as specific hardware logic in a processor or processor core as part of an instruction set accessible by a software component via one or more specific instructions.

[0065] Some portions of the preceding detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulations of physical quantities.

[0066] All of these and similar terms will be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless explicitly stated otherwise as is apparent from the above discussion, it should be understood that throughout the specification, discussions utilizing terms such as those set forth in the following claims refer to the actions and processes of a computer system or similar electronic computing device that converts data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities in the computer system's memories or registers or other such information storage, transfer, or display devices.

[0067] Embodiments of the present disclosure also relate to devices for performing the operations herein. Such a computer program is stored in a non-transitory computer-readable medium. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., computer) readable storage medium, such as a read-only memory ("ROM"), a random access memory ("RAM"), a magnetic disk storage medium, an optical storage medium, a flash memory device.

[0068] The processes or methods depicted in the preceding figures may be performed by processing logic comprising hardware (e.g., circuits, dedicated logic, etc.), software (e.g., contained on a non-transitory computer-readable medium), or a combination of both. Although the processes or methods are described above in terms of some sequential operations, it should be understood that some of the operations described may be performed in a different order. Furthermore, some operations may be performed in parallel rather than sequentially.

[0069] The embodiments of the present disclosure are not described with reference to any particular programming language. It should be appreciated that a variety of programming languages ​​can be used to implement the teachings of the embodiments of the present disclosure as described herein.

[0070] In the foregoing description, embodiments of the present disclosure have been described with reference to specific exemplary embodiments of the present disclosure. It will be apparent that various modifications may be made thereto without departing from the broader spirit and scope of the present disclosure as set forth in the claims that follow. Therefore, the description and drawings are to be considered illustrative rather than restrictive.

Claims

1. A computer-implemented method for training an artificial intelligence (AI) model using a data processing (DP) accelerator, the method comprising: receiving a request from a central processing unit (CPU) for training an AI model based on a set of training data having a plurality of data blocks distributed from the CPU; Generate multiple data blocks representing gradients from a subset of the training data set on a general-purpose processing unit (GPU) for updating the parameters of the AI ​​model; as well as A plurality of general purpose processing units (GPUs) arranged in a logic ring execute a plurality of DP iterations to train the AI ​​model, wherein the plurality of DP iterations include: for each of the DP iterations, During a first DP cycle, each of the GPUs respectively performs a first predetermined DP operation on one of the data blocks in parallel to generate a first corresponding DP result, wherein one of the plurality of GPUs receives a compressed data block from a previous GPU in the logical ring, and performs an operation on the received compressed data block and a compressed data block generated at the GPU to generate the first corresponding DP result; and during a second DP cycle, each of the GPUs transmitting the first respective DP result via an inter-processor link to a downstream GPU in the logical ring for further processing therein; At the end of multiple iterations, the GPU has a compressed copy of the sum of all compressed data blocks; The compressed copy is distributed to each of the GPUs and decompressed within the GPU.

2. The method according to claim 1, further comprising: During the second DP cycle, a second respective DP result is received by each of the GPUs via a respective inter-processor link from an upstream GPU in the logical ring for further processing.

3. The method according to claim 2, further comprising: During a third DP cycle, a second predetermined DP operation is simultaneously performed by each of the GPUs on data blocks processed in the GPU and received from a corresponding upstream GPU.

4. The method according to claim 3, wherein: The second predetermined DP operation is performed on a first data block processed by the current GPU and a second data block processed by a corresponding upstream GPU to generate a first DP result.

5. The method according to claim 4, wherein: The second predetermined DP operation includes a combined operation of the first data block and the second data block.

6. The method according to claim 4, further comprising: During a fourth DP cycle, a third predetermined DP operation is performed on the first DP result simultaneously by each of the GPUs.

7. The method according to claim 6, wherein: The third predetermined DP operation includes a data decompression operation.

8. The method according to claim 1, wherein: The inter-processor link comprises a cache coherent interconnect CCIX connection of the accelerator.

9. The method according to claim 1, wherein: At least a portion of the data blocks represent parameters or gradients generated as part of the training of the AI ​​model.

10. The method according to claim 1, wherein: The first predetermined DP operation includes a data compression operation.

11. The method according to claim 10, wherein: The data compression operation is performed using a zero-value compression algorithm that compresses one or more data blocks into a data structure having a bit mask portion and a compressed data portion, wherein the bit mask portion includes bits indicating locations in the data blocks having non-zero values.

12. Data processing system, including: at least one central processing unit CPU; as well as A plurality of general purpose processing units (GPUs) are connected to the CPU, wherein each of the GPUs is configured to perform artificial intelligence (AI) data processing (DP) operations distributed from the CPU, the operations comprising: receiving a request from a central processing unit (CPU) for training an AI model based on a set of training data having a plurality of data blocks distributed from the CPU, generating a plurality of data blocks representing gradients from a subset of the training data set on the GPU for updating parameters of the AI ​​model; and A plurality of general purpose processing units (GPUs) arranged in a logic ring execute a plurality of DP iterations to train the AI ​​model, wherein the plurality of DP iterations include: for each of the DP iterations, During a first DP cycle, each of the GPUs respectively performs a first predetermined DP operation on one of the data blocks in parallel to generate a first corresponding DP result, wherein one of the plurality of GPUs receives a compressed data block from a previous GPU in the logical ring, and performs an operation on the received compressed data block and a compressed data block generated at the GPU to generate the first corresponding DP result; and during a second DP cycle, each of the GPUs transmitting the first respective DP result via an inter-processor link to a downstream GPU in the logical ring for further processing therein; At the end of multiple iterations, the GPU has a compressed copy of the sum of all compressed data blocks; the compressed copy is distributed to each of the GPUs and decompressed in the GPU.

13. The data processing system according to claim 12, wherein: The operations also include receiving, by each of the GPUs, a second respective DP result from an upstream GPU in the logical ring via a respective inter-processor link for further processing during the second DP cycle.

14. The data processing system according to claim 13, wherein: The operations further include, during a third DP cycle, simultaneously performing, by each of the GPUs, a second predetermined DP operation on a data block processed in the GPU and received from a corresponding upstream GPU.

15. The data processing system according to claim 14, wherein: The second predetermined DP operation is performed on a first data block processed by the current GPU and a second data block processed by a corresponding upstream GPU to generate a first DP result.

16. The data processing system according to claim 15, wherein: The second predetermined DP operation includes a combined operation of the first data block and the second data block.

17. The data processing system according to claim 15, wherein: The operations further include performing, by each of the GPUs simultaneously, a third predetermined DP operation on the first DP result during a fourth DP cycle.

18. The data processing system according to claim 17, wherein: The third predetermined DP operation includes a data decompression operation.

19. A non-transitory machine-readable medium having instructions stored therein, wherein when the instructions are executed by a processor, the processor is caused to perform an operation of artificial intelligence (AI) training, wherein the operation comprises: receiving a request from a central processing unit (CPU) for training an AI model based on a set of training data having a plurality of data blocks distributed from the CPU; generating a plurality of data blocks representing gradients from a subset of the training data set on a general processing unit (GPU) for updating parameters of the AI ​​model; as well as A plurality of general purpose processing units (GPUs) arranged in a logic ring execute a plurality of DP iterations to train the AI ​​model, wherein the plurality of DP iterations include: for each of the DP iterations, During a first DP cycle, each of the GPUs respectively performs a first predetermined DP operation on one of the data blocks in parallel to generate a first corresponding DP result, wherein a GPU among the plurality of GPUs receives a compressed data block from a previous GPU in the logical ring, and performs an operation on the received compressed data block and a compressed data block generated at the GPU to generate a first corresponding DP result; and During a second DP cycle, each of the GPUs transmits the first respective DP result via an inter-processor link to a downstream GPU in a logical ring for further processing therein; At the end of multiple iterations, the GPU has a compressed copy of the sum of all compressed data blocks; The compressed copy is distributed to each of the GPUs and decompressed within the GPU.

20. The non-transitory machine-readable medium of claim 19, wherein: The operations also include receiving, by each of the GPUs, a second respective DP result from an upstream GPU in the logical ring via a respective inter-processor link for further processing during the second DP cycle.

Images