In-switch embedded packet pooling

Abstract

An apparatus and method for reducing memory bandwidth for executing a machine learning model. A computing system includes two or more processing nodes, each processing node including at least one or more processors and a corresponding local memory. The switch circuitry is in communication with at least the local memory and a system memory of the computing system. The switch includes a plurality of direct memory access (DMA) interfaces. Each of the one or more processing nodes stores a plurality of embedded rows of an embedded table. A processor of the processing node identifies two or more embedded rows as source operands of the reduction operation. The switch executes a memory access request to retrieve data of two or more embedded rows from corresponding local memories, and generates a result by performing a reduction operation. The switch sends the result to a local memory.

Description

Background Technology

[0001] Description of related technologies Neural networks are used in a variety of applications across many fields, such as physics, chemistry, biology, engineering, social media, and finance. A neural network uses one or more layers of nodes to classify data in order to provide an output value representing a prediction given a set of inputs. Weights are used to determine the amount of influence a change in a particular input data value will have on a specific output data value within one or more layers of the neural network. The cost of using a trained neural network includes providing hardware resources capable of handling a relatively high number of computations and supporting data storage and memory bandwidth for accessing parameters. Parameters include input data values, weights, biases, and activation values.

[0002] To improve efficiency, recommender systems utilizing neural networks skip matrix multiplication or other combination operations between the encoded input vector and the first hidden layer, instead using lookup operations with one or more embedding tables. Each entry in an embedding table stores a vector of weights to be used in the first hidden layer. These weights are determined during the training of the neural network. Matrix multiplication or other combination operations are replaced by lookup operations with one or more embedding tables. The lookup operation uses the encoded vector as an index. However, as the number of features increases, as the number of users increases, and as the amount of available content increases (e.g., the number of songs in an online music business using a recommender system), the number and size of the embedding tables also increase. For example, the number of embedding rows (or columns) in each embedding table can reach millions.

[0003] The large number and size of embedding tables mean that most of their contents are stored in system memory rather than in on-die cache. Furthermore, memory access to embedding tables often involves irregular memory access operations, making spatial and temporal locality of data unsuitable for generating efficient memory accesses. All these factors combined result in a significant increase in memory bandwidth. This increase in memory bandwidth degrades system performance while increasing power consumption. If an organization cannot afford the cost of using trained neural networks, it cannot benefit from them.

[0004] In view of this, there is a need for methods and apparatus for reducing memory bandwidth for executing machine learning models. Attached Figure Description

[0005] Figure 1 This is a generalized diagram of a computing system that reduces the memory bandwidth required to execute machine learning models.

[0006] Figure 2 This is a generalized diagram of embedded row reduction to reduce memory bandwidth for executing machine learning models.

[0007] Figure 3 This is a generalized illustration of a device for reducing memory bandwidth required to execute machine learning models.

[0008] Figure 4 This is a generalized illustration of a device for reducing memory bandwidth required to execute machine learning models.

[0009] Figure 5 This is a generalized diagram illustrating methods for reducing memory bandwidth when executing machine learning models.

[0010] Figure 6 This is a generalized diagram illustrating methods for reducing memory bandwidth when executing machine learning models.

[0011] While the invention may have various modifications and alternatives, specific embodiments are shown by way of example in the accompanying drawings and are described in detail herein. However, it should be understood that the drawings and the detailed description thereof are not intended to limit the invention to the specific forms disclosed, but rather, the invention covers all modifications, equivalents, and alternatives falling within the scope of the invention as defined by the appended claims. Detailed Implementation

[0012] In the following description, numerous specific details are set forth to provide a thorough understanding of the invention. However, those skilled in the art will recognize that the invention can be practiced without these specific details. In some instances, well-known circuits, structures, and techniques have not been shown in detail to avoid obscuring the invention. Furthermore, it should be understood that, for the sake of simplicity and clarity, the elements shown in the figures are not necessarily drawn to scale. For example, the dimensions of some elements are magnified relative to other elements.

[0013] Apparatus and methods for reducing memory bandwidth for executing machine learning models are envisioned. In various embodiments, the computing system includes two or more processing nodes in a multi-node partition. Each processing node includes at least one or more processors and corresponding local memory. A switch circuitry (“switch”) of the multi-node partition communicates with at least the local memory and the system memory of the computing system. In various embodiments, the switch includes multiple direct memory access (DMA) interfaces. Each processing node is capable of sending direct memory access (DMA) requests, as well as other commands, messages, responses, and tasks, to another processing node and the switch. Each of the one or more processing nodes stores multiple entries of a data array.

[0014] In various implementations, the data array is an embedding table used in one of several machine learning models, and the entries are embedding rows. The processing node stores the data array in its local memory. The processor of the processing node identifies two or more entries of the data array as source operands for a reduction operation. Examples of reduction operations are summation, maximum value, one of several Boolean logic operations, mathematical mean or average, etc. The processor sends the reduction operation to the switch, and the switch executes a memory access request to retrieve the data for two or more entries.

[0015] A switch retrieves data from two or more entries and generates a result by performing a reduction operation using the data from the two or more entries. The switch sends the result to the local memory of the processing node. By having the switch, rather than any processor on the processing node, perform the reduction operation, the switch reduces memory bandwidth requirements and power consumption in multi-node partitioning. While the switch performs memory accesses and performs the reduction operation, the processor is able to perform other tasks. The processor is also able to perform memory accesses to the local memory using a separate link or bus different from the link or bus used between the switch and the local memory. In some implementations, any combination of reduction operations and collective communication operations (data transfer operations) is fused into a single combined operation that performs the steps of the individual operations used to create the single combined operation. Using this fused operation, one or more results generated by the switch are sent to processors other than the processor from which the input data array is loaded. The fused operation reduces data movement between the local memories of individual processors, which reduces memory bandwidth, power consumption, and latency. Figures 1 to 6 Further details of these techniques for reducing memory bandwidth when executing machine learning models are provided in the following description.

[0016] Turn now Figure 1This diagram illustrates a generalized representation of a computing system 100 that reduces memory bandwidth for executing machine learning models. The computing system 100 includes an index generator 110, a data retrieval stage 120, and a data processing stage 160 combined to handle a workload. The workload involves sparse access to data items stored in data item tables 132A-132B (or tables 132A-132B). The workload receives input 102, and the data processing stage 160 generates a result 162 based on a plurality of sparsely located data items targeted by a lookup request 124. In various specific implementations, circuitry of processor 146 or another processor (not shown) performs the functions of the index generator 110, data retrieval stage 120, and data processing stage 160. One example of input 102 is a user query including a user identifier (ID) and a movie name with a corresponding item ID, and the result 162 is the probability of selecting (mouse click) another movie name presented on a webpage. Several other examples are also possible and conceivable. A data item represents a data unit that the data processing stage 160 operates on during the initial data processing steps. In some implementations, the index generator 110, the data retrieval stage 120, and the data processing stage 160 are combined to form a multi-layer network (or data model). The multi-layer network classifies the data to provide outputs, such as result 162, which represents a prediction given an index 112 generated from input 102.

[0017] In various implementations, the functionality of one or more of the index generator 110, interface 122, switch 140, and data processing stage 160 is implemented on an integrated circuit. Examples of integrated circuits are a central processing unit (CPU), a CPU processor core, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), a field-programmable gate array (FPGA), etc. In other implementations, the functionality of one or more of the index generator 110, interface 122, switch 140, and data processing stage 160 is implemented on a separate integrated circuit (such as processor 146, different processor cores of a processor (such as processor 146), different semiconductor dies on a system-on-a-chip (SoC), a multi-chip module (MCM), or others).

[0018] Index generator 110 receives input 102 and generates one or more indexes 112. Indexes in index 112 identify specific entries in specific tables of tables 132A-132B. Although two tables 132A-132B are shown, any number of tables storing data items is possible and conceivable. Interface 122 generates one or more lookup requests 124 based on index 112. The multiple data items targeted by lookup requests 124 are sparsely located on entries in one or more tables of tables 132A-132B. Therefore, although data items may be stored contiguously in tables 132A-132B, target data items are typically stored non-contiguously in one or more tables of tables 132A-132B.

[0019] In one embodiment, Table 130 is stored in low-level memory, such as a low-level cache (e.g., a Level 3 (L3) cache), local node memory implemented by one type of dynamic random access memory (DRAM), system memory implemented by one type of DRAM, disk storage device, or remote memory accessed via a network. Memory 144 is one of these types of low-level memory. Switch 140 includes circuitry for routing packets between one or more sources and one or more destinations. In some embodiments, switch 140 is used within a communications texture. In various embodiments, packets include memory access requests for data stored in memory 144, processor 146, or another processor (not shown). Switch 140 also transmits packets between processors, such as at least with processor 146. Switch 140 includes one or more direct memory access (DMA) circuitry to perform data transfers.

[0020] In one example, input 102 is a user query that includes a user identifier (ID) and an identifier for a specific type of webpage on a social media website, and result 162 is the probability of a selection (mouse click) on a web link to a news post or another user's profile presented on the webpage. Another example of input 102 is an image file ID with an image file containing an image to be identified, and result 162 is an indication of the image's content, such as a dog, a specific type of vehicle, or something else. Multilayer networks can be used in a variety of applications, such as one of several recommendation models. In one implementation, a data item is a weight vector of known length operated on by the first hidden layer of the multilayer network, where the first hidden layer is the initial data processing step of data processing stage 160. In other implementations, index generator 110, data retrieval stage 120, and data processing stage 160 are combined to handle other types of workloads that still utilize sparse lookup operations on data items.

[0021] Index generator 110 receives input 102 and generates one or more indexes 112. In some implementations, the indexes in index 112 include table identifiers. In other implementations, the indexes in index 112 include identifiers mapped to table identifiers by interface 122. The table identifier identifies one of the tables 132A-132B in table 130. Additionally, the indexes in index 112 identify specific entries in the identified table. Interface 122 uses these identifiers when generating lookup request 124. In one of the cases where computation system 100 is a recommender system, input 102 is a user query, and index 112 is a categorical sparse input feature identifier.

[0022] Index 112 is generated from the categorical sparse input features of input 102 (such as user queries at a website). For each feature in the categorical sparse input features, the corresponding entry (or row) in one of the tables 132A-132B is identified by an index in index 112. Each entry (or row) in tables 132A-132B includes a weight vector, which is called an "embedded row". In this design, tables 132A-132B are referred to as "embedded tables". Therefore, through a lookup operation performed on tables 132A-132B, the input vector, which serves as the sparse encoding of index 112, is transformed into multiple weight vectors (multiple embedded rows).

[0023] The lookup operation performed by data retrieval level 120 is a sparse lookup operation. In one example, an online music service uses a recommendation system, such as computational system 100, which uses a multi-layered network to send song recommendations to users. The online music service offers millions of songs available, but a single user only accesses and possibly ranks a few hundred songs. Therefore, the interaction between a single user and the available songs is sparse. Consequently, categorical sparse input feature identifiers result in irregular memory accesses to tables 132A-132B.

[0024] To generate data item 150, one or more reduction operations are performed on the embedded rows read from tables 132A-132B. These reduction operations are also referred to as “pooling operations” and “embedded bag operations.” Examples of reduction operations are summation operations, maximum value operations, mathematical mean or average operations, one of several Boolean logic operations, etc. Instead of having a processor (such as processor 146) perform reduction operations on the retrieved embedded rows, switch 140 includes reduction circuitry 142 that performs reduction operations on the retrieved embedded rows.

[0025] As previously described, switch 140 includes circuitry for routing packets. In addition to routing packets using this circuitry, switch 140 also includes reduction circuitry 142. Therefore, switch 140 is capable of performing various types of collective communication operations. Collective communication operations include operations such as sending a single data item from a process executed by a first processor to one or more other processes executed by one or more other processors, collecting data from multiple processes executed by multiple processors, etc. In various specific implementations, reduction operations (pooling operations or packet embedding operations) and various types of collective communication operations have the syntax and semantics of point-to-point operations. These operations typically involve one or more transmitters (sources) and one or more receivers (destination). Because switch 140 can perform operations other than routing packets between sources and destinations, it may also be referred to as "intelligent switch 140" or "computational switch 140".

[0026] Due to the large data size of the embedded rows and the large number of embedded rows in Tables 132A-132B, most of Tables 132A-132B is stored in low-level memory such as memory 144. In other embodiments, one or more tables (or portions of Tables 132A-132B) are stored in remote memory located on a network. Due to the irregular memory accesses performed on Tables 132A-132B and the large data size of the embedded rows and Tables 132A-132B, the reduction operation becomes memory bandwidth-constrained when the processor performs the reduction operation. In contrast, the reduction circuit 142 reduces memory bandwidth requirements and power consumption by performing the reduction operation as part of retrieving the requested embedded rows.

[0027] Before providing further details of the reduction circuitry 142, a further description of the computing system 100 is provided. Tables 132A-132B use one of a variety of static random access memories (SRAMs) and / or one of a variety of dynamic random access memories (DRAMs) to store data. Copies of Tables 132A-132B reside in one or more of the processing node's local memory, system memory, and / or remote memory accessed via a network. Interface 122 includes queues for storing requests and responses, and circuitry for generating lookup requests 124, scheduling lookup requests 124 to be published to Tables 132A-132B, and scheduling responses to be published to data processing level 160. In another embodiment, another interface (not shown) is used between Tables 132A-132B and data processing level 160, which schedules responses to lookup requests 124 to be published to data processing level 160 and supports any communication protocol with data processing level 160. The circuitry of interface 122 also supports specific communication protocols for communicating with tables 132A-132B and any network.

[0028] In addition to reduction circuitry 142, switch 140 includes queues for storing requests and responses, arbitration circuitry, programmable configuration registers for storing thresholds and other values, and circuitry for supporting communication protocols with memory 144, any network, and one or more processors. Recommendation systems such as computing system 100 use both categorical sparse input feature identifiers and continuous dense input feature identifiers (not shown). Examples of categorical sparse input features are the type of user's computing device, the user's preference for content genres (e.g., movies, songs, clothing), user-provided content rankings, and other users' rankings of similar content. Examples of continuous dense input features are user profile information, such as the user's age, event data and time, ticket prices, etc. The continuous dense input features (not shown) of input 102 bypass tables 132A-132B and are sent as another input (not shown) to data processing stage 160.

[0029] Recommendation systems such as computing system 100 use data processing stage 160 to combine transformed, continuously dense input features (not shown) with data items 150. Data processing stage 160 combines the received inputs, performs further data processing, and generates a result 162. In this case, result 162 is a predicted click-through rate (CTR). Click-through rate prediction provides the probability that a user clicks a webpage link indicating specific content (such as an advertisement, recommended song or movie, recommended clothing, appliances, or others). Although switch 140 is shown as being used in a recommendation system, in other embodiments, switch 140 is used in other systems such as other types of deep learning recommendation models (DLRM), natural language processing (NLP) applications, sentiment analysis models, graph neural networks, etc. In each of these embodiments, the reduction circuitry 142 of switch 140 performs a reduction operation, which offloads another processor from performing the reduction operation. Therefore, in each of these embodiments, switch 140 reduces the memory bandwidth requirements and power consumption of the corresponding system.

[0030] Turn now Figure 2A generalized diagram of embedding row reduction 200 for reducing memory bandwidth of machine learning models is shown. The previously described circuitry is similarly numbered. As shown, embedding table 210 includes multiple embedding rows, each with an embedding row size 212. Embedding row size 212 includes data from multiple cache rows, each with a corresponding cache row size 214 of the computing system. Switch 140 is capable of transmitting packets including memory access requests to each of memory 144 and processor 146. Reduction circuitry 142 of switch 140 is capable of generating rows in output row 220 based on specific reduction operations performed on the multiple embedding rows of embedding table 210. In various specific implementations, embedding table 210 is used for one type of machine learning (ML) model among multiple types of machine learning (ML) models.

[0031] The embedded rows of embedded table 210 are also referred to as entries of embedded table 210 or embedded vectors of embedded table 210. Therefore, the embedded row size 212 can also be referred to as the embedded vector size. In some implementations, each embedded row in the embedded rows of embedded table 210 includes a specific number of elements, and these elements can also be referred to as dimensions. Each element (or dimension) includes precision. In some implementations, when the number of elements (or the number of dimensions) is 192 and the precision is in 32-bit floating-point data format, the embedded row size 212 is 192 dimensions × 4 bytes, or 768 bytes. If the cache line size 214 is 64 bytes, then the embedded row size 212 includes 12 cache lines. In other implementations, the sizes of embedded row size 212 and cache line size 214 have other values ​​based on design requirements.

[0032] The rows of output row 220 may be referred to as entries of output row 220. The reduction circuit 142 of switch 140 is capable of generating rows in output row 220 based on specific reduction operations (or pooling operations) performed on multiple embedded rows of embedded table 210. The reduction operation (or pooling operation) is indicated as "Op" within the reduction circuit 142. As previously described, each reduction operation can be one of a summation operation, a maximum value operation, one of various Boolean logic operations, and a mathematical mean or average operation, etc. By having the reduction circuit 142 of switch 140, rather than processor 146 or any other processor, perform the reduction operations, the reduction circuit 142 reduces memory bandwidth requirements and power consumption.

[0033] In one embodiment, the embedded rows of embedded table 210 include at least the embedded rows labeled "0" through "17". In some embodiments, each row of output row 220 has the same data size as the embedded row size 212. As shown, the rows of output row 220 include at least the rows labeled "0" through "3". Other values ​​for the indices of embedded table 210 and output row 220 are possible and conceivable. Reduction circuit 142 receives an indication of the type of reduction operation. In addition to this indication, reduction circuit 142 also receives a specific embedded row of embedded table 210. Access circuitry (not shown) accesses embedded table 210 and provides a copy of the specific embedded row of embedded table 210 to reduction circuit 142 based on offset 202 and index 204.

[0034] Each row in output row 220 provides a reduction circuit 142 with a specific number of embedded rows from embedded table 210 to perform a reduction operation for computation. Offset 202 indicates the number of embedded rows to be used by reduction circuit 142, and index 204 identifies which specific embedded rows from embedded table 210 are to be provided to reduction circuit 142. In one implementation, offset 202 includes values ​​“0” and “2”, indicating that the number of embedded rows to be used by reduction circuit 142 is 2. In one implementation, each element in offset 202 and index 204 has a data size of bytes, although different data sizes are used in other implementations. The two indices in index 204 that begin with a zero offset include indices “3” and “9”. Therefore, the embedded rows marked “row 3” and “row 9” are read from embedded table 210 by the access circuit and provided to reduction circuit 142. The reduction circuit 142 performs the corresponding reduction operation using the data values ​​of copies of "row 3" and "row 9", and writes the resulting output to the row marked "row 0" of output row 220.

[0035] In a similar manner, the reduction circuit 142 writes the resulting data values ​​into the remaining rows of output line 220. The next consecutive offset pair of offset 202 includes the values ​​"2" and "5", indicating three embedded rows to be used by the reduction circuit 142. The three indices in index 204 starting with offset two include indices "11", "0", and "4". Therefore, the embedded rows labeled "line 0", "line 4", and "line 11" are read from the embedded table 210 by the access circuit and provided to the reduction circuit 142. For clarity, arrows showing access to the embedded rows labeled "line 0", "line 4", and "line 11" are not shown. The reduction circuit 142 performs the corresponding reduction operation using copies of the data values ​​labeled "line 0", "line 4", and "line 11" and writes the resulting output into the row labeled "line 1" of output line 220.

[0036] The next consecutive offset pair after offset 202 includes the values ​​"5" and "10", indicating five embedded rows to be used by the reduction circuit 142. The five indices in index 204 starting at offset five include indices "13", "7", "6", "14", and "17". Therefore, the access circuit reads the embedded rows labeled "row 6", "row 7", "row 13", "row 14", and "row 17" from the embedded table 210 and provides them to the reduction circuit 142. The reduction circuit 142 performs the corresponding reduction operation using copies of the data values ​​from "row 6", "row 7", "row 13", "row 14", and "row 17" and writes the resulting output to the row labeled "row 2" in output row 220. The number of reduction operations performed simultaneously by the reduction circuit 142 varies depending on the specific implementation.

[0037] In several embodiments, the size of embedding table 210 is too large to fit within the cache of a single processor, such as processor 146. While a single embedding table is shown in the illustrated embodiment, in various embodiments there are numerous (e.g., thousands) embedding tables from which embedding rows are selected for a specific reduction operation. Reduction circuit 142. Additionally, the value of index 204 identifying a set of embedding rows for one reduction operation can include long distances between other sets of embedding rows in the same embedding table identified by other values ​​of index 204 for another reduction operation. The required embedding rows may reside in other memory regions and on separate memory devices. Furthermore, the embedding row size 212 may be too large to fit within a single packet transmitted across one or more memory buses and communication texture buses. Therefore, the reduction operation performed for a specific machine learning model is a memory bandwidth-intensive operation.

[0038] Switch 140 is one of several switches used in a computing system that performs operations on a specific machine learning model that utilizes memory bandwidth-intensive operations. Switch 140 (and other switches) perform data acquisition operations within the computing system. By having the reduction circuitry 142 of switch 140 perform the reduction operation instead of processor 146 or any other processor, the reduction circuitry 142 reduces memory bandwidth requirements and power consumption.

[0039] Turn now Figure 3This diagram illustrates a generalized representation of an apparatus 300 for reducing memory bandwidth for executing machine learning models. Apparatus 300 includes processing element 310, processing element 320, switch 330, processor 340, and system memory 342. Processing element 310 includes hardware such as at least a processor 316 for performing data processing, and local memory including at least an embedded table 312 and an output array 314. Similarly, processing element 320 includes a processor 326 for performing data processing, and local memory including at least an embedded table 322 and an output array 324. Switch 330 includes circuitry supporting communication between processing elements 310 and 320, communication with processor 340, communication with system memory 342, and communication with any remote memory via a network (not shown).

[0040] For ease of illustration, power controllers, interrupt controllers, phase-locked loops (PLLs), or other clock generation circuitry are not shown in device 300. In some embodiments, the functionality of device 300 is implemented by a single integrated circuit. In other embodiments, the functionality of device 300 is implemented on a separate integrated circuit on a system-on-a-chip (SoC), multi-chip module (MCM), or otherwise. In one embodiment, the processor 316 of processing element 310 is a processor with a highly parallel data microarchitecture, such as a graphics processing unit (GPU), and the local memory is off-chip memory, such as one type of synchronous random access memory (SRAM) among various types. In some embodiments, the circuitry of processor 316 includes a local memory controller that supports one type of graphics dual data rate (GDDR) communication protocol among various types. In various embodiments, the components of processing element 320 are instantiations of the components of processing element 310.

[0041] In some embodiments, device 300 includes a processor 340 acting as a host processor that assigns tasks to processing nodes in a multi-node partition, wherein device 300 is a multi-node partition and processing elements 310 and 320 are processing nodes. Each processing node includes at least one or more processors and local memory, such as processor 316 of processing element 310 and local memory storing embedded tables 312 and output array 314. In one embodiment, host processor 340 is a general-purpose central processing unit (CPU), and processor 316 is a highly parallel data processor, such as a graphics processing unit (GPU).

[0042] In some implementations, the processing nodes, which can be implemented by processing elements 310 and 320, are non-uniform memory access (NUMA) processing nodes that utilize GPUs to process parallel data tasks. For example, the processing node performs tasks for machine learning applications. The compiler extracts parallelized tasks from program code for parallel execution on the system hardware. In one implementation, the processing node is an expansion card inserted into an expansion slot or socket on a motherboard. In other implementations, the processing node is an integrated circuit placed on the same expansion card in a socket or soldered onto the motherboard. For example, device 300 is a multi-node partition of an MCM placed in a slot on a multi-slot motherboard in a desktop computer or server. In other implementations, one or more of processors 316, 326, and 340 are one of various types of application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc. Memory 340 is system memory implemented as one of various types of dynamic random access memory (DRAM), disk storage devices, or remote memory accessed via a network.

[0043] Switch 330 includes circuitry for routing packets between a source and a destination. In some embodiments, switch 330 is used within a communication texture, and switch 330 includes circuitry for performing collective communication operations. In various embodiments, packets include memory access requests for data stored in one or more of processing elements 310, 320, and memory 340. Furthermore, switch 330 includes reduction circuitry 332 that performs one of a variety of reduction operations (pooling operations or packet embedding operations). Therefore, in various embodiments, reduction circuitry 332 includes ( Figures 1 to 2 The function of the reduction circuit 142, and the switch 330 has ( Figures 1 to 2 The functions of switch 140.

[0044] Processing element 310 stores (or is assigned) at least a portion of embedded table 312 and output array 314. Processing element 320 stores (or is assigned) at least a portion of embedded table 322 and output array 324. In addition to the local cache memory subsystems (not shown) of processors 316 and 326, each of processing elements 310 and 320 also includes local memory, which is off-chip memory, such as one type of synchronous random access memory supported by a communication protocol, such as one type of graphics dual data rate (GDDR) communication protocol. These local memories of processing elements 310 and 320 store copies of embedded tables 312 and 322 and output arrays 314 and 324.

[0045] In one implementation, the first reduction operation is a direct memory access (DMA) packet with a header containing storage information such as an identifier (ID) for one or more of the processors 316 and processing element 310, multiple starting addresses corresponding to the local memory of processing element 310 storing multiple specific embedded rows of embedded table 312 to be used in the reduction operation, the data size of the elements (such as the data size of the embedded rows of embedded table 312), and a destination address corresponding to the local memory of processing element 310 storing output array 314. The second reduction operation includes similar information, but the identifier and address correspond to processing element 320. It should also be noted that although two processing elements 310 and 320 are shown in device 300, in other implementations, a different number of processing elements are used based on design requirements. Therefore, the DMA packets are modified to support any number of sources and destinations for reduction operations performed by switch 330.

[0046] Furthermore, the reduction operation includes one or more instructions, which include those similar to ( Figure 2 The information includes offset 202 and index 204. Packets, such as DMA packets, are modified to include this information and an indication of one of several types of reduction operations, and the reduction operation is performed using an embedding row from an embedding table from a single processor, or using an embedding row from a separate embedding table from a separate processor. In some implementations, the interface circuitry of switch 330 and processing elements 310 and 320 support communication protocols such as xGMI (Chip-to-GPU Global Memory Interconnect) from Advanced Micro Devices. However, in other implementations, other types of communication protocols are possible and conceivable.

[0047] In one embodiment, the bus on the motherboard of device 300 uses communication protocols such as PCIe (Peripheral Component Interconnect Fast), Infinity Fabric from Advanced Micro Devices, and Infinity Architecture from Advanced Micro Devices. In one embodiment, copies of embedded tables 312 and 322 are stored in one or more of system memory such as memory 340 and another processor (not shown) such as a CPU. In one embodiment, each of processors 316 and 326 is a GPU, and at least a portion of embedded tables 312 and 322 is adaptable to the local memory of processing elements 310 and 320. The combination of offloading reduction operations from processors 316 and 326 to switch 330 and transferring data between switch 330 and the local memory of processing elements 310 and 320 using dedicated point-to-point interconnects (such as chip-to-chip interconnects (e.g., xGMI interconnects)) reduces the memory bandwidth requirements and power consumption of device 300. While switch 330 performs memory accesses and performs reduction operations, processors 316 and 326 are able to perform other tasks. Processors 316 and 326 are also capable of performing memory accesses to the corresponding local memory using a separate link or bus, different from the link or bus used between switch 330 and local memory.

[0048] In one implementation, the entire reduction operation (such as the embedding packet operation) is offloaded from processors 316 and 326 to switch 330. As shown, switch 330 receives specific embedding rows from embedding table 312, reduction circuitry 332 performs reduction operation 350 on these received embedding rows to generate result 352, and sends result 352 to output array 314. Similarly, switch 330 receives specific embedding rows from embedding table 322, reduction circuitry 332 performs reduction operation 360 on these received embedding rows to generate result 362, and sends result 362 to output array 324. It should be noted that although a single reduction operation, such as reduction operation 350, is shown, in other implementations, the generation of result 352 is performed by multiple reduction operations, where each reduction operation operates on a specific portion of the data elements of multiple embedding rows, rather than operating on all of the multiple embedding rows. Similarly, the generation of result 362 is performed by multiple reduction operations rather than a single reduction operation.

[0049] It should also be noted that although a specific number of embedded rows and a specific number of data elements in each embedded row are shown for embedded tables 312 and 322 and output arrays 314 and 324, it is possible and conceivable to use other numbers of embedded rows and data elements in other embodiments. In another embodiment, the reduction operation is partially offloaded to switch 330, such that switch 330 performs the reduction operation that generates a portion of the result of output array 314, while processor 316 performs the reduction operation that generates another portion of the result of output array 314. Similarly, processor 326 performs the reduction operation that generates a portion of the result of output array 324. At least a portion of one or more of the output arrays in output arrays 314 and 324 is also stored in system memory such as memory 340.

[0050] refer to Figure 4 This diagram illustrates a generalized representation of an apparatus 400 for reducing memory bandwidth for executing machine learning models. The previously described circuits and data structures are numbered identically. Here, initial output arrays 414 and 424 are shown above processing elements 310 and 320. Initial output arrays 414 and 424 store data values ​​of the output arrays prior to the execution of reduction operations 450 and 460 and collective communication operations. Final output arrays 434 and 444 store data values ​​after the execution of reduction operations 450 and 460 and one or more collective communication operations that transfer data between processing elements 310 and 320. An example of a collective communication operation (data transfer operation) is sending a copy of the data from the bottom row of the initial output array 414 to the top row of the final output array 444.

[0051] An initial description is provided of performing a collective communication operation (data transfer operation) followed by a separate reduction operation. A description of a fusion operation, which combines the steps of performing these two separate operations, is then provided. Compared to performing the collective communication operation and the reduction operation separately, the fusion operation reduces memory bandwidth, power consumption, and latency. When the first collective communication operation (data transfer operation) is performed separately followed by a separate reduction operation, switch 330 performs the first collective communication operation, which transfers a copy of the data from the bottom row of the initial output array 414 to the top row of the final output array 444. This copy of the data from the bottom row of the initial output array 414 is shown as result 472. Next, reduction circuitry 332 performs two reduction operations 450 and 460. Without performing the fusion operation, switch 330 stores result 462 in processing element 320. Switch 330 then performs a second collective communication operation to transfer result 462 from processing element 320 to the bottom row of the final output array 434 of processing element 310. It should be noted that although the terms "left," "right," "horizontal," "vertical," "row," "column," "top," and "bottom" can be used to describe device 400 (and Figure 3 (The device 300), but the meaning of the term may change as the device 400 is rotated or flipped.

[0052] In various implementations, any combination of reduction operations and collective communication operations is merged into a single combined operation, which performs the steps of the individual operations used to create the single combined operation. The single combined operation (merged operation) transfers result 472 from processing element 310 to processing element 320 during a data retrieval stage, which retrieves source data for one or more reduction operations used in the combined operation. The single combined operation (merged operation) also writes results 452 and 462 directly from switch 330 to final output array 434 without data transfer of result data between processing elements 310 and 320. This implementation eliminates the latency overhead associated with data transfer of result data between processing elements 310 and 320, reduces data movement to memory 340, and lowers power consumption.

[0053] During the fusion operation, switch 330 retrieves data from specific embedded rows of embedded table 312, specific data from the initial output array 414, and specific embedded rows of embedded table 322. For this fusion operation, switch 330 retrieves copies of two rows of data from embedded table 312 of processing element 310. Additionally, switch 330 retrieves copies of data from three rows of data from embedded table 322 of processing element 320. Furthermore, switch 330 retrieves a copy of the data from the bottom row of the initial output array 414. Afterward, switch 330 transmits a copy of the retrieved data from the bottom row of the initial output array 414 (indicated as result 472) to the top row of the final output array 444.

[0054] The reduction circuit 332 performs a reduction operation 450 on the two embedded rows retrieved from the embedding table 312 to produce a result 452. The reduction circuit 332 also performs a reduction operation 460 on the three embedded rows retrieved from the embedding table 322 to produce a result 462. In various specific implementations, the reduction circuit 332 writes the result 452 directly to the top row of the final output array 434 and the result 462 directly to the bottom row of the final output array 434. By retrieving source data for multiple operations during a single data retrieval and data transfer stage, and by writing the resulting data directly to the output array of any available processing element (e.g., the final output array 434 of processing element 310) even when the source data of the embedded rows is stored in another processing element, the fusion operation reduces memory bandwidth, lowers power consumption, and reduces latency.

[0055] In one implementation, the first combination (fusion) operation is a direct memory access (DMA) packet having a header containing storage information such as a source identifier (ID) of one or more of the processors 316 and processing element 310, one or more starting addresses corresponding to the local memory of processing element 310 (which stores one or more entries (embedded rows) of embedding table 312 to be moved to the local memory of processing element 320), the data size of the elements (such as the data size of the embedded row of embedding table 312), one or more destination addresses corresponding to the local memory of processing element 320 (which stores the one or more embedded rows that have been moved), a destination identifier (ID) identifying processing element 320, and information corresponding to the reduction operation of the first combination (fusion) operation. The second combination (fusion) operation includes similar information, but the source and destination may be changed from the values ​​used for the first combination (fusion) operation.

[0056] refer to Figure 5 A generalized diagram of a method 500 for reducing memory bandwidth for executing machine learning models is shown. For discussion purposes, the specific implementation is shown in sequential order (and...). Figure 6The steps in the diagram are shown. However, in other specific implementations, some steps occur in a different order than those shown, some steps are executed concurrently, some steps are combined with other steps, and some steps are not present.

[0057] Two or more processing nodes are placed in a multi-node partition of the computing system. Each processing node includes at least one or more processors and local memory. The switch of the multi-node partition communicates with at least the local memory and the system memory of the computing system. Tasks are assigned to the partition by the host processor or by the processing nodes themselves. Each processing node is capable of sending direct memory access (DMA) requests, as well as other commands, messages, responses, and tasks to another processing node and the switch. Each of the one or more processing nodes stores multiple entries of a data array (box 502).

[0058] In various implementations, the data array is an embedding table used in one of several machine learning models, and the entries are embedding rows. The processing node stores the data array in its local memory. The processor of the processing node identifies two or more entries of the data array as source operands for a reduction operation (box 504). Examples of reduction operations are summation, maximum value, one of several Boolean logic operations, mathematical mean or average, etc. The processor sends the reduction operation to a switch, and the switch executes a memory access request to retrieve the data for two or more entries (box 506).

[0059] The switch retrieves data from two or more entries (box 508). The switch generates a result by performing a reduction operation using the data from the two or more entries (box 510). The switch sends the result to the local memory of the processing node (box 512). By having the switch, rather than any processor on the processing node, perform the reduction operation, the switch reduces memory bandwidth requirements and power consumption in multi-node partitions. While the switch performs memory accesses and performs reduction operations, the processor is able to perform other tasks. The processor is also able to perform memory accesses to the local memory using a separate link or bus, different from the link or bus used between the switch and the local memory.

[0060] Turn now Figure 6This diagram illustrates a generalized representation of a method 600 for reducing memory bandwidth for executing machine learning models. Two or more processing nodes are placed in a multi-node partition of a computing system. In various embodiments, the data array is an embedding table used in one of several machine learning models, and the entries are embedding rows. Each processing node stores multiple entries of the data array in its local memory. A first processing node stores multiple entries of a first data array (box 602). A second processing node stores multiple entries of a second data array (box 604). The processor of the first processing node sends a combination operation (fusion operation) to a switch in the computing system (box 606). In some embodiments, the combination operation includes a collective communication operation followed by a reduction operation.

[0061] In various implementations, in block 611, the switch concurrently executes the steps of blocks 608 and 610. The switch transfers data between the first and second processing nodes based on the type of the combined operation (block 608). The switch also retrieves data from the first and second data arrays based on the type of the combined operation (block 610). In other implementations, the switch executes the steps of block 610 prior to the steps of block 608 based on the type of the combined operation. The switch uses the retrieved data to generate a result by performing a reduction operation indicated by the combined operation (block 612). The switch sends the result directly to one or more of the first and second processing nodes (block 614). By retrieving source data for multiple operations during a single data retrieval and data transfer stage, and by writing the resulting data directly to the output array of any available processing node even when the source data is stored in another processing node, the combined operation (fusion operation) reduces memory bandwidth, lowers power consumption, and reduces latency.

[0062] It should be noted that one or more of the above-described embodiments include software. In such embodiments, program instructions for implementing the method and / or mechanism are transmitted or stored on a computer-readable medium. Many types of media configured to store program instructions are available and include hard disks, floppy disks, CD-ROMs, DVDs, flash memory, programmable ROMs (PROMs), random access memory (RAMs), and various other forms of volatile or non-volatile storage devices. Generally, computer-accessible storage media include any storage medium that can be accessed by a computer during use to provide instructions and / or data to the computer. For example, computer-accessible storage media include storage media such as magnetic or optical media, such as magnetic disks (fixed or removable), magnetic tape, CD-ROMs or DVD-ROMs, CD-Rs, CD-RWs, DVD-Rs, DVD-RWs, or Blu-ray discs. Storage media also include volatile or non-volatile storage media, such as RAM (e.g., Synchronous Dynamic RAM (SDRAM), Dual Data Rate (DDR, DDR2, DDR3, etc.) SDRAM, Low Power DDR (LPDDR2, etc.) SDRAM, Rambus DRAM (RDRAM), Static RAM (SRAM), etc.), ROM, flash memory, and non-volatile memory (e.g., flash memory) accessible via peripheral device interfaces (such as Universal Serial Bus (USB) interfaces). Storage media include microelectromechanical systems (MEMS), and storage media accessible via communication media such as networks and / or wireless links.

[0063] Additionally, in various specific implementations, program instructions include behavioral-level or register-transfer-level (RTL) descriptions of hardware functionality in high-level programming languages ​​(such as C) or design languages ​​(HDLs) (such as Verilog, VHDL, or database formats such as GDS II streaming format (GDSII)). In some cases, the description is read by a synthesis tool that synthesizes the description to produce a netlist that includes a list of gates from a synthesis library. The netlist includes a set of gates, which also represents the functionality of the hardware comprising the system. The netlist is then placed and routed to produce a dataset describing the geometry to be applied to a mask. The mask is then used in various semiconductor manufacturing steps to produce semiconductor circuitry corresponding to the system. Alternatively, computer-accessible instructions on a storage medium are, as desired, netlists (with or without synthesis libraries) or datasets. Furthermore, these instructions are used by tools such as Cadence... ® EVE ® and Mentor Graphics ® The purpose of this type of supplier's hardware-based type simulator is to perform simulation.

[0064] Although the specific embodiments described above have been described in considerable detail, many variations and modifications will become apparent to those skilled in the art once the foregoing disclosure is fully understood. The following claims are intended to be interpreted as covering all such variations and modifications.

Claims

1. An integrated circuit, the integrated circuit comprising: The switch includes circuitry configured to: Retrieve data from two or more of the multiple entries of the first data array requested by the first processor of the first processing node from a first memory configured to store data of a multiple entries of the first data array; First result data is generated by performing a first type of reduction operation using the data from the two or more entries; as well as The first result data is stored in the first data array so that it can be accessed by the first processor.

2. The integrated circuit of claim 1, wherein the first data array is an embedding table of a machine learning model.

3. The integrated circuit of claim 1, wherein in response to receiving a first combined operation from the first processor of the first processing node, the switch is further configured to: Transmit the first entry of the first data array of the first processing node to the second data array of the second processing node, which is different from the first processing node; and The data of the first entry is replaced with second result data generated by performing a second type of reduction operation using the data of the first data array.

4. The integrated circuit of claim 3, wherein in response to receiving a second combined operation from the first processor of the first processing node, the switch is further configured to: Transmit the data of the second entry of the second data array of the second processing node to the first data array of the first processing node; and The data in the second entry is replaced with the resulting data generated by performing a third type of reduction operation using the data in the second data array.

5. The integrated circuit of claim 3, wherein the reduction operation of the first type includes one of a summation operation and a maximum value operation.

6. The integrated circuit of claim 3, wherein the switch is configured to communicate with each of the first memory and the second memory, the first memory and the second memory each using a different communication protocol.

7. The integrated circuit of claim 6, wherein the first processor is configured to process a task and access data in the first memory and the second memory when the switch generates one or more of the first result data and the second result data.

8. A method, the method comprising: The first memory stores the data of multiple entries of the first data array; The switch retrieves data from the first memory for two or more of the plurality of entries in the first data array requested by the first processor of the first processing node; The switch generates first result data by performing a first type of reduction operation using the data from the two or more entries; as well as The switch stores the first result data in the first data array so that it can be accessed by the first processor.

9. The method of claim 8, wherein the first data array is an embedding table of a machine learning model.

10. The method of claim 8, wherein in response to receiving a first combined operation from the first processor of the first processing node, the method further comprises: The switch transmits the first entry of the first data array of the first processing node to the second data array of the second processing node, which is different from the first processing node. as well as The switch replaces the data of the first entry with second result data generated by performing a second type of reduction operation using the data of the first data array.

11. The method of claim 10, wherein in response to receiving a second combined operation from the first processor of the first processing node, the method further comprises: The switch transmits the data of the second entry of the second data array of the second processing node to the first data array of the first processing node; as well as The switch replaces the data of the second entry with the resulting data generated by performing a third type of reduction operation using the data of the second data array.

12. The method of claim 10, wherein the reduction operation of the first type includes one of a summation operation and a maximum value operation.

13. The method according to claim 10, further comprising: The switch communicates with each of the first and second memories, each using a different communication protocol.

14. The method according to claim 13, further comprising: The task is processed by the first processor, and the first processor accesses the data in the first memory and the second memory when the switch generates one or more of the first result data and the second result data.

15. A computing system, the computing system comprising: A first processing node, comprising: First processor; A first memory, the first memory including circuitry configured to store data of a plurality of entries of a first data array; and The switch includes circuitry configured to: Retrieve data from two or more of the plurality of entries in the first data array requested by the first processor from the first memory; First result data is generated by performing a first type of reduction operation using the data from the two or more entries; and The first result data is stored in the first data so that it can be accessed by the first processor.

16. The computing system of claim 15, wherein the first data array is an embedding table of a machine learning model.

17. The computing system of claim 15, wherein the computing system further comprises a second processing node, wherein in response to receiving a first combined operation from the first processor of the first processing node, the switch is further configured to: Transmit the first entry of the first data array of the first processing node to the second data array of the second processing node; and The data of the first entry is replaced with second result data generated by performing a second type of reduction operation using the data of the first data array.

18. The computing system of claim 17, wherein in response to receiving a second combined operation from the first processor of the first processing node, the switch is further configured to: Transmit the data of the second entry of the second data array of the second processing node to the first data array of the first processing node; and The data in the second entry is replaced with the resulting data generated by performing a third type of reduction operation using the data in the second data array.

19. The computing system of claim 17, wherein the reduction operation of the first type includes one of an averaging operation and a Boolean logic operation.

20. The computing system of claim 17, wherein the computing system further comprises a second memory, wherein the switch is configured to communicate with each of the first memory and the second memory, the first memory and the second memory each using a different communication protocol.

Images