Fused data generation and related communications

Abstract

Fusion data generation and associated communication techniques are described. In one embodiment, a system includes a processing system having a plurality of processors. A data generation and communication tracking module is configured to track data generation and associated communications defined in programs executed by the plurality of processors. A target communication module is configured to trigger targeted communication of data between the plurality of processors based on the tracked program-defined data generation and associated communications.

Description

[Technical Field]

[0001] (Related Applications) This application claims priority to U.S. Patent Application No. 18 / 190,620, filed March 27, 2023, which claims priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 63 / 387,434, filed December 14, 2022, the entire disclosure of which is incorporated herein by reference. [Background technology]

[0002] Processing systems can be configured to include multiple processors to improve computational efficiency, e.g., through the use of multiple cores in a central processing unit, a graphics processing unit, etc. For example, computations can be performed using multiple processors by alternating computations with associated communication of data resulting from the computations between the processors. Thus, scenarios involving increased communication between processors (e.g., machine learning) have a direct impact on overall device operation and computational efficiency.

[0003] The detailed description is made with reference to the accompanying drawings. [Brief explanation of the drawings]

[0004] [Figure 1] FIG. 1 is a block diagram of a non-limiting example infrastructure configured to use fused data generation and related communication techniques. [Figure 2] FIG. 1 is a block diagram of a non-limiting example of data generation and related communications in a machine learning training example. [Figure 3] 2 is a block diagram of a non-limiting example of the data generation and communication tracking module, the target communication module, and the updates convergence unit of FIG. 1; [Figure 4]FIG. 1 is a block diagram of a non-limiting example of data generation and communication tracking supporting a fused reduce-scatter operation. [Figure 5] FIG. 1 is a block diagram of a non-limiting example of unordered data generation. [Figure 6] FIG. 1 is a block diagram of a non-limiting example of ordering data generation. [Figure 7] FIG. 1 is a flow diagram of a non-limiting example of fusion data generation and communication. [Figure 8] FIG. 1 is a block diagram of a non-limiting example of a baseline system versus a fine-grained in-memory reduction-based aggregation system. [Figure 9] FIG. 10 is a block diagram of a non-limiting example of fine-grained all-to-all and gather-all operations. DETAILED DESCRIPTION OF THE INVENTION

[0005] (overview) In real-world scenarios, it is common practice across domains to split computations (e.g., deep learning training) across multiple processors (e.g., GPUs) and alternate computations (e.g., computing weight gradients via GEMM computations) with associated communications, e.g., performing a reduce-scatter computation to reduce weight gradients across GPUs. Scaling along multiple dimensions (e.g., neural network size, dataset) continues to increase communications, and therefore communication optimization has a direct impact on overall device behavior.

[0006] Large-scale deep neural networks (DNNs), for example, typically rely on distributed training, which involves partitioning parameters and activations across nodes, which, along with techniques such as data-parallel training, involves node-wide reduction of these structures at each training iteration. That is, each node independently generates data for these structures, and at each training iteration, this generated data is communicated and reduced among participating processors.

[0007] To solve these problems, fusion data generation and related communication techniques are described. These techniques are configured through the use of extended components, examples of which include a target communication module, a data generation and communication tracking module, and an update convergence unit. This supports various technical advantages, including concurrent computation / network utilization, improved performance, energy efficiency, avoidance of launching separate kernels for computation / communication, etc. Various other cases are also contemplated, examples of which are described below and illustrated using corresponding figures.

[0008] In some aspects, the techniques described herein relate to a system including a processing system including a plurality of processors, at least one of the plurality of processors configured to track program-defined data generation and associated communications as executed by the plurality of processors, and to trigger targeted communications of data between the plurality of processors based on the tracked program-defined data generation and associated communications.

[0009] In some aspects, the technology described herein relates to a system in which programmatically defined data generation and associated communications include generation of data by at least one processor and target updating by the at least one processor to transmit the data to another processor of a plurality of processors.

[0010] In some aspects, the techniques described herein relate to a system in which a target update is triggered upon completion of generation of data by at least one processor.

[0011] In some aspects, the techniques described herein relate to a system, wherein a target update is triggered based on a remote communication event received at at least one processor to be implemented by a data mover engine by another processor of a plurality of processors.

[0012] In some aspects, the technology described herein relates to a system in which a remote communication event is part of a bulk operation involving the communication of data.

[0013] In some aspects, the technology described herein relates to a system in which programmatically defined data generation and associated communications are defined using a single fused data generation and associated communications operation.

[0014] In some aspects, the technology described herein relates to a system in which fused data generation and associated communication operations identify another processor of a plurality of processors to receive the data.

[0015] In some aspects, the technology described herein relates to a system in which fusion data generation and related communication operations identify address ranges that are sources of data or destinations for sending data.

[0016] In some aspects, the technology described herein relates to a system in which at least one processor is further configured to support concurrent updates to data in physical memory.

[0017] In some aspects, the technology described herein relates to a system, wherein a processor-in-memory component of a memory module including physical memory is configured to implement concurrent updates.

[0018] In some aspects, the technology described herein relates to a system in which programmatically defined data generation and associated communications are configured to control the order of data generation by each processor of a plurality of processors.

[0019] In some aspects, the techniques described herein relate to a device including a processing system including a plurality of processors, wherein at least one of the plurality of processors is configured to trigger targeted communication of data between the at least one of the plurality of processors and another processor as part of program-defined data generation and related communication, and to resolve concurrent updates to data in physical memory.

[0020] In some aspects, the technology described herein relates to a device, wherein a processor-in-memory component of a memory module including a physical memory is configured to resolve concurrent updates to data in the physical memory.

[0021] In some aspects, the techniques described herein relate to a device, wherein at least one processor is further configured to track program-defined data generation and associated communications as executed by the multiple processors, and to trigger target communications based on the tracked program-defined data generation and associated communications.

[0022] In some aspects, the techniques described herein relate to a device, wherein the target communication is configured to be performed based on a single fused data generation and associated communication operation performed by at least one processor, and identifies another processor among a plurality of processors to which the data is to be sent.

[0023] In some aspects, the techniques described herein relate to a method that includes tracking program-defined data generation and associated communications as they occur between multiple processors of a processing system, triggering targeted communications of data between the multiple processors as part of the program-defined data generation and associated communications, and resolving concurrent updates to physical memory that includes data generated by the multiple processors.

[0024] In some aspects, the technology described herein relates to methods in which programmatically defined data generation and associated communications are configured to control the order of data generation by each processor of a plurality of processors.

[0025] In some aspects, the technology described herein relates to methods in which programmatically defined data generation and associated communications are configured to identify a particular processor among a plurality of processors that is to receive the data.

[0026] In some aspects, the technology described herein relates to methods in which programmatically defined data generation and associated communications are configured to identify address ranges that are sources of data or destinations to send data to.

[0027] In some aspects, the techniques described herein relate to a method in which programmatically defined data generation and associated communication includes generation of data by a first processor of a plurality of processors and target updating for transmitting the data by the first processor to a second processor of the plurality of processors.

[0028] FIG. 1 is a block diagram of a non-limiting exemplary infrastructure 100 configured to use fused data generation and related communications. The infrastructure 100 includes a device 102 having a processing system 104. The processing system 104 includes a data mover engine 106, a memory controller 108, and a memory module 110 having a physical memory 112 and an in-memory processing component 114. An example of the physical memory 112 includes a random access memory (e.g., double data rate synchronous dynamic random access memory) implemented using one or more integrated circuits. The in-memory processing component 114 can be configured as an integrated circuit including both a processing component and a memory component implemented in hardware. The processing system 104 implements multiple processors, an example of which is shown as processor 116. Processor 116 represents at least one processor that implements the functionality represented by the data mover engine 106 and the memory controller 108. The memory module 110, in one example, is configured as a printed circuit board on which the physical memory 112 and the in-memory processing component 114 are implemented. The memory module 110 is communicatively coupled to a processor, for example, via one or more buses on a motherboard that implements at least a portion of the device 102. The processor may be configured as a central processing unit, a subsidiary processing unit such as a graphics processing unit, or the like.

[0029] Examples of device 102 configurations include, by way of example only, but not limited to, computing devices, servers, mobile devices (e.g., wearables, mobile phones, tablets, laptops), processors (e.g., graphics processing units, central processing units and accelerators), digital signal processors, interference accelerators, disk array controllers, hard disk drive host adapters, memory cards, solid state drives, wireless communication hardware connections, Ethernet hardware connections, switches, bridges, network interface controllers, and other equipment configurations. Additional examples include artificial intelligence training accelerators, cryptography and compression accelerators, network packet processors, and video coders and decoders.

[0030] The techniques described herein implement mechanisms and primitives to efficiently support the fusion of data generation and associated communication. In real-world scenarios, it is common practice across domains to split computations (e.g., deep learning training) across multiple processors (e.g., GPUs) and alternate computations (e.g., computing weight gradients via GEMM computations) with associated communication, e.g., performing scatter reduction computations to reduce weight gradients across GPUs. Scaling along multiple dimensions (e.g., neural network size, dataset) continues to increase communication, and therefore communication optimization has a direct impact on overall device operation.

[0031] To this end, infrastructure 100 includes enhanced components, examples of which include a target communication module 118 included as part of data mover engine 106 and a data generation and communication tracking module 120. Infrastructure 100 also supports data synchronization using an updates convergence unit 122 configured to leverage near / in-memory offloading as an efficient synchronization substrate for supporting concurrent data generation and associated communications. Data mover engine 106, target communication module 118, data generation and communication tracking module 120, and updates convergence unit 122 are implemented in either hardware, software, firmware, or a combination thereof. In one example, these modules and units are configured as microcontrollers for performing various operations for fused data management, as described below. In another example, the modules and units are implemented using hardware, such as application-specific integrated circuits (ASICs) or other integrated circuits (ICs), to perform various operations for fused data management, as described below.

[0032] This supports various technical advantages, including concurrent compute / network utilization, improved performance, energy efficiency, and avoidance of launching separate kernels for compute / communication.

[0033] Processing system 104 is configured to support scenarios in which, for example, data generated locally on a first processor (e.g., processor 116) is to be communicated to another of multiple processors involved in an overall computation. In the following description, one such example involves training a machine learning model.

[0034] Large-scale deep neural networks (DNNs), for example, typically rely on distributed training, which involves partitioning parameters and activations across nodes, along with techniques such as data-parallel training, and then reducing these structures across nodes at each training iteration as part of a "reduce-scatter operation." That is, each node independently generates data for these structures, and at each training iteration, this generated data is communicated and reduced among participating processors.

[0035] Data generation and associated communication fusion are used as a single fused operation, performing these operations simultaneously while also reducing redundant memory traffic. This supports several technical advantages, including improved operational performance and energy efficiency of the device 102, concurrent utilization of computational and network resources instead of serialized utilization, reduced number of task / kernel launches, etc. While fusion of data generation and associated communication has several advantages, in some scenarios it is too complex to implement solely using software. An example of this includes data generation via a general matrix-matrix multiplication (GEMM) operation and communication via a scattering reduction operation.

[0036] To address these challenges, infrastructure 100 fuses data generation and associated communications programmatically, i.e., in a programmer-defined manner. This is implemented in infrastructure 100 of FIG. 1 through an extension to memory controller 108 through the use of data generation and communication tracking module 120 for tracking data generation and communications. Memory controller 108 is a digital circuit configured to manage the flow of data into and out of physical memory 112. As implemented in data mover engine 106, target communication module 118 is utilized to trigger target communications, for example, as transmissions to defined address ranges (e.g., contiguous or non-contiguous), defined processors, etc. The address ranges can be configured as address ranges that are sources of data or address ranges to which data is sent. Additionally, update convergence unit 122 is also implemented in the illustrated example to support computational operations often associated with communications (e.g., reduction operations in clutter reduction operations) through the use of near / in-memory processing to use physical memory 112 as a synchronization point. This supports local and remote updates to data with relatively low synchronization costs.

[0037] 2 is a block diagram of a non-limiting example 200 of data generation and associated communication in a machine learning training example. This example shows data generation using GEMM operations followed by associated communication using scattering reduction operations.

[0038] The example scattering reduction primitives of Figure 2 show an array with four partitions that are reduced across four nodes (e.g., example processors 116 shown as "P0," "P1," "P2," and "P3") connected via a ring topology. In the first example 202 of the baseline system, each node (i.e., processors P0-P3) first undergoes a data generation process via a GEMM operation. GEMM operations used in machine learning typically involve large amounts of data that are generated in multiple steps shown as four time steps "T1-T4."

[0039] After data generation, scattering reduction operations are invoked. To accomplish this, nodes P0-P3 communicate a partition's worth of data and invoke a reduction kernel to reduce the received partition using locally available partitions, which takes two time steps in steady state. Overall, for four partitions across four nodes, this is done three times; e.g., each node transmits three partitions, performs three local reductions, and receives three partitions. In the illustrated example, this consumes 10 time steps for data generation (GEMM) and associated communication (scattering reduction) in a four-node system. After the scattering reduction primitives are completed, nodes can also be configured to share the reduced partitions among themselves.

[0040] In contrast, in the second example 204, the target communication module 118, the data generation and communication tracking module 120, and the update convergence unit 122 are used to implement a mechanism for performing communication and reduction as each word (or set of words) of data is generated. As a result, generation and communication overlap in what will be referred to as "fine granularity" in the following description. The techniques described herein also support the ability to program "coarse-grain" updates by the data mover engine 106 (e.g., direct memory access "DMA") as the data is generated. Both scenarios support the programming of these updates to implement a wide range of communication patterns.

[0041] In one example, the techniques described herein track data generation using the data generation and communication tracking module 120 and advantageously transmit (e.g., “push”) the generated data to other nodes as “fine-grained targeted updates.” In a second scenario, the target communication module 118 leverages the tracking performed by the data generation and communication tracking module 120 to trigger updates, e.g., as targeted updates coordinated by the data mover engine 106. The specific actions invoked and address ranges tracked are fully programmable, e.g., by a programmer or as part of the operating system. Using the techniques described herein, both data generation and associated communication are completed in four time steps in the second example 204, compared to ten time steps in the first example 202 of the baseline system. The benefits of these techniques increase as the amount of data being processed also scales.

[0042] These technical advantages increase further as the number of devices and GEMM size increase; for example, a number of "n" devices in the baseline involves "{2(n-1)+n}" steps, while the techniques described herein involve "n" steps. For large "n" and GEMM sizes, this works to reduce the time steps by a factor of three. Furthermore, the techniques described herein can be configured to utilize both computational and network resources simultaneously, rather than in a serialized manner as in the baseline scenario.

[0043] 3 is a block diagram of a non-limiting example 300 of the data generation and communication tracking module 120, the target communication module 118, and the update convergence unit 122. These modules represent three logical parts of the described technology that support the merging and overlapping of data generation and related computations.

[0044] The data generation and communication tracking module 120 represents functionality with operations to implement low-overhead tracking of both local data generation 302 and remote data communication 304, e.g., remote stores, DMA transfers, etc., as part of implementing programmable tracking to communication mapping 306. This supports the programmable ability to effectively coordinate communication of data based on the progress and / or completion of data generation (e.g., local or remote) and other communications, thus enabling data generation and communication fusion. The data generation and communication tracking module 120 supports structures and mechanisms to allow a programmer, for example, to program and map target communications (e.g., defined processors and / or address ranges) to specific data generation and / or communication events.

[0045] The data generation and communication tracking module 120 implements operations for tracking address ranges, for example, and is configured to do the following: Transfer=Y, DMA=N: Issues a read-modify-update locally to the defined processor. ● If transfer=N and DMA=Y: local_counter=remote_counter=threshold, send a signal to the data mover engine, - If transfer=N and DMA=N: local_counter=remote_counter=threshold, signal protocol completion.

[0046] The target communications module 118 represents functionality for performing actions and implementing mechanisms for targeting data communications based on configurable conditions triggered by tracking performed by the data generation and communications tracking module 120. This includes fine-grained remote communications 308 and DMA initiated bulk communications 310, as described above. An example of a direct memory access extension as implemented by the target communications module 118 includes "enable memory control signals for address range x, read address range from local memory, initiate read-modify-update to correct address range y within the defined processor."

[0047] The update convergence unit 122 implements operations to support scenarios involving communication-related computations, such as summation for clutter reduction, through the use of a convergence mechanism to allow simultaneous updates to data from the local store / updates 312 and remote store / updates 314.

[0048] The communication of data may be initiated in a variety of ways, for example, based on the completion of local data generation, a remote communication event, etc. To support this, data generation and communication tracking module 120 uses extensions to memory controller 108 to implement lightweight tracking of data generation and communication, for example, using a table structure. This tracking is utilized to trigger targeted fine-grained memory operations (e.g., pre-programmed updates to remote nodes that are triggered when a local update is generated) and / or targeted DMA coordination memory operations, for example, as programmed into the target communication tracking by data mover engine 106.

[0049] Supporting "fine-grained" and "bulk" operations via direct memory access supports numerous technical advantages. In a first example, fine-grained memory operations support immediately communicating locally generated data to remote nodes in a programmable manner. However, in some examples, this can result in high inter-node traffic. Furthermore, data communication can be contingent on remote communication in addition to local generation. To address this, DMA-coordinated bulk communication is configured to implement multiple communication events, e.g., to support triggering communication transmissions spanning multiple words. Additionally, the programming described herein also supports specific communication patterns triggered upon completion of data generation or communication events, as further described in connection with FIG. 6.

[0050] The data generation and communications tracking module 120, the target communications module 118, and the update convergence unit 122 may be implemented in either hardware, software, firmware, or a combination thereof. In the illustrated example, the data generation and communications tracking module 120 may be configured using a microcontroller 316 operable to execute instructions 318 as a special purpose machine to achieve the result of generating a programmable trace to the communications mapping 306. In another example, the data generation and communications tracking module 120 is configured to generate a programmable trace to the communications mapping 306 at least in part using hardware 320 (e.g., an integrated circuit 322, such as an application specific integrated circuit).

[0051] Similarly, the target communication and tracking module 118 can be configured using a microcontroller 324 operable to execute instructions 326 as a special purpose machine to achieve the result of generating the local store / updates 312 and the remote store / updates 314. In another example, the target communication module 118 is configured at least in part using hardware 328 (e.g., integrated circuit 330, examples of which include application specific integrated circuits) to generate the local store / updates 312 and the remote store / updates 314.

[0052] Additionally, the update convergence unit 122 can be configured using a microcontroller 332 configured to execute instructions 334 as a dedicated machine to implement a convergence mechanism that supports concurrent updates using physical memory 112. In another example, the update convergence unit 122 is configured at least in part using hardware 336 (e.g., an integrated circuit 338, such as an application specific integrated circuit) to implement a convergence mechanism that supports concurrent updates using physical memory 112.

[0053] As mentioned above, in some scenarios, communications associated with data generation involve computation, e.g., reduction. To support this, infrastructure 100 enables concurrent local data generation (with updates) while enabling remote updates to the data. This is achieved in the techniques described herein using near / in-memory processing by update convergence unit 122. Update convergence unit 122 is configured to leverage physical memory 112 (e.g., main memory) as a synchronization point. In one example, to do so, data generation is implemented solely by using updates instead of using stores. Furthermore, update implementation is performed by update convergence unit 122; thus, both local store / updates 312 and remote store / updates 314 can run concurrently with low synchronization costs.

[0054] 4 is a block diagram of a non-limiting example 400 of data generation and communication tracking that supports fusion scattering reduction operations. In the illustrated example, a data generation and communication tracking table 402 is shown implemented by the data generation and communication tracking module 120. Similarly, a target communication tracking table 404 is shown implemented by the target communication module 118. In the example shown in FIG. 4, an entry for each node is included in each of the tables. However, in practice, each node may also be configured to store table entries associated with itself; for example, node "P0" stores the column marked "P0".

[0055] The data generation and communication tracking table 402 is configured for use in tracking address ranges. For each range, the data generation and communication tracking module 120 implemented by the memory controller 108 tracks both local stores / updates and remote stores / updates as shown in FIG. 3 using local and remote counters, respectively. The data generation and communication tracking module 120 also tracks whether fine-grained local and / or remote updates should be triggered for a given address range. The data generation and communication tracking module 120 tracks whether DMA coordination communications (e.g., updates) are triggered based on either the completion of a local data generation or communication event (e.g., "remote update = threshold, remote update = local update = threshold," etc.). This can be used for both contiguous and non-contiguous address ranges in memory, for example, using strided, multidimensional, or indirect access patterns.

[0056] The techniques described herein blend data generation and associated communication. To do so, they support the ability to program fine-grained updates or coarse-grained direct memory access coordinated updates as data is generated. This supports the ability to programmatically implement any desired communication pattern.

[0057] In one such scenario, scattering reduction operations are programmatically implemented on a ring network, as shown in FIG. 2. Specifically, stores to a specific address range are tracked by the data generation and communication tracking module 120 of the memory controller 108 (e.g., address range “1” on node P0) and immediately forwarded to a designated address within address range “A” at a designated node, e.g., P1 in the ring topology. Simultaneously, since scattering reduction involves a reduction operation, the stores are issued as read-modify updates both locally and remotely. Furthermore, the programming is implemented such that when the local and remote updates for a specific address range reach a threshold (e.g., address range “3” on P0, threshold=12), the memory controller 108 is programmed to signal this event to the data mover engine 106. The data mover engine 106, via the target communication module 118, then triggers a pre-programmed target communication event, e.g., updating the DMA address range “C” on P1 using values ​​read from the P0 address range “3.”

[0058] Nodes can be programmed in a variety of ways, such as at boot time for a static network topology or programmed for each communication event. Additionally, while the above description refers to nodes as communicating entities, in alternative embodiments, other components in the system (e.g., switches, programmable accelerators) can also be configured as communicating nodes. Additionally, although also not shown, conditions (e.g., local update = remote update = threshold for scattering reduction) and / or actions (e.g., read-modify-update for scattering reduction) can also be programmable as specific to an operation, application, etc.

[0059] While some data generation and associated communication scenarios involve simply communication (e.g., all-to-all in machine learning training), alternative scenarios are also contemplated in which communication has associated computational operations, e.g., reduction operations in clutter reduction. To support these alternative scenarios, a low overhead synchronization substrate is implemented by the update convergence unit 122 that supports simultaneous data generation and remote updates to the data. The update convergence unit 122 is, for example, configured to handle both local and remote updates to the data.

[0060] In the scattering reduction example above, data-generating stores are represented as updates. In one embodiment, this is achieved via software-level changes, such as page table and / or cache-level mechanisms that direct stores to specific address ranges to bypass the cache, allowing the memory controller to convert local data-generating stores into updates. Additionally, memory operations (via a "forward" flag) or direct memory access coordination to remote nodes are also converted into updates. Each of these updates is offloaded to the update convergence unit 122 for completion.

[0061] Update convergence unit 122 can be configured in various ways. In a first example, update convergence unit 122 is implemented as a dedicated unit at a single level in the memory hierarchy, which may be housed in either a memory-side cache, a memory controller, the base die of a 3D memory stack, near a DRAM bank, etc. In scenarios where update convergence unit 122 is located at multiple levels, update convergence units that handle the same address coordinate with each other to ensure proper application of local / remote updates.

[0062] FIG. 5 is a block diagram of a non-limiting example 500 of unordered data generation. FIG. 6 is a block diagram of a non-limiting example 600 of ordered data generation. Data generation order directly affects the communication pattern used to communicate data. Therefore, controlling the data generation order improves the efficiency of fused data generation and associated communication. In the unordered data generation example 500, where order is not controlled, the fused data generation and associated communication are completed in six time steps. However, in the example 600 of FIG. 6, data generation is ordered to improve efficiency, e.g., the fused data generation and associated communication are completed in four time steps.

[0063] In a further embodiment, priority information can be programmed into a Target Communications Tracking (TCT) table in the data mover engine 106 to prioritize certain communications over others to further shorten the critical path. This is shown in Figure 6 for node P0, which prioritizes communications in address range "4" first instead of range "3."

[0064] While merging data generation with associated communications has performance benefits, such merging can result in higher concurrent memory traffic than serializing data generation and communications. Therefore, mechanisms for managing interference are implemented as part of infrastructure 100. As an example, communications memory traffic is deprioritized while data generation is not complete. While these examples involve clutter reduction operations, the techniques described herein can also be used to forward remote communications to designed nodes in a fine-grained manner.

[0065] 7 is a flow diagram of a non-limiting example 700 of fused data generation and communication. A data generation and communication tracking module tracks program data generation and communication as it executes among multiple processors of a processing system (block 702). By way of example, the data generation and communication tracking module 120 implements the data generation and communication tracking table 402.

[0066] Target communication of data between the multiple processors is triggered as part of the program data generation and communication (block 704). By way of example, the target communication module 118 triggers communication based on tracking performed by the data generation and communication tracking module 120.

[0067] Concurrent updates are resolved to physical memory by an update convergence unit that includes data generated by multiple processors (block 706). By way of example, the update convergence unit 122 resolves local store / updates 312 and remote store / updates 314 to physical memory 112.

[0068] In the above example, the techniques described herein support an infrastructure that effectively blends data production and associated communication in a programmable manner. This implements various technical advantages, including, but not limited to, improved performance, concurrent utilization of both computational and network resources instead of serialized utilization, offloading of communication from the main processor (CPU / GPU) where communication is programmed once and implicitly triggered based on completion of data production and / or communication, reduced kernel startup costs, etc. In the following description, these techniques are used in an example for use with fine-grained in-memory reduced-based aggregation.

[0069] Reduction-based ensembles are utilized as part of training for natural language processing applications in multi-device setups. These ensembles involve communication and reduction of data from multiple devices and are used to aggregate gradients (in data-parallel setups) or activations (in model-parallel setups) during training.

[0070] However, these collectives are often serialized with application execution, potentially becoming a bottleneck, causing performance to scale sub-linearly with the number of devices being trained. However, the data used by these collective operations is typically not generated concurrently. Data generated by matrix multiplication (GEMM) operations, for example, is executed in multiple stages with a set of workgroups per stage. Thus, in one embodiment, the communication and reduction of data from a single GEMM stage overlaps with the execution of the next GEMM stage in a fine-grained manner. This reduces the cost of collective operations with producer kernels.

[0071] Implementing this functionality presents several challenges. For example, producer and collective operations are typically implemented as separate kernels in the graphics processing unit, which involves computationally expensive synchronization when performed in a fine-grained manner. Furthermore, contention for both computational and memory resources by the collective GEMM stage and the producer GEMM stage can degrade overall performance.

[0072] To overcome these challenges, hardware / software mechanisms are described to transparently execute producer and collective operations in a fine-grained manner. This is done by leveraging address space to automatically initiate fine-grained data communication on producer store instructions, and therefore can be performed without modifying the kernel. Furthermore, these techniques leverage near-memory compute units to atomically update memory locations on stores, thus limiting contention with producer operations. This mechanism therefore reduces communication costs and frees up compute resources (e.g., in graphics processing units) from performing reductions. This enables efficient near-linear scaling of training as the number of devices increases. Furthermore, this mechanism accelerates collectives (through fewer memory accesses) while also improving overall utilization of compute and network resources.

[0073] For example, a large-scale network matrix multiplication (GEMM) operation executes and generates data in multiple stages. Furthermore, GEMMs from transformer models often have large output sizes, which are tiled / blocked and involve many workgroups (WGs) or thread blocks (TBs) to compute. These workgroups typically do not execute all at once in practice due to a finite number of graphics processing units or streaming multiprocessors. Instead, they typically execute in multiple stages, with each stage being a set of workgroups or thread blocks accommodated by a graphics processing unit. The number of stages varies with the size and shape of the GEMM and the kernel implementation used. Thus, the output of a GEMM, and therefore a layer, is typically generated in multiple stages rather than all at once. This is true even when operations are partitioned across devices with model parallelism. This is because a GEMM that is partitioned across devices and includes a "total reduction" set is typically partitioned in the "K" dimension. Thus, the work performed by the threads or workgroups in each of the sub-GEMMs is generally smaller (e.g., shorter row and column dot products), but the output matrix size produced by each remains the same as in the original GEMM, meaning the number of threads / WGs, and therefore stages, performed by each of the sub-GEMMs remains similar.

[0074] This insight is exploited by the mechanisms described herein to overlap data reduction / communication (e.g., "all-reduce" operations) with data generation. For example, communication of data generated in one stage overlaps with data generation (computation) in the next stage, and the operation is "hidden."

[0075] The mechanism described herein transparently enables fine-grained execution of collective operations with producer GEMMs, for example, by having GEMM writers automatically trigger communication / reduction of the produced data. This is done by allocating the GEMM's output within an address space without modifying the GEMM kernel. In this example, this reduction is handled entirely in hardware.

[0076] Furthermore, overlapping GEMMs and aggregates can cause contention for graphics processing unit resources, slowing down overall execution. There are two sources of contention between GEMMs and aggregates. The first issue is contention for graphics processing compute units, which can degrade GEMM performance. Second, the reduce operation is memory intensive and can compete with the producer GEMM operations for memory bandwidth. To address this, in one example, a collective operation is automatically initiated for GEMM writes to the address space. Therefore, no additional compute units are involved to perform the aggregate. Furthermore, these writes are converted to updates on the fly and handled by the near-memory arithmetic logic unit, and therefore involve minimal additional memory overhead than the original GEMM writes.

[0077] 8 is a block diagram of a non-limiting example 800 of a baseline system 802 versus a fine-grained in-memory reduction-based collective system 804. The fine-grained in-memory reduction-based collective system 804 is shown as an "all-reduce" collective in a simple two-device system.

[0078] In the baseline system 802, the graphics processing units first execute their respective producer GEMMs and store the output in local memory. The graphics processing units then initiate a clutter reduction operation in which each graphics processing unit reduces a "chunk" of the output array (i.e., the chunk's home node). This requires a direct memory access transfer (or peer-to-peer copy) to ensure that each graphics processing unit has a copy of the chunk for which it is responsible. This is followed by a memory load of the copy by each graphics processing unit, reduction by the graphics processing unit, and a local store of the reduced version. A final transfer (e.g., broadcast) of the reduced version of the chunk to the remaining devices is performed to complete the "gather all" operation. The total number of loads / stores from memory depends on the topology, number of devices, and algorithm (e.g., ring vs. direct) used by the baseline system 802.

[0079] In contrast, in the fine-grained in-memory reduction-based aggregation system 804, aggregation is performed transparently in a fine-grained manner using producer GEMM operations, and the aggregation execution time is "hidden." To perform the scatter reduction operation in this example, instead of directing each GEMM write to local memory, the write is directed to either a local (in the case of the array element's home node) or a remote memory location. Furthermore, writes to designated locations in this example use a near-memory arithmetic logic unit to atomically update the data there. Thus, upon receiving writes from each of the associated devices, each home memory location contains a reduced version of the data in its entirety. Following this, chunks can be transferred to other devices to complete an "all-gather" operation of the data.

[0080] Thus, in this example, data scattering reduction overlaps with data generation. In this example, it is fully orchestrated in hardware, thereby reducing software complexity and further reducing total memory traffic. As shown in Figure 8, for example, data corresponding to each element is read / written to local / remote memory nine times in the baseline system 802, compared to four times in the fine-grained in-memory reduction-based aggregation system 804, due to the concurrent execution of GEMM and aggregation.

[0081] The mechanism described herein includes support for automatically initiating data communication / reduction in response to a producer's write command. The mechanism also leverages near-memory computing to atomically update memory locations. To do so, the mechanism implements an address space for transparent merging of producers and collects.

[0082] To avoid the complexity of fine-grained aggregation in software and to avoid modifying the implementation of hundreds of GEMM kernels from a large library, in this example the producer GEMM and fine-grained execution of collective operations are transparently implemented in hardware. To do this, the output of the producer GEMM is assigned to an address space, such that executing aggregations that require writing to the address space automatically executes.

[0083] As shown in Figure 8, writes can be used to trigger three types of actions: local, remote, and direct memory access (DMA). Furthermore, the memory locations and sequences of these actions can differ for different types of aggregates (e.g., full reduction, scatter reduction) and techniques, e.g., ring, direct. Therefore, systems implementing the mechanisms described herein are configured to support a memory mapping API that can be used to configure the memory allocated for different write-initiated actions for different aggregate types and techniques.

[0084] This mechanism is configurable using a library with predefined memory mappings that are "invoked" by each application. In a GPU all-reduce operation, for example, memory is allocated on each device in the address space by, for example, specifying a collection and a mechanism. This function first allocates an array on each device. For an "all-reduce" operation, a local allocation of the entire array is performed on each device, and a final reduced version of the entire array on each device is assembled. This is followed by an API call to map subarrays of the local allocations to remote allocations of the array for remote writes. The output array on each device in this example is thus mapped to distributed physical memory. This mapping ensures that writes to local subarrays are redirected as remote writes to their respective home nodes. Furthermore, it defines which operations (e.g., updates) are performed by the remote write operation. Once allocated, GEMM is executed, followed by an additional direct memory access (or peer-to-peer copy) of the reduced data from remote memory to local memory.

[0085] In the case of memory allocation to an address space, writes to the address space are not cached by the device because they are not read locally until the reduction is complete. Therefore, writes in this example are written to physical memory 112, e.g., dynamic random access memory (DRAM). Furthermore, stores to these pages are either directed to local physical memory 112 if originating from the home node itself, or directly to remote physical memory 112, to avoid redundant writes and reduce memory bandwidth pressure. This also ensures that there is a single aggregation point for each copy of data. This is implemented by extending the translation lookaside buffer and page table to include both the local and remote physical addresses of the pages in memory, or via a separate hardware structure. Stores to these locations are sent to memory controller 108 if to local physical memory 112, while stores to remote physical memory 112 are directed to the remote graphics processing unit memory controller.

[0086] Physical memory 112 on the home device can be used as an aggregation unit for each copy of the array. Local stores issued from the home device and remote stores from other devices are received and enqueued in memory controller 108 for later transmission to physical memory 112, e.g., dynamic random access memory. Loads to these pages occur only as part of the next graphics processing unit kernel. Each direct memory access to stores and locations is guaranteed to complete by a system range fence, which in one example is inserted as part of the direct memory access function after GEMM completes execution. As a result, the translation lookaside buffer and page table direct loads to the local copy.

[0087] In DRAM architectures with near-memory computation support, each bank is associated with an arithmetic logic unit (ALU) and registers for storing intermediate values. Stores to these banks can therefore be used to update memory locations. Thus, the DRAM banks associated with the address space of the techniques described herein can be programmed to update memory locations on store commands.

[0088] Such updates first write the store value to a register associated with the near-memory arithmetic logic unit, activate the corresponding memory row, read and add the column value from the row buffer to the data in the register, and write the reduced value back to the buffer. Queuing stores or near-memory updates in a queue in memory controller 108 facilitates the atomicity of these updates, so that at a given time, a single instruction is issued to and executed by the arithmetic logic unit corresponding to the memory location. Furthermore, converting these stores to atomic updates on the fly does not violate the memory consistency guarantees of the graphics processing unit. These updates are commutative atomics in nature and therefore can be reordered with respect to other relaxed atomics, which are also stores, just like stores. In one example, these stores / updates in the queue are coalesced by a memory queue coalescer to improve performance. Coalescing multiple updates to the same location helps reduce the number of row activations and / or row buffer reads / writes. Overall, these near-memory update-based reductions reduce and in some cases eliminate memory resource contention with a running GEMM. For direct reduce-scatter operations, the total number of memory operations involved in the reduction is the same as that performed by GEMM alone.

[0089] Figure 9 is a block diagram of a non-limiting example 900 of a fine-grained all-to-all operation 902 and a gather all operation 904. The traffic pattern of the all-to-all operation 902 matches that of an all-reduced set, and therefore can utilize the same configuration as shown in Figure 8, except that the writes do not update memory. The gather all operation 904, on the other hand, is implemented by directing GEMM writes to both local and remote memory.

[0090] Furthermore, collective (e.g., "reduced all") operations in natural language processing applications are typically followed by other memory-intensive operations on each of the participating devices (e.g., parameter updates in a data-parallel setup or residual / dropout layers in a model-parallel setup). However, these operations consume the entire reduced array on each device and are therefore redundant in some cases. Memory reduction capabilities therefore provide an opportunity to limit such redundant operations. Consumer operations, which can also be performed using near-memory arithmetic logic units, operate on the (reduced) subarray of data on the home node before it is "collected all" or broadcast to the remaining devices. This reduces redundant computations and further improves distributed natural language processing performance.

[0091] It should be understood that many variations are possible based on the disclosure herein, and although features and elements are described above in particular combinations, each feature or element can be used alone without the other features and elements, or in various combinations with or without the other features and elements.

[0092] The various functional units illustrated in the figures and / or described herein (including device 102, where appropriate) may be implemented in any of a variety of different ways, such as hardware circuits, software or firmware running on a programmable processor, or any combination of two or more of hardware, software, and firmware. The provided methods may be performed in any of a variety of devices, such as a general-purpose computer, a processor, or a processor core. Suitable processors include, by way of example only, a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), a graphics processing unit (GPU), a parallel-accelerated processor, multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), and / or a state machine.

[0093] In one or more embodiments, the methods and procedures provided herein may be implemented in a computer program, software, or firmware embodied in a non-transitory computer-readable storage medium for execution by a general-purpose computer or processor. Examples of non-transitory computer-readable storage media include read only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs).

[0094] Although the systems and techniques have been described in language specific to structural features and / or methodological acts, it is to be understood that the systems and techniques defined in the appended claims are not necessarily limited to the particular features or acts described. Rather, the particular features and acts are disclosed as exemplary forms of implementing the claimed invention.

Claims

1. 1. A system comprising: a processing system including a plurality of processors; At least one processor of the plurality of processors tracking program-defined data generation and associated communications executed by said plurality of processors; triggering a target communication of data between the plurality of processors based on the traced program-defined data generation and associated communication; configured to: system.

2. the program-defined data generation and associated communication includes generation of data by the at least one processor and target updating by the at least one processor to transmit the data to another processor of the plurality of processors; The system of claim 1.

3. the target update is triggered upon completion of generation of the data by the at least one processor; The system of claim 2.

4. the target update is triggered to be performed by a data mover engine by another of the plurality of processors based on a remote communication event received at the at least one processor. The system of claim 2.

5. the remote communication event is part of a bulk operation that includes communication of the data; The system of claim 4.

6. the program-defined data production and associated communication are defined using a single fused data production and associated communication operation; The system of claim 1.

7. the fused data generation and associated communication operations identify another processor of the plurality of processors to receive the data. The system of claim 6.

8. The fused data generation and associated communication operations identify an address range that is a source of the data or a destination to which the data is sent. The system of claim 6.

9. the at least one processor is configured to support concurrent updates to the data in physical memory. The system of claim 1.

10. a processor-in-memory component of the memory module containing the physical memory configured to perform concurrent updates; The system of claim 9.

11. the program-defined data generation and associated communications are configured to control an order of data generation by each processor of the plurality of processors; The system of claim 1.

12. A device, a processing system including a plurality of processors; At least one processor of the plurality of processors triggering a targeted communication of data between said at least one processor and another processor of said plurality of processors as part of a programmatically defined data generation and related communication; resolving concurrent updates to said data in physical memory; configured to: device.

13. a processor-in-memory component of a memory module including the physical memory configured to resolve concurrent updates to the data in the physical memory; The device of claim 12.

14. the at least one processor is configured to track the program-defined data generation and associated communications executed by the plurality of processors, and to trigger the target communication based on the tracked program-defined data generation and associated communications. The device of claim 12.

15. the target communication is configured to be performed based on a single fused data generation and associated communication operation performed by the at least one processor, the fused data generation and associated communication operation identifying another processor of the plurality of processors to which the data is to be sent; 15. The device of claim 14.

Images