System and method for coalesced multicast data transfer via a memory interface

Abstract

A method and apparatus for controlling digital data transfer between a memory module and a processor over a memory channel is disclosed. At least one of the memory module or the processor coalesces a plurality of short data words into a multicast coalesced block data including a single data block for transfer over the memory channel. Each of the plurality of short data words is associated with one of at least two partitioned memory sub-modules in the memory module. The multicast coalesced block data is communicated over the memory channel.

Description

[Background technology]

[0001] Cache memory is typically organized into cache lines that store information read from or written to main memory. When using information in cache memory, whether reading (or loading) data from cache memory or writing (or storing) data to main memory, the memory interface is designed to read or write the entire cache line at once, even if only a portion of the information in the entire cache line is needed, or if only a portion of the information in the entire cache line needs to be updated in main memory. As a result, when performing narrow data word access, there are often more memory transactions than are needed to read the requested information into the cache or write the information to main memory. Excessive memory transactions not only consume power by increasing overhead, but also degrade performance and cause memory degradation.

[0002] Some conventional memory modules typically use a cache line size of 64 bytes such that each transfer of data to and / or from the memory is performed in a cache line sized burst over a memory bus. A simplified example is shown in FIG. 1, which is a logical representation of a prior art system implementing wide access from a single memory bank. Generally, a memory module 100 has a host interface 102 that couples to a memory interface 106 of a host processor 104, also referred to as a main processor, that coordinates the calculations performed in the load and store operations of the memory module 100 via a memory bus or memory channel 108 of a particular width, in this case 256 bits or 32 bytes. Thus, in some examples, the memory module 100 can support only a coarse granularity single access (i.e., 32 bytes) for any load or store operation. The memory module 100 includes a plurality of memory banks or sub-modules 112, 114, 116, 118 (in this example, having 16 sub-modules), each operatively coupled to the host interface 102 via a separate memory channel 110 (in this example, having a 256-bit channel width). Each memory sub-module can include a memory address register (MAR) for storing the address of the memory location being accessed by a load or store operation, and a memory data register (MDR) for storing the data for which the operation is being performed. Such registers are not shown for simplicity.

[0003] The host processor 104 issues a load request or a store request when it needs to load memory data from a specific memory submodule or store data in that memory submodule. When there is a data transfer between the host processor 104 and the memory module 100, such as when the host processor 104 requests that a specific number of bits of data be sent to the host processor 104 from a specified submodule (in this example, submodule 114), a broad access is performed, and the entire cache line of contiguous data containing the requested number of bits of data is transferred from a single specified submodule to the host interface 102. This can lead to significant bandwidth waste in the case of a workload that accesses data sparsely (e.g., at 16-bit granularity), as only a small portion of the data is accessed whenever it is transferred across memory interfaces.

[0004] Some conventional computing systems attempt to address the inefficiencies of subcache line memory access by using software to statically identify such short accesses that are unlikely to be reused. Some approaches involve subranking, which groups multiple memory chips in a dual in-line memory module (DIMM) or RAM stick into subsets, allowing each subset to serve smaller chunks of data than the original transfer size. However, subranking requires separate commands for each subrank, resulting in reduced performance due to a higher demand on the shared command bus, or increased hardware costs due to the dedicated command path required for each subrank. Subranking is also impractical when each data access is served by a single memory module. Some approaches involve reducing the height and / or width of the DRAM banks, but such approaches involve altering the core DRAM array design, resulting in high overhead.

[0005] Other approaches involve matrix-vector multiplication, sparse matrix algebra, graph analysis, and sparse machine learning, which fall within areas where software can often predict cache reuse or its absence. However, modern memory systems still have the problem of not being able to leverage such information to optimize memory bandwidth utilization, and there is still room for improvement in terms of memory efficiency and effective bandwidth for sparse workloads.

[0006] Therefore, having improved data transfer between memory modules and the host processor would be highly advantageous, as it would enable the transfer of narrower data widths from memory submodules without performing excessive memory transactions.

[0007] Embodiments will be more readily understood with reference to the following description when accompanied by the following figures in which similar reference numerals represent similar elements. [Brief explanation of the drawing]

[0008] [Figure 1] This is a prior art block diagram showing an example of a memory module and host processor in a system where, when the host processor issues a load or storage request, an entire cache line of continuous data is transferred between the memory module and the host processor. [Figure 2] This block diagram shows an example of a memory module and host processor in a system configured to perform multicast memory coalesce operations according to the embodiments described in this disclosure. [Figure 3] This block diagram shows an example of multicast merged block data formed as a result of a multicast memory merger operation performed by the system shown in Figure 2. [Figure 4] This figure shows an example of a system with multiple memory modules coupled to a host processor, as shown in Figure 2. [Figure 5] This flowchart shows an example of a method for performing a multicast memory coalescing operation as described in this disclosure. [Figure 6] This flowchart shows an example of a method for switching between facilitating multicast memory merging operations and facilitating the transfer of continuous block data, as described in this disclosure. [Figure 7] This flowchart shows an example of a method for loading data from a memory submodule, as described in this disclosure. [Figure 8] This flowchart shows an example of a method for storing data in a memory submodule, as described in this disclosure. [Figure 9] This figure shows an example of the system as described in this disclosure. [Figure 10] This figure shows an example of a memory module as described in this disclosure. [Figure 11] This figure shows an example of the system as described in this disclosure. [Figure 12] This figure shows an example of the system as described in this disclosure. [Figure 13] This figure shows an example of the system as described in this disclosure. [Figure 14] This figure shows an example of the system as described in this disclosure. [Modes for carrying out the invention]

[0009] In short, the system and method help reduce data transfer overhead and facilitate fine-grained data transfer by combining or aggregating short data words from multiple heterogeneous memory submodules and simultaneously transferring or communicating the combined data, referred herein as multicast combined block data, over a memory channel in a single block data transfer. In some embodiments, short data words are returned to or loaded from each of the partitioned memory submodules to a host processor at a specific location within a single block data transfer. In some embodiments, short data words are written to or stored in each of the partitioned memory submodules from the host processor at a specific location within a single block data transfer. In some examples, a memory submodule has an associated register that stores short data words from the corresponding submodule or short data words extracted from multicast combined block data received from the processor until the multicast memory combined operation is performed. In some examples, the register may be implemented as part of a near-memory processing technique or an in-memory processing technique.

[0010] According to a particular embodiment, a method for controlling digital data transfer over a memory channel between a memory module and a processor, performed by at least one of a memory module or a processor, involves aggregating a plurality of short data words into multicast aggregated block data containing a single data block for transfer over the memory channel. Each of the plurality of short data words relates to one of at least two divided memory submodules within the memory module. The multicast aggregated block data is communicated over the memory channel.

[0011] In some examples, the method includes the processor detecting a condition that indicates the possibility of short data word amalgamation, and switching, depending on the detected condition, between a first mode that facilitates the transfer of multicast amalgamation block data and a second mode that facilitates the transfer of contiguous block data between the processor and any of the memory submodules.

[0012] In some examples, the method includes the processor sending a coalescing load command to a memory module to cause the memory module to perform a multicast memory coalescing operation to retrieve short data words from memory submodules and coalesce the short data words into multicast coalescing block data, and, upon receiving multicast coalescing block data, extracting each of the short data words from the multicast coalescing block data. In certain examples, the method includes the processor sending one or more location identifiers to the memory module that identify multiple locations associated with short data words in a memory submodule to cause the memory module to retrieve short data words from the identified locations in the memory submodule. In some examples, the method includes the processor sending a single location identifier so that the memory module retrieves short data words from the same location (e.g., address offset or near memory register ID) in multiple memory submodules and coalesces the retrieved short data words into a single data block as multicast coalescing block data. In some examples, multiple locations are associated with multiple location identifiers or identified by multiple location identifiers.

[0013] In some embodiments, the method includes the processor configuring at least one register associated with each of the memory submodules to store short data words accessible by the processor so that they may be aggregated into multicast aggregated block data. In some examples, the method includes the processor generating multicast aggregated block data to be transferred to the memory module and sending an aggregated store command to the memory module. The aggregated store command causes the memory module to perform a multicast memory extract operation to extract short data words from the multicast aggregated block data, distribute the short data words to the memory submodules, and store the short data words in the memory submodules. In certain examples, the method includes the processor sending one or more location identifiers to the memory module that identify multiple locations associated with short data words in the memory submodules so that the memory module stores the short data words in the identified locations in the memory submodules. In another example, the method includes the processor sending one location identifier so that the memory module stores short data words in the aggregated data block at the same location (address offset or near memory register ID) in multiple memory submodules.

[0014] In some examples, the method includes: a processor generating multicast coalesced block data to be transferred to a memory module; configuring at least one register associated with a memory submodule to cause the memory module to extract short data words from the multicast coalesced block data; and storing the short data words in at least one register. In some examples, the method includes: a processor generating multicast coalesced block data to be transferred to a memory module; and configuring a memory module to cause the memory module to determine one or more location identifiers that identify multiple locations associated with short data words in a memory submodule, based on multicast coalesced block data supplied by the processor or information stored in the memory module.

[0015] According to a particular embodiment, the processor includes a memory interface that communicates with memory modules via a memory channel and multicast coalescing logic. The multicast coalescing logic performs data transfer between the processor and memory modules via the memory channel and communicates the multicast coalescing block data via the memory channel by coalescing a plurality of short data words into multicast coalescing block data containing a single data block before transmission. Each of the plurality of short data words is associated with one of at least two divided memory submodules.

[0016] In some examples, the multicast coalescing logic configures the memory controller associated with a memory submodule to detect conditions for switching between a first mode that facilitates the transfer of multicast coalescing block data and a second mode that facilitates the transfer of contiguous block data between the processor and either of the memory submodules, where the conditions indicate the possibility of coalescing short data words.

[0017] In some examples, the multicast merge logic sends a merge load command to a memory module to cause the memory module to retrieve short data words from memory sub-modules and perform a multicast memory merge operation to merge the short data words into multicast merge block data, and in response to receiving the multicast merge block data, extracts each of the short data words from the multicast merge block data. In certain examples, the multicast merge logic sends one or more location identifiers that identify a plurality of locations associated with short data words within a memory sub-module to the memory module to cause the memory module to retrieve the short data words from the identified locations within the memory sub-module.

[0018] In some examples, the multicast merge logic configures at least one register associated with a memory sub-module to cause the memory module to read a short data word from the memory sub-module and perform a multicast memory merge operation to merge the short data word from the register into multicast merge block data. In some examples, the register may be a near-memory register per sub-module or an in-memory register per sub-module, and when performing the multicast memory merge operation, the short data word stored in the register from a previous in-memory operation or near-memory operation is retrieved and merged into a single data block.

[0019] In some examples, the multicast aggregation logic generates multicast aggregation block data to be transferred to a memory module, sends an aggregation memory command to the memory module, causes the memory module to perform a multicast memory extraction operation to extract short data words from the multicast aggregation block data, distribute the short data words to memory sub-modules, and store the short data words in the memory sub-modules. In a particular example, the multicast aggregation logic sends one or more location identifiers that identify a plurality of locations associated with short data words in a memory sub-module to the memory module, causing the memory module to store the short data words in the identified locations in the memory sub-module.

[0020] In some examples, the multicast aggregation logic configures at least one register associated with a memory sub-module to cause the memory module to generate multicast aggregation block data to be transferred to the memory module, extract short data words from the multicast aggregation block data, and store the short data words in a register. The register can be a near-memory register per sub-module or an in-memory register per sub-module.

[0021] According to a particular embodiment, a memory module includes a processor interface that communicates with a processor via a memory channel, a plurality of divided memory sub-modules, and multicast aggregation logic. The multicast aggregation logic performs data transfer between the processor and the memory module via the memory channel by aggregating a plurality of short data words into aggregation block data that includes a single data block before transfer and communicating the multicast aggregation block data via the memory channel. Each of the plurality of short data words is associated with one of the memory sub-modules.

[0022] In some examples, the memory module includes a mode selection component that switches between a first mode that facilitates the transfer of multicast amalgamation data, including but not limited to tristate gates or multiplexers, and a second mode that facilitates the transfer of contiguous block data between the processor and any of the memory submodules. In certain examples, the memory module also includes a memory controller associated with the memory submodule that detects conditions indicating the possibility of short data word amalgamation and switches between the first and second modes.

[0023] In some examples, the memory module includes at least one near-memory processing logic or in-memory processing logic that determines one or more location identifiers that identify multiple locations associated with short data words within a memory submodule, based on multicast aggregated block data supplied by the processor or information stored in the memory module.

[0024] In some examples, the multicast coalescing logic for a memory module includes multiple shifter logic components coupled to a memory submodule. Each of the shifter logic components shifts the location of a short data word based on the address offset of the memory submodule. The short data words from the shifter logic components are concatenated to form multicast coalescing block data. In a particular example, the multicast coalescing logic performs the multicast memory coalescing operation by retrieving a short data word from the memory submodule, shifting the location of the short data word using the shifter logic components, and coalescing the short data word into multicast coalescing block data, in response to receiving a coalescing load command from the processor. In a particular example, the multicast coalescing logic receives from the processor one or more location identifiers that identify multiple locations associated with a short data word within the memory submodule, so that the short data word is retrieved from identified locations within the memory submodule.

[0025] In some examples, multicast coalescing logic further includes multiple selector logic components. Each selector logic component selects a short data word from a portion of the data retrieved from one of the memory submodules, based on the address offset from the memory submodule. The short data words from the selector logic components are concatenated to form multicast coalescing block data.

[0026] In some examples, a memory module includes at least one register associated with a memory submodule. The register stores the short data words associated with the corresponding memory submodule until a multicast memory merge operation is performed to merge the short data words from the register into multicast merged block data. In some examples, each of the short data words may have been previously loaded from the corresponding memory submodule, or may have been computed by in-memory or near-memory processing logic based on data stored in the submodule.

[0027] In some examples, the multicast coalescing logic of a memory module includes multiple subset distribution logic components coupled with memory submodules. Each subset driver logic component extracts one of the short data words from the multicast coalescing block data and distributes the extracted short data words to one of the memory submodules. In some examples, the multicast coalescing logic of a memory module performs a multicast memory extraction operation in response to receiving a coalescing store command from the processor to store the short data words distributed to the memory submodules by the subset distribution logic components within the memory submodules. In certain examples, the multicast coalescing logic further receives from the processor one or more location identifiers that identify multiple locations associated with the short data words within the memory submodules, so that the short data words are stored in identified locations within the memory submodules.

[0028] In some examples, a memory module includes at least one register associated with a memory submodule. The register stores short data words extracted from multicast coalesced block data received from the processor. In some examples, subsequent memory commands may perform memory operations using the short data words in the register and / or in the associated memory submodule. In some examples, the multicast coalesced logic performs a multicast memory extract operation to extract short data words from the register, distribute them to the memory submodule, and store the short data words within the memory submodule.

[0029] According to some embodiments, a system for controlling digital data transfer includes a processor, a memory module containing a plurality of partitioned memory submodules, a memory channel between the processor and the memory module, and multicast amalgamation logic. The multicast amalgamation logic performs data transfer between the processor and the memory module over the memory channel by amalgamating a plurality of short data words into multicast amalgamation block data containing a single data block before transfer, and by communicating the multicast amalgamation block data over the memory channel. Each of the plurality of short data words is associated with one of the memory submodules. In some examples, the system further includes a mode selection component that switches between a first mode that facilitates the transfer of multicast amalgamation data and a second mode that facilitates the transfer of contiguous block data between the processor and one of the memory submodules, and a memory controller associated with the memory submodule. The memory controller detects conditions that indicate the possibility of amalgamation of short data words and controls the mode selection component to switch between the first mode and the second mode based on the detected conditions.

[0030] The following description includes numerous specific details to provide a full understanding of the methods and mechanisms presented herein. However, those skilled in the art should recognize that various embodiments can be implemented without these specific details. In some examples, well-known structures, components, signals, computer program instructions, and techniques are not shown in detail to avoid obscuring the approaches described herein. For simplicity and clarity, it should be understood that the elements shown in the figures are not necessarily drawn to scale. For example, the dimensions of some elements may be exaggerated relative to others.

[0031] Figure 2 shows a logical representation of an example computing system, more specifically, a hardware server, smartphone, wearable, printer, laptop, desktop, or part of any other suitable computing device that utilizes data transfer between the memory module 100 and the host processor 104. For simplicity, a single memory module and a single memory channel are shown, but it should be understood that this disclosure also applies to systems with multiple memory modules and channels. In this example, the memory module 100 may be the main memory of the host processor 104, which may be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), an application-specific integrated circuit (ASIC), or any other suitable integrated circuit that performs calculations and issues memory requests to the memory module 100. Cache storage may reside in the host processor, the memory module, or both. As shown in Figure 4, any number of memory modules can be coupled to the host processor 104, and it should be understood that the system can implement multiple host processors 104 coupled together in an operable manner.

[0032] Furthermore, memory submodules may be referred to as memory banks in this specification. Memory submodules 112, 114, 116, and 118 are heterogeneous and operably coupled to the host interface 102 via multiple memory channels or data links 200, 202, 204, and 206 for the transfer of short data words. In an example with 16 submodules, there are 16 short data word links, each short data word link facilitating, for example, the transfer of 16-bit short data words, and each memory link operates independently of the other links. Each link may occupy a subsection of a shared memory channel, such as a shared memory data bus 207. Data links 200, 202, 204, and 206 have a channel width narrower than memory channel 108, such that the sum of the widths of all data links 200, 202, 204, and 206 is equal to the width of memory channel 108.

[0033] In the illustrated example, memory module 100 contains 16 short word links, each with a width of 16 bits, totaling 256 bits, which is equal to the width of memory channel 108. It should be understood that other links may exist to connect memory submodules to the host interface 102. For example, each submodule may be operably connected to the host interface 102 via memory channel 110, as shown in Figure 1, in which case memory channel 110, having a channel width greater than any of data links 200, 202, 204, and 206, is used for the transfer of sequential block data instead of multicast combined block data, the details of which are further disclosed herein.

[0034] The systems and methods disclosed herein are described in the context of memory submodules sharing access to a memory channel to receive a broadcast instruction stream from a host processor. However, it will be apparent to those skilled in the art that the techniques disclosed herein can be extended to other forms of the classification of memory submodules that receive broadcast instructions.

[0035] Figure 3 shows an example of multicast combined block data 300 transferred between the memory module 100 and the host processor 104 via the memory channel 110. The block data 300 includes data segments 302, 304, 306, and 310 such that each data segment is associated with or related to a separate memory submodule. That is, in the case of a load command, each data segment is a short data word retrieved from any memory submodule in a one-to-one correspondence, i.e., each memory submodule can contribute only one short data word to the block data 300. In the case of a store command, each data segment is a short data word stored in a location within any memory submodule in a one-to-one correspondence, i.e., each memory submodule can receive only one short data word from the block data 300 stored therein. Each short data word may have any suitable word size (e.g., 8 bits, 16 bits, 32 bits, 64 bits, etc.) smaller than the channel width and cache line width, depending on the number of memory submodules in the memory module.

[0036] In the exemplary system shown in Figure 2, the transferred block data 300 contains 16 data segments, each segment having 16 bits, and each segment is assigned to only one of the memory submodules. For example, data segment 302 is assigned to memory submodule 112, data segment 304 is assigned to memory submodule 114, data segment 306 is assigned to memory submodule 116, data segment 308 is assigned to memory submodule 118, and so on. For illustrative purposes only, the CLD facilitates each memory submodule (which may include a PIM unit in some examples) returning 16 bits of data calculated by dividing 256 bits (the data transfer width of a single-channel memory interface) among the 16 memory submodules. Therefore, data from memory submodule 112 occupies bits 0 to 15 of block data 300, data from memory submodule 114 occupies bits 16 to 31 of block data 300, data from memory submodule 116 occupies bits 32 to 47 of block data 300, and so on, with data from memory submodule 118 potentially occupying bits 240 to 255 of block data 300. The block data 300 is then transferred or communicated via the memory channel 108 in a single block data transfer between the memory module 100 and the host processor 104. The block data 300 is also referred to as multicast aggregated block data due to its nature as block data containing multiple individual short data words addressed to multiple individual, independently functioning memory submodules, and the individual short data words are transferred simultaneously in a single block data transfer when the multicast aggregated block data is transmitted via the memory channel.

[0037] Figure 4 shows an example of a system having multiple memory modules 100, 400, 402, and 404, each of which is operably coupled to a host processor 104 via a memory channel 108. Each memory module may include one or more memory dies and one or more logic dies having built-in computing power provided by processing-near-memory (PNM) technology. For example, the computing power of memory module 100 is implemented on a separate logic die 406 that is 3D stacked with the memory die, and the memory die may be implemented together with memory submodules 112, 114, 116, and 118. The memory submodules 112, 114, 116, and 118, also referred to as memory banks, may in some examples be dynamic random access memory (DRAM) devices. Each of the other memory modules 400, 402, and 404 may be constructed similarly. The embodiments described herein are also applicable to cases where computing power (i.e., computing units) is incorporated into each memory bank or memory module, such as in bank-level processing-in-memory (PIM) systems. For example, computing units may be implemented directly on memory dies instead of on individual logical dies. To overcome limitations in command bandwidth between the host and PIM units, a stream of commands may be broadcast to multiple PIM units within a memory module. An exemplary embodiment of such a configuration is that each PIM command is broadcast to all memory banks associated with a single memory channel. Furthermore, the embodiments described herein are also applicable to other system configurations that may consist of multiple host processors and memory modules interconnected in various configurations.

[0038] In certain embodiments, a non-temporary storage medium such as memory includes executable instructions that cause one or more processors, such as a host processor 104 or a memory module 100 having data processing capabilities, to execute a method for controlling digital data transfer over the memory channel disclosed in Figures 5 to 8.

[0039] Figure 5 shows a method 500 for performing a transfer of amalgamated block data, which is performed by a host processor 104 or by a memory module 100 having either a logical die with built-in computing power, such as provided by a PNM, or a memory die with built-in computing power, such as provided by a PIM, where each memory die has its own independent computing power. In step 502, the processor or memory module amalgamates a plurality of short data words into multicast amalgamated block data. The multicast amalgamated block data contains a single data block for transfer over a memory channel, and each of the short data words is associated with one of at least two divided memory submodules within the memory module. In some examples, the short data words are associated with a subset of memory submodules such that the short data words are loaded from or stored at specific locations within the memory submodules. In step 504, the single data block containing the plurality of short data words is transferred over the memory channel in a single data transfer. The transfer may be from the processor to the memory module or from the memory module to the processor.

[0040] Figure 6 shows a method 600 for switching between different modes in the system, one mode facilitating the formation and transfer of multicast amalgamated block data between the processor and at least two of the memory submodules, as described in method 500, and the other mode facilitating the transfer of contiguous block data between the processor and a single memory submodule. In step 602, the processor detects a condition indicating the possibility of amalgamation of short data words. In step 604, depending on the detected condition, the memory module or the processor switches between a first mode that facilitates the transfer of multicast amalgamated block data and a second mode that facilitates the transfer of contiguous block data between the processor and any of the memory submodules.

[0041] In some embodiments, the process of detecting conditions and issuing commands to switch between a first mode and a second mode is performed entirely on the host side at the host processor's memory interface. Multicast memory merging operations for merging short data words associated with multiple memory submodules may be performed on the host memory side of the memory module (i.e., when a merging load command is issued by the processor), but the host processor is responsible for detecting conditions and issuing multicast merging requests to the memory module, which may be issued as part of a merging load command.

[0042] In some embodiments, the condition is explicitly triggered by an application running on the processor or an instruction issued by an application (e.g., a special memory command to “start coalescing,” or an explicit coalescing memory command issued by an application that triggers coalescing). In some embodiments, the condition includes an indicator of sparse memory access behavior in the memory interface. Sparse memory access is defined as sparse access of fewer data bits (e.g., short data words) in two or more memory submodules, as opposed to contiguous memory access where a single contiguous section of more data bits (e.g., an entire cache line or data block) is accessed in a single memory submodule. For example, the memory interface may include a memory controller that stores a set of memory commands from the processor in queues, such that there is a separate queue for each memory submodule. The memory controller detects a hint or indicator that the command at the top of each queue consists of a memory command for sparse bits or short data words, and in response to the detection, the sparse bits or short data words from the queue are concatenated or coalesced into multicast coalescing block data.

[0043] In some embodiments, the memory controller may store a set of memory commands in a single memory queue. In this case, the condition may be detected by periodically searching the queue to find a memory command for a short data word that maps to a different memory submodule. Additionally or alternatively, the queue may be searched to find other commands of the same or similar type as the short data word in a different submodule that it may be coalesced with when inserting a memory command into a short data word. A threshold for the number of coalesced commands may be implemented to trigger the condition. If coalescing is limited to short data words that share some address offset bits (for example, if the short data words are in the same DRAM column index), the address bits are also compared when searching for coalescing opportunities. In such embodiments, the queue entry may also contain information about whether the associated memory command targets a short data word, information about whether multiple short data words have been coalesced into the queue entry, and offset information about the short data word targeted by the queue entry.

[0044] Figure 7 shows a method 700 in which a processor performs a coalescing memory load operation. In step 702, the processor sends a coalescing load command to the memory module. The command causes the memory module to perform a multicast memory coalescing operation to retrieve a short data word from a memory submodule and coalesce the short data word into coalescing block data. In step 704, the processor determines whether the coalescing command targets a memory submodule directly or whether the command targets an in-memory register or a near-memory register. In some examples, the type of command being coalesced specifies whether a memory submodule or an in-memory register or a near-memory register is targeted.

[0045] If the coalesce command targets a memory submodule, in step 706 the processor communicates one or more location identifiers to the memory module. One or more location identifiers identify multiple location identifiers associated with short data words in the memory submodule. The processor may cause the memory module to extract short data words from the identified locations in the memory submodule. The short data words are coalesced into multicast coalesced block data before being transmitted as a single data block over the memory channel. Then, in step 708, the processor extracts (or decouples) each of the short data words from the multicast coalesced block data as it receives the multicast coalesced block data.

[0046] If the merge memory command targets an in-memory register or a near-memory register, step 710 configures at least one register associated with each memory submodule so that the memory module stores the short data words to be merged in the merged block data. In some embodiments, the registers may be implemented per memory submodule and directly accessible. If the same register and the same offset within the register are accessed for all memory submodules, then, according to some examples, it may not be necessary to provide location information for each of the short data words. The arrangement of the short data words in the register may be pre-arranged by the processor by executing a memory-local load command from the memory submodule in some examples. In some examples, the arrangement of the short data words in the register may be arranged by processing each of the short data words (e.g., performing a calculation on the short data words) using the near-memory processing capability of the memory module or in-memory processing capability. In step 712, the multicast memory merge operation is performed by the processor, reading the short data words from the registers and forming the multicast merged block data before proceeding to step 708.

[0047] Figure 8 shows a method 800 in which a processor performs a coalescing memory storage operation. In step 802, the processor generates multicast coalescing block data to be transferred to the memory module. In step 804, the processor determines whether the coalescing command targets a memory submodule or whether the command targets an in-memory register or a near-memory register.

[0048] If the coalesce command targets a memory submodule, in step 806 the processor sends one or more location identifiers to the memory module. The location identifiers identify the locations associated with short data words within the memory submodule, thereby causing the memory module to store the short data words at the identified locations within the memory submodule. In step 808 the processor sends a coalesce store command to the memory module. The command causes the memory module to perform a multicast memory extract operation to extract short data words from the multicast coalesced block data, distribute the short data words to the memory submodule, and store the short data words at the identified locations within the memory submodule. Steps 806 and 808 may be performed sequentially or concurrently. In some examples, the location identifiers may be generated or computed by an in-memory processing logic component or a near-memory processing logic component based on registers per submodule or data stored in the memory submodule.

[0049] If the coalescing memory command targets an in-memory register or a near-memory register, the processor proceeds to step 810 and configures at least one register associated with the memory submodule so that the memory module extracts a short data word from the multicast coalescing block data and stores the short data word in at least one register. For example, after the short data word has been stored in a register specific to the memory submodule, the near-memory computing component or in-memory computing component of the memory submodule may access the register for further calculations and / or data movement.

[0050] In methods 700 and 800, the location identifier may be communicated by using a coalescing load / storage command and / or a few bits in the data bus to transmit the location bits for each submodule. Alternatively, the location identifier may be obtained by calculating the location information for each submodule, for example, by loading the location identifier and / or by calculating the location identifier in near memory and then storing it in a near memory register. In some examples, static location information, or location information common to all memory submodules, may not need to be transmitted individually. For example, if additional per-submodule location bits are added to a common base address and the coalescing command targets a short data word at the same offset in each memory submodule or near memory register, additional location information may not be provided.

[0051] As shown in Figures 9 to 14, any logic component disclosed herein may be implemented as individual logic, one or more state machines, a field programmable gate array (FPGA), or any preferred combination of an instruction-executing processor / processing logic component and other hardware. According to embodiments, a logic device or logic component includes a device or component capable of performing logical operations, such as a device capable of performing Boolean logic operations. Functional blocks are shown, and it will be recognized that various operations can be combined or separated as desired. It will also be recognized that not all functional blocks are required in all embodiments. The arrows shown in the figures indicate the direction of data transfer between components during a specified load or storage operation, which may be implemented using any preferred data path, such as hardwiring or data channels such as data buses. In addition, some embodiments may involve pipelined transfers / operations in which data is buffered in one or more registers. Furthermore, for the sake of simplification, the embodiments described herein relate to a memory module 100 which is a high-bandwidth memory (HBM) from which the entire data access cache line is provided from a single memory module.

[0052] It should be understood that this disclosure is also applicable to other types of memory in which multiple memory modules contribute to a single data access. In such cases, the consolidation of multiple short data elements (i.e., short data words) into a single data block may occur within each involved memory module. Furthermore, although the drawings are presented in the context of an exemplary system having 16 memory banks or submodules and a memory channel interface having a 256-bit data width (i.e., the size of the data transferred in each load or storage operation to the channel) for block data transfer over a single channel, it should be understood that other embodiments of the system with different parameters are possible, among other things, such as more or fewer memory submodules and / or wider or narrower memory channels.

[0053] Figure 9 shows an example of a computing system that utilizes data transfer between a memory module 100 and a host processor 104. In this example, the memory module 100 includes a multicast coalescing logic 900 which includes multiple processing logic components 902, 904, 906, and 908, each having multiple registers 912, 914, 916, and 918, with each processor and register associated with one of the memory submodules 112, 114, 116, and 118. The processing logics 902, 904, 906, and 908 are near-memory computing components or in-memory computing components configured to provide computing power to the memory submodules. The multicast coalescing logic 900 is coupled with data links 200, 202, 204, and 206 so that short data words are transferred via data links 200, 202, 204, and 206 through a shared data bus or data channel 934.

[0054] For illustrative purposes only, short word data received from or sent to processing logic 902 may occupy bits 0-15 of data channel 934; short word data received from or sent to processing logic 904 may occupy bits 16-31 of data channel 934; short word data received from or sent to processing logic 906 may occupy bits 32-47 of data channel 934; and similarly, short word data received from or sent to processing logic 908 may occupy bits 240-255 of data channel 934. In some embodiments, processing logics 902, 904, 906, and 908 may include sufficient processing power to support coalesced load (CLD) and coalesced store (CST) operations, and may be dedicated solely to supporting such operations.

[0055] The memory module 100 further includes a mode selector 922, which is a mode selection component that can be implemented, for example, as a tristate gate or multiplexer. The mode selector 922 operates as a switch between a first mode that facilitates the transfer of multicast coalesced block data and a second mode that facilitates the transfer of sequential block data, as described. The mode selector 922 may be implemented as a programmable logic device such that control bits are used to activate the switching. The mode selector 922 is configured to transfer data through a data channel 934 in the first mode and to transfer data to and from a memory submodule selector 924 in the second mode. The memory submodule selector 924 is shared by all memory submodules and is configured to select which submodule to access based on a provided memory submodule identifier (ID) 910 when the mode selector 922 is operating in the second mode. Each of the mode selector 922 and the submodule selector 924 may be implemented as a single logic component in a single central location (as shown in the figure) according to some embodiments. Alternatively, in some embodiments, one or both of the mode selector 922 and / or submodule selector 924 may be implemented in a distributed manner such that the mode selector 922 and / or submodule selector 924 include a plurality of independently functioning logic components, and at least one logic component is located near each of the memory submodules to control access by the memory submodules to a single shared data channel or data bus. In some examples, the logic components may include, but are not limited to, multiplexers or tristate gates.

[0056] Furthermore, the host processor 104 includes a multicast coalescing logic 926 separate from the multicast coalescing logic 900 of the memory module 100, and therefore has different functionality. An example of performing the transfer of multicast coalescing block data is described in detail below, taking into account the components described above.

[0057] In some embodiments of the CLD operation, for a PIM implementation system, a PIM register identifier is specified for each PIM unit associated with each memory bank or submodule, utilizing PIM support. Each PIM unit brings a short data word (e.g., 16 bits) of the identified register to the combined output. If the register is longer than the length of the short data word, some embodiments may return the lower 16 bits of the register or some other fixed offset of the register. In other embodiments, the CLD operation may have a parameter that allows software on the host to specify which 16 bits of the register each PIM unit should return. In other embodiments, the register may be programmed a priori in the PIM unit before the CLD operation is issued.

[0058] In some of the other embodiments, the CLD operation may specify a memory address within each memory bank or submodule so that each memory submodule reads and returns a short word data (e.g., 16 bits) stored at a specified memory location. In one embodiment, this is achieved by each memory module receiving a broadcast memory address within the module as part of the CLD operation and returning the data at that location within each memory submodule. In other embodiments, this is achieved by utilizing support for communicating additional address information to each memory submodule. This can be achieved through the generation of bank-local addresses or by enhancing the command interface, for example by sending command information using a data bus, or alternatively, by combining commands that share the address bits in question.

[0059] In CLD operation, in response to a command broadcast to a collection of memory submodules, each memory submodule returns a chunk or block of data to the requesting host at a fixed, unique location within the broad load data return of the memory interface. Once each memory submodule has returned data to its specific unique location within the data returned to the host, all involved memory submodules return their respective data in a single-block data transfer over the memory data bus or channel.

[0060] Specifically, during CLD operation as shown in several examples, the host processor 104 specifies submodule-specific registers 912, 914, 916, and 918 associated with the memory submodules 112, 114, 116, and 118 to be accessed. Each of the in-memory processing logic components 902, 904, 906, and 908 outputs to the host interface 102 via data channels 934 and 936, and thus brings forth a short data word of a predetermined word length (e.g., 16 bits in the illustrated example) to be stored in the corresponding register.

[0061] In some embodiments, if registers 912, 914, 916, and 918 are wider than the word length of the short data word, the lower 16 bits (or a number of bits related to the length of the short data word) may be returned, or alternatively, other offsets may be employed to retrieve the short data word from a different location in the register. In some embodiments, when the CLD operation is initiated, the host processor 104 may issue a command specifying which bits of the register each of the processing logics 902, 904, 906, and 908 should return. In some embodiments, the short data word may be captured by the corresponding processing logics 902, 904, 906, and 908 by a priori retrieval of the short data word via the memory channel 930 before the CLD operation is issued by the host processor 104. As shown in the diagram, memory channel 930 is wider than data links 200, 202, 204, and 206 (e.g., 256 bits) because the same channel can be used to transfer block data between submodules 112, 114, 116, and 118 and the memory bank selector 924.

[0062] In some examples, after short data words from their respective corresponding memory submodules 112, 114, 116, and 118 are captured in registers 912, 914, 916, and 918, processing logics 902, 904, 906, and 908 transfer the short data words as multicast coalesced block data via data channel 934. If the mode selector 922 switches to a first mode that facilitates the transfer of multicast coalesced block data, the host processor 104 can retrieve block data via data channel 934. Otherwise, if the mode selector 922 switches to a second mode that facilitates the transfer of continuous block data, the host processor 104 can retrieve continuous block data from any of the memory submodules 112, 114, 116, and 118 via another data channel 932, as determined by the submodule ID 910 provided by the host processor 104.

[0063] Upon receiving multicast coalescing block data via memory channel 108, the host processor 104 transfers the block data via internal data channel 938 to multicast coalescing logic 926, which extracts short data words from the block data to be processed accordingly. Thus, multicast coalescing logic 926 can also operate as an extractor or decoupling component, and multicast coalescing logic 900 in memory module 100 can perform such operations as further described herein with respect to coalescing memory operations.

[0064] In some embodiments of the CST operation, for a PIM implementation system, a PIM register identifier is specified for each PIM unit associated with each memory bank or submodule, utilizing PIM support. Each PIM unit writes a short data word received from the combined input to the identified register. If the register is longer than the length of the short data word (e.g., 16 bits), some embodiments may store or perform the PIM operation on the data in the lower 16 bits and set the remaining bits to zero, for example, through masking. Other embodiments may sign-extend the 16-bit short data word and store or perform the PIM operation with the extended data targeting the specified PIM register. In yet another embodiment, the CST operation may have parameters that allow software on the host to specify which 16 bits of the register each PIM unit should write the corresponding data to. In another embodiment, the register may be programmed a priori in the PIM unit before the CST operation is issued.

[0065] In some of the other embodiments, the CST operation may specify a memory address within each memory bank or submodule so that each memory submodule writes a short word data (e.g., 16 bits) to the designated memory location. In one embodiment, this is achieved by each memory module receiving a broadcast memory address within the module as part of the CST operation and storing the data at that location within each memory submodule. In other embodiments, this is achieved by utilizing support for communicating additional address information to each memory submodule. This can be achieved through the generation of bank-local addresses or by enhancing the command interface, for example, by sending command information using a data bus, or alternatively, by combining commands that share the address bits in question.

[0066] Specifically, in some examples, the host processor 104 forwards the data in the opposite direction to the CLD operation so that the multicast coalescing logic 926 executes the coalescing of multiple short data words to be transferred to the memory module 100 as a single data block (or single write operation), and then the short data words are extracted and distributed across memory submodules 112, 114, 116, and 118. For example, data distribution may be performed as follows: bits 0-15 of the host-provided multicast coalescing block data are sent to submodule 112, bits 16-31 of the host-provided multicast coalescing block data are sent to submodule 114, bits 32-47 of the host-provided multicast coalescing block data are sent to submodule 116, and so on, until bits 240-255 of the host-provided multicast coalescing block data are sent to submodule 118.

[0067] In some embodiments, the CST operation specifies registers 912, 914, 916, and 918 associated with submodules 112, 114, 116, and 118, and the corresponding processing logic 902, 904, 906, and 908 writes a short data word (e.g., 16 bits) extracted from the multicast coalesced block data stored in registers 912, 914, 916, and 918.

[0068] In some embodiments, if registers 912, 914, 916, and 918 are wider than the word length of a short data word, the short data word may be stored in the lower 16 bits (or a number of bits related to the length of the short data word), while leaving the remaining bits zero. In some embodiments, a sign extension operation is performed on the short data word to extend its data length before it is stored or processed by the processor, and the extended data is targeted to the specified registers 912, 914, 916, and 918. In other embodiments, the CST operation may have parameters that allow the host to specify which of the bits stored in registers 912, 914, 916, and 918 should be written by processing logic 902, 904, 906, and 908 to the corresponding memory submodules 112, 114, 116, and 118.

[0069] In both CLD and CST operations, the host processor 104 may provide instructions for the operation of the mode selector 922 and the submodule selector 924. For example, when the multicast coalescing logic 926 provides CLD or CST operation via a command channel or command bus 928, it may initiate a command or instruction 940 to switch the mode selector 922 from a second mode that facilitates the transfer of continuous block data to a first mode that facilitates the transfer of multicast coalescing block data. An instruction to switch to the first mode may be as simple as activating a control bit in the mode selector 922 such that the first mode is activated when the control bit is 1 and the second mode is activated when the control bit is 0. Otherwise, if the host processor 104 intends to switch the mode selector 922 back to the second mode (for example, after the transfer of multicast coalescing block data is complete), this can be achieved by toggling the control bit.

[0070] In some embodiments, near-memory processing logic components 902, 904, 906, 908 can determine one or more location identifiers (e.g., address bits such as column indexes of a memory array) that identify specific locations associated with short data words in memory submodules 112, 114, 116, 118, based on multicast coalesced block data supplied by the host processor 104, or information stored in memory module 100, e.g., registers 912, 914, 916, 918. Furthermore, although Figure 9 shows each memory submodule associated with its own near-memory processor or in-memory processor, in some embodiments, the processing power of a single near-memory processor or in-memory processor may be shared among multiple memory submodules.

[0071] In some embodiments, instructions for switching between the two modes described above may be provided by a near-memory processor or an in-memory processor, indicated by the transfer of aggregate configuration bits 944 from near-memory storage, for example, from any of the processing logic components 902, 904, 906, or 908. The processor may update its near-memory storage values ​​whenever it determines the conditions for switching between aggregate mode and continuous mode. For example, the processor may be able to detect conditions indicating the possibility of transferring multicast aggregate block data, as described herein. Furthermore, some embodiments may also involve pipelined transfers / operations in which the data is buffered in one or more intermediate buffer registers, for example, the block data register 1102 shown in Figures 11 to 14, which may be located between the mode selector 922 and the multicast aggregate logic 900.

[0072] Figure 10 shows different types of registers that may be used by processing logic components 902, 904, 906, and 908 as part of the multicast coalescing logic 900. In some embodiments, offset registers 1000, 1002, 1004, and 1006 may be implemented to store offset information for short data words loaded from or stored in the corresponding submodules. For example, in CLD operation, offset registers 1000, 1002, 1004, and 1006 may store specific address offsets of short data words when short data words coalesce with other short data words to form multicast coalescing block data (e.g., a short data word from memory submodule 112 occupying bits 0-15, a short data word from memory submodule 114 occupying bits 16-31, etc.). Each address offset is different and unique to the corresponding memory submodule to avoid short data word entries overwriting other short data word entries due to unintended duplication of occupied bits.

[0073] Furthermore, in CST operation, the address offset can help the processing logics 902, 904, 906, and 908 identify from which bits of the multicast coalesced block data the retrieved short data word should be stored in the corresponding memory submodules 112, 114, 116, and 118. In some embodiments, the offset registers 1000, 1002, 1004, and 1006 contain multiple offset values ​​so that, when the processor provides a base address, the processing logics 902, 904, 906, and 908 can calculate unique location information associated with each of the multiple memory submodules based on the stored offset information and the provided base address. The offset values ​​stored in the registers may be pre-programmed or stored a priori before the CLD operation or CST operation is issued by the host processor 104.

[0074] In some embodiments, the coalescing configuration registers 1008, 1010, 1012, and 1014 may be implemented to store data relating to the mode selected by the mode selector 922, as determined by the processing logic components 902, 904, 906, and 908 in response to detecting conditions indicating the transfer of multicast coalescing block data. The registers can represent a single bit, where a bit of 1 activates a first mode and a bit of 0 activates a second mode. The short data registers 1016, 1018, 1020, and 1022 are registers that store short data words that are coalesced into multicast coalescing block data during CLD operation, or short data words that are stored in the corresponding memory submodules 112, 114, 116, and 118 during CST operation. In some cases, the CLD operation reads from short data registers 1016, 1018, 1020, 1022 (or one or more near memory registers per submodule) to obtain a short data word, in which case the short data word is already stored in the register as a result of a previous near memory processing operation. Thus, the short data word may be stored a priori in these short data registers before the operation is issued.

[0075] Figure 11 shows an example of a computing system that utilizes data transfer between a memory module 100 and a host processor 104. In this example, the host processor 104 provides instructions to operate the mode selector 922, address offsets for each short data word, and location addresses for each of the memory submodules 112, 114, 116, and 118. The address offsets and location addresses are sent from the host processor 104 as instruction commands 1100 to the multicast coalescing logic 900, which is coupled with a block data register 1102 for storing multicast coalescing block data, and to the memory submodules, respectively. The address offsets define how much the position of the short data word is shifted during multicast coalescing operation, and the location addresses define a specific location within a given memory submodule (e.g., a column index of a memory array within the memory submodule) that is accessed for CLD or CST operation. In some examples, a near-memory integer adder may be implemented to generate the location addresses of the memory submodules. Figures 12 and 14 show the data flow within the system shown in Figure 11 during CLD operation, and Figure 13 shows the data flow within the system during CST operation.

[0076] Figure 12 shows an example of a system that operates CLD operation, where the multicast coalescing logic 900 includes concatenation logic 1200 coupled with an address shift component, in this example the address shift component being shifter / selector logics 1202, 1204, 1206, and 1208, which are programmable and can be configured to receive data loaded from memory submodules 112, 114, 116, and 118. The shifter / selector logics 1202, 1204, 1206, and 1208 are configured to receive contiguous block data from the corresponding memory submodules 112, 114, 116, and 118. The shifter / selector logics 1202, 1204, 1206, and 1208 are then configured to (1) select a short data word from the contiguous block data for storage in the block data register 1102, or (2) shift a short data word by an address offset before storing the short data word in the block data register 1102.

[0077] If selector logic (1) is implemented, short data words are selected using a predetermined address offset without performing an address shift. If shifter logic (2) is used, short data words are shifted separately to a first predetermined number of bits using their respective offsets, and a predetermined number of bits starting from the first bit are selected to obtain the short data word. The stored short data words are joined together using concatenation logic 1200, which performs string concatenation to join the short data words from end to end.

[0078] Although not shown in the diagram, the shifter / selector logic 1202, 1204, 1206, and 1208 may include registers configured to store location addresses (which determine which bits or locations within a contiguous block of data should be selected as a short data word) and / or address offset values ​​(which determine the amount of shift performed on the short data word before coalescing).

[0079] The short data word extraction logic 1210 may be implemented in the host processor 104 to extract individual short data words from multicast amalgamated block data after receiving multicast amalgamated block data via memory channel 108 in a single block data transfer. Each shifter / selector logic 1202, 1204, 1206, 1208 receives address offset information in the instruction command 1100 provided by the host processor 104, which defines where the short data words from each memory submodule should be located within the multicast amalgamated block data. Each memory submodule 112, 114, 116, 118 may receive location address information in the instruction command 1100 to store in a memory address register the location from which the short data words should be retrieved.

[0080] Figure 13 shows an example of a system operating CST, where the host processor 1300 includes concatenation logic 1300 for forming multicast coalescing block data containing short data words to be stored in multiple memory submodules 112, 114, 116, and 118. The multicast coalescing logic 900, in this case, receives the multicast coalescing block data via the decoupling path and includes subset distribution logic 1302, 1304, 1306, and 1308 for distributing the short data words to their respective memory submodules as intended by the host processor 104. In some examples, the subset distribution logic further implements additional address offset bits communicated from the multicast coalescing logic to the memory submodules to indicate which short data words need to be written and to prevent writing other bits in the column index of the memory array.

[0081] The block data register 1102 stores multicast amalgamated block data received from the host processor 104, and the stored data is transferred to the respective subset distribution logics 1302, 1304, 1306, and 1038 via data channels 1310, 1312, 1314, and 1316. Each subset distribution logic receives address offset information in the instruction command 1100 provided by the host processor 104, which defines which bits in the multicast amalgamated block data should be distributed to which memory submodule. In some examples, each subset distribution logic may be used to drive a subset of the data stored in the block data register 1102. Each memory submodule 112, 114, 116, and 118 may receive location address information in the instruction command 1100 to store in the memory address register the location where a short data word should be stored.

[0082] Figure 14 shows an example of a system that operates CLD operation, where the shifter / selector logics 1202, 1204, 1206, and 1208 from Figure 12 are excluded from the multicast coalescing logic 900. Instead, the address offsets of the short data words from the corresponding memory submodules 112, 114, 116, and 118 in the block data register 1102 are either a priori configurable or predetermined. That is, the short data word offsets are static offsets, in which case the data connections to the appropriate bits in the block data register 1102 can be hard-routed (thus forming an address shift component) so that the address bits of the short data words from each memory submodule are automatically shifted so that they are stored in some predetermined or pre-configured bits in the block data register 1102.

[0083] Furthermore, the concatenation logic 1200 is also excluded from the multicast coalescing logic 900. Instead of using the concatenation logic 1200 to control the coalescing of short data words into a single block of data stored in the block data register 1102, as shown in Figure 12, the position-shifted short data words from memory submodules 112, 114, 116, and 118 are concatenated to each other by directly joining the wiring from these memory submodules to the data bus in order to transfer the bits of the short data words to the block data register 1102.

[0084] In some embodiments, CLD and CST operations can be performed by communicating partial command information via a data bus rather than a command bus. For example, in CLD operation with a memory module 100 having PNM or PIM capabilities as shown in Figure 9, the combined short data words occupy a portion of the multicast amalgamated block data, and as a result, the rest of the block data is used to store other information, such as address bits for each of the short data words. Similarly, in CST operation, the address bits for each of the short data words can be implemented in a portion of the multicast amalgamated block data that is not occupied by the short data words stored in the memory submodule.

[0085] In certain embodiments, a non-temporary storage medium, such as memory, includes executable instructions, commands, or address information that cause one or more processors to arrange short data words so that, when executed by one or more processors such as a host processor 104 or a PNM / PIM device, sparse access via CLD and CST operations to the short data words associated with corresponding locations within each memory submodule can be coordinated. This may be applicable in situations where large table entries are partitioned across memory submodules for efficient use of PNM or PIM operations, including but not limited to situations with deterministic sparse access patterns such as access along the sub-axis of a multidimensional matrix or tensor, or applications involving machine learning-based recommendation systems. In some embodiments, coalescence may be explicitly performed via a software embodiment, and the executable instructions for this purpose are stored in the non-temporary storage medium. For example, a processor running software may send a pre-coalescence request that bypasses the cache, and the request is processed by the memory controller without requiring additional hardware for coalescence.

[0086] In some embodiments, when an executable instruction is executed by one or more processors, it causes one or more processors to send independent, fine-grained, sparse accesses, which are then combined via hardware such as, for example, individual logic, a state machine, an FPGA, or any preferred combination of processors executing the instruction. The host processor 104 may be able to dynamically detect opportunities for combination based on monitoring the data channels and memory submodules targeted by independent and concurrent accesses. The host processor or PNM / PIM device may also be able to integrate or combine independent requests (i.e., CLD or CST commands requesting access to memory submodules) and may be able to split responses during CLD or CST operation.

[0087] If sparse commands are capable of traversing the cache hierarchy, then, as some examples show, these requests may need to be handled differently by the cache controller, thus requiring a distinction between sparse access and non-sparse data access, such as contiguous block data access with a single memory submodule. In such situations, the distinction can be facilitated, for example, by implementing additional bits or opcodes within the command. In some embodiments, sparse access is handled differently from non-sparse data access because it does not access the entire cache line of data. Therefore, as some examples show, sparse access can simply bypass the cache (for example, using existing support for uncached address ranges). Alternatively, as some examples show, sparse access may traverse the cache but be prevented from allocating or adding cache blocks in case of a miss. In some embodiments, the cache is enhanced with a sector mask that tracks state information at the granularity of sparse access, allowing the cache to store partially valid cache blocks.

[0088] In multicast memory coalescing operations that utilize register offset information (if PNM / PIM registers are implemented) or address offset information, commands from the host processor may require additional address information compared to sparse access. This can be implemented by splitting the request into more packets using a bitmask or bytemask to indicate which bytes in the cache line are accessed by the sparse command, or by sending part or all of the request along a dedicated data path such as command bus 928.

[0089] In systems where the host processor can dynamically detect opportunities for coalescing, the processor may be implemented with means for merging and / or splitting requests and responses associated with sparse memory accesses. In some embodiments, the coalescing operation is performed within the memory controller, and requests are already sorted into different queues based on the memory submodule of interest. When a sparse memory access is detected at the head of a queue of multiple memory submodules, the memory controller merges the requests together and issues a single CLD or CST operation that includes all requests associated with the memory submodule. In some embodiments, when a sparse memory access reaches the head of a bank queue, elements in other queues (or a subset of elements in other queues) are searched to find a sparse memory access that can be coalesced into a single memory request having the original sparse memory access. In some embodiments, sparse memory access requests are arranged in separate queues, and when a threshold for coalescentially sparse memory accesses to different submodules is reached or a timing threshold is exceeded, the sparse memory access requests are interleaved from the separate queues with dense memory access requests (i.e., requests to access contiguous blocks of data from a single memory submodule).

[0090] When sending a response to a request involving a CLD or CST operation, the memory controller may divide the response and return individual sparse memory access responses to the requesting processor (possibly through the cache hierarchy). This can be achieved by storing response metadata for all pending sparse memory accesses (e.g., requester ID, sparse address offset, or access granularity) in a small structure within the memory controller. When a sparse memory access response is returned from a memory module, the memory controller may divide the data content of the memory module into multiple response packets based on the sparse granularity and append the stored metadata to these packets. When a sparse memory access response is returned to the requester through the cache hierarchy, the cache cannot allocate space for this data if the granularity of valid state tracking is larger than the sparse granularity.

[0091] The advantages of implementing a system capable of performing multicast memory merger / extraction operations as disclosed herein include improved efficiency in reading / loading metadata (e.g., the number of data elements involved in random calculations in each PNM / PIM device) from a collection of PNM / PIM devices associated with a memory channel to a host processor in a single load operation. Metadata (e.g., different loop iteration counts) can also be efficiently written / stored in a collection of PNM / PIM devices associated with a memory channel in a single store operation. Conditional codes (e.g., which PNM / PIM device has data that satisfies dynamically calculated data dependency conditions) can be efficiently read or loaded from each of the collections of PNM / PIM devices associated with a memory channel in a single store operation.

[0092] Furthermore, short data words distributed across a collection of memory banks or submodules coupled to memory channels can be efficiently loaded or stored without the need to unnecessarily transfer full cache lines of data from each memory submodule. This improves the performance of many application domains, such as scientific computing (e.g., high-performance computing, HPC) and graph analysis, as well as the performance of implementing widely used data structures such as hash tables and set membership data structures. The systems and methods disclosed herein also provide the ability to improve the performance of near-memory processing techniques or in-memory processing techniques, which often involve transferring short data words to or from multiple near-memory processing units or in-memory processing units associated with memory banks or submodules. In addition, for load operations that require loading short data words from different memory submodules (e.g., for high-throughput fine-grained atomic access) and combining supplied operands with memory submodules, fine-grained operands can be efficiently provided simultaneously to multiple memory submodules. Furthermore, multicast memory coalescing operations as disclosed herein are also effective in improving the efficiency of memory modules (e.g., DRAM efficiency) for sparse memory access patterns such as those found in graph analysis, sparse matrix algebras, and sparse machine learning models.

[0093] Although features and elements are described above in specific combinations, each feature or element can be used alone without other features and elements, or in various combinations with or without other features and elements. In some embodiments, the devices described herein may be implemented in computer programs, software, or firmware embedded in a non-temporary computer-readable storage medium for implementation by a general-purpose computer or processor. Examples of computer-readable storage media include read-only memory (ROM), random-access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs).

[0094] In the detailed descriptions above of various embodiments, references have been made to the accompanying drawings, which form part of the description and illustrate specific preferred embodiments in which the invention can be carried out. These embodiments are described in sufficient detail to enable those skilled in the art to carry out the invention, and it should be understood that other embodiments may be utilized and logical, mechanical, and electrical modifications may be made without departing from the scope of the invention. In order to avoid unnecessary details in order to enable those skilled in the art to carry out the invention, the description may omit certain information known to those skilled in the art. Furthermore, many other various embodiments incorporating the teachings of this disclosure can be readily constructed by those skilled in the art. Accordingly, the invention is not intended to be limited to any particular form described herein, but rather to encompass such alternative forms, modifications, and equivalents that can reasonably be included within the scope of the invention. Accordingly, the detailed descriptions above should not be construed as restrictive, and the scope of the invention is defined solely by the appended claims. The detailed descriptions above of the embodiments and examples described herein are presented for illustrative and explanatory purposes only, and not as limitations. For example, the operations described may be performed in any preferred order or method. Accordingly, the present invention is intended to encompass any modifications, variations, or equivalents that fall within the scope of the fundamental principles disclosed above and claimed herein.

[0095] As described above, systems and methods disclosed herein help reduce data transfer overhead by combining or aggregating short data words from multiple heterogeneous memory submodules and simultaneously transferring or communicating multicast combined block data over a memory channel in a single-block data transfer. The short data words are either returned to or loaded from each of the partitioned memory submodule sets to a host processor at a specific location within the single-block data transfer, or written to or stored from the host processor at a specific location within the single-block data transfer to each of the partitioned memory submodule sets. Furthermore, in some PIM architectures in which execution units are associated with memory banks, it may be necessary to read and write small amounts of data between PIM units and the main (or host) processor, for example, to report data-dependent conditions or status to the host processor, or to write the number of loop iterations that may vary between PIM units. Systems and methods disclosed herein facilitate the reduction of narrow data access overhead in such examples.

[0096] The above detailed description and the embodiments described herein are provided for illustrative and explanatory purposes only, and not for limitation.

Claims

1. A method for controlling digital data transfer via a memory channel between a memory module divided into a plurality of memory submodules and a processor, At least one of the memory module or the processor switches between a first mode that facilitates the transfer of multicast combined block data and a second mode that facilitates the transfer of continuous block data between the processor and any of the memory submodules, depending on conditions indicating the transfer of short data words. In the first mode, at least one of the memory module or the processor combines a plurality of short data words into multicast combined block data, which includes a single data block for transfer over the memory channel, wherein each of the plurality of short data words relates to one of at least two divided memory submodules within the memory module, and the size of the short data word is smaller than the data width of the memory channel. This includes communicating the multicast aggregated block data via the memory channel, method.

2. The processor sends a merge load command to the memory module, causing the memory module to acquire the short data word from the memory submodule, and to perform a multicast memory merge operation to merge the short data word into the multicast merge block data. The processor further includes, upon receiving the multicast amalgamated block data, extracting each of the short data words from the multicast amalgamated block data. The method according to claim 1.

3. The processor further includes transmitting one or more location identifiers to the memory module that identify a plurality of locations within the memory submodule associated with the short data word, causing the memory module to retrieve the short data word from the identified locations within the memory submodule. The method according to claim 2.

4. The processor further includes configuring at least one register associated with each of the memory submodules to store the short data word accessible to the processor for aggregating into the multicast aggregated block data, The method according to claim 1.

5. The processor generates the multicast aggregated block data to be transferred to the memory module, The processor further includes sending a merged storage command to the memory module, causing the memory module to perform a multicast memory extraction operation to extract the short data word from the multicast merged block data, distribute the short data word to the memory submodule, and store the short data word in the memory submodule. The method according to claim 1.

6. The processor further includes transmitting one or more location identifiers to the memory module that identify a plurality of locations associated with the short data word within the memory submodule, causing the memory module to store the short data word in the identified locations within the memory submodule. The method of claim 5.

7. The processor generates the multicast aggregated block data to be transferred to the memory module, The processor further includes configuring the at least one register associated with the memory submodule in order to cause the memory module to extract the short data word from the multicast coalesced block data and store the short data word in at least one register, The method according to claim 1.

8. The processor generates the multicast aggregated block data to be transferred to the memory module, The processor further comprises configuring the memory module to cause the memory module to determine one or more location identifiers that identify multiple locations associated with the short data word in the memory submodule, based on the multicast combined block data supplied by the processor or information stored in the memory module. The method according to claim 1.

9. It is a processor, A memory interface configured to communicate with a memory module via a memory channel, Equipped with multicast fusing logic, The multicast aggregation logic is as follows: Performing data transfer between the processor and the memory module via the memory channel by combining multiple short data words into multicast combined block data containing a single data block before transmission, wherein each of the multiple short data words relates to at least one of two divided memory submodules, and the size of the short data word is smaller than the data width of the memory channel. Communicating the multicast aggregated block data via the memory channel, It is configured to do the following: The processor switches between a first mode that facilitates the transfer of multicast combined block data and a second mode that facilitates the transfer of continuous block data between the processor and any memory submodule, depending on conditions indicating the transfer of short data words. Processor.

10. The multicast aggregation logic is as follows: Sending a merge load command to the memory module, causing the memory module to acquire the short data word from the memory submodule, and to perform a multicast memory merge operation to merge the short data word into the multicast merge block data, Upon receiving the multicast aggregate block data, each of the short data words is extracted from the multicast aggregate block data. It is configured to do the following: The processor according to claim 9.

11. The multicast aggregation logic is as follows: The configuration is configured to transmit one or more location identifiers, which identify multiple locations associated with the short data word within the memory submodule, to the memory module, causing the memory module to retrieve the short data word from the identified locations within the memory submodule. The processor according to claim 10.

12. The multicast aggregation logic is as follows: The memory module is configured such that it reads the short data word from the memory submodule, performs a multicast memory merging operation, and merges the short data word from at least one register into the multicast merging block data, thereby configuring the at least one register associated with the memory submodule. The processor according to claim 9.

13. The multicast aggregation logic is as follows: To generate the multicast combined block data to be transferred to the memory module, A merge memory command is sent to the memory module, causing the memory module to perform a multicast memory extraction operation to extract the short data word from the multicast merged block data, distribute the short data word to the memory submodule, and store the short data word in the memory submodule. It is configured to do the following: The processor according to claim 9.

14. The multicast aggregation logic is as follows: The memory submodule is configured to transmit one or more location identifiers that identify multiple locations associated with the short data word within the memory submodule to the memory module, causing the memory module to store the short data word in the identified locations within the memory submodule. The processor according to claim 13.

15. The multicast aggregation logic is as follows: To generate the multicast combined block data to be transferred to the memory module, To cause the memory module to extract the short data word from the multicast combined block data and store the short data word in at least one register, the at least one register associated with the memory submodule is configured, It is configured to do the following: The processor according to claim 9.

16. It is a system, A memory module containing multiple divided memory submodules, A processor comprising any of claims 9 to 15, system.

Images