Method and memory module for performing in-memory computation

Abstract

Methods and memory modules for performing in-memory computation are provided. The memory module includes a memory die including a plurality of dynamic random access memory (DRAM) banks, each DRAM bank including an array of DRAM cells arranged in pages, a row buffer storing a value of one of the pages, an input / output (IO) module, and an in-memory computation (IMC) module including an arithmetic logic unit (ALU) receiving operands from the row buffer or the IO module and computing an output based on the operands and one of a plurality of ALU operations, and a result register storing the output of the ALU, and a controller receiving operands and instructions from a host processor, determining a data layout based on the instructions, supplying the operands to the DRAM banks according to the data layout, and controlling the IMC module to perform one of the ALU operations on the operands according to the instructions.

Description

TECHNICAL FIELD

[0001] Aspects of embodiments of the present disclosure relate to systems and methods for data placement for in-memory compute. BACKGROUND

[0002] High bandwidth memory (HBM) is commonly used as a high performance memory for graphics processing units (GPUs). HBM has the advantage of having a very wide bus compared to typical DRAM. Current HBM architectures include multiple stacked DRAM dies (e.g., dice) that can be connected using through silicon vias (TSVs, also known as through-silicon vias) and a logic die that serves as a buffer in the HBM and an HBM memory controller in the GPU. Further performance improvements can be realized by adding processing-in-memory (e.g., in-memory processing) capabilities to the memory system.

[0003] The above information is presented as background information only to assist with an understanding of the embodiments of the present disclosure. No determination has been made, and no assertion can be made, as to whether features described would be a priori SUMMARY

[0004] According to one embodiment of the present disclosure, a memory module includes: a memory die including a plurality of dynamic random access memory (DRAM) banks, each of the plurality of DRAM banks including: an array of DRAM cells arranged in a plurality of pages, each page including a plurality of DRAM cells, each cell storing a bit value; a row buffer configured to store values of open pages of the plurality of pages; an input / output (IO) module; and an in-memory compute (IMC) module, the IMC module including: an arithmetic logic unit (ALU) configured to receive operands from the row buffer or the IO module and compute an output based on the operands and a selected ALU operation of a plurality of ALU operations; and a result register configured to store the output computed by the ALU; and a memory controller configured to: receive a first operand, a second operand, and an instruction from a host processor; determine a data placement from a plurality of data placements based on the instruction; supply the first operand and the second operand to the DRAM bank according to the data placement; and control the IMC module of the DRAM bank to perform an ALU operation of the plurality of ALU operations on the first operand and the second operand according to the instruction.

[0005] The plurality of data placements can include: an operand (1OP) data placement in which the first operand is written to the DRAM cells and the second operand is supplied directly to the IMC module of the DRAM bank from the host processor.

[0006] The IMC compute module can also include an operand register, and the memory controller can be further configured to: store a first chunk of the first operand in the operand register; and perform the ALU operation on the first operand stored in the operand register and each of a plurality of second chunks of the second operand.

[0007] The first operand can be divided into a plurality of first chunks, the second operand divided into a plurality of second chunks, each chunk comprising a plurality of values, and wherein the plurality of data layouts can comprise a same page (SR) data layout, wherein the memory controller stores one or more of the plurality of first chunks and one or more of the plurality of second chunks in a same page of a DRAM unit.

[0008] The IIMC module can also include an operand register, and the memory controller can be further configured to: store a first chunk of the one or more of the plurality of first chunks in the operand register, and perform the ALU operation on the first operand stored in the operand register and each of the one or more of the second chunks stored in the same page of the array of DRAM units as the first chunk.

[0009] The IMC module of the at least one DRAM bank can also include an accumulator comprising an accumulator register configured to store an accumulation value, the accumulator configured to: receive an output computed by the ALU; and update the accumulator register with a sum of the accumulation value and the output, and the instruction can comprise computing an inner product of the first operand and the second operand, wherein the first chunk of the plurality of first chunks stores row data and the second chunk of the plurality of second chunks comprises column data.

[0010] The first chunk can have a first number of values, the second chunk having a second number of values, the IMC module of the at least one DRAM bank can include an output buffer having a size to store a product of the first number of values greater than or equal to the second number of values, and the instruction can comprise computing an outer product of the first operand and the second operand, wherein the first chunk of the plurality of first chunks stores row data and the second chunk of the plurality of second chunks comprises column data.

[0011] The first tile can have a first number of values, a second tile has a second number of values, the IMC module of the at least one DRAM bank can include an output buffer having a size to store a greater of a first number of values and a second number of values, and the instruction can include computing a tensor product of the first operand and the second operand, wherein the first tile of the plurality of first tiles stores row data and a second tile of the plurality of second tiles includes column data.

[0012] The first operand can be divided into a plurality of first tiles, the second operand is divided into a plurality of second tiles, each tile includes a plurality of values, and the data layout can include a different page (DR) data layout in which the memory controller stores a subset of the plurality of first tiles in a first page of an array of DRAM cells and stores a subset of the plurality of second tiles in a second page of the array of DRAM cells.

[0013] The IMC module can further include an operand register, and the memory controller can be further configured to store the first tile of the first operand from the first page in the operand register and perform the ALU operation on the first operand stored in the operand register and each of a plurality of second tiles of the second operand from the second page.

[0014] The IMC module of each DRAM bank can further include a buffer configured to buffer an output computed by the ALU.

[0015] The hardware buffer can be at least four times the size of the result register.

[0016] The IMC module of each DRAM bank can further include an accumulator including an accumulator register configured to store an accumulation value, the accumulator configured to receive an output computed by the ALU and update the accumulator register with a sum of the accumulation value and the output.

[0017] The memory module can be a high bandwidth memory (HBM) module including a stack of a plurality of memory dies connected by through silicon vias, the plurality of memory dies including the memory die.

[0018] According to embodiments of the present disclosure, a method for performing in-memory computation includes receiving, by a memory controller of a memory module, a first operand, a second operand, and an instruction; determining, by the memory controller, a data layout from a plurality of data layouts based on the instruction; supplying the first operand and the second operand to one or more dynamic random access memory (DRAM) banks of the memory module according to the data layout, each of the one or more DRAM banks comprising: an array of DRAM cells arranged in a plurality of pages, each page comprising a plurality of DRAM cells, each DRAM cell storing a bit value; a row buffer configured to store values of an open page of the plurality of pages; an input / output (IO) module; and an in-memory computation (IMC) module, the IMC module comprising: an arithmetic logic unit (ALU) configured to: receive operands from the row buffer or the IO module and compute an output based on the operands and a selected ALU operation of a plurality of ALU operations; and a result register configured to store the output computed by the ALU; and performing, by the IMC module of the DRAM bank, an ALU operation of the plurality of ALU operations on the first operand and the second operand according to the instruction.

[0019] The plurality of data layouts can include an operand (1OP) data layout in which the first operand is written to the DRAM cells and the second operand is directly supplied to the IMC module of the DRAM bank from a host processor.

[0020] The IMC module can further include an operand register, and the memory controller can be further configured to: store a first chunk of the first operand in the operand register; and perform the ALU operation on the first operand stored in the operand register and each of a plurality of second chunks of the second operand.

[0021] The first operand can be divided into a plurality of first chunks and the second operand is divided into a plurality of second chunks, each chunk comprising a plurality of values, and the plurality of data layouts can include a same page (SR) data layout in which the memory controller stores one or more of the plurality of first chunks and one or more of the plurality of second chunks in a same page of the DRAM cells.

[0022] The IMC module can further include an operand register, and the memory controller can be further configured to: store a first chunk of the one or more of the plurality of first chunks in the operand register, and perform the ALU operation on the first operand stored in the operand register and each of the one or more of the second chunks of the same page of the array of the DRAM cells as the first chunk.

[0023] The IMC module of the at least one DRAM bank can further include an accumulator including an accumulator register configured to store an accumulation value, the accumulator configured to: receive an output computed by the ALU; and update the accumulator register with a sum of the accumulation value and the output, and wherein the instruction includes computing an inner product of the first operand and the second operand, wherein the first zone of the plurality of first zones stores row data and the second zone of the plurality of second zones includes column data.

[0024] The first zone can have a first number of values, the second zone can have a second number of values, the IMC module of the at least one DRAM bank can include an output buffer having a size to store a product of the first number of values and the second number of values that is greater than or equal to a value, and the instruction can include computing an outer product of the first operand and the second operand, wherein the first zone of the plurality of first zones stores row data and the second zone of the plurality of second zones includes column data.

[0025] The first zone can have a first number of values, the second zone can have a second number of values, the IMC module of the at least one DRAM bank can include an output buffer having a size to store a product of the first number of values and the second number of values that is greater than or equal to a value, and the instruction can include computing an outer product of the first operand and the second operand, wherein the first zone of the plurality of first zones stores row data and the second zone of the plurality of second zones includes column data.

[0026] The first operand can be divided into a plurality of first zones, the second operand is divided into a plurality of second zones, each zone includes a plurality of values, and the plurality of data layouts can include different page (DR) data layouts, wherein the memory controller stores a subset of the plurality of first zones in a first page of an array of DRAM cells and stores a subset of the plurality of second zones in a second page of the array of DRAM cells.

[0027] The IMC module can further include an operand register, and the memory controller can be further configured to: store a first zone of the first operand from the first page in the operand register, and perform the ALU operation on the first operand stored in the operand register and each of a plurality of second zones of the second operand from the second page. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a block diagram of an architecture of a high bandwidth memory (HBM) system in accordance with embodiments of the present disclosure.

[0029] Figure 2Ais a schematic block diagram of a memory bank of a memory with embedded arithmetic logic unit (ALU) according to one embodiment of the present disclosure.

[0030] Figure 2B is a circuit diagram illustrating an example of a DRAM cell according to one embodiment of the present disclosure.

[0031] Figure 2C is a schematic diagram of a DRAM bank according to one embodiment of the present disclosure.

[0032] Figure 3 is a schematic diagram of an array of DRAM banks according to one embodiment of the present disclosure.

[0033] Figure 4A Placement of data for a general matrix-matrix multiplication (GEMM) is schematically illustrated according to one embodiment of the present disclosure in a case where one operand (matrix A) is stored in DRAM and a second operand (matrix B) is broadcasted externally from a memory module with integrated in-memory computation (IMC).

[0034] Figure 4B Placement of data for a general matrix-matrix multiplication (GEMM) is schematically illustrated according to one embodiment of the present disclosure in a case where two operands (matrix A and matrix B) are stored in the same page of DRAM of a memory module with integrated IMC.

[0035] Figure 4C Placement of data for a general matrix-matrix multiplication (GEMM) is schematically illustrated according to one embodiment of the present disclosure in a case where two operands (matrix A and matrix B) are stored in different pages of DRAM of a memory module with integrated IMC.

[0036] Figure 5A is a schematic diagram of multiplying a first row of matrix A with a first column of matrix B in a one-operand (1OP) data layout according to one embodiment of the present disclosure, where one operand is fed externally and one operand is stored in a DRAM bank with integrated IMC.

[0037] Figure 5B is a schematic diagram of multiplying a first value of a first row of matrix A with a first value of each column of matrix B in a 1OP data layout with data reuse according to one embodiment of the present disclosure, where one operand is fed externally and one operand is stored in a DRAM bank with integrated IMC.

[0038] Figure 6Ais a schematic illustration of multiplying a first row of matrix A with a first column of matrix B in the same row (SR) data layout according to one embodiment of the disclosure, where both operands are stored in the same page of a DRAM bank with integrated IMC.

[0039] Figure 6B is a schematic illustration of multiplying a first value of a first row of matrix A with a first value of each column of matrix B with data reuse in the SR data layout according to one embodiment of the disclosure, where both operands are stored in the same page of a DRAM bank with integrated IMC.

[0040] Figure 7A is a schematic illustration of multiplying a first row of matrix A with a first column of matrix B in the different row (DR) data layout according to one embodiment of the disclosure, where operands are stored in different pages of a DRAM bank with in-memory computation.

[0041] Figure 7B is a schematic illustration of multiplying a first value of a first row of matrix A with a first value of each column of matrix B and multiplying a second value of a first row of matrix A with a second value of each column of matrix B with data reuse in the DR data layout according to one embodiment of the disclosure, where operands are stored in different pages of a DRAM bank with in-memory computation.

[0042] Figure 8 is a schematic block diagram of an in-memory computation (IMC) module of a DRAM bank according to one embodiment of the disclosure, where the IMC module is further connected to a result buffer, an accumulator, a buffer.

[0043] Figure 9 is a schematic illustration of an example of a general matrix-matrix multiplication (GEMM) for explaining the computation using the same row (SR) layout according to some embodiments of the disclosure.

[0044] Figure 10 is a flowchart of a method for controlling placement of data in a DRAM bank with an in-memory computation module according to embodiments of the disclosure. DETAILED DESCRIPTION

[0045] The features and methods of the disclosed concepts can be better understood with reference to the following drawings and detailed description of embodiments. Embodiments will be described in greater detail below with reference to the drawings, in which like reference numerals refer to like elements throughout. The disclosure may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided as examples so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. Therefore, the description of the embodiments should not be interpreted as a limitation on the further aspects and features of the disclosure. Thus, the scope of the disclosure should be determined by the appended claims and their legal equivalents, rather than by the description of the embodiments. Throughout the drawings and the detailed description, like reference numerals refer to like elements, and the description of the same elements will not be repeated across the drawings and the text. In the drawings, the relative sizes of elements, layers, and regions can be exaggerated for clarity.

[0046] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments. It is apparent, however, that various embodiments can be practiced without using these specific details or with one or more equivalent arrangements. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the various embodiments.

[0047] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" "comprising," "includes" and / or "including," when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0048] The electronic or electrical devices and / or any other related devices or components according to embodiments of the present disclosure described herein can be implemented utilizing any suitable hardware, firmware (e.g., application specific integrated circuits), software, or combinations of software, firmware, and hardware. For example, in some cases, various components of these devices can be formed on one integrated circuit (IC) chip or on different IC chips. Further, various components of these devices can be implemented on a flexible printed circuit film, a tape carrier package (TCP), a printed circuit board (PCB), or formed on a substrate. Further, various components of these devices can be a process or thread run on one or more processors in one or more computing devices and interacting with other system components for performing various functions described herein. The computer program instructions can be stored in memory and accessed by the processors using standard memory device, such as random access memory (RAM). The computer program instructions can also be stored in other non-transitory computer-readable media, such as, for example, a CD-ROM, a flash drive, etc. Further, those skilled in the art will appreciate that functions of various computing devices can be combined or integrated into a single computing device, or functions of a particular computing device can be distributed across one or more other computing devices, without departing from the spirit and scope of the example embodiments of the present disclosure.

[0049] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and / or the present specification and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0050] Embodiments of some aspects of the present disclosure generally relate to management of placement of data in memory in the context of in-memory-compute. One example of a context for in-memory-compute is in a high bandwidth memory (HBM) system that includes a dynamic random access memory (DRAM) die containing memory and an arithmetic logic unit (ALU) and a memory controller on an HBM logic die, where the ALU on the HBM logic die can perform in-memory-compute. The memory controller of the HBM controls storing data into the memory of the DRAM die and reading data from the DRAM die.

[0051] For clarity, as used herein, the term "in-memory computing" means performing computations within a memory module (such as a high-bandwidth memory module) using data stored in the DRAM die without traversing an external data bus. In a comparative computer system, a processor can be coupled to main memory (e.g., DRAM) via an external DRAM data bus, and accessing data from main memory is significantly slower (e.g., orders of magnitude slower) than accessing data in a register file within the processor and / or data in a hardware cache (e.g., LI cache, L2 cache) closer to the processor. By placing an additional processor (e.g., an "in-memory processor") at or near the memory, the latency due to traversing an external bus can be avoided, leading to higher performance computations.

[0052] Aspects of embodiments of the present disclosure relate to placing compute circuits (such as ALUs) on the same die as DRAM banks (e.g., directly connected to the sense amplifiers or row buffers of individual DRAM banks).

[0053] Due to the design and performance characteristics of DRAM modules, the particular arrangement of data in memory can affect the performance of in-memory computations. Accordingly, aspects of embodiments of the present disclosure relate to systems and methods for placing data within a DRAM module of a memory module (e.g., an HBM memory module), where the placement of data can be controlled based on the characteristics of computations to be performed by an in-memory computation (IMC) module.

[0054] Figure 1 is a block diagram of an architecture of a memory (e.g., HBM) system according to embodiments of the present disclosure.

[0055] Referring to Figure 1Embodiments of the present disclosure provide systems for a function-in-memory (FIM) memory system 100. The memory system 100 (or HBM system) supports additional compute resources integrated in a memory module (HBM module) 110. For example, in various embodiments, the memory system 100 allows some data computation and movement to be performed in-memory, and also provides a high-capacity memory scratchpad. The memory system 100 includes at least one memory module 110 connected to a host processor 170 (such as a graphics processor (GPU) or central processing unit (CPU)). In various embodiments, the memory module 110 includes one or more dynamic random access memory (DRAM) dies 120 connected to a memory controller 140 (e.g., on a logic die) via an internal memory bus 130. In various embodiments, the host processor 170 includes a host memory controller (or host controller) 180 for interfacing with the memory module 110. However, embodiments of the present disclosure are not limited thereto. For example, the host memory controller 180 can be separate from the host processor 170 (e.g., on a separate die or on the same die as the host processor 170).

[0056] According to various embodiments, the memory controller 140 is configured to coordinate execution of instructions from the host processor 170. The instructions can include both regular instructions and FIM instructions. For example, regular instructions (e.g., legacy load and store functions, as opposed to function-in-memory instructions) are sent by the host memory controller 180 and received by the memory controller 140, and are executed in a regular manner. For example, regular instructions can include instructions to store data received over the external bus 190 in the DRAM dies 120, and instructions to obtain data from the DRAM dies 120 and transfer the data to the host processor 170 via the external bus 190. In some embodiments, regular instructions and FIM instructions can include operations to store data in a particular location (e.g., a particular page of a particular bank) in the DRAM dies. These data can include two different operands, where each operand can include multiple values (e.g., floating point values or integer values), and the values of these operands can be distributed across different locations of the DRAM dies according to various data placement policies, as discussed in more detail below.

[0057] Aspects of embodiments of the present disclosure relate to the use of in-memory computing (IMC). Some comparative HBM systems include an arithmetic logic unit (ALU) external to the DRAM dies 120 (e.g., located at the memory controller 140) and shared by the memory banks of the DRAM dies 120, such that the ALU can perform operations (e.g., arithmetic operations) on data stored in one or more of the DRAM dies 120 without traversing the external bus 190. For example, the memory controller 140 can perform data movement operations (e.g., load / store pair instructions) to move or copy data between different portions of the DRAM dies 120. For example, the memory controller 120 can execute FIM instructions (such as by coordinating execution of compute FIM instructions (e.g., atomic instructions and ALU instructions) that utilize the ALU) that are initially a plurality of regular instructions. As another example, in some cases, FIM instructions received from the host processor 170 cause the memory module with integrated IMC to perform a particular computation (such as a compute inference) using a trained machine learning model (e.g., a neural network) based on supplied input, perform backpropagation during training of the neural network, or multiply two matrices. In these cases, the memory controller 140 can coordinate execution of these instructions by storing received data (e.g., operands of the instructions) in particular pages of the DRAM banks and by providing particular ALU operations to the DRAM banks storing data related to the FIM instructions. In performing particular FIM instructions, the memory controller 140 can place received data (operands) in particular pages of the DRAM banks in a manner that improves performance of the computation. Results are then stored in the DRAM dies 120 or can be returned to the host processor 170 via the external bus 190.

[0058] Some aspects of embodiments of the present disclosure relate to accelerating memory bounded operations by integrating in-memory computing (IMC) modules into the memory banks of the DRAM dies 120, thereby avoiding traversal of the internal memory bus 130 between the DRAM dies 120 and the memory controller 140. For example, the IMC modules are on the same physical semiconductor die as the DRAM banks holding the data. Each DRAM bank can have an associated IMC module, such that computations can be performed on data stored in the DRAM bank without traversing the internal memory bus 130 to the memory controller (e.g., without transferring the data outside of the DRAM bank), while also enabling parallelization of computations across the IMC modules of the DRAM banks.

[0059] While aspects of embodiments of the present disclosure will be described in the context of high bandwidth memory, embodiments are not so limited, and can also be applicable to DRAM dies that integrate IMC modules into other types of DRAM systems.

[0060] Figure 2A is a schematic block diagram of a bank of a memory with embedded arithmetic logic units (ALUs) according to one embodiment of the present disclosure. As shown in Figure 2A The DRAM bank 200 includes an array of DRAM cells 210 arranged in rows and columns (or pages and columns). In Figure 2A In the example shown in In the example shown in

[0061] Figure 2B is a circuit diagram showing an example of a DRAM cell according to one embodiment of the present disclosure. Each DRAM cell 210 can generally be modeled as including a capacitor 212 for storing a data voltage (e.g., a bit value, where each capacitor can store a voltage representing a 0 bit or a voltage representing a 1 bit) and a switch 214 for sending the data voltage to the capacitor 212. Figure 2B The particular DRAM cell 210 shown in Figure 2B The switch 214 of the DRAM cell 210 is connected between the ith bit line Bi and one terminal of the capacitor 212, and the other terminal of the capacitor 212 is connected to ground. Figure 2B The gate electrode of the switch 214 of the DRAM cell 210 shown in

[0062] Referring back to Figure 2ADRAM bank 200 includes a row decoder 220 connected to row enable lines R1 through Rn, and row decoder 220 is configured to supply a row enable signal to a particular one of the row enable lines corresponding to a row address supplied from, for example, memory controller 140. When data is written to or read from a particular row (or page) of DRAM cells, row decoder 220 supplies a row enable signal to the row enable line corresponding to the particular row (or page). When data is written, voltages corresponding to the data to be written are supplied to bit lines B1 through Bm while the particular row (or page) is enabled. Similarly, when data is read from a particular row (or page) of the array of DRAM cells 210, voltages corresponding to the voltages stored in capacitors 212 are transferred along bit lines B1 through Bm and read by an input / output sense amplifier layer 230 (or IOSA) including sense amplifiers 232. Each of sense amplifiers 232 is connected to a corresponding one of the bit lines (e.g., sense amplifiers 232 include m sense amplifiers). For example, in some embodiments, the array of DRAM cells 210 includes 8192 columns and 8192 corresponding bit lines (e.g., bit lines B1 through B8192) connected to 8192 corresponding sense amplifiers 232 (e.g., each page can store 8192 bits or 8 Kibits (Kib) of data). Because sense amplifiers 232 store data read from the current row (or page) until they are cleared by a “precharge” command, sense amplifiers 232 can also be referred to as “row buffers.”

[0063] Column decoder 240 can be used to select a subset of columns of data using multiplexer 234, and the read data can be supplied to an in-memory compute (IMC) module 250 through a global IO layer (e.g., a 256-bit global IO layer (e.g., global IO (256)) 236 to perform computations on the data. For example, in some embodiments, column decoder 240 and multiplexer 234 can allow 256 bits (256b) of data to be selected from 8192 columns of DRAM cells 210.

[0064] When loading data from a page of DRAM bank 200 that is different from the page currently stored in sense amplifiers 232, a “precharge” (PRE) command is used to close the current page and prepare DRAM bank 200 for the next access. An “activate” (ACT) command is then used to open a particular row or page of the bank, and then the data is stored in sense amplifiers 232. The data can then be read (READ) from sense amplifiers 232 to be transferred to IMC module 250.

[0065] On the other hand, when loading data into the IMC module 250 from a page that has already been opened, then the READ command can be sufficient to load the data (such as by using the column decoder 240 to select the appropriate subset of data already stored in the sense amplifiers 232) (the PRE command and the ACT command can be skipped).

[0066] According to various embodiments, the IMC module 250 (or ALU & Reg) includes an arithmetic logic unit (ALU) 252 and one or more registers. In Figure 2A In the embodiment shown in FIG. 2, the IMC module 250 includes an operand register Rop 254 (or input buffer) and a result register Rz 256. Multiplexers 257 and 258 can be used to control the flow of data to the two inputs of the ALU 252 (e.g., as the first operand and the second operand of the ALU 252). For example, in the embodiment shown in FIG. 2, the operand register Rop 254 is connected to the first operand input of the ALU 252, and the first multiplexer 257 is connected to write data from the sense amplifiers 232 through the global IO layer 236 or from an external source (e.g., the host processor) through the input output (IO) module 260 (or write input / output and read input / output, or WIO and RIO) to the operand register Rop 254. Figure 2A In the embodiment shown in FIG. 2, the second multiplexer 258 is configured to provide data directly to the second operand input of the ALU 252 again from the sense amplifiers 232 through the global IO layer 236 or from an external source through the IO module 260. The ALU 252 can output its computation to the result register Rz 256, and the data can be written back from the result register Rz 256 through the global IO layer 236 into the DRAM unit 210, or sent out to the host processor 170 through the IO module 260 (or WIO and RIO). Figure 2A In the embodiment shown in FIG. 2, the second multiplexer 258 is configured to provide data directly to the second operand input of the ALU 252 again from the sense amplifiers 232 through the global IO layer 236 or from an external source through the IO module 260. The ALU 252 can output its computation to the result register Rz 256, and the data can be written back from the result register Rz 256 through the global IO layer 236 into the DRAM unit 210, or sent out to the host processor 170 through the IO module 260 (or WIO and RIO).

[0067] According to some embodiments, the ALU 252 is configured to perform various computing operations (e.g., simple computing commands). For example, the ALU 252 can be a 16-bit ALU, a 32-bit ALU, a 64-bit ALU, a 128-bit ALU, or a 256-bit ALU (e.g., an ALU(256)) configured to perform arithmetic operations, bitwise operations, shift operations, and the like. In various embodiments, the ALU 252 can include circuitry for performing integer operations, floating point operations, or both. For example, the ALU 252 can be configured to perform arithmetic operations such as addition (+), subtraction (-), multiplication (*), and division (÷), bitwise operations such as AND (&) operations, OR (|) operations, exclusive OR (^) operations, and NOT (~) operations, and tensor operations. Further, in some embodiments, the ALU 252 can implement single instruction multiple data (SIMD) or vector instructions for performing operations on data vectors in parallel. Examples of vector operations implemented by the ALU 252 according to embodiments of the present disclosure include dot product (·), outer product linear rectifier unit (ReLU), square (vsSqr), and square root (vsSqrt). The ALU 252 can be used for atomic and non-atomic operations. Table 1 below lists operations supported by the ALU 252 according to some embodiments of the present disclosure.

[0068] Table 1

[0069]

[0070]

[0071] Figure 2C is a schematic diagram of a DRAM bank 200 according to one embodiment of the present disclosure, including an array of DRAM cells 210 such as discussed above, a row decoder 220, an IOSA 230, an IMC module 250, and a column decoder 240.

[0072] Figure 3 is a schematic diagram of an array of DRAM banks according to one embodiment of the present disclosure. In the embodiment shown in Figure 3 , 16 DRAM banks 200 are arranged in a four-by-four array and labeled DRAM banks A through P, and are arranged into four bank groups labeled BG0 (including DRAM banks A, B, C, and D), BG1 (including DRAM banks E, F, G, and H), BG2 (including DRAM banks I, J, K, and L), and BG3 (including DRAM banks M, N, O, and P). As discussed above with respect to Figure 2C Figure 3 ​Each DRAM bank 200 shown in FIG. 1 can include an IMC module 250 for performing computations within the DRAM die 120 (e.g., without traversing an external bus). Further, as shown in FIG. 1, the DRAM die 120 also includes a multiplexer 300 (e.g., a 4: 1 multiplexer (MUX)) configured to multiplex data from an external source (e.g., in a through-silicon via or TSV fashion that connects the DRAM die 120 to a plurality of other stacked DRAM dies 120 and the memory controller 140) to the four columns of the block. For example, the multiplexer 300 can broadcast a 256-bit (256b) data vector to each DRAM bank 200, or can be used to supply the data vector to a particular column (BG0, BG1, BG2, or BG3) of the DRAM bank 200. Figure 3

[0073] Because data does not need to traverse the bottleneck of an external bus (e.g., the bus 190) to perform computations, a memory system 100 that includes in-memory computing (IMC) integrated into the DRAM die 120, such as the IMC module 250, accelerates the performance of memory bounded host operations. However, the IMC can still encounter computational overhead in the form of ALU pipelines and DRAM processes, and memory overhead in the form of data placement and DRAM timing.

[0074] Accordingly, aspects of embodiments of the present disclosure relate to systems and methods for placing data within a DRAM to avoid or reduce the impact of DRAM timing overhead when performing in-memory computations by the integrated IMC module 250. Some aspects of embodiments of the present disclosure relate to a software and hardware co-design for achieving improved performance.

[0075] ​In various embodiments, the memory controller 140 provides operations and manages the input and output of data to and from the DRAM dies 120. Accordingly, some aspects of embodiments of the present disclosure relate to the memory controller 140 configured to place data within the DRAM dies 120 according to instructions provided to the memory controller 140 of the memory system 100 by the host processor 170. For example, some aspects of embodiments of the present disclosure relate to providing an application programming interface (API) for interacting with an HBM having in-memory computation and a compiler (e.g., a data compiler) configured to generate commands that control the memory controller 140 to place data according to calls to the API when a source code of a program is compiled or interpreted using the API. For example, the API can provide a function call for performing a general matrix-matrix multiplication (GEMM), and the compiler can generate a sequence of commands to control the memory controller 140 to place data representing operand matrices into the DRAM dies 120 in a manner that improves performance relative to simple placement of data (e.g., placement of data in a row-major order) based on factors including the operation to be performed on the data (e.g., an inner product, an outer product, a matrix multiplication, etc.), the size of the data (e.g., whether the data fits within a page of memory), etc. When software is written to utilize a DRAM having an IMC through an API, a compiler or data compiler according to some embodiments of the present disclosure can convert at least some of a source code of the software into commands executed by the memory controller 140 to place data at specific locations of the DRAM dies 120 and control the IMC modules of the DRAM dies 120 to perform computations and store results.

[0076] For ease of discussion, according to one embodiment, a 16-lane ALU can achieve a peak performance of 8 GFLOPS (Giga floating point operations per second) of half-precision floating point (FP-16) computation. (Although performance of the IMC modules according to embodiments of the present disclosure is described herein in terms of floating point operations per second (FLOPS), embodiments of the present disclosure are not limited to performing floating point operations, and the relative performance of various data layouts can be similar when performing, for example, integer operations.) Accordingly, an implementation of in-memory computation (IMC) according to one embodiment of the present disclosure utilizing four dies (4H or 4-Hi) stacked using a second generation high bandwidth memory standard (HBM2) is capable of achieving 8 TFLOPS (Tera floating point operations per second) of FP-16 computation (256 banks per die multiplied by a stack of 4 dies = 1024 banks, each bank having a corresponding 16-lane ALU).

[0077] ​Peak computational performance, as measured in TFLOPS, can vary under different data placement scenarios. In one (first) scenario (denoted as 2OP), both operands from outside the memory module 110 are fully fed to the in-memory compute ALU, and the result is fully buffered and accumulated, resulting in the 8 TFLOPS peak computational performance discussed above.

[0078] In another (second) scenario (denoted as 1OP), one operand is fully fed to the IMC from outside the HBM, while the second operand is read from a random location in the DRAM die 120. This results in reduced peak computational performance of about 6.5 TFLOPS.

[0079] In a third scenario (denoted as DR), both operands are in different pages of DRAM, and the result is written back to DRAM. In this case, the measured computational performance is about 0.8 TFLOPS (e.g., an order of magnitude slower than the 2OP case).

[0080] In a fourth scenario (denoted as SR), both operands are in the same row or page of the DRAM block. This results in a significant performance improvement, exceeding the DR case to about 3.3 TFLOPS.

[0081] Thus, various data placement tradeoffs can be made based on considerations of the data. For example, when the first operand is in DRAM and the second operand is broadcast from outside the HBM in the 1OP case, the performance is high (about 6.5 TFLOPS, as noted above), but this imposes an overhead on the host processor 170 that must provide the second operand to the HBM.

[0082] Figure 4A Placement of data for a general matrix-matrix multiplication (GEMM) is schematically illustrated in the case where one operand (matrix A) is stored in DRAM and the second operand (matrix B) is broadcast from outside a memory module with integrated in-memory compute (IMC), according to one embodiment of the disclosure. For ease of illustration, the DRAM bank 200-0 is shown in more detail. As Figure 4A As shown in the 2OP case, data associated with the first operand matrix A is placed in a page 401 (e.g., a first row or page) of the DRAM bank 200-0, and data associated with the second operand matrix B is broadcast from outside the DRAM die 120 (e.g., via the multiplexer 300). The result of the computation C can be placed in a different page 402 of the DRAM bank 200-0.

[0083] When both operands are in DRAM, placing the operands in the same page or same row (SR) improves the computational performance (e.g., to about 3.3 TFLOPS) by partially reducing the number of PRE and ACT operations that must be performed as discussed in more detail below, but imposes more constraints on placing data in the correct part of the DRAM.

[0084] Figure 4B The placement of data for a general matrix-matrix multiplication (GEMM) is schematically illustrated according to one embodiment of the disclosure in the case where the two operands (matrix A and matrix B) are stored in different pages of a DRAM of a memory module with an integrated IMC. For ease of illustration, the DRAM bank 200-0 is shown in more detail. As Figure 4B illustrated in FIG. 21 A, the data associated with the first operand matrix A is placed in page 421 and the data associated with the second operand matrix B is placed in page 422. The result C is placed in page 423. Figure 4A

[0085] On the other hand, placing the operands in different pages is more flexible and reduces the constraints on the layout (e.g., can be applied to data having a size that does not completely fit in a page of a fixed size of the memory), but typically comes at the expense of reduced computational performance.

[0086] Figure 4C The placement of data for a general matrix-matrix multiplication (GEMM) is schematically illustrated according to one embodiment of the disclosure in the case where the two operands (matrix A and matrix B) are stored in different pages of a DRAM of a memory module with an integrated IMC. For ease of illustration, the DRAM bank 200-0 is shown in more detail. As Figure 4C illustrated in FIG. 21 A, the data associated with the first operand matrix A is placed in page 421 and the data associated with the second operand matrix B is placed in page 422. The result C is placed in page 423.

[0087] The various performance impacts of the different data placement strategies (1OP, SR and DR) discussed above for Figure 4A , Figure 4B and Figure 4C will be discussed below for Figure 5A ​、 Figure 5B 、 Figure 6A 、 Figure 6B 、 Figure 7A and Figure 7B are described in more detail. For example, as shown in Figure 5A , matrix A is an M x K matrix and matrix B is a K x N matrix, so the product of matrix A and matrix B (matrix C) has dimensions M x N. For ease of discussion, the following examples describe the case where K = 5, but embodiments of the present disclosure are not limited thereto. According to standard matrix multiplication, the top-left value of the resulting matrix C, C00, is computed by the pairwise multiplication of the first row of matrix A (each position Aij represents a vector or "tile" of sixteen-bit half-precision floating-point values in horizontal order and Figure 5A 、 Figure 5B 、 Figure 6A 、 Figure 6B 、 Figure 7A and Figure 7B , A00, A01, A02, A03, and A04) with the first column of matrix B (each position Bij represents a vector or "tile" of sixteen-bit half-precision floating-point values in vertical order). In other words, C00 stores A00- B00 + A01- B10 + A02- B20 + A03- B30 + A04- B40. In Figure 5A 、 Figure 5B 、 Figure 6A 、 Figure 6B 、 Figure 7A and Figure 7B , shading is used to identify operands that are multiplied together. In more detail, two operands that are shaded using the same pattern are multiplied together as part of the computation shown. While aspects of embodiments of the present disclosure are described in the context of an IMC configured to perform floating-point operations on floating-point operands, embodiments of the present disclosure are not limited thereto and can also apply to an IMC configured to perform integer operations on integer operands, for example.

[0088] Figure 5A is a schematic diagram of multiplying the first row of matrix A with the first column of matrix B in an operand (1 OP) data layout according to one embodiment of the present disclosure, where one operand is fed from outside and one operand is stored in a DRAM bank with in-memory computation. As shown in Figure 5A , the tiles A00, A01, A02, A03, and A04 of the first row of matrix A are stored in the same row 401 of DRAM bank 200, the vectors or tiles B00, B10, B20, B30, and B40 of matrix B are supplied from outside, and the result (e.g., C00) is stored in another page 402 of DRAM bank 200.

[0089] The process of computing the product can begin by reading the value A00 from the DRAM bank 200 to compute A00-B00. This will involve opening the page 401, so a precharge (PRE) command to prepare the sense amplifier 232 will be needed, followed by an activate (ACT) command to load the page 401 into the sense amplifier 232, and a read (READ) command to load the value A00 from the sense amplifier 232 into the IMC module 250. As noted above, B00 is input from outside, so no DRAM operations are needed to obtain this value. The ALU 252 can then compute the product A00-B00, and store the temporary result in an output buffer (e.g., output register Rz).

[0090] Next, the ALU 252 computes the product A01-B10 by reading A01 from the DRAM. This also requires a PRE command, an ACT command, and a READ command. The vector or block B10 is provided as input from outside, so A01-B10 can then be computed and added to the temporary result in the buffer (e.g., output register Rz) that stores A00-B00. This process can then be repeated for the remaining values of matrix A and matrix B. As a result, each computation (e.g., multiplication of two operands such as blocks A00 and B00) requires one PRE, one ACT, and one READ. As discussed below with respect to FIG. 6, the number of DRAM operations can be reduced by reusing data. Figure 8 In more detail, in some embodiments, the IMC also includes accumulators configured to store values and update the stored values with the sum of previously stored values and newly received values.

[0091] Figure 5B is a schematic diagram of multiplying a first value of a first row of matrix A with a first value of each column of matrix B in a 1OP data layout with data reuse, according to one embodiment of the disclosure, where one operand is fed from outside and one operand is stored in a DRAM bank with in-memory computation. Figure 5B The process shown in FIG. 5 is similar to the process shown in FIG. 4, except that the first value of the first row of matrix A is fed from outside. Figure 5AThe difference in the processing shown in FIG. 6 is that the data loaded from the DRAM (e.g., stored in operand register Rop 254) is reused for different columns of matrix B. Specifically, when two matrices are multiplied, the jth element of each row of matrix A is multiplied with each element of the jth row of matrix B. Thus, by loading each element of matrix A once and multiplying it with all N columns of matrix B, the cost of loading data from DRAM bank 200 is amortized across the N columns. In more detail, as before, when matrix A is multiplied with matrix B, tile A00 can be read from DRAM bank 200 using a PRE command, an ACT command, and a READ command, and tile B00 can be received as input from outside. ALU 252 computes A00- B00 to compute part of the sum of C00. Rather than loading tile A01 from the DRAM (e.g., using another PRE, ACT, and READ sequence), however, tile A00 can be reused and multiplied with B01 (received from outside) to compute A00- B01, which is the product in the product used to compute C01. As a result, each computation requires 1 / Nth PRE, 1 / Nth ACT, and one READ (because the PRE command and ACT command to load another portion of matrix A can be amortized over the N columns of matrix B).

[0092] Figure 6A is a schematic diagram of multiplying a first row of matrix A with a first column of matrix B in the same row (SR) data layout, in which both operands are stored in the same page of a DRAM bank with in-memory computation, according to one embodiment of the disclosure. As Figure 6A As shown in FIG. 6, tiles A00, A01, and A02 of the first row of matrix A and tiles B00, B10, and B20 of the first column of matrix B are stored in the same page 611 of DRAM bank 200, and tiles A03 and A04 of matrix A and tiles B30 and B40 of matrix B are stored in page 612 of DRAM bank 200. The results (e.g., C00) are stored in page 613 of DRAM bank 200.

[0093] The processing of the result of the computation (e.g., the inner product) can begin by computing A00- B00 via reading block A00 from DRAM bank 200. This will involve opening page 411, so a precharge (PRE) command will be needed, followed by an activate (ACT) command and a read (READ) command to load block A00 from the sense amplifiers 232 into the IMC module 250. Block B00 can then be read from the DRAM. However, because block B00 is in the same page 411 as A00, its value is already stored in the sense amplifiers 232, so a READ command is sufficient (no further PRE and ACT need to be performed to read block B00 into the IMC module 250). Thus, once blocks A00 and B00 are read, the ALU 252 computes A00- B00, and the temporary result can be stored in a buffer. Similarly, because blocks A01 and B10 are both in page 411 and thus previously stored in the sense amplifiers 232 when page 411 was initially opened to read A00, reading blocks A01 and B10 to compute A01- B10 can likewise use READ commands without PRE and ACT being performed. Thus, each computation can take 1 / r PREs, 1 / r ACTs, and 2 read operations, where r is the number of matching pairs of values stored in the same page of the DRAM bank 200. For example, as described above, Figure 6A The case where blocks A00, A01, and A02 of the first row of matrix A and blocks B00, B10, and B20 of the first column of matrix B are stored in page 411 of the DRAM bank is depicted. Thus, because page 411 contains three pairs of blocks (e.g., r = 3), the computations A00- B00, A01- B10, and A02- B20 will each take (dole out) 1 / 3 of a PRE, 1 / 3 of an ACT, and 2 READ commands. When computing A03- B30 and A04- B40, because page 412 contains two pairs of values (e.g., r = 2), these computations will take (dole out) 1 / 2 of a PRE, 1 / 2 of an ACT, and 2 READ commands, respectively. The reduction in the number of PRE commands and ACT commands needed for each computation improves the performance of the overall computation.

[0094] Figure 6B is a schematic of multiplying the first value of the first row of matrix A with the first value of each column of matrix B in an SR data layout with data reuse, according to one embodiment of the disclosure, where both operands are stored in the same page of a DRAM bank with in-memory computation. In a similar manner as the comparison between the arrangements of Figure 5A and Figure 5B Figure 6B The computation process shown in Figure 6A ​The computational processing shown differs in that values ​​loaded from DRAM are reused. More specifically, based on the understanding that matrix multiplication involves multiplying the i-th element of a given row of the first operand with the i-th element of each column of the second operand, the elements of matrices A and B are arranged in DRAM memory 200 such that the data for each element of a given row of matrix A and the value of that row of matrix B to be multiplied are placed on the same page.

[0095] For example, such as Figure 6B As shown, when matrix A is multiplied by matrix B, in the processing of the first row of the resulting matrix C (e.g., as one of the terms C00, C01, C02, C03, C04, C05, ...), block A00 of matrix A will be multiplied by the first block of each column of matrix B (e.g., ...). Figure 6B Each element in the first row of matrix B shown (e.g., B00, B01, B02, B03, B04, B05, ...) is multiplied. Similarly, the value A01 of matrix A is multiplied by the second block of each column of matrix B (e.g., ...). Figure 6B The elements of the second row of matrix B shown are multiplied together (e.g., B10, B11, B12, B13, B14, B15, ...).

[0096] In this way, at least one value of matrix A is stored on the same page as the corresponding value of matrix B. Figure 6B In the specific example shown, page 411 stores block A00 from matrix A and blocks B00, B01, B02, B03, B04, B05, ... from matrix B, and page 412 stores block A01 from matrix A and blocks B10, B11, B12, B13, B14, B15, ... from matrix B. Due to this arrangement of data within the DRAM, assuming a DRAM page can store at least N+1 entries, each computation will require 1 / N PRE commands, 1 / N ACT commands, and (N+1) / N READ commands (because PRE and ACT commands can be amortized across the N columns of matrix B). This reduction in the number of PRE and ACT commands required per computation improves the overall computational performance.

[0097] Figure 7A This is a schematic diagram illustrating the multiplication of the first row of matrix A with the first column of matrix B in different row (DR) data layouts according to an embodiment of the present disclosure, wherein the operands are stored in different pages of a DRAM memory bank with in-memory computation. Figure 7A As shown, blocks A00, A01, A02, A03 and A04 of the first row of matrix A are stored in page 421, while blocks B00, B10, B20, B30 and B40 of the first column of matrix B are stored in a different page 422.

[0098] Because block B00 is in a different row 422 than block A00, the process of computing C00 can start with the computation of A00-B00 by reading A00 from row 421 of DRAM bank 200 using a PRE, ACT, and READ command sequence, followed by reading block B00 using a PRE, ACT, and READ command sequence. The ALU 252 can then compute A00-B00 and store the result in a temporary buffer. To continue the computation of C00, block A01 can be read from row 421, followed by block B10 from row 421, where each value involves performing a PRE, ACT, and READ sequence on DRAM bank 200. Thus, each computation takes 2 PRE commands, 2 ACT commands, and 2 READ commands.

[0099] Figure 7B is a diagram of multiplying a first value of a first row of matrix A with a first value of each column of matrix B and a second value of the first row of matrix A with a second value of each column of matrix B in a DR data layout with data reuse according to one embodiment of the disclosure, where the operands are stored in different pages of a DRAM bank with in-memory computation.

[0100] In a similar manner as described above for Figure 5B and 1 In a similar manner as described above for Figure 7A discussed above, the ith value of a given row of matrix A is multiplied with each value in the ith row of matrix B. Thus, if all values in a given row of matrix B are stored in the same page of DRAM bank 200, the number of PRE and ACT commands can be reduced, thereby improving performance.

[0101] For example, the process of multiplying matrix A with matrix B can start with multiplying block A00 of matrix A with each of the N blocks (B00, B01, B02,..., B0N) in the first row of matrix B to compute N partial sums (e.g., partials of C00, C01, C02,..., C0N). The process can start by loading block A00 from DRAM bank 200, which can involve a PRE command, an ACT command, and a READ command to open row 421 and load block A00 of matrix A into operand register Rop 254. After loading A00, blocks B00, B01, B02,..., B0N of the first row of matrix B can be loaded to be multiplied with block A00. As shown in Figure 7B When these values of matrix B are all in the same page (e.g., B00, B01, B02, B03, B04,..., in row 421 of DRAM bank 200), the number of PRE and ACT commands can be reduced, thereby improving performance.Figure 7B When accessing page 422 of the DRAM memory bank 200 shown, the PRE and ACT commands used to access that page are amortized across the N values ​​stored in that page. Therefore, multiplying a value from matrix A by each value in a row of matrix B requires 2 PRE commands, 2 ACT commands, and N+1 READ commands. When these commands are amortized across the N values ​​in that row, it results in 2 / N PRE commands, 2 / N ACT commands, and (N+1) / N READ commands per computation. In a similar manner to the example given above, the reduction in the number of DRAM commands improves the overall performance of computations (e.g., matrix multiplication).

[0102] The data placement options discussed above were experimentally tested using a cycle-level high-bandwidth memory-in-memory function (HBM-FIM) simulator customized for in-memory computation (IMC) with general matrix-matrix multiplication (GEMM) tracing. The experimental architecture included an HBM with four 4H HBM2 modules, where the DRAM had a matrix size of 16384 (16 Kib) rows and 8192 (8 Kib) columns (e.g., 8192 bits per page), and the row buffers had a size of 8192 bits (8 Kib). The IMC modules included 16-channel FP-16 vector units with delay cycles and 768-bit buffers, enabling pipelined operation. In various embodiments, the buffers can be smaller (e.g., 512 bits, including 256-bit input buffers and 256-bit output buffers) or larger (e.g., a 1280-bit buffer with 256-bit input buffers and 1024-bit output buffers). In some embodiments, the input buffers are larger than 256 bits (e.g., 512 bits).

[0103] Some aspects of embodiments of this disclosure relate to including additional buffers and accumulators at the level of DRAM memory bank 200 (e.g., additional hardware provided for each DRAM memory bank).

[0104] Figure 8 This is a schematic block diagram of an in-memory computing (IMC) module of a DRAM memory according to an embodiment of the present disclosure, wherein the IMC module is also connected to a result buffer, an accumulator, and a buffer.

[0105] like Figure 8 As shown, the ALU 252 can receive input operand A (in Figure 8 The operands are labeled as 256-bit operand A[0:255] and input operand B (in...). Figure 8 The operands are labeled as 256-bit operands B[0:255]). The ALU performs operations (e.g., addition, multiplication, dot product, outer product, etc.) on the two input operands and computes the result C (in...).Figure 8 The result is marked as 256 bits (C[0:255]).

[0106] like Figure 8 As shown, the IMC module 250 may further include an accumulator 802 connected to the output of the ALU 252. For example, the output of the ALU 252 may be connected to both the result register Rz 256 and the accumulator 802, such that the result C is stored in the result register Rz 256 and provided to the accumulator 802. The accumulator 802 includes an accumulator register that stores the accumulated value (e.g., a 256-bit value). When the accumulator 802 receives a new result from the ALU 252, the new result is added to the accumulated value already stored in the accumulator register (e.g., added to the accumulated value already stored in the accumulator register) (e.g., the value stored in the accumulator register of the accumulator 802 is updated or set to the sum of the new result and the value previously stored in the accumulator 802). In some embodiments, the accumulator 802 may be configured to reset the accumulator register in response to a reset command (e.g., setting the accumulated value stored in the accumulator register to zero). The accumulator 802 is particularly useful when calculating inner products or dot products (such as when calculating matrix multiplication (e.g., when each value in the resulting matrix is ​​the inner product of the rows of the first operand and the columns of the second operand)). Figure 8 In the embodiment shown, the IMC module 250 further includes a 1024-bit output buffer, shown as a first 512-bit buffer 812 and a second 512-bit buffer 814. The first 512-bit buffer 812 and the second 512-bit buffer 814 can receive the output of the accumulator 802 (e.g., D[0:16]). The larger output buffer, capable of storing multiple result values, allows the memory controller 140 to control the DRAM memory bank to calculate multiple results at once before opening another page of the DRAM array to store these results. For example, in... Figure 6B In the case of data reuse in the SR data layout shown, the value from a row of the first operand matrix A is multiplied by the different corresponding value from a row of the second operand matrix B to calculate the page (e.g., page 411) that is different from the page of the storage operand written to the DRAM memory bank 200. Figure 6A and 6B The portion of page 413 shown would require additional PRE and ACT commands to open the page used to store the results. However, a larger output buffer reduces the number of page switches required to complete the computation (since each page switch requires PRE and ACT), thus improving computational performance. Although Figure 8The IMC module 250 is depicted with both larger output buffers 812 and 814 and the accumulator 802, but embodiments of the present disclosure are not so limited and can also include embodiments in which the IMC module 250 includes the accumulator 802 without the additional output buffers 812 and 814, and embodiments in which the IMC module 250 includes the additional output buffers 812 and 814 without the accumulator 802. In some embodiments of the present disclosure, the IMC module 250 can include multiple accumulators 802 arranged in parallel (e.g., so that multiple values can be accumulated simultaneously). Embodiments of the present disclosure are also not limited to the case of two 512-bit output buffers 812 and 814, but can include buffers that are larger or smaller than 512 bits and / or more or fewer than two output buffers. The various portions of the IMC module 250 can have different amounts of latency. For example, the ALU 252 that performs the multiplication operation can introduce a four-cycle latency, while the accumulation operation can involve a one-cycle latency.

[0107] Aspects of embodiments of the present disclosure will be described in greater detail below in the context of implementing a general matrix-matrix multiplication (GEMM).

[0108] Figure 9 is a schematic diagram of an example of a general matrix-matrix multiplication (GEMM) computation to explain using a same row (SR) layout, in accordance with some embodiments of the present disclosure. In more detail, Figure 9 Matrix A is depicted as being multiplied by matrix B, and the result C stored in the accumulator (e.g., accumulator 802) is updated by adding the product of matrix A and matrix B to the value currently stored in the accumulator 802 (result C += matrix A x matrix B (i.e., result C = result C + matrix A x matrix B)). In Figure 9 The computation shown in illustrates multiplying 16 blocks of matrix A with 16 blocks of matrix B, where each block has 16 FP-16 elements (256 bits). A particular internal block organization layout will be discussed in more detail below, but the overall complexity of the GEMM remains the same for different layouts. As illustrated in Figure 9 As represented in, the blocks shaded with a similar pattern are the blocks that are multiplied together. As noted above, Figure 9 A same row (SR) data layout is depicted in which all 32 values of matrix A and matrix B are stored in the same page 414 of the DRAM bank 200, such that both operands of each computation (e.g., block A00 and blocks B00, B01, B02, and B03) are located in the same page 414. The results of the respective computations can be accumulated in the result register Rz 256 to compute the result C.

[0109] As a specific example of block-level multiplication, when computing the inner product (e.g., dot product) of two blocks A00 and B00 loaded from the same page of DRAM bank 200, block A00 can be dot-multiplied with block B00 of matrix B. As discussed above, in some embodiments, each block includes 16 FP-16 elements. For example, block A00 can include elements a00, a01,..., a15, block B00 can include elements b00, b01,..., b15, and the inner product of blocks A00 and B00 is the sum of the product pairs: a00 x b00 + a01 x b01 +... + a15 x b15. Thus, the inner product (or dot product) of two blocks produces a single element or single value, and thus, an in-memory compute (IMC) module according to embodiments of the present disclosure can compute the inner product using fewer buffer registers (e.g., accumulator registers 802) for storing the result. Thus, an IMC module including accumulator registers according to some embodiments of the present disclosure can be well suited to perform computations involving the accumulation of values, such as computing inner products. In some embodiments, an IMC module having accumulators suitable for computing inner products is used to perform inference (or forward propagation) using a trained neural network (such as the first operand can represent an input (e.g., activations from a previous layer) and the second operand can represent weights associated with neurons of a layer of the trained neural network).

[0110] As another example of block-level multiplication, when computing the outer product of two blocks loaded from the same page of DRAM bank 200, each value of a first block is multiplied with each value of a second block, and the two blocks can be loaded simultaneously. For example, where each block includes 16 values, as discussed above, the outer product of two blocks has 16 x 16 = 256 output values. Because there is no tree-adder accumulation needed and because the 256 output values can be computed in parallel, the outer product can be computed using simpler hardware than the inner product. However, according to embodiments of the present disclosure, computing the outer product requires multiple output buffer registers (e.g., output buffers 812 and 814) in the in-memory compute module 250 (e.g., to store all of the values of the result of the outer product). For example, assuming each block includes up to 16 FP-16 values, an IMC module 250 configured to compute the outer product can include an output buffer large enough to store 256 values (e.g., a 256 x 16-bit = 4096-bit buffer). Thus, an IMC module including an output buffer according to some embodiments of the present disclosure can be well suited to perform computations involving the outer product of operands, the output buffer having a size to store a product greater than or equal to the number of values in the blocks of the first and second operands (in the example above, because each of the blocks of the operands includes 16 values, the output buffer stores at least 256 values).

[0111] As a third example of block-level multiplication, when computing the tensor product of two blocks loaded from the same page of DRAM storage 200, the two blocks can include values arranged in a matrix. For example, where each block includes 16 FP-16 values, as described above, each block can be arranged as a 4x4 matrix of values. Computing the tensor product of these blocks generates a 4x4 result matrix of values. For example, assuming each block includes up to 16 FP-16 values, an IMC module 250 configured to compute the tensor product of two 16-value 4x4 blocks can have an output buffer large enough to store 16 values (e.g., a 16x16-bit = 256-bit buffer). Thus, an IMC module including an output buffer large enough to store the quantity of values equal to the quantity of values in the larger of the first and second operands can be well suited to perform computations involving the tensor product of operands according to embodiments of the present disclosure. Implementing the tensor product in a memory-in-compute module according to embodiments of the present disclosure presents a tradeoff with simpler hardware and an intermediate number of buffer registers compared to IMC module hardware suitable for computing inner products and computing outer products.

[0112] In more detail, performing a tensor product between two 4x4 matrices A and B involves 64 multiplication operations, 48 addition operations, and transposition of matrix A / matrix B. When using a 16- lane e-ALU 252 according to one embodiment of the present disclosure, 16 operations can be performed in parallel. As such, 64 + 48 = 112 operations can be performed in seven cycles of the ALU (assuming multiplication can be performed in one cycle). In another embodiment of the present disclosure, to improve performance, a 64-lane ALU 252 can be used to compute the tensor product of two blocks representing two 4x4 matrices in 2-3 cycles or 4-6 nanoseconds. Thus, increasing the number of lanes in the ALU 252 can increase the amount that can be performed in parallel between computations, thereby also improving performance.

[0113] Accordingly, aspects of embodiments of the present disclosure relate to systems and methods for data placement in a dynamic random access memory (DRAM) system with memory-in-compute. In the above three examples of computing inner products, outer products, and tensor products in the context of SR (single page) data layout, a single row includes a first block of (matrix of) row data in the same page as a second block of (matrix of) column data.

[0114] According to some embodiments of the present disclosure, the memory controller 140 (or client memory controller) of the memory module 110 controls placement of data (operands) received from the host processor 170 according to instructions received from the host processor 170. For example, based on whether the instruction is to compute a dot product, an outer product, or a tensor product and based on the size of the operands (e.g., the dimension of the matrices), the memory controller 140 can use a 1OP, SR, or DR data placement and control the DRAM bank to store the data accordingly.

[0115] Figure 10 is a flowchart of a method for controlling placement of data in a DRAM bank with an in-memory compute module according to embodiments of the present disclosure. In operation 1110, the memory controller 140 receives operands (e.g., a first operand and a second operand) and instructions (e.g., functions) to be applied to the operands. These instructions can include, for example, computing an inner product, computing an outer product, performing an inference using a trained neural network and an input vector, performing a step of a backpropagation algorithm for training a neural network, etc.

[0116] In operation 1130, the memory controller 140 determines which data placement (e.g., 1OP, SR, or DR) to use to store one or both operands in the DRAM bank on the DRAM die based on the instructions. In some embodiments, the instructions (e.g., from the host memory controller 180) explicitly specify which data placement to use. In some embodiments, the placement is selected based on the type of computation to be performed, and the memory or hardware requirements for accelerating such computation in the DRAM bank (e.g., an instruction to compute a dot product can result in data being placed in a DRAM bank with accumulators, while an instruction to compute an outer product can result in data being placed in a DRAM bank with more output buffers).

[0117] In operation 1150, the memory controller 140 supplies the first and second operands to the DRAM bank based on the selected placement. For example, in the case of 1OP, the memory controller 140 controls the DRAM bank 200 to store at least a first chunk of the first operand, and to supply a second chunk of the second operand directly to the IMC module of the DRAM bank. As another example, in the case of SR, the memory controller 140 controls the DRAM bank 200 to store chunks corresponding to the first and second operands in the same row or page of the DRAM bank.

[0118] In operation 1170, the memory controller 140 controls the in-memory compute module of the DRAM bank 200 to perform operations based on the input instructions. For example, in the case of instructions for performing an inference using a trained model, the operations can include preparing an input vector based on one operand, and weighting values of the input vector based on stored parameters in a second operand.

[0119] While specific example embodiments have been set forth, it will be understood that the disclosure is not limited to the embodiments disclosed, but is intended to cover any modifications and equivalent arrangements included within the spirit and scope of the appended claims and their equivalents.

Claims

1. A memory module comprising: a memory die comprising a plurality of dynamic random access memory (DRAM) banks, each of the plurality of DRAM banks comprising: an array of DRAM cells arranged in a plurality of pages, each page comprising a plurality of DRAM cells, each DRAM cell storing a bit value; a row buffer configured to store values of an open page of the plurality of pages; an input / output module; and an in-memory compute module, wherein the in-memory compute module comprises: an arithmetic logic unit configured to: receive operands from the row buffer or the input / output module and compute an output based on the operands and a selected arithmetic logic operation of a plurality of arithmetic logic operations; and a result register configured to store the output computed by the arithmetic logic unit; and a memory controller configured to: receive a first operand, a second operand, and an instruction from a host processor; determine one data layout from a plurality of data layouts based on the instruction; supply the first operand and the second operand to the DRAM bank according to the one data layout; and control the in-memory compute module of the DRAM bank to perform one arithmetic logic operation of the plurality of arithmetic logic operations on the first operand and the second operand according to the instruction, wherein the data layout indicates an arrangement of the first operand and the second operand among the array of DRAM cells and the in-memory compute module.

2. The memory module of claim 1, wherein, the plurality of data layouts comprises: an operand data layout, wherein the first operand is written to the DRAM cells and the second operand is supplied directly from the host processor to the in-memory compute module of the DRAM bank.

3. The memory module of claim 2, wherein, the in-memory compute module further comprises an operand register, and wherein the memory controller is further configured to: store a first chunk of the first operand in the operand register; and perform the one arithmetic logic operation on the first operand stored in the operand register and each of a plurality of second chunks of the second operand.

4. The memory module of claim 1, wherein, the first operand is divided into a plurality of first chunks and the second operand is divided into a plurality of second chunks, each chunk comprising a plurality of values, and wherein the plurality of data layouts comprises a same-page data layout, wherein the memory controller stores one or more of the plurality of first chunks and one or more of the plurality of second chunks in a same page of the DRAM cells.

5. The memory module of claim 4, wherein, the in-memory compute module further comprises an operand register, and wherein the memory controller is further configured to: store one of the one or more of the plurality of first chunks in the operand register and perform the one arithmetic logic operation on the first operand stored in the operand register and each of the one or more of the second chunks stored in the same page of the array of DRAM cells as the one of the first chunks.

6. The memory module of claim 5, wherein, the in-memory compute module of the plurality of DRAM banks further comprises an accumulator comprising an accumulator register configured to store an accumulated value, the accumulator configured to: receive the output computed by the arithmetic logic unit; and ​ updating an accumulator register with a sum of the accumulated value and the output, and wherein the instruction comprises computing an inner product of the first operand and the second operand, wherein the one of the plurality of first tiles stores row data, wherein one of the plurality of second tiles comprises column data.

7. The memory module of claim 5, wherein, the one first tile has a first number of values, the second tile has a second number of values, wherein the in-memory compute module of the plurality of DRAM banks comprises an output buffer, wherein the output buffer has a size that is greater than or equal to a product of the first number of values and the second number of values, and wherein the instruction comprises computing an outer product of the first operand and the second operand, wherein the one of the plurality of first tiles stores row data, wherein one of the plurality of second tiles comprises column data.

8. The memory module of claim 5, wherein, the one first tile has a first number of values, the second tile has a second number of values, wherein the in-memory compute module of the plurality of DRAM banks comprises an output buffer, the output buffer has a size that is greater than or equal to a larger of the first number of values and the second number of values, and wherein the instruction comprises computing a tensor product of the first operand and the second operand, wherein the one of the plurality of first tiles stores row data, and one of the plurality of second tiles comprises column data.

9. The memory module of claim 1, wherein, the first operand is divided into a plurality of first tiles, and wherein the second operand is divided into a plurality of second tiles, wherein each tile comprises a plurality of values, and wherein the plurality of data layouts comprises different page data layouts, wherein the memory controller stores a subset of the plurality of first tiles in a first page of an array of DRAM cells and stores a subset of the plurality of second tiles in a second page of the array of DRAM cells.

10. The memory module of claim 9, wherein, the in-memory compute module further comprises an operand register, and wherein the memory controller is further configured to: store the first tile of the first operand from the first page in the operand register, and perform the one arithmetic-logic operation on the first operand stored in the operand register and each of the plurality of second tiles of the second operand from the second page.

11. The memory module of any one of claims 1 to 10, wherein, the in-memory compute module of each DRAM bank further comprises a buffer configured to buffer an output computed by the arithmetic-logic unit.

12. The memory module of claim 11, wherein, the buffer is at least four times the size of the result register.

13. The memory module of any one of claims 1 to 10, wherein, the in-memory compute module of each DRAM bank further comprises an accumulator comprising an accumulator register configured to store an accumulated value, the accumulator is configured to: receive the output computed by the arithmetic-logic unit; and update the accumulator register with a sum of the accumulated value and the output.

14. The memory module of any one of claims 1 to 10, wherein, the memory module is a high-bandwidth memory module, the high-bandwidth memory module comprises a stack of a plurality of memory dies connected by through-silicon vias, wherein the stack of the plurality of memory dies comprises the memory die.

15. A method for performing in-memory computation, comprising: receiving, by a memory controller of a memory module, a first operand, a second operand, and an instruction; determining, by a memory controller, one data layout from a plurality of data layouts based on the instruction; supplying a first operand and a second operand to one or more dynamic random access memory (DRAM) banks of a memory module in accordance with the one data layout; and performing, by an in-memory compute module of the DRAM banks, one arithmetic logic operation from a plurality of arithmetic logic operations on the first operand and the second operand in accordance with the instruction, wherein each of the one or more DRAM banks comprises: an array of DRAM cells arranged in a plurality of pages, each page comprising a plurality of DRAM cells, each DRAM cell storing a bit value; a row buffer configured to store values of an open page from the plurality of pages; an input / output module; and an in-memory compute module, wherein the in-memory compute module comprises: an arithmetic logic unit configured to receive operands from the row buffer or the input / output module and compute an output based on the operands and a selected arithmetic logic operation from the plurality of arithmetic logic operations; and a result register configured to store the output computed by the arithmetic logic unit, wherein the data layout indicates an arrangement of the first operand and the second operand among the array of DRAM cells and the in-memory compute module.

16. The method of claim 15, wherein, the plurality of data layouts comprises: an operand data layout in which the first operand is written to the DRAM cells and the second operand is supplied directly to the in-memory compute module of the DRAM banks from a host processor.

17. The method of claim 16, wherein, the in-memory compute module further comprises an operand register, and wherein the memory controller is further configured to: store a first chunk of the first operand in the operand register; and perform the one arithmetic logic operation on the first operand stored in the operand register and each second chunk of a plurality of second chunks of the second operand.

18. The method of claim 15, wherein, the first operand is divided into a plurality of first chunks and the second operand is divided into a plurality of second chunks, each chunk comprising a plurality of values, and wherein the plurality of data layouts comprises a same-page data layout in which the memory controller stores one or more first chunks from the plurality of first chunks and one or more second chunks from the plurality of second chunks in a same page of the plurality of DRAM cells.

19. The method of claim 18, wherein, the in-memory compute module further comprises an operand register, and wherein the memory controller is further configured to: store a first chunk of the one or more first chunks in the operand register, and perform the one arithmetic logic operation on the first operand stored in the operand register and each second chunk of the one or more second chunks stored in the same page of the array of DRAM cells as the one first chunk.

20. The method of claim 19, wherein, the in-memory compute module of at least one DRAM bank further comprises an accumulator, wherein the accumulator comprises an accumulator register configured to store an accumulated value, wherein the accumulator is configured to: receive the output computed by the arithmetic logic unit; and update the accumulator register with a sum of the accumulated value and the output, and wherein the instruction comprises computing an inner product of the first operand and the second operand, wherein the one of the plurality of first tiles stores row data, wherein the one of the plurality of second tiles comprises column data.

21. The method of claim 19, wherein, The one of the first tiles has a first number of values, the second tile has a second number of values, wherein the in-memory compute module of the at least one DRAM bank comprises an output buffer having a size that is greater than or equal to a product of the first number of values and the second number of values, and wherein the instruction comprises computing an outer product of the first operand and the second operand, wherein the one of the plurality of first tiles stores row data, wherein the one of the plurality of second tiles comprises column data.

22. The method of claim 19, wherein, The one of the first tiles has a first number of values, the second tile has a second number of values, wherein the in-memory compute module of the at least one DRAM bank comprises an output buffer having a size that is greater than or equal to a product of the first number of values and the second number of values, and wherein the instruction comprises computing a tensor product of the first operand and the second operand, wherein the one of the plurality of first tiles stores row data, wherein the one of the plurality of second tiles comprises column data.

23. The method of claim 15, wherein, The first operand is divided into a plurality of first tiles, the second operand is divided into a plurality of second tiles, each tile comprises a plurality of values, and wherein the plurality of data layouts comprises different page data layouts, wherein the memory controller stores a subset of the plurality of first tiles in a first page of an array of DRAM cells and stores a subset of the plurality of second tiles in a second page of the array of DRAM cells.

24. The method of claim 23, wherein, The in-memory compute module further comprises an operand register, and wherein the memory controller is further configured to: store the one of the first tiles of the first operand from the first page in the operand register, and perform the one arithmetic-logic operation on the first operand stored in the operand register and each of the plurality of second tiles of the second operand from the second page.

Images