Calculation device in memory module and control method thereof

Abstract

Provided are a calculation device in a memory module and a control method thereof. According to an embodiment of the present invention, the memory module comprises: a memory that stores data; an RCD that receives a memory command from a host, controls the memory and a DB, and transmits a near-memory operation command to the DB; and the DB that performs an operation according to the near-memory operation command received from the RCD and stores an operation result in the memory, wherein the memory command distinguishes between memory control commands and near-memory operation commands by address. Accordingly, the performance of an application requiring a large memory capacity at a low cost can be improved by performing an arithmetic operation inside the memory module even at a high memory speed by maintaining the structure of the existing LRDIMM as much as possible.

Description

Computing unit and control method in memory module

[0001] The present invention relates to a technology for controlling operations within a memory, and more particularly, to a memory module structure for supporting arithmetic operations, etc., in a Load Reduced DIMM (LRDIMM), which is one of the memory modules, and a technology for controlling the same.

[0002] Conventional DIMMs (Dual In-line Memory Modules), such as Unbuffered DIMM (UDIMM), Registered DIMM (RDIMM), and Load Reduced DIMM (LRDIMM), consist of a memory device for storing data and a Registered Clock Driver (RCD) or Data Buffer (DB) for controlling it. These DIMMs use specific DIMMs depending on the characteristics of the system, taking into account cost, storage capacity, speed, etc. In common, these DIMMs can only store data in memory and cannot perform arithmetic operations such as addition and multiplication using the data stored in the memory.

[0003] However, with the recent increase in programs requiring large memory capacities, such as deep learning, technologies that perform computations within or near memory are gaining prominence. In particular, many technologies based on LRDIMM memory modules are being proposed for performing computations within memory modules.

[0004] LRDIMM-based technology proposes configuring the LRDIMM's RCD and DB into a single chip, utilizing this to perform computations near the memory. This memory module structure, however, suffers from the drawbacks of connecting multiple memory devices, leading to signal integrity issues between each memory, hindering high-speed operation. Furthermore, the single, large chip increases the cost of the memory module.

[0005] Additionally, some technologies require dedicated memory controllers to exclude these memory modules, making them difficult to apply to commercial systems.

[0006] The present invention has been devised to solve the above problems, and the purpose of the present invention is to provide a memory module and a control method therefor for improving the performance of applications requiring a large memory capacity at a low cost by performing arithmetic operations within the memory module even at high memory speeds while maintaining the structure of the existing LRDIMM as much as possible.

[0007] Another object of the present invention is to provide a method that enables performing operations directly within a memory module without the need to modify the memory controller.

[0008] According to one embodiment of the present invention for achieving the above object, a memory module includes: a memory in which data is stored; an RCD (Registered Clock Driver) that receives a memory command from a host, controls the memory and a DB (Data Buffer), and transmits a memory-near operation command to the DB; and a DB that performs an operation according to the memory-near operation command received from the RCD and stores the operation result in the memory; wherein the memory command distinguishes a memory control command and a memory-near operation command by an address.

[0009] RCD controls memory according to the memory control command if the memory command is a memory control command, controls memory and DB according to the memory near operation command if the memory command is a memory near operation command, and can transmit the memory control command to DB.

[0010] RCD can control memory to transfer data required for a memory-near operation instruction to the DB if the memory instruction is a memory-near operation instruction.

[0011] Memory commands can be distinguished between memory control commands and memory vicinity operation commands based on the row address of the RCD.

[0012] The DB may include a memory-near operator for executing a memory-near operation instruction; the memory-near operator may include a command memory in which the operation instruction is stored; a command decoder for decoding the stored operation instruction; a data memory in which data to be operated is stored; and an arithmetic operator for executing the operation instruction decoded with data stored in the data memory and storing the operation execution result in the memory.

[0013] The instruction decoder may include multiple instruction decoders for decoding instructions in parallel.

[0014] Memory-near operation instructions may include instructions for memory of a memory-near operation unit, instructions for data memory, and instructions for arithmetic operations.

[0015] The memory-near operation commands may further include commands for DB configuration.

[0016] The host can change the mode of the memory-near operator to operation mode through a command for the DB environment settings when executing a memory-near operator command.

[0017] According to another aspect of the present invention, a memory module control method is provided, including: a step in which an RCD receives a memory command from a host; a step in which the RCD controls a memory and a DB in which data is stored based on the received memory command, and transmits a memory-near operation command to the DB; a step in which the DB performs an operation according to the memory-near operation command received from the RCD, and stores the operation result in the memory; and the memory command is characterized in that the memory control command and the memory-near operation command are distinguished by an address.

[0018] According to another aspect of the present invention, a memory-near operation system is provided, comprising: a host for generating a memory command; and a memory module for receiving and processing a memory command from the host; wherein the memory module comprises: a memory in which data is stored; an RCD for receiving a memory command from the host, controlling a memory and a DB, and transmitting a memory-near operation command to the DB; and a DB for performing an operation according to the memory-near operation command received from the RCD and storing the operation result in the memory; wherein the memory command is characterized in that the memory control command and the memory-near operation command are distinguished by an address.

[0019] According to another aspect of the present invention, a memory module control method is provided, including a step in which a host generates a memory command and transmits it to an RCD; a step in which the RCD controls a memory and a DB in which data is stored based on the memory command received from the host and transmits a memory-near operation command to the DB; a step in which the DB performs an operation according to the memory-near operation command received from the RCD and stores the operation result in the memory; and the memory command is characterized in that the memory control command and the memory-near operation command are distinguished by an address.

[0020] As described above, according to embodiments of the present invention, by maintaining the structure of the existing LRDIMM as much as possible and performing arithmetic operations within the memory module even at high memory speeds, the performance of applications requiring a large memory capacity can be improved at a low cost.

[0021] Additionally, according to embodiments of the present invention, it is possible to perform operations directly within a memory module without modifying the memory controller.

[0022] Figure 1. Deep learning model size trends

[0023] Figure 2. (a) General computing system and (b) memory-near computing system.

[0024] Fig. 3. (a) LRDIMM (100) and (b) LRDIMM (100) capable of memory-near operations.

[0025] Figure 4. Memory module structure for LRDIMM (100)-based memory-near operation.

[0026] Figure 5. Example of a memory address space for performing memory-near operations.

[0027] Figure 6. Example of extended BCOM instructions for near-memory operations.

[0028] Figure 7. Example of memory-near operators and control flow diagram within the DB.

[0029] Hereinafter, the present invention will be described in more detail with reference to the drawings.

[0030] Computing systems adopting the von Neumann architecture are divided into computing modules, such as CPUs or GPUs, and memory modules that store data. To perform computations, these computing systems retrieve data from the memory modules to the computing modules, perform the computations, and then store the results back in the computing modules.

[0031] However, applications such as big data and databases that require large amounts of data are on the rise. In particular, the size of Large Language Models (LLMs), which are increasingly important among deep learning models, as shown in Figure 1, is growing exponentially. Consequently, the emergence of these applications is also significantly increasing the memory capacity and bandwidth required for memory modules in existing computing systems.

[0032] To overcome the limitations of the von Neumann architecture, Near-Data Processing (NDP) technology emerged. As shown in Figure 2 (a), a typical computing system retrieves data from a memory module (i.e., DIMM) that stores data to an xPU (e.g., a CPU or GPU) that has computational capabilities, performs computations using the data, and then stores it back in the memory module.

[0033] On the other hand, as shown in (b) of Fig. 2, the near-memory computation technology adds a computation module (i.e., PE) with computational functions to the memory module, so that data can be brought to the PE inside the memory module without being brought from the memory module to the xPU, and then, after the computation, stored directly in the memory of the memory module. This near-memory computation technology has the advantage of not only improving computational response speed but also reducing energy consumption because it does not perform computations on data without passing through the off-chip bus in the memory module and does not pass through the internal memory system of the xPU (e.g., Cache, etc.).

[0034] Computing systems, such as CPUs, primarily use memory modules like DIMMs. These memory modules come in various types, including Unbuffered DIMM (UDIMM), Registered DIMM (RDIMM), and Load Reduced DIMM (LRDIMM). These memory modules are selected based on factors such as cost, memory capacity, and bandwidth required by the computing system. In particular, LRDIMMs offer high memory capacity and bandwidth, making them widely used in environments such as servers.

[0035] Figure 3 (a) illustrates the structure of an LRDIMM. An LRDIMM is largely comprised of a memory device (i.e., DRAM) and a Registered Clock Driver (RCD) or Data Buffer (DB) for controlling and buffering data for the memory device. The RCD receives DRAM commands from the CPU's memory controller and transmits them to the DB and DRAM, while the DB acts as a buffer between the memory controller and DRAM.

[0036] Memory-near-computation technology is based on these LRDIMMs. Memory-near-computation technology integrates the RCD and DB functions of an LRDIMM with the computational functions (PE) into a single chip. In other words, memory-near-computation technology retrieves data from the memory device to this chip, performs operations on it, and then stores it back in the memory device.

[0037] Single-chip memory-near-computing technology has various limitations. The LRDIMM's DB decouples the memory controller and the memory devices on the memory module, reducing the load on the memory controller's off-chip bus. This increases memory bandwidth and allows for increased memory capacity by increasing the number of memory devices on the memory module. Furthermore, by ensuring a uniform distance between the DB and memory devices, the data pin speeds of the DB and DRAM can be increased, further increasing bandwidth. However, memory-near-computing technology requires the RCD, DB, and PE to be integrated into a single chip. This single chip and the uneven length of the data pins on each memory module hinder high-speed operation of the memory module. Furthermore, there are limitations, such as the need for a dedicated memory controller to enhance the memory module's computational performance.

[0038] Accordingly, an embodiment of the present invention proposes a memory module structure including an operator capable of overcoming the limitations of LRDIMM-based memory peripheral operation technology and a method for controlling the operator.

[0039] FIG. 4 is a diagram illustrating the structure of a memory module for LRDIMM-based memory-near operations according to an embodiment of the present invention. The presented structure maintains the structures of the RCD (110), DB (130), and memory devices (121 to 124) in which data is stored of the LRDIMM (100). The DB (130) and RCD (110) each include an NDP Unit (135), which is an operation unit for memory-near operations, and an NDP Command Generator (115), which is a control command generation block for the corresponding operations.

[0040] In embodiments of the present invention, memory-near operations are driven based on memory commands (e.g., RD ​​or WR DRAM commands). To distinguish between memory read / write commands and memory-near operation commands, embodiments of the present invention distinguish them based on the address of the memory command.

[0041] Figure 5 illustrates an example of a memory address space for memory-near operations, in which operations for memory read / write commands and memory-near operations are distinguished based on row addresses among addresses of memory commands. As illustrated, the RCD (110) includes address spaces for storing commands for data memory (NDP Data Memory), instruction memory (NDP Instruction Memory), and operation execution (NDP Execution) of the NDP Unit (135), which is a memory-near operation unit, for memory-near operations.

[0042] Accordingly, the command generation block (115) of the RCD (110) receives a memory command such as read / write from the memory controller (250) based on the memory address space and transmits a memory vicinity operation command to the DB (130) according to the address of the command.

[0043] The RCD (110) of the LRDIMM (100) controls the DB (130) via the Data Buffer Command Bus (BCOM). At this time, the command for controlling the DB (130) is called a BCOM command. Since the memory vicinity operation follows the basic structure of the LRDIMM (100), the NDP Unit (135) in the DB (130) is controlled via this BCOM command.

[0044] Figure 6 illustrates an example of an extended BCOM command for memory-near operations in a BCOM command. The memory-near operation command generator (115) of the RCD (110) generates a read / write command as a WR / RD or NDP Write / Read BCOM command based on the memory address space and transmits it to the DB (130).

[0045] The RCD (110) transmits the metadata of the BCOM command through the same bus. For example, in the case of WR / RD, the memory rank address, etc. are transmitted after the BCOM command. In the case of NDP Write / Read, the memory operation type, memory rank, and bank address for memory-near operations are also transmitted from the RCD (110) to the DB (130) after the BCOM command. The memory operation type includes data memory and command memory access and operation execution of the NDP Unit (135).

[0046] Fig. 7 shows a detailed block diagram of an NDP Unit (135) provided inside a DB (130). As illustrated, the NDP Unit (135) includes a command memory (135-1), a command decoder (135-2), an arithmetic operator (135-3), and a data memory (135-4).

[0047] The NDP Unit (135) performs an operation using the BCOM instruction as a signal. For example, assuming that addition is performed in the memory module, the addition instruction is pre-stored in the instruction memory (135-1) and some of the data required for the operation is stored in the data memory (135-4). At this time, the CPU (200) changes the mode of the NDP Unit (135) to the operation mode. The operation mode change is performed by accessing the NDP DB CONF area in the above memory address space. After the mode of the NDP Unit (135) is changed to the operation mode, the CPU (200) reads the data required for the addition. At this time, the memory controller (250) transmits the memory read command to the memory module (100). The RCD (110) receives the command, generates an NDP Read BCOM command, and transmits it to the DB (130). On the other hand, the RCD (110) transmits the memory read command to the memory devices (121 to 124) regardless of this. In the NDP Unit (135), the memory operation command decoded by the command decoder (135-2) is executed using the NDP Read BCOM command as a signal. At this time, the command brings the data required for addition from the data memory to the calculator and then waits until the remaining data arrives from the memory devices (121 to 124). The NDP Unit (135), which receives data from the memory devices (121 to 124), performs the addition operation and then stores the result back in the data memory.

[0048] In this way, in the embodiment of the present invention, operations are performed based on BCOM commands and data (and data strobe signals). Therefore, there is an advantage in that the CPU (200) or xPU does not need to track the status of the NDP Unit (135) or the like by controlling both the start and the end of the operation of the memory module. In addition, in the embodiment of the present invention, the memory controller (250) is used without modification, but a plurality of command decoders (135-2) are used to increase the bandwidth of the operation near the memory.

[0049] So far, preferred embodiments of the computational device and control method within the memory module have been described in detail.

[0050] In the above embodiment, the structure of the existing LRDIMM is maintained as much as possible to perform arithmetic operations within the memory module even at high memory speeds, thereby improving the performance of applications requiring a large amount of memory capacity at a low cost, and enabling operations to be performed directly within the memory module without modifying the memory controller.

[0051] In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and various modifications can be made by a person having ordinary skill in the art to which the present invention pertains without departing from the gist of the present invention as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present invention.

Claims

1. Memory where data is stored; RCD (Registered Clock Driver) that receives memory commands from the host, controls the memory and DB (Data Buffer), and transmits memory-area operation commands to the DB; Includes a DB that performs operations according to memory-area operation commands received from the RCD and stores the operation results in memory; Memory commands are, A memory module characterized by distinguishing memory control instructions and memory vicinity operation instructions by address.

2. In claim 1, RCD is, If the memory command is a memory control command, the memory is controlled according to the memory control command. A memory module characterized in that if a memory command is a memory-near operation command, it controls memory and DB according to the memory-near operation command and transmits the memory control command to the DB.

3. In claim 2, RCD is, A memory module characterized in that, if a memory command is a memory-near operation command, it controls the memory to transmit data required for the memory-near operation command to the DB.

4. In claim 2, Memory commands are, A memory module characterized by distinguishing between memory control commands and memory vicinity operation commands based on the row address of the RCD.

5. In claim 2, DB is, A memory-near arithmetic unit for executing a memory-near arithmetic instruction; The memory vicinity operator is, Instruction memory where operation instructions are stored; An instruction decoder that decodes stored operation instructions; Data memory where data to be operated is stored; and A memory module characterized by including an arithmetic operator for performing an operation command decoded into data stored in a data memory and storing the result of performing the operation in the memory.

6. In claim 5, The command decoder is, A memory module characterized by including a plurality of instruction decoders for decoding instructions in parallel.

7. In claim 5, Memory vicinity operation instructions are, A memory module characterized by including instructions for a memory-near-arithmetic unit, instructions for a data memory, and instructions for an arithmetic unit.

8. In claim 7, Memory vicinity operation instructions are, A memory module characterized by further including commands for DB environment settings.

9. In claim 8, The host, A memory module characterized in that, when a memory vicinity operation command is issued, the mode of the memory vicinity operation unit is changed to the operation mode through a command for DB environment settings.

10. RCD receives a memory command from the host; A step in which the RCD controls the memory and DB where data is stored based on the received memory command, and transmits a memory vicinity operation command to the DB; A step of DB performing an operation according to a memory-local operation command received from RCD and storing the operation result in memory; Memory commands are, A memory module control method characterized by distinguishing memory control commands and memory vicinity operation commands by address.

11. A host that generates memory commands; and A memory module that receives and processes a memory command from a host; The memory module is, Memory where data is stored; RCD that receives memory commands from the host, controls memory and DB, and transmits memory-area operation commands to DB; Includes a DB that performs operations according to memory-area operation commands received from the RCD and stores the operation results in memory; Memory commands are, A memory-near operation system characterized by distinguishing memory control instructions and memory-near operation instructions by address.

12. The host generates a memory command and transmits it to the RCD; A step in which RCD controls the memory and DB where data is stored based on a memory command received from the host, and transmits a memory vicinity operation command to the DB; A step of DB performing an operation according to a memory-local operation command received from RCD and storing the operation result in memory; Memory commands are, A memory module control method characterized by distinguishing memory control commands and memory vicinity operation commands by address.

Images