Systems and methods for computing

Abstract

A system and method for computing are disclosed. In some embodiments, the system includes a memory including one or more in-memory function circuits and a cache coherence protocol interface circuit having a first interface and a second interface. An in-memory function circuit of the one or more in-memory function circuits can be configured to perform an operation on operands including a first operand fetched from the memory to form a result. The first interface can be connected to the memory, and the second interface can be configured as a cache coherence protocol interface on a bus interface.

Description

Technical Field

[0001] One or more aspects of embodiments of this disclosure relate to function-in-memory computation, and more specifically, to a system and method for performing function-in-memory computation using a cache coherence protocol interface. Background Technology

[0002] The background information provided in this section is included for contextual purposes only. The content of this section is not intended to be considered prior art. In-memory function computing can have advantages over other computing configurations because the total bandwidth of the data path between memory and multiple function-in-memory circuits can be significantly greater than the bandwidth of the data path between memory and a central processing unit (CPU) or graphics processing unit (GPU). However, implementing in-memory function computing can be challenging, partly because the execution of operations by the in-memory function circuits affects the latency of memory operations.

[0003] Therefore, there is a need for improved systems and methods for performing in-memory functional computations. Summary of the Invention

[0004] According to an embodiment of the present invention, a system for computation is provided, the system comprising: a memory including one or more in-memory functional circuits; and a cache coherence protocol interface circuit having a first interface and a second interface, wherein the in-memory functional circuits of the one or more in-memory functional circuits are configured to perform operations on operands including a first operand obtained from the memory to form a result, the first interface being connected to the memory, and the second interface being configured as a cache coherence protocol interface on a bus interface.

[0005] In some embodiments, the in-memory functional circuitry is arranged in a single instruction multiple data configuration; or the in-memory functional circuitry is arranged in a pulsed configuration.

[0006] In some embodiments, the cache coherence protocol interface circuit is a compute fast link (CXL) interface circuit, and the bus interface is a peripheral component interconnect fast (PCIe) endpoint interface.

[0007] In some embodiments, the in-memory functional circuitry of the one or more in-memory functional circuits is located on a semiconductor chip having dynamic random access memory.

[0008] In some embodiments, the first interface is configured to operate according to a protocol selected from a group consisting of DDR2, DDR3, DDR4 and DDR5.

[0009] In some embodiments, the in-memory functional circuitry in the one or more in-memory functional circuits includes: a plurality of registers, a plurality of multiplexers, and an arithmetic logic unit.

[0010] In some embodiments, the in-memory functional circuits in the one or more in-memory functional circuits are configured to perform arithmetic operations selected from a group consisting of addition, subtraction, multiplication, and division.

[0011] In some embodiments, the in-memory functional circuits in the one or more in-memory functional circuits are configured to perform arithmetic operations selected from a group consisting of floating-point addition, floating-point subtraction, floating-point multiplication, and floating-point division.

[0012] In some embodiments, the in-memory functional circuits in the one or more in-memory functional circuits are configured to perform logical operations selected from a group consisting of bitwise AND, bitwise OR, bitwise XOR, and bitwise complement.

[0013] In some embodiments, the in-memory functional circuits of the one or more in-memory functional circuits are configured to: store the result in memory in a first state, and send the result to a cache coherence protocol interface circuit in a second state.

[0014] In some embodiments, the system further includes host processing circuitry connected to the second interface.

[0015] In some embodiments, the host processing circuitry includes a PCIe root complex having a root port connected to the second interface.

[0016] According to an embodiment of the present invention, a system for computation is provided, the system comprising: a memory; and a cache coherence protocol interface circuit having a first interface and a second interface, the cache coherence protocol interface circuit being configured to perform arithmetic operations on data stored in the memory, the first interface being connected to the memory, and the second interface being configured as a cache coherence protocol interface on a bus interface.

[0017] In some embodiments, the memory includes one or more in-memory functional circuits, and the in-memory functional circuits of the one or more in-memory functional circuits are configured to perform operations on operands including a first operand obtained from the memory to form a result.

[0018] In some embodiments, the in-memory functional circuitry is arranged in a single instruction multiple data configuration; or the in-memory functional circuitry is arranged in a pulsed configuration.

[0019] In some embodiments, the cache coherence protocol interface circuit is a compute fast link (CXL) interface circuit, and the bus interface is a peripheral component interconnect fast (PCIe) endpoint interface.

[0020] In some embodiments, the in-memory functional circuitry of the one or more in-memory functional circuits is located on a semiconductor chip having dynamic random access memory.

[0021] In some embodiments, the first interface is configured to operate according to a protocol selected from a group consisting of DDR2, DDR3, DDR4, DDR5, GDDR, HBM, and LPDDR.

[0022] In some embodiments, the in-memory functional circuits of the one or more in-memory functional circuits are configured to: store the result in memory in a first state, and send the result to a cache coherence protocol interface circuit in a second state.

[0023] According to an embodiment of the present invention, a method for computation is provided, the method comprising: sending one or more CXL packets to a CXL interface circuit via a host processing circuit; in response to receiving a CXL packet, sending an instruction via the CXL interface circuit to an in-memory functional circuit in a memory connected to the CXL interface circuit; and performing an operation on an operand including a first operand retrieved from the memory according to the instruction via the in-memory functional circuit to form a result. Attached Figure Description

[0024] The accompanying drawings are included to provide examples only and should not be construed as limiting the scope of the invention. These and other features and advantages of this disclosure will be appreciated and understood by referring to the specification, claims, and drawings, in which:

[0025] Figure 1A This is a block diagram of a memory including multiple memory bank groups according to embodiments of the present disclosure, each memory bank group including multiple memory banks, and the memory also including multiple in-memory functional circuits.

[0026] Figure 1B This is a block diagram of a memory bank having multiple sense amplifiers, global input / output registers, and in-memory functional circuitry according to embodiments of the present disclosure;

[0027] Figure 1C It is a table of instructions for execution by in-memory functional circuitry according to embodiments of the present disclosure;

[0028] Figure 2AThis is a block diagram of an in-memory functional configuration using multiple standard dynamic random access memory (DRAM) modules according to embodiments of the present disclosure, each DRAM module being connected to in-memory functional circuitry, and the multiple DRAM modules sharing a channel to the host processing circuitry.

[0029] Figure 2B This is a block diagram of an in-memory functional configuration using multiple standard DRAM modules arranged in multiple rows according to embodiments of the present disclosure, each row being connected to in-memory functional circuitry.

[0030] Figure 2C This is a block diagram of an in-memory functional configuration using multiple modified DRAM chips according to embodiments of the present disclosure, each chip including in-memory functional circuitry. Figure 2D This is a block diagram of an in-memory functional configuration using multiple modified DRAM chips according to embodiments of the present disclosure, each chip including multiple DRAM memory banks and in-memory functional circuitry for each memory bank;

[0031] Figure 3A This is an in-memory functional configuration according to an embodiment of the present disclosure, wherein a plurality of in-memory functional circuits are arranged as a single instruction multiple data processor;

[0032] Figure 3B This is an in-memory functional configuration according to an embodiment of the present disclosure, wherein a plurality of in-memory functional circuits are arranged in a pulsating array;

[0033] Figure 4A This is a block diagram of a system for computation according to embodiments of the present disclosure;

[0034] Figure 4B It is a block diagram of a system for performing in-memory functional calculations; and

[0035] Figure 5 This is a flowchart of a method for calculation according to an embodiment of the present disclosure. Detailed Implementation

[0036] The specific embodiments described below with reference to the accompanying drawings are intended as exemplary descriptions of systems and methods for performing in-memory functional computations provided in this disclosure, and are not intended to represent the only forms in which this disclosure may be constructed or utilized. This description, in conjunction with the illustrated embodiments, illustrates the features of this disclosure. However, it should be understood that the same or equivalent functions and structures may be implemented by different embodiments intended to be included within the scope of the disclosure. As indicated elsewhere herein, the same element numbers are intended to indicate the same elements or features.

[0037] In some embodiments, the in-memory functional circuitry is processing circuitry integrated with the memory circuitry, or processing circuitry located closer to the memory than, for example, a CPU or GPU connected to the memory via a standard memory bus. Therefore, the total bandwidth between the multiple in-memory functional circuits and the memory can be significantly greater than the total bandwidth of the memory bus, potentially enabling greater processing throughput.

[0038] If memory including in-memory functional circuitry is connected to the CPU or GPU using interfaces such as Double Data Rate 2 (DDR2), DDR3, etc., in some cases, the memory including in-memory functional circuitry may fail to operate correctly. This can happen in part because the latency of responses generated by the in-memory functional circuitry can violate the latency assumptions of the memory controller, which relies on maintaining cache coherence (the consistency of shared resource data ultimately stored in multiple local caches).

[0039] In some embodiments, a cache coherence computing protocol (or "cache coherence protocol") (such as a compute fast link (CXL) interface) can be used to connect memory to the CPU or GPU to mitigate or address this problem. Although some embodiments are described herein as using the CXL protocol, the invention is not limited to such embodiments. For example, any other protocol suitable for maintaining cache coherence (which may be referred to herein as a "cache coherence protocol") may be used instead of CXL.

[0040] Reference Figure 1A In some embodiments, the memory (e.g., high-bandwidth memory (HBM) or dual in-line memory module (DIMM)) may be arranged as multiple memory bank groups (BG0, BG1, BG2, BG3), each memory bank group including multiple memory banks 105 (e.g., BG0 includes memory banks 105 labeled A, B, C, and D, BG1 includes memory banks 105 labeled E to H, BG2 includes memory banks 105 labeled I to L, and BG3 includes memory banks 105 labeled M to P). Figure 1ASome features (such as through-silicon vias (TSVs)) are specific to HBM; however, other forms of memory (e.g., DIMMs) can operate in a similar manner. DRAM memory can be organized into rank, chips, and banks 105. A “rank” can be part of a memory with a shared chip select pin. Each rank may include eight chips, and each chip may include 16 banks 105. The banks 105 of the chips can be organized into “mega-banks” such that, for example, a set of banks 105 consisting of bank groups BG0 from each of the eight chips in a rank can be mega-bank 0 (not shown). The chips can be read in parallel onto a 256-bit (256b) wide bus, where each of the eight chips provides 32 bits (32B) of the 256 bits of data. Figure 1A In the WRIO, write / read input / output can be indicated, and tCCDL can be related to column access timing.

[0041] The memory may be connected to host processing circuitry 205 (e.g., a CPU or GPU, or a CPU or GPU with additional components such as a memory controller (MC)) and provide storage for host processing circuitry 205. In some embodiments, host processing circuitry 205 is on the host side of the network path (e.g., it is a host server). In an in-memory computing (IMC) system, each memory bank 105 may include an input / output sense amplifier 112 (IOSA) and in-memory function (FIM) circuitry 110 (also referred to as "in-memory computing circuitry" or "in-memory processing circuitry"). As used herein, in-memory function circuitry is processing circuitry capable of performing arithmetic or logical operations and is connected to memory more directly than host processing circuitry 205 (and also more directly than accelerators to memory). For example, in a system where the memory is connected to the host processing circuit 205 via a DDR bus, the processing circuit on the memory side of the DDR bus can be considered as an in-memory functional circuit, while the processing circuit on the host processing circuit side of the DDR bus (e.g., an accelerator on the host processing circuit side of the DDR bus to which the host processing circuit 205 can delegate computational tasks) is not considered as an in-memory functional circuit. Figure 1B The structure of such a memory bank 105 is shown in some embodiments. Figure 1C This is a table showing a list of operations that can be performed by the in-memory functional circuitry 110. In some embodiments, the host processing circuitry 205 sends numbers (e.g., with...) to the in-memory functional circuitry 110. Figure 1CThe table lists the numbers between 0 and 9 in a row, and then the in-memory functional circuit 110 performs the corresponding operation. The instruction (or equivalently, the number that identifies the instruction) can be sent by the host processing circuit 205 to the in-memory functional circuit 110 via a reserved for future use (RFU) bit (e.g., the RFU bit of a DDR interface).

[0042] like Figure 1B As shown, the in-memory functional circuitry 110 may include registers 115 (e.g., Rop and Rz), an arithmetic logic unit (ALU) 120, and a multiplexer 125 (in... Figure 1B All of them are marked as "MUX", and they can be used together to execute instructions (e.g., Figure 1C (Instructions listed in the table). The in-memory functional circuitry 110 may also include FIM logic 220, a controller 210, and a memory-mapped register 225 (discussed in further detail below). Figure 1C As shown in the table, the instructions can cause the in-memory functional circuit 110 to copy the contents of one register to another (e.g., instructions 0-5 and 9) or perform an operation (“op”) on the contents of two registers and store the result in a third register (in Figure 1C In the case of the instruction set of the table, the result is stored in register Rz). Operations can be arithmetic operations (e.g., +, -, ×, or / performed according to, for example, IEEE-754) or logical operations (e.g., bitwise & (AND), bitwise | (OR), bitwise ^ (XOR), or bitwise ~ (two's complement)). When the instruction is Figure 1C When one of instructions 6, 7, and 8 in the table is selected, a register (e.g., one of memory-mapped registers 225) can specify the operation to be performed (e.g., a specific arithmetic or logical operation). Return to Figure 1B The arithmetic logic unit 120 may include a 16-channel, 16-bit floating-point (FP-16) vector unit or an 8-channel, 32-bit floating-point (FP-32) vector unit, enabling various operations. In one example, the in-memory functional circuitry 110 may perform at least one of floating-point addition, floating-point subtraction, floating-point multiplication, and floating-point division. Non-limiting examples may include tensor operations (e.g., dot product, outer product, ReLU (rectifier or corrected linear unit), vsSqr (squaring the elements of a vector), and vsSQrt (taking the square root of each element of a vector)). To efficiently utilize the in-memory functional circuitry 110, data may be arranged in memory such that multiple operands are simultaneously available in an open row. As used herein, an "open row" refers to data in the sense amplifier 112 (after row activation is issued). An open row may, for example, include 8192 bits of data from which the ALU can read multiple operands (e.g., 32-bit operands). Figure 1BIn this context, WIO indicates write input / output, while RIO indicates read input / output.

[0043] The memory controller 206 (MC) of the host processing circuit 205 may be a memory controller conforming to the DRAM interface standard issued by the Joint Electron Device Engineering Council (JEDEC) and the BIOS of the host processing circuit 205; in this case, the memory controller 206 may implement no cache or a limited cache. In some embodiments, the memory controller 206 may implement different communication protocols that may not conform to JEDEC (e.g., timing constraints may be different, or the data bus or address and control bus, or both, may be divided into two or more parts to provide multiple buses with reduced widths). In some embodiments, the memory controller 206 is transactional, i.e., instead of guaranteeing that the result of any memory access will return at a specific time, the host processing circuit 205 may wait until the memory controller 206 indicates that the requested data is ready. Instead of a cache hierarchy, the host processing circuit 205 may only have registers (registers may not require cache coherence). In some embodiments, the host processing circuit 205 is connected to more than one memory (e.g., connected to a first memory that includes in-memory functional circuitry 110 and has no cache, and a second memory that lacks in-memory functional circuitry but has a cache).

[0044] During operation, the host processing circuit 205 can first write the operation value into the memory. For example... Figure 1A As shown, this could involve broadcasting the value to multiple memory banks (e.g., memory bank 105). Such broadcasting reduces the number of write loops used when operands are reused multiple times (e.g., in matrix multiplication, each row of the first matrix can be multiplied by each column of the second matrix). The host processing circuit 205 can then send the address of the operand to memory (causing the contents of the addressed memory location to be read into a global input / output (global I / O, GIO) register) and pass the instruction (e.g., an identifier) ​​to memory. Figure 1C The instruction in the table (a number between 0 and 9) is sent to the functional circuit 110 in the memory to cause the processing to be executed in the memory.

[0045] For example, as described below, the memory-in-processing circuit 110 can perform multiplication of the first operand and the second operand, and return the product to the host processing circuit 205. The host processing circuit 205 can send the address of the first operand to the memory (causing the first operand to be read into the global I / O register), and set the number 0 (identifying...) Figure 1CInstruction 0 from the table is sent to the in-memory functional circuit 110. Then, the in-memory functional circuit 110 can store the first operand in the Rop register upon receiving instruction 0 (e.g., copy the first operand from the global I / O register to the Rop register). Then, the host processing circuit 205 can send the address of the second operand to memory (causing the second operand to be read into the global I / O register) and set the number 6 (identifying...) to... Figure 1C Instruction 6 from the table is sent to the in-memory functional circuit 110. Then, upon receiving instruction 6, the in-memory functional circuit 110 can calculate the product of the two operands (the first operand is in the Rop register, and the second operand is in the global I / O register) (in this case, "op" is multiplication) and store the product in register Rz. Finally, the host processing circuit 205 can convert the number 5 (identifying...) into... Figure 1C Instruction 5) in the table is sent to the functional circuit 110 in memory, so that the product (stored in the Rz register) is written to the DQ output (i.e., returned to the host processing circuit 205).

[0046] As another example, the in-memory functional circuit 110 can perform multiplication of the first operand and the second operand, and store the product in memory by following the same sequence of steps, except that the final instruction can be instruction number 3 (identifying...). Figure 1C Instruction 3 in the table causes the product to be written back to memory by the host processing circuit 205 and simultaneously sent to the address specified in the memory (instead of returning to the host processing circuit 205).

[0047] Figure 2A and Figure 2B Two configurations of implementing the in-memory function circuitry 110 using standard dynamic random access memory (DRAM) chips (i.e., without modifying the DRAM chips for use with the in-memory function circuitry 110) are shown. Although such configurations may be referred to as “function near memory” in some contexts, as used herein, the term “in-memory function” includes configurations where the in-memory function circuitry 110 is located on a semiconductor chip separate from the memory (e.g., Figure 2A and 2B (Configuration). Figure 2A In some embodiments, multiple (e.g., two) DIMM modules share a channel to host processing circuitry 205 (which includes a CPU and a memory controller (MC)). Each DIMM module includes in-memory functional circuitry 110 (or a "FIM module"). The DIMM modules may be low-load DIMM (LR-DIMM) modules to facilitate channel sharing. Figure 2BIn one embodiment, each of the multiple rows of the memory module is associated with a corresponding in-memory functional circuit 110. Figure 2A and Figure 2B Each FIM module may include a controller 210 and an intermediate buffer 215. Figure 1B The Rop register can be an example of intermediate buffer 215, FIM logic 220, and memory-mapped register 225. Figure 2B The memory can be M.2 or DIMM form factor. Figure 2B In this configuration, the in-memory functional circuitry 110 can be fabricated on a buffer chip, which can be the chip that primarily performs retiming in a DIMM without in-memory functional circuitry.

[0048] Figure 2C and Figure 2D Two different configurations are shown, in which the in-memory functional circuitry 110 is located on the same chip as the DRAM (e.g., fabricated on the same silicon chip). Figure 2C In one embodiment, each chip 230 includes in-memory functional circuitry 110. Figure 2C The configuration does not affect the DRAM cores, and partly for this reason, Figure 2C The configuration is comparable Figure 2D The configuration is easier to implement. Furthermore, in Figure 2C In the configuration, wiring (in Figure 2D Using a limited number of metal layers to implement wiring in a configuration can be challenging; it may be simpler. Figure 2C The configuration is logically similar to Figure 2B In each of these two configurations, multiple DRAM banks are connected to and used by the in-memory functional circuitry 110. (Compared to a configuration where the in-memory functional circuitry 110 is fabricated on a buffer chip.) Figure 2C and Figure 2D This configuration reduces the complexity of the buffer chip. Figure 2C and Figure 2D In this embodiment, each chip 230 may be only slightly larger than a standard memory chip, and because there is no separate chip for the in-memory functional circuitry 110, it is comparable to... Figure 2A and Figure 2B Compared to embodiments where the in-memory functional circuitry 110 is located on a separate chip from the DRAM, chip 230 can be more easily accommodated in a standard form factor (e.g., on a DIMM), thus allowing the chips (DRAM chip and chip including the in-memory functional circuitry 110) to occupy more board space. Figure 2CIn this embodiment, each in-memory functional circuit 110 accesses only one memory chip, and cache lines can be entirely within a single chip 230 (i.e., data can be avoided by not striping across multiple chips 230, which would make it difficult for the in-memory functional circuit 110 to perform useful operations). As used herein, "cache line" refers to the granularity of the host processing circuit 205's access to memory (i.e., reading from and writing to memory). For example, for a CPU, a cache line could be 64 bytes, and for a GPU, a cache line could be 128 bytes.

[0049] exist Figure 2D In one embodiment, each memory bank 105 has an associated in-memory functional circuitry 110, such that each chip 230 includes a plurality (e.g., 16) of in-memory functional circuitry 110. Figure 2D Implementation examples may include more Figure 2C The embodiments have a greater number of in-memory functional circuits 110, and therefore can exhibit more than Figure 2C The implementation offers better performance. Changes to the I / O path for each memory bank (such as...) Figure 1B As shown, Figure 1B It also shows a configuration where each memory bank has an in-memory functional circuit 110, which can consume, for example... Figure 2C The embodiments of the present invention have a larger chip area and the design complexity can be greater than, for example, Figure 2C The complexity of the design of this embodiment stems in part from the challenge of routing signals using a limited number of metal layers. Figure 2D In this embodiment, because too few DRAM control bits are available to make independent address selection feasible, the in-memory functional circuitry 110 in each memory bank 105 can operate at the same address at any time.

[0050] Data flow between functional circuits 110 within the memory can occur in various ways. In some embodiments, such as Figure 3A As shown, the in-memory functional circuitry 110 and its associated memory portion can be configured as a single-instruction multiple-data (SIMD) parallel processor. Each of the in-memory functional circuits 110 can execute the same instructions as the other in-memory functional circuits 110 at any given time using one or more different operands. After each operation, as described above... Figure 1B As discussed in the context, the result of the operation can be returned to the host processing circuit 205 or stored in memory.

[0051] In some embodiments, such as Figure 3BAs shown, in-memory functional circuits 110a, 110b, and 110c (collectively referred to as in-memory functional circuit 110) and their associated memory portions can be configured as a systolic array, which can represent a homogeneous network of tightly coupled data processing circuits. In such an embodiment, the result of each operation of the first in-memory functional circuit 110a can be passed as an argument to a subsequent second in-memory functional circuit 110b in the network. In some embodiments, such as Figure 3B As shown, each memory bank group can be connected to a corresponding chain of in-memory functional circuitry 110, and there are no connections between the multiple chains. Although the logic for communication between the connected multiple in-memory functional circuits 110 may not exist, the data paths between memory banks 105 within each memory bank group may already exist in a standard memory architecture (e.g., DIMM or HBM); if to use Figure 3B In this configuration, such logic can be added. This logic may include additional conductors between the connected in-memory functional circuits 110, for example, via the first in-memory functional circuit 110a, to notify its downstream neighbor (the second in-memory functional circuit 110b) that data on the common bus is available for the downstream neighbor. The in-memory functional circuits 110 can be connected to the common bus, and only one in-memory functional circuit 110 can drive the bus at a time. In this way, communication between the in-memory functional circuits 110 can be implemented using appropriate logic and arbitration, while avoiding bus contention. Figure 3B The implementation examples may not be well-suited for some calculations. However, for Figure 3B The calculations suitable for the embodiments, Figure 3B The embodiments may have the following advantages: the host processing circuit 205 does not handle intermediate results, while if... Figure 3A In alternative embodiments, similar calculations are performed, in which case the host processing circuitry 205 handles the intermediate results. In some embodiments, a method based on... Figure 3A Or according to Figure 3B The system executes either Level 1 (BLAS1) or Level 2 (BLAS2) of the Basic Linear Algebra Subroutines (BLAS) or the General Matrix Multiplication (GEMM) (GEMM can be part of BLAS3), or executes a portion of Level 1 (BLAS1) or Level 2 (BLAS2) of the Basic Linear Algebra Subroutines (BLAS) or the General Matrix Multiplication (GEMM). To perform GEMM computations, the system can select the order in which the loops are executed to maximize parallelism. Figure 3A Or according to Figure 3B The system can also perform operations on transpose operands (for example, it can calculate matrix products AB, A). T B or AB TThis eliminates the need for the host processing circuit 205 to first reorder the data in the memory.

[0052] Figure 4A A system for computation is shown in some embodiments. The system for computation includes a CPU 405 (which can operate as a host processing circuit), which is connected via a switch 410 to multiple (e.g., two) hardware accelerators 415 and a system 420 for performing in-memory functional computations. Each of the hardware accelerators 415 is connected to a corresponding memory 425 (e.g., low-power DDR5 memory) and may include a GPU, CPU, application-specific integrated circuit (ASIC), or field-programmable gate array (FPGA). Each hardware accelerator 415 may be configured to receive computational tasks delegated to it by the CPU 405, perform the delegated computational tasks, and (i) return the results to the CPU 405 or (ii) store the results for additional subsequent processing or for the CPU 405 to retrieve later. Similarly, the system 420 for performing in-memory functional computations (hereinafter referred to as...) Figure 4B (Discussed in further detail in the context of this document) can be configured to receive computational tasks delegated to it by CPU 405, execute the delegated computational tasks, and (i) return the results to CPU 405 or (ii) store the results for additional subsequent processing or for later retrieval by CPU 405. As used herein, a “computational task” is any set of operations to be performed by the computer; thus, a computational task may consist of or include multiple smaller computational tasks.

[0053] The system for computation may include a network interface card (NIC) 445 connected to the CPU 405, enabling computational tasks to be delegated to the system for computation by, for example, another processing system via remote procedure calls (RPC). The CPU 405 may include one or more (e.g., two) memory controllers 430, each connected to a corresponding memory (e.g., DDR5 memory) 435. The CPU 405 may also include a PCIe root complex 440 (e.g., a PCIe 5 root complex as shown), and the root port of the PCIe root complex 440 may be connected to a switch 410. The switch 410 may be configured to exchange PCIe packets, and the switch 410 may be aware of CXL to be able to handle CXL packet sizes and formats (which may differ from conventional PCIe), and enable the switch 410 to perform routing and forwarding based on 64-byte packets. The switch 410 may be compatible with the PCIe version (e.g., PCIe 5) used by the PCIe root complex 440 of the CPU 405.

[0054] Communication between CPU 405 and hardware accelerator 415 via switch 410, and with system 420 for performing in-memory functional computation, follows the Compute Fast Link (CXL) protocol. CXL is an open interconnect standard for high-speed CPU-to-device and CPU-to-memory connections. CXL is a layer on top of the PCIe protocol, meaning a CXL packet can be a PCIe packet. In some embodiments, CXL is a transaction protocol overlaid on the PCIe electrical PHY layer. Connecting hardware accelerator 415 and system 420 for performing in-memory functional computation to CPU 405 using the CXL interface (via switch 410) offers the following advantages: CPU 405 can maintain a cached copy of the memory regions in the CXL device, enabling fine-grained sharing between CPU 405 and accelerator 415. On the other hand, the accelerator can also access the host cache region, helping it complete processing faster. PCIe has variable latency, so this concept facilitates memory acceleration with indeterminate latency while ensuring that CPU 405 can still use memory accelerator 415 as a conventional memory device. Furthermore, by delegating computational tasks to hardware accelerators 415 and system 420 for performing in-memory functional computations via CPU 405, cache coherency can remain unaffected.

[0055] Figure 4B A system 420 for performing in-memory functional computations is shown in some embodiments. The system 420 for performing in-memory functional computations may include a near-data accelerator or “CXL interface circuitry” 445, and a memory module 450 including in-memory functional circuitry 110 (e.g., according to…). Figures 2A to 2D (One of the embodiments shown). The CXL interface circuit 445 may have a first interface 455 (e.g., a DDR2, DDR3, DDR4, DDR5, GDDR, HBM, or LPDDR interface) for communicating with the memory module 450, and a second interface (e.g., a CXL interface on a bus interface (such as a PCIe endpoint interface)) 460 for communicating with the CPU 405 (e.g., via a switch 410).

[0056] CXL interface circuitry 445 can be used as an interface adapter circuit between CXL interface 460 and first interface 455, enabling CPU 405 to delegate computational tasks to in-memory functional circuitry 110 of memory module 450. In some embodiments, control flow can be executed first. Instruction streams can be written to a contiguous portion of memory and a start pointer, then the size can be written to a register, and a doorbell can be rung (i.e., an interrupt register is set). The device recognizes the instruction stream and acknowledges it after ensuring data integrity using CRC. The device can then operate on the instructions and memory regions while continuing to provide processor responses for regular memory instructions. Processing engine 470 provides all auxiliary functions (such as DMA, power management, JTAG, and driver API). DMA enables the device to communicate with other I / O devices in the system (such as a network card or GPU or another in-memory processing device). Once the operation is complete, the doorbell register that the CPU is waiting for is set (using interrupts or polling). The CPU then reads back the result and acknowledges receipt. The continuous instruction stream from the CPU is pipelined and has priorities attached to them to aid in efficient execution on the in-memory processing unit.

[0057] The CXL interface circuit 445 can also be used as an accelerator (e.g., to perform computational tasks (e.g., arithmetic operations) where a portion of the computational task delegated to it by the CPU 405 (by the CPU 405 or by the CXL interface circuit 445) is further delegated to the in-memory functional circuitry 110 of the memory module 450, or where a portion of the computational task delegated to it by the CPU 405 (by the CPU 405 or by the CXL interface circuit 445) is not further delegated to the in-memory functional circuitry 110 of the memory module 450). To achieve such operation, the CXL interface circuit 445 may also include a processing core 465 and a processing engine 470, which may be designed to perform specific computational tasks (e.g., BLAS1 or BLAS2 operations) that may be well-suited to being delegated by the CPU 405 (e.g., because the CPU may be relatively less suited to performing BLAS1 or BLAS2 operations). High-speed interconnect 475 connects processing core 465 and processing engine 470 to host manager 480, SRAM controller 485 (connected to static random access memory (SRAM) module 486), and DRAM controller 490. The SRAM controller can also be modified to issue cache snoop requests to the host (which is CXL enabled). It can also respond to host snoops (i.e., requests from the CPU to invalidate rows, requiring the host to migrate the rows to itself and modify them, i.e., M state or S state). Host manager 480 implements the CXL stack (including the PCIe stack). In some embodiments, the CXL stack is responsible for decoding CXL type 2 or type 3 memory and accelerator transaction requests. At the link layer, it implements link protection and flow control. At the PHY layer, it is similar to PCIe 5.0. In some embodiments, the circuitry enabling CXL interface circuitry 445 to function as an accelerator is absent, and the circuitry enabling CXL interface circuitry 445 to function as an accelerator is only used as an interface circuitry to memory module 450 and the in-memory functional circuitry 110 included in memory module 450.

[0058] Figure 5 A flowchart of a method for computation in some embodiments is shown. The method includes: at 510, sending a plurality of CXL packets to a CXL interface circuit via host processing circuitry; and at 520, in response to receiving a CXL packet, sending an instruction via the CXL interface circuitry to in-memory functional circuitry (connected to a memory of the CXL interface circuitry). The method further includes: at 530, performing an operation on operands according to the instruction via the in-memory functional circuitry to form a result, the operands including a first operand retrieved from memory.

[0059] As used herein, the term "or" should be interpreted as "and / or," such that, for example, "A or B" means either "A" or "B" or "A and B." It will be understood that although the terms "first," "second," "third," etc., may be used herein to describe various elements, components, regions, layers, and / or portions, these elements, components, regions, layers, and / or portions should not be limited by these terms. These terms are used only to distinguish one element, component, region, layer, or portion from another. Therefore, without departing from the spirit and scope of the inventive concept, the first element, first component, first region, first layer, or first portion discussed herein may be referred to as a second element, second component, second region, second layer, or second portion.

[0060] (For example, in any system diagram included herein) any component or combination of components described may be used to perform one or more of the operations in any flowchart included herein. Furthermore, (i) the operations are example operations and may involve various additional steps not explicitly included, and (ii) the temporal order of the operations may be changed.

[0061] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the inventive concept. As used herein, the terms “substantially,” “approximately,” and similar terms are used as approximate terms rather than terms of degree and are intended to take into account the inherent deviations of measurements or calculations that will be recognized by one of ordinary skill in the art.

[0062] As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of the stated features, integrals, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. When expressions such as “at least one of…” follow a list of elements, they modify the entire list of elements, not individual elements of the list. Furthermore, the word “may” is used when describing embodiments of the inventive concept to mean “one or more embodiments of this disclosure.” Additionally, the term “exemplary” is intended to indicate an example or illustration. As used herein, the term “use” and variations thereof may be considered synonymous with the term “utilize” and variations thereof.

[0063] It will be understood that when an element or layer is referred to as being "on" another element or layer, "connected to", "bonded to", or "adjacent to" another element or layer, the element or layer may be directly on, connected to, bonded to, or adjacent to the other element or layer, or one or more intermediate elements or layers may be present. Conversely, when an element or layer is referred to as being "directly on" another element or layer, "directly connected to", "directly bonded to", or "immediately adjacent to" another element or layer, no intermediate elements or layers are present.

[0064] Any numerical range listed herein is intended to include all subranges with the same numerical precision included within the listed range. For example, the range “1.0 to 10.0” or “between 1.0 and 10.0” is intended to include all subranges between the listed minimum value 1.0 and the listed maximum value 10.0 (and includes both the listed minimum value 1.0 and the listed maximum value 10.0) (i.e., a minimum value equal to or greater than 1.0 and a maximum value equal to or less than 10.0, for example, 2.4 to 7.6). Any maximum numerical limit listed herein is intended to include all lower numerical limits included therein, and any minimum numerical limit listed in this specification is intended to include all higher numerical limits included therein.

[0065] The term "processing circuitry" is used herein to refer to any combination of hardware, firmware, and software for processing data or digital signals. Processing circuitry hardware may include, for example, application-specific integrated circuits (ASICs), general-purpose or special-purpose central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), and programmable logic devices (such as field-programmable gate arrays (FPGAs)). In processing circuitry, as used herein, each function is performed by hardware configured (i.e., hardwired) to perform said function, or by more general-purpose hardware (such as a CPU) configured to execute instructions stored in a non-transitory storage medium. Processing circuitry may be fabricated on a single printed circuit board (PCB) or distributed across multiple interconnected PCBs. Processing circuitry may include other processing circuitry; for example, processing circuitry may include two processing circuits (FPGA and CPU) interconnected on a PCB.

[0066] Although exemplary embodiments of systems and methods for performing in-memory functional computations have been specifically described and illustrated herein, many modifications and variations will be apparent to those skilled in the art. Therefore, it will be understood that systems and methods for performing in-memory functional computations constructed in accordance with the principles of this disclosure may be embodied in ways different from those specifically described herein. The invention is further defined in the claims and their equivalents.

Claims

1. A system for computation, the system comprising: Host processing circuitry; Memory, which includes one or more internal functional circuits; as well as The cache coherence protocol interface circuit has a first interface for communicating with the memory and a second interface for communicating with the host processing circuit via a switch. The in-memory functional circuit of the one or more in-memory functional circuits is configured to perform operations on operands including a first operand retrieved from memory to form a result. The first interface is connected to the memory, and the second interface is connected to the host processing circuitry. The second interface is configured as a cache coherence protocol interface on the bus interface, and the bus interface is a Peripheral Component Interconnect Fast PCIe endpoint interface. Among them, the cache coherence protocol interface circuit is the compute fast link (CXL) interface circuit. The cache coherence protocol interface circuit serves as an interface adapter circuit between the second interface and the first interface, enabling the host processing circuit to delegate computing tasks to one or more memory-in-memory functional circuits of the memory. The switch is configured to exchange PCIe packets and is aware of CXL in order to be able to handle the packet size and format of CXL.

2. The system according to claim 1, wherein, The functional circuitry within the memory is arranged in a single instruction multiple data configuration; or The functional circuitry within the memory is arranged in a pulsating configuration.

3. The system according to claim 1, wherein, The in-memory functional circuits of the one or more memory functional circuits are located on a semiconductor chip having dynamic random access memory.

4. The system according to claim 1, wherein, The first interface is configured to operate according to a protocol selected from a group consisting of DDR2, DDR3, DDR4 and DDR5.

5. The system according to any one of claims 1 to 4, wherein, The in-memory functional circuits of the one or more in-memory functional circuits include: Multiple registers, Multiple multiplexers, and Arithmetic logic unit.

6. The system according to any one of claims 1 to 4, wherein, The memory-in-memory functional circuits in the one or more memory-in-memory functional circuits are configured to perform arithmetic operations selected from a group consisting of addition, subtraction, multiplication, and division.

7. The system according to any one of claims 1 to 4, wherein, The in-memory functional circuit of the one or more in-memory functional circuits is configured to perform arithmetic operations selected from a group consisting of floating-point addition, floating-point subtraction, floating-point multiplication and floating-point division.

8. The system according to any one of claims 1 to 4, wherein, The memory-in-memory functional circuits in the one or more memory-in-memory functional circuits are configured to perform logical operations selected from a group consisting of bitwise AND, bitwise OR, bitwise XOR, and bitwise complement.

9. The system according to any one of claims 1 to 4, wherein, The memory-in-memory functional circuit of the one or more memory-in-memory functional circuits is configured to: store the result in memory in a first state, and send the result to the cache coherence protocol interface circuit in a second state.

10. The system according to claim 1, wherein, The host processing circuitry includes a PCIe root complex with a root port connected to the second interface.

11. A system for computation, the system comprising: Host processing circuitry; Memory, including one or more functional circuits within the memory; as well as The cache coherence protocol interface circuit has a first interface for communicating with the memory and a second interface for communicating with the host processing circuit via a switch. The cache coherence protocol interface circuit is configured to perform arithmetic operations on the data stored in memory. The first interface is connected to the memory, and the second interface is connected to the host processing circuitry. The second interface is configured as a cache coherence protocol interface on the bus interface, and the bus interface is a peripheral component interconnect (PCIe) endpoint interface. Among them, the cache coherence protocol interface circuit is the compute fast link (CXL) interface circuit. The cache coherence protocol interface circuit serves as an interface adapter circuit between the second interface and the first interface, enabling the host processing circuit to delegate computing tasks to one or more memory-in-memory functional circuits of the memory. The switch is configured to exchange PCIe packets and is aware of CXL in order to be able to handle the packet size and format of CXL.

12. The system according to claim 11, wherein, The memory-in-memory functional circuit of the one or more memory-in-memory functional circuits is configured to perform operations on operands including a first operand obtained from memory to form a result.

13. The system according to claim 12, wherein, The functional circuitry within the memory is arranged in a single instruction multiple data configuration; or The functional circuitry within the memory is arranged in a pulsating configuration.

14. The system according to claim 12, wherein, The in-memory functional circuits of the one or more memory functional circuits are located on a semiconductor chip having dynamic random access memory.

15. The system according to any one of claims 12 to 14, wherein, The first interface is configured to operate according to a protocol selected from a group consisting of DDR2, DDR3, DDR4, DDR5, GDDR, HBM, and LPDDR.

16. The system according to any one of claims 12 to 14, wherein, The memory-in-memory functional circuit of the one or more memory-in-memory functional circuits is configured to: store the result in memory in a first state, and send the result to the cache coherence protocol interface circuit in a second state.

17. A method for calculation, the method comprising: One or more CXL packets are sent from the host processing circuitry to the compute fast link CXL interface circuitry. In response to receiving one or more CXL packets, an instruction is sent to the memory-in-memory functional circuit in the memory connected to the CXL interface circuit via the CXL interface circuit; as well as The memory's functional circuitry performs operations on operands, including a first operand retrieved from memory, according to instructions to form a result. The CXL interface circuit has a first interface for communicating with the memory and a second interface for communicating with the host processing circuit via a switch. The second interface is configured as a cache coherence protocol interface on the bus interface, and the bus interface is a peripheral component interconnect (PCIe) endpoint interface. The CXL interface circuit serves as an interface adapter circuit between the second interface and the first interface, enabling the host processing circuit to delegate computing tasks to the memory-in-memory functional circuitry of the memory. The switch is configured to exchange PCIe packets and is aware of CXL in order to handle the packet size and format of CXL.

Images