Method for in-memory computation and system for computation

Abstract

A method for in-memory computing and a system for computing are provided. In some embodiments, the method includes: executing, by a first in-memory function circuit, a first instruction to produce a resulting first value, wherein a first computing task includes a second computing task and a third computing task, the second computing task including the first instruction; storing, by the first in-memory function circuit, the first value in a first buffer; reading, by a second in-memory function circuit, the first value from the first buffer; and executing, by the second in-memory function circuit, a second instruction, the second instruction using the first value as an argument, the third computing task including the second instruction, wherein: storing, by the first in-memory function circuit, the first value in the first buffer includes storing the first value directly in the first buffer.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and interest in U.S. Provisional Application No. 62 / 986,326, filed March 6, 2020, and U.S. Application No. 16 / 914,151, filed June 26, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] One or more aspects of embodiments of this disclosure relate to data processing, and more specifically, to adjustable in-memory computation. Background Technology

[0004] This background section is for setting the context only and is not intended to limit the scope of the invention, nor does it constitute any endorsement of the prior art. In-memory computation (“in-memory computation” or “in-memory compute”) can be used to reduce performance degradation caused by memory access bottlenecks. However, efficient use of in-memory computation may involve efficiently passing intermediate results between function-in-memory circuits and may involve using in-memory function circuits specifically adapted to the computational task to be performed.

[0005] Therefore, there is a need for improved systems and methods for in-memory computing. Summary of the Invention

[0006] In some embodiments, using a producer-consumer model can improve the efficiency of in-memory function circuits performing computations with multiple stages. For example, if the product of three matrices is to be calculated, a producer-consumer model can be used to allow the overall product to be calculated before the entire product of the first two factors is completed.

[0007] Depending on the specific circumstances, different methods of performing a particular computation can be advantageous. In some embodiments, different "islands" in memory can contain in-memory function circuits with different capabilities, and the scheduler can delegate different computational tasks to the appropriate islands by host processing circuitry connected to the islands, based on the nature of the computational task and the capabilities of the in-memory function circuits in each island.

[0008] According to an embodiment of the present invention, a method for in-memory computation is provided, the method comprising: executing a first instruction by a first in-memory function circuit to generate a first value as a result, wherein the first computation task includes a second computation task and a third computation task, the second computation task including the first instruction; storing the first value in a first buffer by the first in-memory function circuit; reading the first value from the first buffer by a second in-memory function circuit; and executing a second instruction by the second in-memory function circuit, the second instruction using the first value as an argument, the third computation task including the second instruction, wherein: storing the first value in the first buffer by the first in-memory function circuit includes directly storing the first value in the first buffer, reading the first value from the first buffer by the second in-memory function circuit includes directly reading the first value from the first buffer, the first in-memory function circuit being in a first amount of memory; the second in-memory function circuit being in a first amount of memory; and the first buffer being in a first amount of memory.

[0009] In some embodiments, the method further includes adjusting a pointer by a first memory function circuit that identifies a boundary of a first portion of a first buffer in the memory of the first amount, the adjustment including moving the boundary, wherein the movement of the boundary has the effect of including the first value within the first portion.

[0010] In some embodiments, the method further includes: executing a third instruction by a first in-memory function circuit to produce a second value as a result, the second computation task including the third instruction; and executing a fourth instruction by a second in-memory function circuit, the fourth instruction using the second value as an argument, the third computation task including the fourth instruction.

[0011] In some embodiments: a first computational task includes calculating the product of a first matrix, a second matrix, and a third matrix; a second computational task includes calculating the product of the first matrix and the second matrix to form a first product matrix; and a third computational task includes calculating the product of the first product matrix and the third matrix.

[0012] In some embodiments, the method further includes executing a third instruction by a third in-memory function circuit, wherein: the fourth computation task includes the third instruction, the first in-memory function circuit is in a first amount of memory, the third in-memory function circuit is in a second amount of memory, and the third in-memory function circuit has capabilities different from those of the first in-memory function circuit.

[0013] In some embodiments, the fourth computational task includes: calculating the product of a first matrix stored in a second amount of memory and a second matrix stored in a second amount of memory; and calculating the product of a third matrix and a fourth matrix such that the storage order of the elements of the stored first matrix and the second matrix remains unchanged, wherein the third matrix is ​​a transpose of the first matrix, or the fourth matrix is ​​a transpose of the second matrix.

[0014] In some embodiments: the first computational task includes calculating the product of the fifth matrix, the sixth matrix, and the seventh matrix; the second computational task includes calculating the product of the fifth matrix and the sixth matrix to form a first product matrix; and the third computational task includes calculating the product of the first product matrix and the seventh matrix.

[0015] In some embodiments, the first amount of memory is substantially isolated from the second amount of memory.

[0016] In some embodiments, calculating the product of the fifth matrix and the sixth matrix includes calculating the product of submatrices of the fifth matrix and submatrices of the sixth matrix.

[0017] In some embodiments, calculating the product of the fifth and sixth matrices includes calculating the dot product of the rows of the first matrix and the columns of the second matrix.

[0018] In some embodiments, calculating the product of the fifth and sixth matrices includes calculating the outer product of the columns of the first matrix and the rows of the second matrix.

[0019] In some embodiments, the second computational task includes computing a function selected from a group consisting of the sigmoid function, the rectified linear unit function, the hyperbolic tangent function, and combinations thereof.

[0020] In some embodiments, the third computational task includes calculating the product of the fifth and sixth matrices, where the result of the function is an element of the fifth matrix.

[0021] According to an embodiment of the present invention, a system for computation is provided, the system comprising: a host processing circuit; and a first memory including a first in-memory function circuit, a second in-memory function circuit, and a first buffer, the host processing circuit being configured to program the first in-memory function circuit and the second in-memory function circuit such that: the first in-memory function circuit: executes a first instruction to produce a first value as a result, wherein a first computation task includes a second computation task and a third computation task, the second computation task including the first instruction; and stores the first value in the first buffer; and the second in-memory function circuit: reads the first value from the first buffer; and executes a second instruction, the second instruction using the first value as an argument, the third computation task including the second instruction, wherein: storing the first value in the first buffer by the first in-memory function circuit includes storing the first value directly in the first buffer, and reading the first value from the first buffer by the second in-memory function circuit includes reading the first value directly from the first buffer.

[0022] In some embodiments, the programming further causes a first memory-in-memory function circuit to adjust a pointer that identifies a boundary of a first portion of a first buffer in the memory of the first amount, the adjustment including moving the boundary, wherein the movement of the boundary has the effect of including the first value within the first portion.

[0023] In some embodiments, the programming further causes: after executing the second instruction, the first memory-in-memory function circuit executes a third instruction to produce a second value as a result, the second computation task including the third instruction; and after executing the third instruction, the second memory-in-memory function circuit executes a fourth instruction that uses the second value as an argument, the third computation task including the fourth instruction.

[0024] In some embodiments: a first computational task includes calculating the product of a first matrix, a second matrix, and a third matrix; a second computational task includes calculating the product of the first matrix and the second matrix to form a first product matrix; and a third computational task includes calculating the product of the first product matrix and the third matrix.

[0025] In some embodiments, the host processing circuit is further configured to program a third memory-in-memory function circuit, the programming of the third memory-in-memory function circuit causing the third memory-in-memory function circuit to execute a third instruction, wherein: the fourth computation task includes the third instruction, the third memory-in-memory function circuit is in a second amount of memory, and the third memory-in-memory function circuit has capabilities different from those of the first memory-in-memory function circuit.

[0026] In some embodiments, the fourth computational task includes: calculating the product of a first matrix stored in a second amount of memory and a second matrix stored in a second amount of memory; and calculating the product of a third matrix and a fourth matrix such that the storage order of the elements of the stored first and second matrices remains unchanged, wherein the third matrix is ​​a transpose of the first matrix, or the fourth matrix is ​​a transpose of the second matrix.

[0027] According to an embodiment of the present invention, a system for computation is provided, the system comprising: means for processing; and a first memory including a first in-memory function circuit, a second in-memory function circuit, and a first buffer, the means for processing being configured to program the first in-memory function circuit and the second in-memory function circuit such that: the first in-memory function circuit: executes a first instruction to produce a first value as a result, wherein a first computation task includes a second computation task and a third computation task, the second computation task including the first instruction; and stores the first value in the first buffer; and the second in-memory function circuit: reads the first value from the first buffer; and executes a second instruction, the second instruction using the first value as an argument, the third computation task including the second instruction, wherein: storing the first value in the first buffer by the first in-memory function circuit includes storing the first value directly in the first buffer, and reading the first value from the first buffer by the second in-memory function circuit includes reading the first value directly from the first buffer. Attached Figure Description

[0028] It should be understood that the accompanying drawings are merely examples and other embodiments not explicitly shown can be constructed. These and other features and advantages of this disclosure will be understood and appreciated by referring to the specification, claims, and drawings, in which:

[0029] Figure 1A A block diagram of a memory including multiple banks according to an embodiment of the present disclosure is shown, each bank including multiple banks, and the memory also including multiple in-memory function circuits;

[0030] Figure 1B A block diagram of a memory bank having multiple sense amplifiers, global input / output registers and in-memory function circuitry according to an embodiment of the present disclosure is shown.

[0031] Figure 1C A table of instructions executed by in-memory function circuitry according to embodiments of the present disclosure is shown;

[0032] Figure 2A A block diagram of two in-memory function circuits in a producer-consumer configuration according to an embodiment of the present disclosure is shown;

[0033] Figure 2BA flowchart of a producer-consumer operation according to an embodiment of the present disclosure is shown;

[0034] Figure 3A A general data layout diagram is shown for the operation of portions of two matrices according to embodiments of the present disclosure;

[0035] Figure 3B A data layout diagram is shown for a dot product operation involving portions of two matrices according to an embodiment of the present disclosure;

[0036] Figure 3C A data layout diagram is shown that involves a partial outer product operation of two matrices according to an embodiment of the present disclosure;

[0037] Figure 3D A data layout diagram is shown representing a tensor product operation involving portions of two matrices according to an embodiment of this disclosure; and

[0038] Figure 3E A data layout diagram of an element-wise vector product operation involving portions of two matrices according to an embodiment of the present disclosure is shown. Detailed Implementation

[0039] The detailed description set forth below with reference to the accompanying drawings is intended as a description of exemplary embodiments of an adaptable in-memory function system provided according to this disclosure, and is not intended to represent the only form in which this disclosure can be constructed or utilized. This description illustrates features of this disclosure in conjunction with the illustrated embodiments. However, it should be understood that identical or equivalent functions and structures can be implemented by different embodiments, also intended to be included within the scope of this disclosure. As shown elsewhere herein, similar element designations are intended to denote similar elements or features.

[0040] In some embodiments, using a producer-consumer model can improve the efficiency of function-in-memory (FIM) circuits performing computations with multiple stages. Generally, a producer-consumer model can be used when there is an asynchrony between the data source and the data sink. The producer keeps pushing partially ready input without waiting for the consumer's confirmation, and the consumer begins processing anything in the queue. The queue has start and end pointers to keep track of efficient data. Benefits include better latency and avoidance of needing to block fully ready input (as long as they can be processed independently). For example, such a model can be used if the product of three matrices is to be computed (“calculate” or “compute”). A first set of one or more FIM circuits can compute the product of a first matrix and a second matrix of the three matrices to form a first product matrix, and a second set of one or more FIM circuits can compute the product of the first product matrix and a third matrix of the three matrices. In some embodiments, the elements of the first product matrix can be computed one at a time. As used herein, the terms “in-memory function” and “in-memory computation” are used interchangeably.

[0041] The use of the producer-consumer model may make it unnecessary to complete the calculation of the first product matrix before starting to calculate the product of the first product matrix and the third matrix of the three matrices. Instead of waiting until all elements of the first product matrix have been calculated, a first set of one or more in-memory function circuits can make the elements of the first product matrix available to a second set of one or more in-memory function circuits when they are being calculated. The second set of one or more in-memory function circuits can then proceed to partial calculations of the product of the first product matrix and the third matrix of the three matrices, provided that sufficient data is available for each such partial calculation. For example, if the second set of one or more in-memory function circuits calculates the product of the first product matrix and the third matrix as multiple dot products of the rows of the first product matrix and the columns of the third matrix, then the second set of one or more in-memory function circuits can begin using each row of the first product matrix when it becomes available.

[0042] Depending on the specific circumstances, different methods of performing particular calculations may be advantageous. For example, the matrix product of the first and second matrices can be computed as the set of dot products of the rows of the first matrix and the columns of the second matrix. This method may require relatively little register space because each such dot product produces a single number. Alternatively, the matrix product can be computed by forming the product of a submatrix of the first matrix and a submatrix of the second matrix. This method may require more register space, but its advantage is that it allows for the efficient computation of other products (such as the product of the transpose of any matrix and another matrix) without reordering the data in memory.

[0043] In some cases, there are multiple options for performing matrix multiplication based on trade-offs between data locality, available buffer space, and the layout of the input matrix in memory (e.g., row-major or column-major). In some embodiments, the system re-evaluates these trade-offs because the data may be arranged within main memory without a suitable cache.

[0044] In some embodiments, different “islands” in memory may contain in-memory function circuits with different capabilities, and the scheduler may delegate different computing tasks to the appropriate islands from the host processing circuitry connected to each island, based on the nature of the computing task and the capabilities of the in-memory function circuits in each island. As used herein, a “computing task” is any set of operations to be performed by a computer; thus, a computing task may include or consist of multiple smaller computing tasks.

[0045] refer to Figure 1A In some embodiments, the memory (e.g., high bandwidth memory (HBM) or dual in-line memory module (DIMM)) can be arranged as multiple memory bank groups (BG0, BG1, BG2, BG3), each memory bank group including multiple memory banks 105 (e.g., BG0 includes memory banks 105 labeled A, B, C and D). Figure 1ASome features (such as through-silicon vias, TSVs) are specific to HBM; other forms of memory (e.g., DIMMs) can operate in a similar manner. DRAM memory can be organized into rank, chips, and banks 105. A "rank" can be part of a memory with shared chip select pins. Each rank can include eight chips, and each chip can include 16 banks 105. The banks 105 of the chips can be organized into "megabanks," such that a collection of banks 105, for example, consisting of bank 0 from each of the eight chips in a rank, can be a megabank 0. The chips can be read in parallel onto a 256-bit wide bus, with each of the eight chips providing 32 bits of the 256 bits of data. Furthermore, Figure 1A In this context, "ROWDEC" can represent a row decoder, "COLDEC" can represent a column decoder, "WRIO" can represent write input / output, "tCCDL" can represent delay time, and "tCK" can represent clock cycle time.

[0046] The memory can be connected to and provide storage for the host processing circuitry 205 (e.g., a CPU or GPU, or a CPU or GPU with additional components such as a memory controller (MC)). In some embodiments, the host processing circuitry 205 is located on the host side of the network path (e.g., it is a host server). In an in-memory compute (IMC) system, each memory bank 105 may include an input / output sense amplifier 112 (IOSA) and in-memory function (FIM) circuitry 110 (also referred to as "in-memory compute circuitry" or "process-in-memory circuitry"). As used herein, in-memory function circuitry is processing circuitry capable of performing arithmetic or logical operations and connected to the memory more directly than the host processing circuitry 205 (and also more directly than an accelerator). As used herein, "more directly" means involving fewer intermediate components (e.g., connecting in a way that does not involve a host-processor memory interface and a narrow bus that multiplexes data from multiple memory banks and rank onto the same channel bus). For example, in a system where the memory is connected to the host processing circuit 205 via a DDR bus, the processing circuitry on the memory side of the DDR bus can be considered as in-memory function circuitry, while the processing circuitry on the host processing circuitry side of the DDR bus (e.g., the accelerator on the host processing circuitry side of the DDR bus to which the host processing circuitry 205 delegates computational tasks) is not considered as in-memory function circuitry. Figure 1B The structure of such a memory 105 in some embodiments is shown, and Figure 1C This is a table showing a list of operations that can be performed by the in-memory function circuitry 110. In some embodiments, the host processing circuitry 205 sends numbers (e.g., with...) to the in-memory function circuitry 110. Figure 1C The corresponding number between 0 and 9 in one row of the table is used to perform the corresponding operation in the memory function circuit 110. The host processing circuit 205 can send instructions to the memory function circuit 110 by reserving a bit for future use (RFU) (e.g., the RFU bit of a DDR interface).

[0047] like Figure 1BAs shown, the in-memory function circuitry 110 may include registers 115 (e.g., Rop and Rz), an arithmetic logic unit (ALU) 120, and multiplexers 125 (each in...). Figure 1B Marked as "MUX" in Chinese, they can be used together to execute instructions (e.g., Figure 1C (Instructions listed in the table). The in-memory function circuitry 110 may also include FIM logic 220, a controller 210, and a memory-mapped register 225 (discussed in further detail below). Figure 1C As shown in the table, instructions can cause the in-memory function circuit 110 to copy the contents of one register to another (e.g., instructions 0-5 and 9), or to perform an operation (“op”) on the contents of two registers and store the result in a third register (in...). Figure 1C In the case of the instruction set in the table, it is in register Rz). The operation can be an arithmetic operation (e.g., +, -, X, or / performed according to IEEE-754), or a logical operation (e.g., bitwise & (AND), | (OR), ^ (exclusive OR), or ~ (two's complement). When the instruction is Figure 1C For instructions 6, 7, and 8 in the table, a register (e.g., one of memory-mapped registers 225) can specify the operation to be performed (e.g., a specific arithmetic or logical operation). Return Figure 1B The arithmetic logic unit 120 may include a 16-lane, 16-bit floating-point (FP-16) vector unit or an 8-lane, 32-bit floating-point (FP-32) vector unit, enabling a variety of operations. Non-limiting examples may include tensor operations (e.g., dot product, outer product, ReLU (rectifier or rectifier linear unit), vsSqr (squaring the elements of a vector), and vsSQrt (taking the square root of each element of a vector). For efficient use of the in-memory function circuitry 110, data may be arranged in memory such that multiple operands are simultaneously available in an open row. As used herein, an "open row" refers to data in the readout amplifier 112 (after row activation is issued). An open row may, for example, include 8192 bits of data from which the ALU can read multiple operands (e.g., 32-bit operands).

[0048] As described above, in some embodiments, a producer-consumer model can be used to improve the efficiency of in-memory function circuitry 110 in performing computations with multiple stages. Figure 2AA block diagram of a first in-memory function circuit 110a and a second in-memory function circuit 110b in a producer-consumer configuration is shown. In this configuration, both the first in-memory function circuit 110a and the second in-memory function circuit 110b are connected to the memory locations or registers of buffer 230 and storage pointer 235. In one example, the first computation task includes the product of three matrices (a first matrix, a second matrix, and a third matrix). Computing the product of the first and second matrices can be identified as a second computation task (included in the first computation task), and calculating (i) the product of the first and second matrices (computed by the second computation task) and (ii) the product of the third matrix can be identified as a third computation task (also included in the first computation task). Figure 2B As shown, in this example, the first in-memory function circuit 110a (which can operate as a producer in this example) can execute a first instruction at 240 to produce a first value as a result. The first instruction may include (for example, consisting of) calculating the dot product of the rows of a first matrix and the columns of a second matrix, and the first value may be the value of the calculated dot product. At 245, the first in-memory function circuit 110a may store the first value in a buffer 230, such as in a first portion of a buffer used to store intermediate results generated by the first in-memory function circuit 110a (such as the product of the first and second matrices). A pointer 235 may identify the first portion of the buffer used to store such results (discussed in further detail below).

[0049] Once the first value is available in buffer 230, the second in-memory function circuit 110b (which can act as a consumer operation in this example) can read the first value from buffer 230 at 250 and execute a second instruction at 255 using the first value as an argument. The second instruction can be part of a third computational task, and it can include, for example, the dot product of (i) the first matrix and the second matrix (which may include the first value) and (ii) the columns of the third matrix. In this way, as a result of using a producer-consumer model, the second in-memory function circuit 110b may not need to wait for the entire second computational task (computing the product of the first and second matrices) to complete before starting the third computational task.

[0050] As described above, the first portion of buffer 230 can store intermediate results for use by the second in-memory function circuit 110b. The boundary of the first portion of buffer 230 can be identified by a pointer 235 pointing to buffer 230. The first portion of buffer 230 can be used to store intermediate values. The boundary of the first portion of buffer 230 is defined by pointers (one of which is defined by pointer 235). This facilitates the use of a lock (or other synchronization) in this area and ensures that only the producer can write to this area after the producer releases the lock (effectively ensuring that only the consumer can process it). After storing the first value in buffer 230, the first in-memory function circuit 110a can adjust pointer 235 such that after adjusting pointer 235, the first value (which was not in the first portion of buffer 230 before adjusting pointer 235) is in the first portion of buffer 230. The adjustment of the pointer can indicate to the second in-memory function circuit 110b that the first value is an intermediate result ready to be used by the second in-memory function circuit 110b. Therefore, the operation mode of the function circuit 110b in the second memory can be (i) to perform calculations using data in the first part of the buffer 230 that is not yet used, or (ii) if no such data is available, to wait until more data becomes available in the first part of the buffer 230 (e.g., when the boundary of the first part of the buffer 230 is moved as a result of the adjustment of the pointer 235). As used herein, the “boundary” of a part or region of memory is (i) the maximum address in the part or region of memory, or (ii) the minimum address in the part or region of memory. Thus, each part or region of memory has two boundaries.

[0051] The first in-memory function circuit 110a and the second in-memory function circuit 110b can directly access the buffers and pointers. In this sense, both can directly read and write the buffers and pointers without the involvement of the host processing circuit 205. As used herein, “directly reading” memory by the in-memory function circuit 110 means reading memory without the involvement of the host processing circuit 205, and “directly writing” memory by the in-memory function circuit 110 means writing to memory without the involvement of the host processing circuit 205.

[0052] In another example using the producer-consumer model, neural network inference computation could involve computing the activation of the first layer of the neural network and multiplying that activation by the weights in subsequent layers. In this case, the producer-in-memory function circuitry could compute the activation function, and the consumer-in-memory function circuitry could multiply each activation by one or more weights as each activation has been generated. Computing the activation function could involve evaluating a sigmoid function, a rectifier linear unit (ReLU) function, a hyperbolic tangent function, or any other suitable function.

[0053] As described above, in some embodiments, memory connected to the host may include one or more memory "islands". Each such island may be a quantity of memory. As used herein, "quantity of memory" is an arbitrary set of memory cells within a contiguous range of physical addresses. Each island may include in-memory function circuits 110 with different capabilities. For example, a first island may include multiple in-memory function circuits 110, each configured to perform computations on a gated recurrent unit, which may include computations of two sigmoid functions and a hyperbolic tangent. A second island may, for example, include multiple in-memory function circuits 110, each having an arithmetic logic unit 120 configured to perform matrix multiplication. The host processing circuitry 205 may delegate different computational tasks to different islands based on the nature of each of the different computational tasks and the corresponding capabilities of the in-memory function circuits 110 in the different islands. In this way, the system may be "adaptable" or "tunable" in its ability to adjust or adapt to different computational tasks.

[0054] These islands can be substantially isolated from each other. As used herein, if (i) the in-memory function circuit 110 of the first amount of memory cannot read or write to the memory cells of the second amount of memory without the participation of the host processing circuit 205, and (ii) the in-memory function circuit 110 of the second amount of memory cannot read or write to the memory cells of the first amount of memory without the participation of the host processing circuit 205, then the first amount of memory including the in-memory function circuit 110 is "substantially isolated" from the second amount of memory including the in-memory function circuit 110.

[0055] In some embodiments, the in-memory function circuits 110 in different islands may differ in such an order that they use data from open cells to perform (i) vector operations (which may be part of matrix operations) or (ii) matrix operations. Figure 3A –3E illustrates several ways to perform vector or matrix operations. (Reference) Figure 3A To form a matrix product of two matrices (the first and second matrices), the result of which can be added to the contents of a third matrix, as shown in the figure, part A of the first matrix can be stored in a first group of 16 tiles, and part B of the second matrix can be stored in a second group of 16 tiles. Parts A and B can be matrices themselves. Each tile can include sixteen FP-16 (16-bit floating-point) values. Thus, each tile occupies 256 bits, and the 32 tiles shown can fill an 8192-bit row of DRAM. Each tile in the first row of A can be multiplied by all the tiles in the corresponding column of B; for example, A00 can be multiplied by each of B00, B01, B02, and B03, A01 can be multiplied by each of B10, B11, B12, and B13, and so on. Similarly, A10 can be multiplied by each of B00, B01, B02, and B03. Open rows can be read one column at a time, each column having 256 bits, and corresponding to... Figure 3A One of the tiles shown.

[0056] Figure 3B This illustrates a data configuration that can be used to calculate matrix products by taking the dot product of the rows forming matrix part A and the columns forming matrix part B. Figure 3B In the example, the first tile A00 of matrix part A stores rows including elements a00, a01, etc., and the first tile B00 of matrix part B stores columns including elements b00, b01, b02, etc. In the terminology used herein, a lowercase "a" or "b" followed by a number indicating its position in a vector (which can be an FP-16 number) represents an element of the matrix, and an uppercase "A" or "B" followed by a number indicating its position represents multiple matrix elements stored in the tile (e.g., 16 matrix elements). In operation, rows of A are read from the first column of the open rows into a first register 115 (e.g., the Rop register). Then, columns of B are read from the second column of the open rows along with the contents of the first register into an arithmetic logic unit 120, where the arithmetic logic unit 120 calculates the dot product and stores it in a second register of the in-memory function circuit 110. This method of calculating matrix products has the advantage of requiring very little space to store intermediate products (i.e., dot products, each dot product being an element of the product matrix).

[0057] Figure 3C This illustrates a data configuration that can be used to calculate matrix products by using the outer product of the columns forming matrix part A and the rows forming matrix part B. Figure 3CIn the example, the first tile A00 of matrix part A stores columns including elements a00, a01, etc., and the first tile B00 of matrix part B stores rows including elements b00, b01, b02, etc. In operation, columns of A are read from the first column of the open rows into a first register 115 (e.g., the Rop register). Then, rows of B are read from the second column of the open rows along with the contents of the first register into an arithmetic logic unit 120, and the arithmetic logic unit 120 calculates the outer product and stores it in the second register of the in-memory function circuit 110. Figure 3B Compared to the previous method, this method of calculating matrix products has the advantage of being suitable for implementation in simpler and faster hardware (because an adder tree can be avoided), but at the cost of using larger registers (large enough to store the outer product matrix) to store the intermediate products. The elements of the matrix product can be calculated by summing each element of the product array.

[0058] Figure 3D This illustrates a data configuration that can be used to compute matrix products by forming a matrix (or "tensor") product between submatrices of matrix part A and submatrices of matrix part B. Figure 3D In the example, the first tile A00 of matrix part A stores a submatrix including elements a00, a01, etc., and the first tile B00 of matrix part B stores a submatrix including elements b00, b01, b02, etc. In operation, submatrix A is read from the first column of the open row into the first register 115 (e.g., the Rop register) in the in-memory function circuit 110. Then, submatrix B, along with the contents of the first register, is read from the second column of the open row into the arithmetic logic unit 120, and the arithmetic logic unit 120 calculates the matrix product of submatrix A and submatrix B and stores it in the second register of the in-memory function circuit 110. This method of calculating matrix products is compared to... Figure 3B The method has the advantage of using a smaller adder tree and is able to compute products involving transpose (using a suitable arithmetic logic unit 120) without reordering the data in memory (e.g., the in-memory function circuit 110 can compute AB, A from an open row). T B and AB T (one or more).

[0059] Figure 3E This illustrates a data configuration that can be used to compute the element-wise vector product of rows of matrix A with rows of matrix B (or, similarly, columns of matrix A with columns of matrix B). For example, this capability could be useful for computing convolutions. Figure 3EIn the example, the first tile A00 of matrix part A stores rows including elements a00, a01, etc., and the first tile B00 of matrix part B stores rows including elements b00, b01, b02, etc. In operation, rows of A are read from the first column of the open rows into the first register 115 of the in-memory function circuit (e.g., the Rop register). Then, rows of B are read from the second column of the open rows along with the contents of the first register into the arithmetic logic unit 120, whereby the arithmetic logic unit 120 calculates the vector product of the element directions and stores it in the second register of the in-memory function circuit 110.

[0060] In some embodiments, the scheduler can schedule various computational tasks to be performed by in-memory function circuitry 110, for example by instrumenting code to be executed by host processing circuitry 205, so that host processing circuitry 205 programs the in-memory function circuitry to perform the specific computational tasks thus delegated to the in-memory function circuitry. Programming the in-memory function circuitry may require writing commands to the in-memory function circuitry to cause it to execute instructions or perform read or write operations. The scheduler can be executed by processing circuitry (e.g., by host processing circuitry 205 or another processing circuitry) at compile time (e.g., using static analysis of source code or object code), or it can be executed at runtime (by processing circuitry).

[0061] The scheduler can analyze the framework graph (e.g., Tensorflow). TM The scheduler can allocate multiple FIM islands (as shown in the figure), each with different characteristics (register size, clock rate, lane, etc.). The scheduler can further (i) maintain the occupancy of different islands (e.g., ensure that all islands are always performing computations as much as possible, (ii) identify computational tasks that can be executed in parallel and schedule them to be executed by different islands, and (iii) use the producer-consumer model as described above to reduce the idle time of function circuits in memory while waiting for other computations to complete.

[0062] The scheduler may also include an adaptive placement multiplexer that identifies computational tasks suitable for delegation to in-memory function circuitry and uses a cost function to analyze the expected benefits of delegating any computational task to each of various types of islands. For example, if the computational task involves one or more matrix multiplications, the adaptive placement multiplexer may evaluate the cost function for each of the following: (i) Figure 3B (ii) Configuration Figure 3C Configuration, and (iii) Figure 3DThe configuration then delegates the computational task to an island containing an in-memory function circuit with a configuration that minimizes the cost function.

[0063] In some embodiments, the methods described herein can be used to perform convolutions, or for gated recurrent units (GRUs) or multilayer perceptrons (MLPs), or for computation of scatter and gather operations.

[0064] As used herein, “part of something” means all or less of that thing. Therefore, for example, “part of a matrix” means all or less of the matrix. As used herein, a vector (e.g., rows or columns of a matrix) is an example of a matrix (e.g., it is a 1xn matrix or an nx1 matrix). As used herein, the term “or” should be interpreted as “and / or”, so for example, “A or B” means either “A” or “B” or “A and B”.

[0065] It should be understood that although the terms "first," "second," "third," etc., may be used herein to describe various elements, components, regions, layers, and / or portions, these elements, components, regions, layers, and / or portions should not be limited by these terms. These terms are used only to distinguish one element, component, region, layer, or portion from another. Therefore, without departing from the spirit and scope of the inventive concept, the first element, component, region, layer, or portion discussed herein may be referred to as the second element, component, region, layer, or portion.

[0066] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the invention. As used herein, the terms “substantially,” “approximately,” and similar terms are used as approximate terms rather than terms of degree and are intended to describe inherent biases in measurements or calculations that will be recognized by one of ordinary skill in the art.

[0067] As used herein, the singular forms “a” and “an” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that, when used herein, the term “comprising” specifies the presence of stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. When preceding a list of elements, expressions such as “at least one” modify the entire list of elements, rather than a single element in the list. Furthermore, when describing embodiments of the inventive concept, the use of “may” means “one or more embodiments of this disclosure.” Additionally, the term “exemplary” is intended to indicate an example or illustration. As used herein, the terms “use,” “used,” and “being used” can be considered synonymous with the terms “utilize,” “exploited,” and “being exploited,” respectively.

[0068] It should be understood that when an element or layer is referred to as "on another element or layer," "connected to," "coupled to," or "adjacent to" another element or layer, it can be directly on, connected to, coupled to, or adjacent to that other element or layer, or one or more intermediate elements or layers may exist. Conversely, when an element or layer is referred to as "directly on another element or layer," "directly connected to," "directly coupled to," or "directly adjacent to" another element or layer, no intermediate elements or layers exist.

[0069] Any numerical range described herein is intended to include all subranges containing the same numerical precision within the stated range. For example, the range “1.0 to 10.0” or “between 1.0 and 10.0” is intended to include all subranges between (and inclusive of) the minimum value of 1.0 and the maximum value of 10.0, i.e., a minimum value equal to or greater than 1.0 and a maximum value equal to or less than 10.0, such as 2.4 to 7.6. Any maximum numerical limit described herein is intended to include all lower numerical limits contained therein, while any minimum numerical limit described in this specification is intended to include all higher numerical limits contained therein.

[0070] As used herein, the term "processing circuitry" refers to any combination of hardware, firmware, and software used to process data or digital signals. Processing circuitry hardware may include, for example, application-specific integrated circuits (ASICs), general-purpose or special-purpose central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), and programmable logic devices such as field-programmable gate arrays (FPGAs). As used herein, in processing circuitry, each function is executed by hardware configured (i.e., hardwired) to execute that function, or by more general-purpose hardware (such as a CPU) configured to execute instructions stored in a non-transitory storage medium. Processing circuitry may be fabricated on a single printed circuit board (PCB) or distributed across several interconnected PCBs. Processing circuitry may contain other processing circuitry; for example, processing circuitry may include two processing circuits, an FPGA and a CPU, interconnected on a PCB.

[0071] Although exemplary embodiments of adaptive in-memory function systems have been specifically described and illustrated herein, many modifications and alterations will be apparent to those skilled in the art. Therefore, it should be understood that adaptive in-memory function systems constructed according to the principles of this disclosure may be implemented in ways other than those specifically described herein. The invention is also defined in the following claims and their equivalents.

Claims

1. A method for computation within a memory connected to a processor, the memory comprising a first amount of memory and a second amount of memory, the method comprising: Based on the nature of the first computing task, the processor selects a first memory-in-memory function circuit, a second memory-in-memory function circuit, and a first buffer from either the first amount of memory or the second amount of memory. The first instruction is executed by the function circuit in the first memory to produce a first value as a result, wherein the first calculation task includes a second calculation task and a third calculation task, and the second calculation task includes the first instruction; The first value is stored in the first buffer by the function circuit in the first memory; The first value is read from the first buffer by the function circuit in the second memory; as well as The second instruction is executed by the function circuit in the second memory, and the second instruction uses the first value as an argument. The third computation task includes the second instruction. in: Storing the first value in the first buffer by the function circuit within the first memory includes directly storing the first value in the first buffer. Reading the first value from the first buffer by the function circuit in the second memory includes reading the first value directly from the first buffer. The function circuit in the first memory is located in the first amount of memory. The second memory function circuit is located in the first amount of memory, and The first buffer is in the memory of the first amount.

2. The method of claim 1, further comprising adjusting a pointer by a function circuit within the first memory, the pointer identifying a boundary of a first portion of the first buffer in the memory of the first amount, the adjustment comprising moving the boundary, wherein the movement of the boundary has the effect of including the first value within the first portion.

3. The method according to claim 1, further comprising: The third instruction is executed by the function circuit in the first memory to produce a second value as a result, and the second calculation task includes the third instruction; as well as The fourth instruction is executed by the function circuit in the second memory, the fourth instruction using the second value as an argument, and the third computation task includes the fourth instruction.

4. The method according to claim 1, wherein: The first computational task includes calculating the product of a first matrix, a second matrix, and a third matrix; The second computational task includes calculating the product of the first matrix and the second matrix to form a first product matrix; and The third computational task includes calculating the product of the first product matrix and the third matrix.

5. The method according to claim 1, further comprising executing a third instruction by a function circuit in a third memory, wherein: The fourth computational task includes the third instruction. The function circuit in the first memory is located in the first amount of memory. The function circuit in the third memory is located in the memory of the second quantity, and The third memory-in-memory function circuit has capabilities different from those of the first memory-in-memory function circuit.

6. The method according to claim 5, wherein the fourth computational task comprises: Calculate the product of the first matrix stored in the memory of the second quantity and the second matrix stored in the memory of the second quantity; as well as Calculate the product of the third and fourth matrices while maintaining the original storage order of the elements in the first and second matrices. The third matrix is ​​the transpose of the first matrix, or The fourth matrix is ​​the transpose of the second matrix.

7. The method according to claim 6, wherein: The first computational task includes calculating the product of the fifth, sixth, and seventh matrices; The second computational task includes calculating the product of the fifth matrix and the sixth matrix to form a first product matrix; and The third computational task includes calculating the product of the first product matrix and the seventh matrix.

8. The method of claim 7, wherein the first amount of memory is substantially isolated from the second amount of memory.

9. The method of claim 7, wherein calculating the product of the fifth matrix and the sixth matrix comprises calculating the product of a submatrix of the fifth matrix and a submatrix of the sixth matrix.

10. The method of claim 7, wherein calculating the product of the fifth matrix and the sixth matrix comprises calculating the dot product of the rows of the first matrix and the columns of the second matrix.

11. The method of claim 7, wherein calculating the product of the fifth matrix and the sixth matrix comprises calculating the outer product of the columns of the first matrix and the rows of the second matrix.

12. The method of claim 6, wherein the second computational task comprises computing a function selected from the group consisting of the sigmoid function, the rectified linear unit function, the hyperbolic tangent function, and combinations thereof.

13. The method of claim 12, wherein the third computational task comprises computing the product of the fifth matrix and the sixth matrix, and the result of the computation of the function is an element of the fifth matrix.

14. A system for computation, the system comprising: Host processing circuitry; and The memory includes a first amount of memory and a second amount of memory. The host processing circuit is configured as follows: Based on the nature of the first computational task, a function circuit in the first memory, a function circuit in the second memory, and a first buffer are selected from either the first amount of memory or the second amount of memory. The first memory function circuit and the second memory function circuit are programmed such that: The function circuit within the first memory: Execute a first instruction to produce a first value as a result, wherein the first computation task includes a second computation task and a third computation task, and the second computation task includes the first instruction; and Store the first value in the first buffer; as well as The function circuit in the second memory: Read the first value from the first buffer; as well as The second instruction is executed, which uses the first value as an argument. The third computational task includes the second instruction. in: Storing the first value in the first buffer by the function circuit within the first memory includes directly storing the first value in the first buffer. Reading the first value from the first buffer by the function circuit in the second memory includes reading the first value directly from the first buffer.

15. The system of claim 14, wherein the programming further causes a function circuit in the first memory to adjust a pointer that identifies a boundary of a first portion of the first buffer in the memory of the first amount, the adjustment comprising moving the boundary, wherein the movement of the boundary has the effect of including the first value within the first portion.

16. The system of claim 14, wherein the programming further causes: After executing the second instruction, the function circuit in the first memory executes a third instruction to produce a second value as a result, the second computation task including the third instruction; and After executing the third instruction, the second memory-in-function circuit executes a fourth instruction, which uses the second value as an argument, and the third computation task includes the fourth instruction.

17. The system according to claim 14, wherein: The first computational task includes calculating the product of a first matrix, a second matrix, and a third matrix; The second computational task includes calculating the product of the first matrix and the second matrix to form a first product matrix; and The third computational task includes calculating the product of the first product matrix and the third matrix.

18. The system of claim 14, wherein the host processing circuitry is further configured to program a function circuitry in a third memory, the programming of the function circuitry in the third memory causing the function circuitry in the third memory to execute a third instruction, wherein: The fourth computational task includes the third instruction. The function circuit in the third memory is located in the memory of the second quantity, and The third memory-in-memory function circuit has capabilities different from those of the first memory-in-memory function circuit.

19. The system of claim 18, wherein the fourth computing task comprises: Calculate the product of the first matrix stored in the memory of the second quantity and the second matrix stored in the memory of the second quantity; as well as Calculate the product of the third and fourth matrices while maintaining the original storage order of the elements in the first and second matrices. The third matrix is ​​the transpose of the first matrix, or The fourth matrix is ​​the transpose of the second matrix.

20. A system for computation, the system comprising: Device for processing; and The memory includes a first amount of memory and a second amount of memory. The processing apparatus is configured as follows: Based on the nature of the first computational task, a function circuit in the first memory, a function circuit in the second memory, and a first buffer are selected from either the first amount of memory or the second amount of memory. The first memory function circuit and the second memory function circuit are programmed such that: The function circuit within the first memory: Execute a first instruction to produce a first value as a result, wherein the first computation task includes a second computation task and a third computation task, and the second computation task includes the first instruction; and Store the first value in the first buffer; as well as The function circuit in the second memory: Read the first value from the first buffer; as well as The second instruction is executed, which uses the first value as an argument. The third computational task includes the second instruction. in: Storing the first value in the first buffer by the function circuit within the first memory includes directly storing the first value in the first buffer. Reading the first value from the first buffer by the function circuit in the second memory includes reading the first value directly from the first buffer.

Images