Method and system for memory search mechanism and memory bare chip
By using a lookup table methodology on the memory die to perform multiplication operations and then transferring the data to the logic die for further computation, the high bandwidth requirements of storage and computing resources are addressed, memory utilization and computational efficiency are improved, and training time is shortened.
Patent Information
- Application Number
- CN201811590387.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-03-06
- Filing Date
- 2018-12-25
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2038-12-25
AI Technical Summary
In existing technologies, the demand for storage and computing resources increases in high-performance computing and deep neural network training. In particular, the high bandwidth requirements for storage weights and intermediate results are not being fully utilized.
The method employs a lookup table approach on the memory die, performing multiplication operations by accessing the lookup table and sending the results to the logic die for further calculations, including floating-point full product multiplication, partial product multiplication, tensor operations, and fusion layer operations. Buffers and accumulators are used for the transfer and accumulation of results.
It improves memory utilization and computational efficiency, reduces energy consumption, shortens training time, and expands the computational scale.
Smart Images

Figure CN109976808B_ABST
Abstract
Description
[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS]
[0002] This application claims priority to and the rights of U.S. Provisional Patent Application No. 62 / 610,439 filed on December 26, 2017, the contents of which are incorporated herein by reference in their entirety. Technical Field
[0003] One or more aspects of the embodiments disclosed herein relate to a high bandwidth memory lookup mechanism system. Background Art
[0004] Emerging applications (e.g., deep neural networks) use massive computing and storage capabilities to train different data sets and learn with high accuracy. In addition, as applications (e.g., High Performance Computing (HPC) graph algorithms) become computationally intensive, energy efficiency and low latency become increasingly important.
[0005] As a result, the demand for storage and computational resources (e.g., memory for storing weights and intermediate results, and computation mainly for acceleration of Basic Linear Algebra Subprograms (BLAS) kernels and General Matrix to Matrix Multiplication (GEMM) kernels) has increased significantly. Summary of the invention
[0006] Various aspects of the disclosed embodiments relate to a memory lookup mechanism in a high-bandwidth storage system.
[0007] According to some exemplary embodiments of the present disclosure, a method for a memory lookup mechanism in a high-bandwidth memory system is provided, the method comprising: using a memory die, utilizing a lookup table (LUT) methodology to perform a multiplication operation by accessing a lookup table stored on the memory die, the lookup table comprising a floating-point operation result; sending the result of the multiplication operation by the memory die to a logic die, the logic die comprising a processor and a buffer; and performing a matrix multiplication operation by the logic die using a computing unit.
[0008] According to some exemplary embodiments, the lookup table methodology includes a floating-point full product multiplication method, which includes: retrieving a full scalar multiplication result from the lookup table using a full scalar floating-point input by the memory die, the lookup table including multiple full product multiplication results; sending the full scalar multiplication result from the lookup table to the buffer by the memory die; sending the full scalar multiplication result to an accumulator on the logic die by the buffer; and accumulating the full scalar multiplication result by the accumulator.
[0009] According to some exemplary embodiments, the lookup table methodology includes a floating-point partial product multiplication method, which includes: retrieving a partial product multiplication result from the lookup table including multiple partial product multiplication results using a fractional scalar floating-point input by the memory die; sending the partial product multiplication result to the buffer by the memory die; sending the partial product multiplication result to an adjuster on the logic die by the buffer; sending the product multiplication result to an accumulator on the logic die by the adjuster; and accumulating the product multiplication result by the accumulator.
[0010] According to some exemplary embodiments, the lookup table methodology includes a floating-point tensor operation method, which includes: retrieving a tensor multiplication result from the lookup table using a tensor floating-point input by the memory die, the lookup table including multiple tensor multiplication results; sending the tensor multiplication result to the buffer by the memory die; and sending the tensor multiplication result to a tensor post-processor on the logic die by the buffer.
[0011] According to some exemplary embodiments, the lookup table methodology includes a floating-point fusion layer operation method, which includes: retrieving a complete product multiplication result from the lookup table including multiple complete product multiplication results using full scalar floating point by the memory die; sending the complete product multiplication result to a buffer die by the memory die, the buffer die including multiple buffers separate from the buffer of the logic die; sending the complete product multiplication result to the buffer of the logic die by the buffer die; and sending the complete product multiplication result to a tensor processor on the logic die by the buffer of the logic die.
[0012] According to some example embodiments, the method further comprises: storing the lookup table associated with a first type of operation on the memory die; and storing the lookup table associated with a second type of operation different from the first type of operation on the memory die.
[0013] According to some exemplary embodiments, the method further includes: reading each possible position in the first layer input feature map; convolving the first layer input feature map with the first layer kernel; and generating a result of the second layer input feature map based on the convolution.
[0014] According to some exemplary embodiments of the present disclosure, a high-bandwidth memory lookup mechanism system includes: a logic die including a processor and a buffer, wherein the processor uses a computing unit to implement a matrix multiplication operation; and a memory die including a lookup table (LUT), wherein the lookup table includes floating-point operation results.
[0015] According to some example embodiments, the memory die performs operations including: accessing the lookup table; performing a multiplication operation using a lookup table methodology; and sending a result of the multiplication operation to the logic die.
[0016] According to some example embodiments, the memory die performs operations further comprising: retrieving a full scalar multiplication result from the lookup table including a plurality of full product multiplication results using a full scalar floating point input; and sending the full scalar multiplication result from the lookup table to the buffer.
[0017] According to some example embodiments, the logic die further includes an accumulator that performs operations including: receiving the complete scalar multiplication result from the buffer; and accumulating the complete scalar multiplication result.
[0018] According to some example embodiments, the memory die performs operations further comprising: retrieving a partial product multiplication result from the lookup table including a plurality of partial product multiplication results using a fractional scalar floating point input; and sending the partial product multiplication result to the buffer.
[0019] According to some example embodiments, the logic die further includes: an adjuster that receives the partial product multiplication result from the buffer; and an accumulator that performs operations including: receiving the partial product multiplication result from the adjuster; and accumulating the partial product multiplication result.
[0020] According to some example embodiments, the memory die performs operations further comprising: retrieving a tensor multiplication result from the lookup table including a plurality of tensor multiplication results using a tensor floating point input; and sending the tensor multiplication result to the buffer.
[0021] According to some example embodiments, the logic die further includes a tensor post-processor that receives the tensor multiplication result from the buffer.
[0022] According to some example embodiments, the memory die performs operations further comprising: retrieving a full product multiplication result from the lookup table including a plurality of full product multiplication results using full scalar floating point; and sending the full product multiplication result to a buffer die.
[0023] According to some example embodiments, the logic die further includes a tensor processor that receives the full product multiplication result from the buffer die.
[0024] According to some example embodiments, the logic die further includes an accelerator die.
[0025] According to some example embodiments, the memory die includes a dynamic random access memory (DRAM) die.
[0026] According to some exemplary embodiments of the present disclosure, a memory die includes a lookup table (LUT) stored on the memory die and is configured to perform: a floating-point full product multiplication method, including: retrieving a full scalar multiplication result from the lookup table using a full scalar floating-point input by the memory die, the lookup table including a plurality of full scalar multiplication results; sending the full scalar multiplication result to an accumulator by a buffer; and accumulating the full scalar multiplication result by the accumulator; a floating-point partial product multiplication method, including: retrieving a partial product multiplication result from the lookup table using a fractional scalar floating-point input by the memory die, the lookup table including partial product multiplication results; sending the partial product multiplication result to the buffer by the memory die; sending the partial product multiplication result to an adjuster by the buffer; The accumulator accumulates the partial product multiplication results; a floating-point tensor operation method, comprising: the memory die uses tensor floating-point input to retrieve the tensor multiplication result from the lookup table, the lookup table including multiple tensor multiplication results; the memory die sends the tensor multiplication result to the buffer; the buffer sends the tensor multiplication result to a tensor post-processor; and a floating-point fusion layer operation method, comprising: the memory die uses full scalar floating point to retrieve the full product multiplication result from the lookup table including multiple full product multiplication results; the memory die sends the full product multiplication result to a buffer die including multiple buffers; the buffer die sends the full product multiplication result to the buffer; and the buffer sends the full product multiplication result to a tensor processor. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Some embodiments may be understood in more detail by reading the following description in conjunction with the accompanying drawings, in which:
[0028] Figure 1A and Figure 1B is a block diagram of the architecture of a second generation high bandwidth memory HBM2 system according to one or more embodiments of the present disclosure;
[0029] Figures 2A to 2D is a block diagram illustrating design options according to one or more embodiments of the present disclosure;
[0030] Figure 3 Provides an illustration of how a memory lookup table performs scalar-vector multiplication;
[0031] Figure 4 A flowchart 400 is shown outlining the different components of a floating point (FP), such as a floating point exponent, fraction, or sign;
[0032] Figure 5 shows a block diagram 500 showing a redundant portion of a lookup table and a table 510 showing the number of bits used by the row index, column index, and result size;
[0033] Figure 6 A visual representation of an algorithm that can save memory space by "folding" a lookup table;
[0034] Figure 7 7 shows a block diagram 700 representing a lookup table and a table 710 for showing the number of bits used by the row index, column index and result size of a lookup table storing only partial product multiplication results;
[0035] Figure 8 shows a block diagram 800 representing a lookup table containing fractional partial product results;
[0036] Fig. 9 Provides a visual representation of layer-by-layer convolution operations;
[0037] Description of Figure Numbers:
[0038] 100: Second generation high bandwidth memory system / HBM2 system;
[0039] 110: host bare chip;
[0040] 120: 3D HBM2 memory stacking / HBM2 memory stacking / memory stacking / stacking;
[0041] 130: HBM2 memory die / memory die;
[0042] 150: logic die;
[0043] 160: Computing accelerator die;
[0044] 210A, 210B, 210C, 210D: DRAM dies;
[0045] 220A, 220B, 220C, 220D: lookup table;
[0046] 230A, 230B, 230C, 230D: buffers;
[0047] 240A, 240B: accumulator;
[0048] 250: Regulator;
[0049] 260: Tensor post-processing unit / TPU;
[0050] 270: Buffer die;
[0051] 280A, 280B, 280C, 280D: logic die;
[0052] 290: Fusion layer post-processor / FPU;
[0053] 310: memory lookup table;
[0054] 320: line buffer;
[0055] 400: Flowchart;
[0056] 500, 700, 800: block diagram;
[0057] 410, 510, 710: table;
[0058] 900: layer 1 kernel;
[0059] 910: layer 1 input feature map / input feature map;
[0060] 920: layer 2 kernel;
[0061] 930: layer 2 input feature map / input feature map;
[0062] 950: layer 3 input feature map / input feature map;
[0063] A: input value / tensor / scalar;
[0064] A', B': input values;
[0065] B: input value / tensor;
[0066] B 1 , B 2 , …B n :vector;
[0067] C': real valued result / value / result / tensor / tensor result;
[0068] C”: real value result;
[0069] A.fraction: input value;
[0070] B.fraction: input value;
[0071] C.fraction: result;
[0072] C 1 , C 2 ,…C n : Vector product / lookup result / result. DETAILED DESCRIPTION
[0073] By referring to the following detailed description of the embodiments and the accompanying drawings, the features of the inventive concept and its implementation method can be more easily understood. Hereinafter, the embodiments will be described in more detail with reference to the accompanying drawings, wherein the same reference numerals in all the accompanying drawings refer to the same elements. However, the present disclosure may be implemented in various different forms and should not be considered to be limited to the embodiments shown herein. Specifically, these embodiments are provided as examples in order to make the present disclosure thorough and complete, and to fully convey the various aspects and features of the present disclosure to those skilled in the art. Therefore, the processes, elements and technologies that are not necessary for the general technicians in the art to fully understand the various aspects and features of the embodiments of the present disclosure may no longer be described. Unless otherwise stated, the same reference numerals throughout all the drawings and written descriptions represent the same elements, and therefore, they will no longer be repeated. In the figure, for clarity, the relative sizes of each element, each layer and each zone may be exaggerated.
[0074] In the following description, for the purpose of explanation, various specific details are set forth to provide a thorough understanding of various embodiments. However, it is apparent that various embodiments may be practiced without these specific details or with one or more equivalent configurations. In other examples, well-known structures and devices are shown in block diagram form to avoid unnecessary confusion of various embodiments.
[0075] It should be understood that when an element, layer, region or component is referred to as being "on", "connected to" or "coupled to" another element, layer, region or component, the element, layer, region or component may be directly on, directly connected to or directly coupled to the other element, layer, region or component, or one or more intermediate elements, layers, regions or components may exist. However, "directly connected / directly coupled" means that one component is directly connected or directly coupled to another component without any intermediate components. At the same time, other expressions that describe the relationship between components, such as "between", "immediately between", "adjacent to" and "directly adjacent to", may be interpreted similarly. In addition, it will also be understood that when an element or layer is referred to as being “between” two elements or layers, it can be the only element or layer between the two elements or layers, or one or more intervening elements or layers may also be present.
[0076] The terms used herein are only for the purpose of illustrating specific embodiments and are not intended to limit the present disclosure. Unless the context clearly indicates otherwise, the singular forms "a and an" used herein are intended to also include plural forms. It should also be understood that when the terms "comprises, comprising", "have, having" and "includes, including" are used in this specification, it is to indicate the presence of stated features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. The term "and / or" used herein includes any and all combinations of one or more items in the relevant listed items.
[0077] As used herein, the terms "substantially", "about", "approximately", and the like are used as terms of approximation, not as terms of degree, and are intended to take into account the inherent deviations of measured or calculated values that would be known to one of ordinary skill in the art. As used herein, "about" or "approximately" includes the stated value and means being within an acceptable range of deviations of a particular value as determined by one of ordinary skill in the art, taking into account the measurements involved and the errors associated with measurements of a particular quality (i.e., the limitations of the measurement system). For example, "about" may mean being within one or more standard deviations, or within ±30%, ±20%, ±10%, ±5% of the stated value. In addition, the use of "may" when describing embodiments of the present disclosure refers to "one or more embodiments of the present disclosure". As used herein, the terms "use", "using", and "used" may be considered synonymous with the terms "utilize", "utilizing", and "utilized", respectively. Additionally, the word "exemplary" is intended to refer to an example or illustration.
[0078] When a certain embodiment can be implemented in different ways, a specific process order can be performed differently from the order described. For example, two processes described in succession can be performed substantially simultaneously or in the reverse order of the order described.
[0079] Various embodiments are described herein with reference to cross-sectional views, which are schematic illustrations of embodiments and / or intermediate structures. Therefore, it is expected that changes relative to the illustrated shapes may occur due to, for example, manufacturing techniques and / or tolerances. In addition, the specific structural descriptions or functional descriptions disclosed herein are merely illustrative and are intended to illustrate embodiments according to the concepts of the present disclosure. Therefore, the embodiments disclosed herein should not be considered to be limited to the specific illustrated shapes of the various regions, but should include shape deviations caused by, for example, manufacturing. For example, an implanted region illustrated as a rectangle should generally have circular features or curvilinear features and / or a gradient of implant concentration at its edge rather than a binary change from an implanted region to a non-implanted region. Similarly, a buried region formed by implantation may cause some implantation in the region between the buried region and the surface passed through when implanting. Therefore, the regions illustrated in the figure are schematic and their shapes are not intended to illustrate the actual shape of the region of the device and are not intended to be limiting.
[0080] The electronic device or electrical device and / or any other related device or component according to the embodiments of the present disclosure described herein may be implemented using any suitable hardware, firmware (e.g., application-specific integrated circuit), software, or a combination of software, firmware, and hardware. For example, the various components of these devices may be formed on an integrated circuit (IC) chip or formed on separate integrated circuit chips. In addition, the various components of these devices may be implemented on a flexible printed circuit film, a tape carrier package (TCP), a printed circuit board (PCB), or formed on a substrate. In addition, the various components of these devices may be processes or threads that are run by one or more processors in one or more computing devices, execute computer program instructions, and interact with other system components for performing various functionalities described herein. The computer program instructions are stored in a memory, which may be implemented in a computing device using standard memory devices such as, for example, random access memory (RAM). The computer program instructions may also be stored in other non-transitory computer readable media such as, for example, a compact disc read only memory (CD-ROM), a flash drive, or the like. In addition, those skilled in the art will appreciate that the functionality of various computing devices may be combined or integrated into a single computing device, or the functionality of a particular computing device may be distributed among one or more other computing devices without departing from the spirit and scope of the exemplary embodiments of the present disclosure.
[0081] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present disclosure belongs. It should also be understood that terms (such as those defined in commonly used dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and / or in this specification, and should not be interpreted as having an idealized or overly formal meaning unless explicitly defined herein.
[0082] Use large data batch sizes to reduce total training time without losing accuracy. This approach can scale up by improving compute / storage utilization on a single high-performance process. Alternatively, this approach can scale out by using model parallelism and data parallelism to distribute learning across multiple high-performance processors.
[0083] Deep learning applications exhibit various compute-to-memory ratios. Convolutional layers have high compute-to-memory ratios. Fully connected layers / recurrent layers have medium-low compute-to-memory ratios. Element-by-element / scalar operations have low compute-to-memory ratios. The compute-to-memory ratio of individual applications depends on the above combination.
[0084] For low compute-to-memory ratio workloads, compute-centric throughput processors can appropriately use large memory capacity and bandwidth. Chip memory cannot contain large batch sizes of data, nor can it contain large layer sizes of data. Prior art processors (e.g., graphics processing units (GPUs), central processing units (CPUs), and field-programmable gate arrays (FGPAs)) use high bandwidth memory (HBM). For high compute-to-memory ratio workloads, if the workload is compute-constrained, memory bandwidth and memory capacity are not fully utilized.
[0085] Figure 1A and Figure 1B is a block diagram of the architecture of a second generation high-bandwidth memory (“HBM2”) system according to one or more embodiments of the present disclosure. Figure 1A Provides a top view of the architecture. Figure 1B Provides a side view of the architecture.
[0086] Reference Figure 1A, embodiments of the present disclosure provide a system for a memory lookup computing mechanism, or a second generation high bandwidth memory (HBM2) system 100. The HBM2 system 100 includes a plurality of host dies 110 connected to a plurality of three-dimensional HBM2 memory stacks 120. According to various embodiments, the plurality of host dies 110 may be connected to up to four HBM2 memory stacks 120.
[0087] Reference Figure 1B , the three-dimensional HBM2 memory stack 120 includes a plurality of HBM2 memory dies 130. As shown in the figure, the stack 120 may have 4 to 8 memory dies 130 per stack 120. Each stack 120 may include up to eight channels per stack 120. In some embodiments, the memory stack 120 may be positioned on a logic die 150. In some embodiments, an operational accelerator die 160 may be placed on top of the stack 120.
[0088] A portion of the memory stack 120 is configured as a lookup table ("LUT") to exploit bank-level parallelism. The lookup table is stored on the memory die 130. The lookup table contains the results of floating-point operations (e.g., pre-computed results). The results are real values. The results contained in the lookup table on the memory die 130 can be streamed to the logic die 150.
[0089] Figures 2A to 2D is a block diagram illustrating design options according to one or more embodiments of the present disclosure.
[0090] Figure 2A One embodiment of a memory lookup mechanism is shown. According to the present embodiment, the lookup table 220A is located on the DRAM die 210A. The lookup table 220A contains one or more pre-calculated real value results C' of floating point operations. In the current embodiment, the real value result C' may include a complete scalar floating point value. The DRAM die 210A retrieves the value / result C' from the lookup table 220A. The DRAM die 210A uses input value A and input value B to identify the result C' in the lookup table 220A. The input value A represents the row address of the lookup table 220A, and the input value B represents the column address of the lookup table 220A. According to this embodiment, the input value A and the input value B may include a complete scalar floating point value.
[0091] DRAM die 210A transfers result C' to buffer ("BUF") 230A located on logic die 280A. Buffer 230A stores result C'. Buffer 230A allows faster access to result C' for matrix multiplication operations performed by logic die 280A in conjunction with DRAM die 210A. Buffer 230A transfers result C' to accumulator ("ACC") 240A. In various embodiments, accumulator 240A collects result C' for calculation of intermediate results of matrix multiplication operations performed by logic die 280A and DRAM die 210A.
[0092] Figure 2B A second embodiment of a memory lookup mechanism is shown. According to this embodiment, the lookup table 220B is located on the DRAM die 210B. The lookup table 220B stores the partial product multiplication result "C.fraction" of the pre-calculated floating point operation. The DRAM die 210B uses the input values "A.fraction" and "B.fraction" to identify the result C.fraction in the lookup table. The A.fraction value represents the row address of the lookup table and is the fractional portion of the input value A. B.fraction represents the column address of the lookup table and is the fractional portion of the input value B. According to this embodiment, the A.fraction value and the B.fraction value may include fractional scalar floating point values.
[0093] In this embodiment, the lookup table does not implement sign logic and exponent addition and subtraction logic to achieve a reduced size of the lookup table. Since the input values A.fraction and B.fraction are symmetrical, it is appropriate to store only half of the result in the lookup table 220B. The reduced size of the lookup table 220B enables more storage space on the DRAM die 210B.
[0094] DRAM die 210B transmits result C.fraction to buffer 230B located on logic die 280B. Buffer 230B stores result C.fraction. Buffer 230B allows faster access to result C.fraction for matrix multiplication operations performed by logic die 280B. Buffer 230B streams result C.fraction to adjustor (“ADJ”) 250. Adjustor 250 aligns / processes result C.fraction to generate result C’. After alignment, adjuster 250 transmits result C’ to accumulator 240B, which can effectively implement the same Figure 2A Accumulator 240B will collect result C' for use in calculations performed by logic die 280B on intermediate results of the matrix multiplication operation.
[0095] Figure 2C A third embodiment of a memory lookup mechanism is shown. According to this embodiment, a lookup table 220C is located on a DRAM die 210C. The lookup table 220C contains a tensor value result tensor C' of a pre-calculated floating-point operation. The DRAM die 210C retrieves the tensor value result tensor C' from the lookup table 220C. The DRAM die 210C uses the row address and column address of the lookup table 220C to identify the tensor value result tensor C'. For the multiplication of tensor A and tensor B, the row address is the concatenation of each element in each row of the input value A, and the column address is each element in each column of the input value B. The input value A and the input value B may include tensor floating-point values. The row address is equal to the number of elements multiplied by the length of the element. It is assumed that the row index and the column index are unsigned integers. Since the input value A is symmetrical with the input value B, the lookup table 220C only needs to store half of the tensor value result tensor C'.
[0096] In some embodiments, the size of the lookup table 220C can be further reduced. Before cascading, the control engine on the DRAM die 210C can classify the tensor values in the rows and columns according to their absolute values and according to their pre-calculated sign bit logic. If the row index is greater than the column index, the control engine on the DRAM die 210C can switch the row index and the column index. The control engine on the DRAM die 210C can assume that the row index and the column index are unsigned integers. In addition, the control engine on the DRAM die 210C can use the pre-calculated sign bit logic to select the lookup table 220C for cascading of row and column address lookups.
[0097] DRAM die 210C transmits the tensor value result tensor C' to buffer 230C located on logic die 280C. Buffer 230C stores the tensor value result tensor C'. Buffer 230C allows faster access to the tensor value result tensor C' for matrix multiplication operations performed by logic die 280C. Buffer 230C transmits the tensor result C' to a tensor post processing unit (TPU) 260, which may include one or more accumulators, such as Figure 2A The accumulator 240A and Figure 2BAccumulator 240B is shown. TPU 260 acts as an accumulator for tensor results (eg, tensor result C'). In various embodiments, TPU 260 will collect tensor result C' for computation on intermediate results of operations performed by logic die 280C.
[0098] Figure 2D A fourth embodiment of a memory lookup mechanism is shown. According to this embodiment, the lookup table 220D is located on the DRAM die 210D. In addition, the present embodiment also includes a buffer die 270. The lookup table 220D contains a pre-calculated real value result C". of a floating-point operation. In the current embodiment, the real value result C" may include a complete scalar floating-point value. The DRAM die 210D retrieves the real value result C". from the lookup table 220D. The DRAM die 210D uses input value A' and input value B' to identify the real value result C". in the lookup table 220D. The input value A' represents the row address of the lookup table 220D, and the input value B' represents the column address of the lookup table 220D. According to this embodiment, the input value A' and the input value B' may include a complete scalar floating-point value.
[0099] DRAM die 210D transfers the real-valued result C" to a buffer 230D located on logic die 280D. Buffer 230D stores the real-valued result C". Buffer 230D allows faster access to the real-valued result C" for matrix multiplication operations implemented by logic die 280D. Buffer 230D transfers the real-valued result C" to a fused-layer post processor (FPU) 290. In the case where the real-valued result C" is an intermediate value in a series of operations implemented by logic die 280D, FPU 290 may transfer the real-valued result C" to buffer die 270. In some embodiments, buffer die 270 may include multiple buffers similar to buffer 230D of logic die 280D. A separate buffer die 270 will enable wider storage of intermediate results, which is suitable for larger and more complex operations.
[0100] Figure 3 An illustration of how a memory lookup table performs scalar-vector multiplication is provided.
[0101] Reference Figure 3 In the process of this embodiment, the scalar A which is unrelated to the input value A is multiplied by the vector B 1 -B n The result of a vector-scalar multiplication is given by the vector product C 1 -C n Scalar A is used as a row address to activate a row in the memory lookup table 310. Vector B 1 -B nis used as a column address to read the corresponding search result C from the row buffer 320 1 -C n Result C 1 -C n is transferred to logic die 280A, 280B, 280C or 280D for accumulation, as previously described in Figures 2A to 2D as shown in .
[0102] Figure 4 A flow chart 400 is shown summarizing the different components of a floating point (eg, a floating point exponent, a fraction, or a sign).
[0103] Reference Figure 4 , table 410 shows the number of bits used by each component of the floating point. For example, a 64-bit floating point will have 52 bits for the fraction component and 11 bits for the exponent component.
[0104] Figure 5 Shown are a block diagram 500 showing a redundant portion of a lookup table and a table 510 showing the number of bits used by the row index, column index, and result size.
[0105] Reference Figure 5 In the example of this embodiment, the 64-bit floating point has a 63-bit row index, a 63-bit column index, and a 63-bit result size.
[0106] Figure 6 Shows a visual representation of an algorithm that can save memory space by "folding" the lookup table.
[0107] Reference Figure 6 , in the example of the present embodiment, the input values are represented as A and B. As in the previous figures, the input value A represents the row address of the lookup table and the input value B represents the column address of the lookup table. In some cases, the input value B may be greater than the input value A. In these cases, the control engine on the DRAM die may switch the input value A and the input value B. In addition, if the first bit of the input value B is equal to 1, the input value A is equal to the inverted form of all bits of the input value A, and the input value B is equal to the inverted form of all bits of the input value B. The control engine on the DRAM die may use the final value of A as the row index and the final value of B as the column index. This algorithm ensures that the actual lookup table access is half of the original table.
[0108] Figure 7 Shown are a block diagram 700 representing a lookup table and a table 710 for showing the number of bits used by the row index, column index, and result size of a lookup table that stores only partial product multiplication results.
[0109] Reference Figure 7In the example of this embodiment, the 64-bit floating point has a 52-bit row index, a 52-bit column index, and a 54-bit result size.
[0110] Figure 8 A block diagram 800 is shown representing a lookup table containing fractional partial product results. The fractional portion may be divided into multiple segments of equal size (e.g., fraction_length = fraction_segment * fraction_part). It may be desirable to have the lookup table implement only a multiplication table for fractional segment multiplication. Intermediate results are not rounded and normalized. Instead, the full precision of the intermediate results is maintained and sent to an adder to shift or sum the final result. An example of fractional segment multiplication is provided below:
[0111] (A 1 *2 12 +A 0 )*(B 1 *2 12 +B 0 )=2 24 *A 1 *B 1 +2 12 *(A 1 *B 0 +B 1 *A 0 )+A 0 *B 0
[0112] The normalized expression can be shown as follows:
[0113] 24=12+12
[0114] 2 12 =4K
[0115] Fig. 9 Provides a visual representation of the layer-by-layer convolution operation.
[0116] Reference Fig. 9 , the representation of this embodiment is the same as Figure 2D In the traditional convolution operation, each layer must be calculated layer by layer. Fig. 9As shown, the layer 1 kernel 900 will slide along every possible position in the layer 1 input feature map 910 (e.g., read every possible position), convolve with the layer 1 input feature map 910 and produce a result in the layer 2 input feature map 930. Thus, each element in the layer 2 input feature map 930 is the result of the convolution of the layer 1 kernel 900 and the layer 1 input feature map 910. In the next step, the layer 2 kernel 920 will similarly slide along every possible position in the layer 2 input feature map 930 to produce a result in the layer 3 input feature map 950. This convolution operation will continue layer by layer for N layers. However, the storage of each of these intermediate results (e.g., layer 1, layer 2, layer 3... layer N) becomes a problem. Storing all the intermediate results on the DRAM die may require a relatively large amount of energy and storage space.
[0117] To solve the energy consumption and storage problems, this embodiment focuses on a small computation pyramid. The small computation pyramid is represented by the shaded area of the input feature map (910, 930, 950). As each layer proceeds, the shaded area decreases. When storing intermediate results, only the shaded area of the input feature map is stored. Each of these intermediate results is stored on the buffer die instead of the DRAM die. The buffer die is Figure 2D 2 is shown as a buffer die 270. Retrieving intermediate results from a buffer die uses less energy than retrieving intermediate results from a DRAM die. In some embodiments, the buffer may be a buffer on a memory die, a buffer on a logic die, or a separate buffer die.
[0118] The above is an illustration of an exemplary embodiment and should not be regarded as a limitation of the exemplary embodiment. Although several exemplary embodiments have been described, it will be readily understood by those skilled in the art that many modifications may be made in the exemplary embodiments without substantially departing from the novel teachings and advantages of the exemplary embodiments. Therefore, all of these modifications are intended to be included within the scope of the exemplary embodiments defined by the claims. In the claims, the means-plus-function clause is intended to encompass the structures described herein as performing the functions, and not only structural equivalents but also equivalent structures. Therefore, it should be understood that the above is an illustration of an exemplary embodiment and should not be regarded as limited to the specific embodiments disclosed, and modifications to the disclosed exemplary embodiments and other exemplary embodiments are also intended to be included within the scope of the appended claims. The inventive concept is defined by the above claims, wherein equivalent forms of the claims are included herein.
Claims
1. A method for a memory lookup mechanism in a storage system, include: performing, by a memory die, a multiplication operation using a lookup table methodology by accessing a lookup table stored on the memory die, the lookup table including a floating point operation result, wherein the lookup table methodology includes one of a floating point full product multiplication method, a floating point partial product multiplication method, a floating point tensor operation method, and a floating point fusion layer operation method; The memory die sends the result of the multiplication operation to a logic die, wherein the logic die includes a processor and a buffer; as well as A matrix multiplication operation is performed by the logic die using the result of the multiplication operation performed on the memory die.
2. The method according to claim 1, wherein the floating point full product multiplication method include: retrieving, by the memory die, a full scalar multiplication result from the lookup table using a full scalar floating point input, the lookup table comprising a plurality of full product multiplication results; sending, by the memory die, the complete scalar multiplication result from the lookup table to the buffer; sending the complete scalar multiplication result from the buffer to an accumulator on the logic die; as well as The complete scalar multiplication results are accumulated by the accumulator.
3. The method according to claim 1, wherein the floating point partial product multiplication method include: retrieving, by the memory die, a partial product multiplication result from the lookup table including a plurality of partial product multiplication results using a fractional scalar floating point input; sending, by the memory die, the partial product multiplication result to the buffer; sending the partial product multiplication result by the buffer to a regulator on the logic die; The adjuster sends the partial product multiplication result to an accumulator on the logic die; as well as The partial product multiplication results are accumulated by the accumulator.
4. The method according to claim 1, wherein the floating point tensor operation method include: retrieving, by the memory die, a tensor multiplication result from the lookup table using a tensor floating point input, the lookup table comprising a plurality of tensor multiplication results; sending, by the memory die, the tensor multiplication result to the buffer; as well as The tensor multiplication result is sent by the buffer to a tensor post-processor on the logic die.
5. The method according to claim 1, wherein the floating point fusion layer operation method include: retrieving, by the memory die, a full product multiplication result from the lookup table using full scalar floating point, wherein the lookup table includes a plurality of full product multiplication results; sending, by the memory die, the full product multiplication result to a buffer die, the buffer die comprising a plurality of buffers separate from the buffers of the logic die; sending, by the buffer die, the full product multiplication result to the buffer of the logic die; as well as The full product multiplication result is sent by the buffer of the logic die to a tensor processor on the logic die.
6. The method according to claim 1, further comprising: include: storing the lookup table associated with a first type of operation on the memory die; as well as The lookup table associated with a second type of operations different from the first type of operations is stored on the memory die.
7. The method according to claim 1, further comprising: include: Read every possible position in the first layer input feature map; Convolve the first layer input feature map with the first layer kernel; as well as The result of generating a second layer input feature map is based on the convolution.
8. A memory search mechanism, include: a logic die including a processor and a buffer, wherein the processor utilizes a computation unit to perform a matrix multiplication operation; as well as a memory die comprising a lookup table, wherein the lookup table comprises floating point operation results, and wherein the memory die performs operations including: accessing the lookup table; Implementing the multiplication operation using a lookup table methodology, wherein the lookup table methodology includes one of a floating point full product multiplication method, a floating point partial product multiplication method, a floating point tensor operation method, and a floating point fusion layer operation method; and A result of the multiplication operation is sent to the logic die for use by the logic die in performing a matrix multiplication operation.
9. The memory lookup mechanism of claim 8, wherein in response to the lookup table methodology being a floating point full product multiplication method, the memory die performs operations further comprising: retrieving a full scalar multiplication result from the lookup table using the full scalar floating point input, wherein the lookup table includes a plurality of full scalar multiplication results; and The complete scalar multiplication result is sent from the lookup table to the buffer.
10. The memory lookup mechanism of claim 9, wherein the logic die further comprises an accumulator, the accumulator performing operations comprising: receiving the complete scalar multiplication result from the buffer; and The complete scalar multiplication results are accumulated.
11. The memory lookup mechanism of claim 8, wherein in response to the lookup table methodology being a floating point partial product multiplication method, the memory die performs operations further comprising: retrieving a partial product multiplication result from the lookup table using a fractional scalar floating point input, wherein the lookup table includes a plurality of partial product multiplication results; and The partial product multiplication result is sent to the buffer.
12. The memory lookup mechanism of claim 11, wherein the logic die further comprises: include: an adjuster receiving the partial product multiplication result from the buffer; as well as An accumulator that performs operations including: receiving the partial product multiplication result from the adjuster; as well as The partial product multiplication results are accumulated.
13. The memory lookup mechanism according to claim 8, wherein in response to the lookup table methodology being a floating point tensor operation method, the memory die performs operations further comprising: retrieving a tensor multiplication result from the lookup table comprising a plurality of tensor multiplication results using the tensor floating point input; and Send the tensor multiplication result to the buffer.
14. The memory lookup mechanism of claim 13, wherein the logic die further comprises a tensor post-processor that receives the tensor multiplication result from the buffer.
15. The memory search mechanism according to claim 8, wherein in response to the lookup table methodology being a floating point fusion layer operation method, the memory die performs operations further comprising: retrieving a full product multiplication result from the lookup table using full scalar floating point, wherein the lookup table includes a plurality of full product multiplication results; and The full product multiplication result is sent to a buffer die.
16. The memory lookup mechanism of claim 15, wherein the logic die further comprises a tensor processor that receives the full product multiplication result from the buffer die.
17. The memory lookup mechanism of claim 8, wherein the logic die further comprises an accelerator die.
18. The memory lookup mechanism of claim 8, wherein the memory die comprises a dynamic random access memory die.
19. A memory die, wherein the memory die comprises a lookup table stored on the memory die and is configured to perform: Floating point full product multiplication method, include: retrieving, by the memory die, a full scalar multiplication result from the lookup table using a full scalar floating point input, the lookup table comprising a plurality of full scalar multiplication results; sending the complete scalar multiplication result to an accumulator from a buffer; as well as accumulating the complete scalar multiplication result by the accumulator; Floating-point partial product multiplication methods, including: retrieving, by the memory die, a partial product multiplication result from the lookup table using a fractional scalar floating point input, the lookup table comprising a plurality of partial product multiplication results; sending, by the memory die, the partial product multiplication result to the buffer; The buffer sends the partial product multiplication result to the adjuster; Accumulating the partial product multiplication results by the accumulator; Floating-point tensor operations, including: retrieving, by the memory die, a tensor multiplication result from the lookup table using a tensor floating point input, the lookup table comprising a plurality of tensor multiplication results; sending, by the memory die, the tensor multiplication result to the buffer; The buffer sends the tensor multiplication result to a tensor postprocessor; and a floating point fusion layer operation method, comprising: retrieving, by the memory die, a full product multiplication result from the lookup table using full scalar floating point, wherein the lookup table includes a plurality of full product multiplication results; sending, by the memory die, the full product multiplication result to a buffer die comprising a plurality of buffers; sending, by the buffer die, the full product multiplication result to the buffer; and The complete product multiplication result is sent by the buffer to a tensor processor.
