Calculation method and device for universal matrix vector multiplication

Abstract

The invention provides a calculation method and device for universal vector matrix multiplication, and the method comprises the steps: distributing an alignment part of a weight matrix to each in-memory calculation execution unit of an in-memory calculation device according to a first granularity, and controlling the in-memory calculation device to execute multiplication calculation of an input vector and the alignment part; the non-aligned part of the weight matrix is distributed to each in-memory calculation execution unit of the in-memory calculation device according to a second granularity, the in-memory calculation device is controlled to execute multiplication calculation of the input vector and the non-aligned part, and the size of the first granularity is larger than that of the second granularity.

Description

Technical Field

[0001] This application relates to the field of General Matrix-Vector Multiplication (GEMV) computation technology, and more specifically, to a GEMV computation method and apparatus. Background Technology

[0002] With the rapid development of deep learning technologies (e.g., Generative Artificial Intelligence, GenAI), applications based on modern deep neural networks (DNNs) have placed higher demands on off-chip memory bandwidth, exhibiting memory-intensive characteristics. Processing-in-Memory (PIM) technology, by embedding a Programmable Computing Unit (PCU) on the memory module, keeps computational operations on the memory module, thereby avoiding data transfer with the host (e.g., CPU / GPU), reducing the need for off-chip memory bandwidth, and thus lowering power consumption.

[0003] For example, the on-chip bandwidth of Dynamic Random Access Memory (DRAM) can be increased through bank-level parallelism of the PIM device, and the computation can be performed using the Programmable Computing Unit (PCU) built into the PIM device to reduce data movement, thereby improving energy efficiency.

[0004] However, in existing technologies, when performing GEMV calculations using a PIM device, if the input vector and weight matrix do not meet the requirements of the PIM device, the input vector and weight matrix must first be aligned by padding with zeros, and then GEMV is performed based on the aligned input vector and weight matrix. This causes some PIM execution units to perform matrix multiplications corresponding to zeros, resulting in low hardware resource utilization.

[0005] Therefore, for unaligned input vectors and / or weight matrices, how to improve the hardware utilization of in-memory computing devices and reduce computation latency is a problem that this invention urgently needs to solve. Summary of the Invention

[0006] The purpose of this invention is to provide a general matrix-vector multiplication calculation method and an apparatus for performing the method, so as to at least solve the problems in the above-mentioned related technologies, or without solving any of the above-mentioned problems.

[0007] According to one aspect of an embodiment of the present invention, a method for calculating a general matrix multiplication (GEMV) is provided, comprising: distributing the aligned portion of a weight matrix to each in-memory computation execution unit of an in-memory computing device according to a first granularity, and controlling the in-memory computing device to perform a multiplication calculation of an input vector with the aligned portion; distributing the unaligned portion of the weight matrix to each in-memory computation execution unit of the in-memory computing device according to a second granularity, and controlling the in-memory computing device to perform a multiplication calculation of the input vector with the unaligned portion, wherein the size of the input vector is [1, K], and the size of the weight matrix is ​​[K, N], where K = k m, where m is a positive integer. , where, and column interval The corresponding part is the unaligned part of the weight matrix, and the remaining part of the weight matrix is ​​the aligned part of the weight matrix, where k and n are values ​​corresponding to the type of the in-memory computing device, and the size of the first granularity is greater than the size of the second granularity.

[0008] According to embodiments of this disclosure, when processing unaligned GEMV calculations, the alignment granularity is reduced, resulting in less padding data. Therefore, when executing unaligned GEMVs, invalid data moved from the host to the in-memory computing device is reduced. Furthermore, since the amount of zero-padding data is reduced, the amount of invalid calculations performed by the PIM execution unit is reduced, thereby improving hardware utilization and reducing computational latency.

[0009] Optionally, the in-memory computing device is an HBM2-PIM device, where k=256 and n=4096.

[0010] Optionally, the size of the first granularity is [128,8], and the size of the second granularity is [128,1].

[0011] Optionally, the step of distributing the unaligned portion of the weight matrix to each in-memory compute execution unit of the in-memory computing device according to a second granularity includes: preferentially distributing the unaligned portion of the weight matrix on the storage banks of all in-memory compute execution units with a size of [128, 1] such that consecutive data blocks within each bank are ordered and each in-memory compute execution unit's bank is allocated at most a weight data of size [128, ceil(j / 512)], where ceil indicates an up-rounding function.

[0012] According to embodiments of this disclosure, the computing resources of each PIM execution unit can be fully utilized.

[0013] Optionally, the method further includes: generating a first in-memory computation instruction for the aligned portion and generating a second in-memory computation instruction for the unaligned portion, wherein the step of performing the multiplication of the input vector with the aligned portion includes: triggering the first in-memory computation instruction to control the in-memory computation device to perform the multiplication computation of the input vector with the aligned portion, and the step of performing the multiplication of the input vector with the unaligned portion includes: triggering the second in-memory computation instruction to control the in-memory computation device to perform the multiplication computation of the input vector with the unaligned portion.

[0014] Optionally, each in-memory computation execution unit includes eight first general-purpose vector registers for storing input vectors and eight second general-purpose vector registers for storing multiplication and accumulation results. For the aligned portion, each in-memory computation execution unit uses eight second general-purpose vector registers, and for the unaligned portion, each in-memory computation execution unit uses at most ceil(j / 512) second general-purpose vector registers, where ceil indicates the round-up function.

[0015] Optionally, the step of performing the multiplication calculation between the input vector and the unaligned portion includes: The method further includes: caching the multiplication and accumulation result in a second general-purpose vector register; and writing the multiplication and accumulation result cached in the second general-purpose vector register into the bank.

[0016] According to another aspect of the embodiments of this disclosure, a computational apparatus for General Vector Matrix Multiplication (GEMV) is provided, comprising: a first distribution unit configured to distribute an aligned portion of a weight matrix to each in-memory computation execution unit of an in-memory computing device according to a first granularity; a first control unit configured to control the in-memory computing device to perform a multiplication calculation of an input vector with the aligned portion; a second distribution unit configured to distribute an unaligned portion of the weight matrix to each PIM execution unit of the in-memory computing device according to a second granularity; and a second control unit configured to control the in-memory computing device to perform a multiplication calculation of the input vector with the unaligned portion, wherein the size of the input vector is [1, K], and the size of the weight matrix is ​​[K, N], where K = k m, where m is a positive integer. , where, and column interval The corresponding part is the unaligned part of the weight matrix, and the remaining part of the weight matrix is ​​the aligned part of the weight matrix, where k and n are values ​​corresponding to the type of the in-memory computing device, and the size of the first granularity is greater than the size of the second granularity.

[0017] Optionally, the in-memory computing device is an HBM2-PIM device, where k=256 and n=4096.

[0018] Optionally, the size of the first granularity is [128,8], and the size of the second granularity is [128,1].

[0019] Optionally, the second distribution unit is configured to distribute the unaligned portion of the weight matrix on the storage banks of all in-memory computation execution units with a size of [128, 1], such that consecutive data blocks within each bank are ordered and each in-memory computation execution unit's bank is allocated at most a weight data of size [128, ceil(j / 512)], where ceil indicates the rounding up function.

[0020] Optionally, the apparatus further includes: a generation unit configured to generate a first in-memory computation instruction for the aligned portion and a second in-memory computation instruction for the unaligned portion, wherein a first control unit is configured to: trigger the first in-memory computation instruction to control the in-memory computation device to perform a multiplication calculation of the input vector and the aligned portion, and a second control unit is configured to: trigger the second in-memory computation instruction to control the in-memory computation device to perform a multiplication calculation of the input vector and the unaligned portion.

[0021] Optionally, each in-memory computation execution unit includes eight first general-purpose vector registers for storing the input vector and eight second general-purpose vector registers for storing the multiplication and accumulation results. For the aligned portion, each in-memory computation execution unit uses eight second general-purpose vector registers, and for the unaligned portion, each PIM execution unit uses at most ceil(j / 512) second general-purpose vector registers, where ceil indicates the round-up function.

[0022] Optionally, the second control unit is configured to: cache the multiplication and accumulation result in a second general-purpose vector register; and write the multiplication and accumulation result cached in the second general-purpose vector register into the bank, wherein the apparatus further includes: a reduction and summation unit configured to perform reduction and summation on the column cells in the bank according to the number of second general-purpose vector registers used by each in-memory computation execution unit to obtain the multiplication result of the input vector and the unaligned portion.

[0023] According to another aspect of embodiments of the present disclosure, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, the processor causes the processor to implement the GEMV calculation method as described herein. Attached Figure Description

[0024] The above and other objects and features of the invention will become clearer from the following description, taken in conjunction with the accompanying drawings, which exemplarily illustrate one example, wherein: Figure 1 This is a schematic diagram illustrating the microstructure of an example PIM device according to an embodiment of the present disclosure; Figure 2 This is a schematic diagram illustrating the microarchitecture of a PIM execution unit according to an embodiment of the present disclosure; Figure 3 This is a flowchart illustrating an example of a PIM device performing GEMV operation according to an embodiment of the present disclosure; Figure 4 This is a schematic diagram illustrating the block division and rearrangement of the weight matrix according to an embodiment of the present disclosure; Figure 5 This is a schematic diagram illustrating an example of performing an unaligned GEMV operation according to an embodiment of the present disclosure; Figure 6 This is a graph illustrating the computational latency of GEMV and the corresponding PCU utilization according to an embodiment of the present disclosure; Figure 7 This is a flowchart illustrating a method for calculating GEMV according to an embodiment of the present disclosure; Figure 8 This is a schematic diagram illustrating the division of aligned and unaligned portions of the weighted data; Figure 9 This is a schematic diagram illustrating the weight matrix partitioning and rearrangement and the use of registers according to embodiments of the related technology and this disclosure; Figure 10 This is a schematic diagram illustrating an example of the execution process and reduction operation of a Multiply Accumulate (MAC) instruction according to an embodiment of the present disclosure; and Figure 11 This is a block diagram illustrating the structure of a computing device for GEMV according to an embodiment of the present disclosure. Detailed Implementation

[0025] Hereinafter, various embodiments of the present disclosure are described with reference to the accompanying drawings, wherein the same reference numerals are used to denote the same or similar elements, features, and structures. However, it is not intended that the present disclosure be limited to the specific embodiments described herein, and it is intended that the present disclosure cover all modifications, equivalents, and / or substitutions of the present disclosure, provided they fall within the scope of the appended claims and their equivalents. The terms and words used in the following description and claims are not limited to their dictionary meanings, but are used only to enable a clear and consistent understanding of the present disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is for illustrative purposes only and is not intended to limit the purpose of the present disclosure as defined by the appended claims and their equivalents.

[0026] It should be understood that, unless the context clearly indicates otherwise, the singular form includes the plural form. The terms “comprising,” “including,” and “having” as used herein indicate the presence of a disclosed function, operation, or element, but do not exclude other functions, operations, or elements.

[0027] For example, the expression “A or B” or “at least one of A and / or B” can indicate A and B, or A or B. For example, the expression “A or B” or “at least one of A and / or B” can indicate (1) A, (2) B or (3) both A and B.

[0028] In various embodiments of this disclosure, it is intended that when a component (e.g., a first component) is referred to as being "coupled" or "connected" to, or being "coupled" or "connected" to, another component (e.g., a second component), the component may be directly connected to, or may be connected via, another component (e.g., a third component). Conversely, when a component (e.g., a first component) is referred to as being "directly coupled" or "directly connected" to, or being directly coupled to or directly connected to, another component (e.g., a second component), there is no other component (e.g., a third component) between the component and the other component.

[0029] The expression “configured as” used in describing the various embodiments of this disclosure may be used interchangeably, for example, with expressions such as “suitable for,” “capable of,” “designed to,” “suitable for,” “manufactured as,” and “capable,” depending on the context. The term “configured as” may not necessarily indicate that the hardware is “specifically designed for.” Rather, in some cases, the expression “a device configured as…” may indicate that the device and another device or portion are “capable of….” For example, the expression “a processor configured to perform A, B, and C” may indicate a dedicated processor (e.g., an embedded processor) for performing the respective operations or a general-purpose processor (e.g., a central processing unit CPU or application processor (AP)) for performing the respective operations by executing at least one software program stored in a memory device.

[0030] The terminology used herein is intended to describe certain embodiments of this disclosure but is not intended to limit the scope of other embodiments. Unless otherwise indicated herein, all terms used herein (including technical or scientific terms) are to have the same meaning as commonly understood by one of ordinary skill in the art. Generally, terms as defined in dictionaries should be considered to have the same meaning as in the context of the relevant art and should not be interpreted differently or as having an overly formal meaning unless expressly defined herein. In no event should the terminology defined in this disclosure be construed as excluding embodiments of this disclosure.

[0031] As those skilled in the art will understand, in-memory computing integrates storage and computation into a single die during chip manufacturing, enabling the memory itself to possess a certain computational capability. Compared to near-memory computing, in-memory computing brings the "storage" and "computation" closer together. Therefore, the in-memory computing device described herein refers to a memory that includes the aforementioned die or a memory that utilizes in-memory computing technology.

[0032] Figure 1 This is a schematic diagram illustrating the microstructure of an example PIM device according to an embodiment of the present disclosure. (Refer to...) Figure 1 PIM devices can be implemented in a high-bandwidth memory (HBM) stack as multiple stacked PIM DRAM dies. A PIM DRAM die may include PIM execution units and memory banks (e.g., even memory banks and odd memory banks).

[0033] Figure 2 This is a schematic diagram illustrating the microarchitecture of a PIM execution unit according to an embodiment of the present disclosure.

[0034] Reference Figure 1 and Figure 2The PIM device may include 64 pseudo channels, each pseudo channel includes 8 PIM execution units, therefore the PIM device includes a total of 512 PIM execution units.

[0035] As an example, each PIM execution unit may include: a 16-way floating-point type 16 multiplier (FP16 MULT) and an adder (FP16 ADD); two banks: an even bank and an odd bank, for storing data; register resources, including 32 32-bit PIM command registers (CRF), 16 256-bit general vector registers (GRF), namely 8 256-bit GRF_A and 8 256-bit GRF_B; 16 16-bit scalar registers (SRF); and a PIM controller.

[0036] Figure 3 This is a flowchart illustrating an example of a PIM device performing GEMV operation according to an embodiment of the present disclosure.

[0037] Reference Figure 3 In step S301, the input vector and the weight matrix are aligned respectively.

[0038] First, define the two operands of the GEMV operation: the shape of the input vector: [1, K], and the shape of the weight data: [K, N], where K and N are the size of the reduction dimension and the output dimension, respectively. As those skilled in the art will understand, the shape of the input vector can also be referred to as the size of the input vector, and the shape of the weight data can be referred to as the size of the weight data.

[0039] Those skilled in the art will understand that a data block of size [S1, S2] can represent a data block of S1 rows multiplied by S2 columns.

[0040] During the execution of GEMV by the PIM device, there are two constraints: 1) The input vector is aligned to 256 in the reduction dimension (i.e., the size of K should be an integer multiple of 256), and 2) The weight data is aligned to 256 in the reduction dimension and to 4096 in the output dimension (i.e., the size of N should be an integer multiple of 4096).

[0041] Therefore, for GEMVs that do not meet the above two constraints, i.e., unaligned GEMVs, when executing GEMVs using a PIM device, the input vector and weight matrix can be aligned separately. As those skilled in the art will understand, the alignment described herein involves padding the newly added dimension with zeros so that the zero-padding input vector and weight data satisfy the above two constraints, in order to execute aligned GEMVs.

[0042] In step S302, the weighted data is divided into blocks and rearranged.

[0043] Figure 4 This is a schematic diagram illustrating the block division and rearrangement of the weight matrix according to an embodiment of the present disclosure.

[0044] Reference Figure 4 The weight data is divided into blocks of 256x4096, and each block is arranged in the even bank and odd bank of all PIM execution units in the order of [128,8].

[0045] In step S303, a PIM instruction is generated for the GEMV operation; In step S304, the PIM kernel is executed. Specifically, the host processor (e.g., an AMD GPU) triggers the corresponding PIM instructions to be executed on the PIM device in sequence through DRAM read / write memory access operations.

[0046] Figure 5 This is a schematic diagram illustrating an example of performing an unaligned GEMV operation according to an embodiment of the present disclosure.

[0047] Reference Figure 5 The input vector has a shape of [1, 256], and the weight data has a shape of [256, 2048]. Since the shape of the weight data does not meet constraint 2), i.e., it is not aligned in the output dimension, an alignment operation needs to be performed. After the alignment operation, the aligned weight data has a shape of [256, 4096]. Then, the aligned weight data is divided into blocks and rearranged.

[0048] The rearrangement of the weight data of the unaligned GEMV shows that only the PIM execution units of the first 32 pseudo-channels are used to perform valid GEMV operations, while the other PIM execution units perform invalid calculations due to padding.

[0049] As an example, the utilization rate of the PIM execution unit (or computing unit) or programmable computing unit (PCU) within a PIM device is defined by the following equation: ): ,in, The output dimension indicating the weights This represents the function for rounding up.

[0050] therefore, Figure 5 In =50%.

[0051] Figure 6 This is a graph illustrating the computational latency of GEMV and the corresponding PCU utilization of embodiments of the present disclosure.

[0052] Reference Figure 6 Define the unaligned interval: The area between the dashed vertical lines represents the unaligned interval. It can be seen that for a given unaligned interval, as the output dimension N decreases, the utilization rate of the PCU becomes increasingly lower, while the computation latency of the corresponding GEMV remains essentially unchanged due to padding. Therefore, if weight data is simply aligned using a "padding" operation, there is a problem of low PIM computation unit utilization when calculating unaligned GEMV operations.

[0053] The following describes a method for performing GEMV calculation by rearranging weighted data with finer granularity, which improves the utilization of PIM calculation units and thus reduces the computational latency of unaligned GEMV.

[0054] Figure 7 This is a flowchart illustrating a method for calculating the general matrix multiplication GEMV according to an embodiment of the present disclosure.

[0055] As an example, Figure 7 The method shown can be executed by the host.

[0056] Reference Figure 7 In step S701, the aligned portion of the weight matrix is ​​distributed to each PIM execution unit of the in-memory computing device according to the first granularity, and the in-memory computing device is controlled to perform the multiplication calculation of the input vector and the aligned portion. The size of the input vector is [1, K], and the size of the weight matrix is ​​[K, N], where K = k × m, and m is a positive integer. .

[0057] As an example, the part corresponding to the column interval [n×i, N] of the weight matrix (i.e., the part corresponding to the n×i+1th column to the Nth column of the weight matrix) is the unaligned part of the weight matrix, and the remaining part of the weight matrix is ​​the aligned part of the weight matrix, where k and n are values ​​corresponding to the type of the in-memory computing device, and the size of the first granularity is greater than the size of the second granularity.

[0058] As understood by those skilled in the art, granularity refers to the size of the unit used to divide data (e.g., aligned or unaligned portions).

[0059] In step S702, the unaligned portion of the weight matrix is ​​distributed to each PIM execution unit of the in-memory computing device according to the second granularity, and the in-memory computing device is controlled to perform the multiplication calculation of the input vector and the unaligned portion.

[0060] As understood by those skilled in the art, steps S701 and S702 can be executed sequentially or in reverse order, i.e., S702 can be executed first, followed by S701.

[0061] As an example, the in-memory computing device is an HBM2-PIM device, where k=256 and n=4096.

[0062] As an example, the size of the first granularity is [128,8], and the size of the second granularity is [128,1].

[0063] As an example, the step of distributing the unaligned portion of the weight matrix to each PIM execution unit of the in-memory computing device according to a second granularity includes: distributing the unaligned portion of the weight matrix in a priority of size [128,1] across the banks of all PIM execution units such that consecutive data blocks within each bank are ordered and each bank of a PIM execution unit is allocated at most weight data of size [128, ceil(j / 512)], where ceil indicates the rounding up function.

[0064] As understood by those skilled in the art, ceil(j / 512) represents the smallest integer greater than or equal to j / 512.

[0065] As an example, each PIM execution unit includes eight first general-purpose vector registers (e.g., GRF-A described below) for storing input vectors and eight second general-purpose vector registers (e.g., GRF-B described below) for storing multiplication and accumulation results. For the aligned portion, each PIM execution unit uses eight second general-purpose vector registers, and for the unaligned portion, each PIM execution unit uses at most ceil(j / 512) second general-purpose vector registers.

[0066] Figure 8 This is a schematic diagram illustrating the division of aligned and unaligned portions of weight data according to an embodiment of the present disclosure.

[0067] Reference Figure 8 The left side of the weight data corresponds to the aligned part of the weight data. It can be rearranged based on the first granularity; the unaligned portion of the weight data on the right side of the weight data. It can be rearranged based on a second granularity smaller than the first to take advantage of the parallel computing capabilities of PIM.

[0068] Figure 9 This is a schematic diagram illustrating the block division and rearrangement of weight matrices and the use of registers according to embodiments of the related technology and this disclosure.

[0069] The relevant techniques first perform an alignment operation on the weighted data of size [256, 2560] by padding with zeros (see [reference]). Figure 9 (a) In this context, the weight matrix is ​​then divided into blocks of size [256, 4096] based on the aligned weight data. These blocks are then further divided into blocks of size [128, 8] and sequentially distributed across the even and odd banks of all PIM execution units. This results in [128, 8] data blocks being stored in two consecutive rows within each bank. This weight data layout fully utilizes the PIM PCU's computational resources and GRF cache resources. However, as mentioned above... Figure 5 As mentioned above, the introduction of padding for alignment operations leads to insufficient PCU utilization in existing weighted data layout methods. For example, referring to... Figure 9 In (b), the data for the non-aligned portion is only allocated to a portion of the PIM execution units.

[0070] During GEMV computation, each GPU thread sequentially triggers eight PIM MultiplyAccumulate (MAC) instructions to fully utilize register resources. This disclosure rearranges the weight data layout in a fine-grained manner, offloading some of the sequentially executed MAC instructions to PCUs that perform ineffective computations on padding data, thereby achieving load balancing across all PCUs and improving PCU utilization. For example, see [reference to...] Figure 9 (c) The weight data is evenly distributed to all PIM execution units.

[0071] To maximize the utilization of the PCU, the weighted data layout rearrangement method disclosed herein aims to address the problem of uneven distribution of weighted data across all banks. It rearranges the weighted data layout in a fine-grained manner to maximize the utilization of the PCU's parallelism.

[0072] The core MAC instruction of the GEMV operation uses GRF_A to cache the input vector and GRF_B to cache the accumulated result of the MAC instruction.

[0073] For unaligned GEMV operations, the weighted data layout rearrangement method of this disclosure reduces the granularity of alignment, resulting in less data needing to be padded. Therefore, when performing unaligned GEMV, it reduces the amount of invalid data moved from the host to the in-memory computing device. Furthermore, since the amount of zero-padding data is reduced, the amount of invalid computation performed by the PIM execution unit is reduced, thereby improving hardware utilization and reducing computational latency. In addition, according to embodiments of this disclosure, for the unaligned portion... It is not necessary to use all 8 GRF_Bs; the maximum number of GRF_Bs that can be effectively used in practice is... The definition is as follows: .

[0074] This disclosure proposes a fine-grained layout rearrangement for unaligned weighted data, namely, prioritizing the distribution across all banks with a block size of [128, 1] while ensuring that consecutive data blocks within each bank are ordered. Figure 9 As shown in (c). Thus, the maximum storage size on each bank is [128, ...]. The data blocks, using GRF_B on demand, maximize the parallelism of the PCU.

[0075] For example, for an unaligned GEMV operation: [1,512]x[512,2560]=[1,2560], the weight matrix output dimension N=j=2560, that is, the unaligned part corresponds to... .

[0076] As those skilled in the art will understand, since 2560 is an integer multiple of 512, each PIM execution unit is assigned exactly weight data of size [128, 5]. However, if the unaligned GEMV operation is: [1, 512] x [512, 2561] = [1, 2561], then... 6. In this scenario, the first PIM execution unit is assigned a weight data block of size [128, 6], and the remaining 511 PIM execution units are assigned weight data of size [128, 5]. Due to the PIM execution unit calculation mechanism, zero-padding is required to ensure that the data size in the remaining 511 PIM execution units is also [128, 6]. Specifically, for each of the remaining 511 PIM execution units, a matrix of size [128, 1] filled with zeros (i.e., a matrix where all elements are 0) is added. During calculation, the 6 weight data blocks in each PIM execution unit... Used. Of the 511 PIM execution units, one for each PIM execution unit. Regarding invalid calculations, specifically 511... Used for invalid computations (i.e., computations involving the input vector and 0).

[0077] However, if the alignment granularity in the existing technology is used, a zero-padding operation is required to distribute each of the 512 PIM execution units with data of size [128, 8]. In this case, each PIM execution unit is used with 8 [missing information - likely referring to a specific number of data units]. .

[0078] Clearly, the hardware utilization rate of the PIM device can be improved according to the calculation method of this disclosure.

[0079] Table 1 below shows information related to the execution of unaligned GEMV operations by a PIM-based device (containing 512 PIM execution units) according to relevant technologies and embodiments of the present disclosure: [1,512]x[512,2560]=[1,2560].

[0080] Table 1

[0081] Furthermore, the data layout method disclosed herein reduces the granularity of alignment, resulting in less zero-padding data, which in turn reduces the amount of data moved from the host to the PIM device, thereby reducing the time overhead of the process.

[0082] The weighted data layout rearrangement process requires dividing the weighted data into blocks (256x4096) and then copying them block by block (128x8) from the host memory to the even bank and odd bank of the PIM execution unit. For unaligned GEMV operations, existing methods require padding the weighted data at a granularity of 4096 due to alignment constraints, resulting in the copying of a significant amount of invalid data. The method disclosed in this paper, however, uses a granularity of 512 for padding, which reduces the amount of invalid data copied and thus significantly reduces the latency of the rearrangement process.

[0083] As an example, Figure 7 The method may further include: generating a first PIM instruction for the aligned portion and generating a second PIM instruction for the unaligned portion, wherein the step of controlling the PIM device to perform a multiplication calculation of the input vector and the aligned portion includes: triggering the first PIM instruction to control the PIM device to perform a multiplication calculation of the input vector and the aligned portion, and the step of performing a multiplication calculation of the input vector and the unaligned portion includes: triggering the second PIM instruction to control the PIM device to perform a multiplication calculation of the input vector and the unaligned portion.

[0084] As an example, PIM instructions are stored in the command register file (CRF). In PIM computing mode, the host processor triggers the PIM instructions in the CRF sequentially through standard DRAM memory access operations (Read / Write).

[0085] Table 2 below shows the general form of the PIM command corresponding to GEMV calculation.

[0086] Table 2

[0087] Table 3 illustrates the PIM instructions involved in the above GEMV calculation. Table 3

[0088] Among them, the number of loops in the JUMP instruction ( ) can be represented as:

[0089] The formula above shows the number of MAC instruction iterations during the execution of an unaligned GEMV operation. It is by The number of GRF_Bs used determines the weight data. Since the weight data is divided into aligned and unaligned parts, these two parts use different numbers of GRF_Bs when performing GEMV calculations. Therefore, different PIM instructions need to be generated for these two parts.

[0090] For example, for an unaligned GEMV operation: [1,512]x[512,6656]=[1,6656], where the output dimension 6656=4096x1+2560. The weighted data alignment part corresponds to the GEMV operation performed on it. The value is 15, corresponding to the GEMV operation performed on the unaligned portion. It is 5. It is 9.

[0091] Table 4 shows the PIM instructions for unaligned and aligned portions.

[0092] Table 4

[0093] As an example, the steps for performing multiplication of the input vector with the unaligned portion include: caching the multiplicative summation result in a second general-purpose vector register, and writing the multiplicative summation result cached in the second general-purpose vector register into a bank, wherein... Figure 7 The method shown may also include performing a reduction summation on the column cells in the bank based on the number of second general-purpose vector registers used by each PIM execution unit.

[0094] As an example, PIM calculation instructions and reduction summation operations can be triggered and executed on demand for aligned and unaligned portions of weighted data.

[0095] The PIM kernel execution process of GEMV operations includes the following two steps: 1) Trigger and execute PIM instructions. The host processor (e.g., AMD GPU) iteratively triggers MAC instructions to perform GEMV operations via DRAM memory access operations, and writes the multiply-accumulate result cached in GRF_B back to the bank; 2) The Host thread performs a reduction and summation operation. The host processor performs a reduction and summation operation on the accumulated results in the bank.

[0096] The number of memory access operations required to be triggered in step 1) as follows:

[0097] For unaligned GEMV operations, since the weight data is divided into aligned and unaligned parts, and these two parts use different amounts of GRF_B when performing GEMV calculation, the two steps of the above GEMV operation need to be performed separately for the aligned and unaligned parts of the weight data.

[0098] As an example, for the unaligned portion, the execution of MAC instructions and the execution of reduction summation operations need to be triggered iteratively on demand based on the number of GRF_B.

[0099] As understood by those skilled in the art, the PIM MAC instruction corresponds to a 16-way multiply-accumulate unit, which outputs a 16-way multiply-accumulate result (i.e., 16 FP16 elements, which are cached in one GRF_B during PIM computation and written to the bank after PIM computation). PIM cannot perform reduction summation on the 16-way accumulated result. The reduction summation operation is performed on the host side (e.g., AMD GPU). Therefore, reduction summation is to sum the 16 FP16 elements to obtain a single FP16 element as the output.

[0100] More specifically, the reduction and summation process sums the 16 FP16 elements within a single column unit of the bank. The sum is then used as one output dimension, and different column units are independent of each other (and different output dimensions are independent of each other). When `triggered_RD_instruction_cnt=5`, it indicates that the bank has 5 column units. For each of these 5 column units, the reduction and summation of its own 16 FP16 elements is performed independently to obtain 5 FP16 elements, i.e., 5 output dimensions. In other words, each PIM execution unit can obtain the input vector and the result of multiplication of the 5 columns of weighted data.

[0101] Figure 10 This is a schematic diagram illustrating an example of the MAC instruction execution process and the reduction summation operation according to an embodiment of the present disclosure.

[0102] Reference Figure 10 For unaligned GEMV operations: [1, 256] x [256, 2560] = [1, 2560], firstly, based on the number of GRF_Bs used, the corresponding memory access READ operation is executed as needed to trigger the MAC instruction. In this example, the number of... Then, during the reduction and summation phase, reduction and summation are performed on the column units in the PIM bank as needed. To ensure the correctness of the result, the operation is performed because... Therefore, this operation is performed on the first 5 column cells.

[0103] Reference above Figures 1 to 10 A method for calculating GEMV according to embodiments of the present disclosure is described below, with reference to... Figure 11 A computing device for GEMV according to embodiments of the present disclosure will be described.

[0104] Figure 11 This is a block diagram illustrating the structure of a computing device for GEMV according to an embodiment of the present disclosure.

[0105] Reference Figure 11The computing device 1100 includes a first distribution unit 1110, a first control unit 1120, a second distribution unit 1130, and a second control unit 1140.

[0106] As an example, the first distribution unit 1110 may be configured to distribute the aligned portion of the weight matrix to each in-memory computing execution unit of the in-memory computing device at a first granularity.

[0107] As an example, the first control unit 1120 may be configured to control the in-memory computing device to perform a multiplication calculation of the input vector and the aligned portion; As an example, the second distribution unit 1130 may be configured to distribute the unaligned portion of the weight matrix to each PIM execution unit of the in-memory computing device according to a second granularity.

[0108] As an example, the second control unit 1140 may be configured to control the in-memory computing device to perform a multiplication calculation of the input vector and the unaligned portion.

[0109] As an example, the size of the input vector is [1, K], and the size of the weight matrix is ​​[K, N], where K = k m, where m is a positive integer. , where, and column interval The corresponding part is the unaligned part of the weight matrix, and the remaining part of the weight matrix is ​​the aligned part of the weight matrix, where k and n are values ​​corresponding to the type of the in-memory computing device, and the size of the first granularity is greater than the size of the second granularity.

[0110] As an example, the in-memory computing device is an HBM2-PIM device, where k=256 and n=4096.

[0111] As an example, the size of the first granularity is [128,8], and the size of the second granularity is [128,1].

[0112] As an example, the second distribution unit 1130 is configured to distribute the unaligned portion of the weight matrix on the storage banks of all in-memory computation execution units with a size of [128, 1], such that consecutive data blocks within each bank are ordered and each in-memory computation execution unit's bank is allocated at most a weight data of size [128, ceil(j / 512)], where ceil indicates the rounding up function.

[0113] As an example, the computing device 1100 further includes: a generation unit (not shown) configured to generate a first in-memory computation instruction for the aligned portion and a second in-memory computation instruction for the unaligned portion, wherein a first control unit 1120 may be configured to: trigger the first in-memory computation instruction to control the in-memory computing device to perform a multiplication calculation of the input vector and the aligned portion, and a second control unit 1140 may be configured to: trigger the second in-memory computation instruction to control the in-memory computing device to perform a multiplication calculation of the input vector and the unaligned portion.

[0114] As an example, each in-memory computation execution unit includes eight first general-purpose vector registers for storing the input vector and eight second general-purpose vector registers for storing the multiplication and accumulation results. For the aligned portion, each in-memory computation execution unit uses eight second general-purpose vector registers, and for the unaligned portion, each PIM execution unit uses at most ceil(j / 512) second general-purpose vector registers, where ceil indicates the round-up function.

[0115] As an example, the second control unit 1140 may be configured to: cache the multiplication and accumulation result in a second general-purpose vector register; and write the multiplication and accumulation result cached in the second general-purpose vector register into the bank, wherein the computing device 1100 may further include: a reduction and summation unit (not shown) configured to perform reduction and summation on the column cells in the bank according to the number of second general-purpose vector registers used by each in-memory computing execution unit to obtain the multiplication result of the input vector and the unaligned portion.

[0116] According to embodiments of this disclosure, a computer-readable storage medium storing instructions is also provided, wherein when the instructions are executed by at least one processor, the at least one processor causes the at least one processor to perform a GEMV calculation method according to embodiments of this disclosure. Examples of computer-readable storage media herein include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0117] According to embodiments of this disclosure, a computer program product may also be provided, wherein the instructions in the computer program product are executable by a processor of a computer device to perform the GEMV calculation method described herein.

[0118] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

Claims

1. A general method for calculating vector-matrix multiplication, comprising: The aligned portion of the weight matrix is ​​distributed to each in-memory computation execution unit of the in-memory computing device according to the first granularity, and the in-memory computing device is controlled to perform multiplication calculation of the input vector and the aligned portion. as well as The unaligned portion of the weight matrix is ​​distributed to each in-memory computation execution unit of the in-memory computing device according to the second granularity, and the in-memory computing device is controlled to perform the multiplication calculation of the input vector and the unaligned portion. The input vector has a size of [1, K], and the weight matrix has a size of [K, N], where K = k m, where m is a positive integer. , where, and column interval The corresponding part is the unaligned portion of the weight matrix, and the remaining part of the weight matrix is ​​the aligned portion of the weight matrix, where k and n are values ​​corresponding to the type of the in-memory computing device. The size of the first particle size is larger than the size of the second particle size.

2. The method as described in claim 1, wherein, The in-memory computing device is an HBM2-PIM device, where k=256 and n=4096.

3. The method as described in claim 2, wherein, The size of the first granularity is [128, 8], and the size of the second granularity is [128, 1].

4. The method of claim 3, wherein, The step of distributing the unaligned portion of the weight matrix to each in-memory computing execution unit of the in-memory computing device according to a second granularity includes: The unaligned portion of the weight matrix is ​​distributed across the storage banks of all in-memory computation execution units with a priority of [128, 1], such that consecutive data blocks within each bank are ordered and each in-memory computation execution unit's bank is allocated at most a weight data of size [128, ceil(j / 512)], where ceil indicates the rounding up function.

5. The method of claim 1, further comprising: Generate a first in-memory computation instruction for the aligned portion, and generate a second in-memory computation instruction for the unaligned portion. The step of performing the multiplication of the input vector with the aligned portion includes: A first in-memory computation instruction is triggered to control the in-memory computing device to perform a multiplication calculation between the input vector and the aligned portion. The steps of performing the multiplication of the input vector with the unaligned portion include: A second in-memory computation instruction is triggered to control the in-memory computation device to perform a multiplication calculation of the input vector and the unaligned portion.

6. The method of claim 3, wherein, Each in-memory computation execution unit includes eight first general-purpose vector registers for storing input vectors and eight second general-purpose vector registers for storing multiplication and accumulation results. For the aligned portion, each in-memory computation execution unit uses eight second general-purpose vector registers, and for the unaligned portion, each in-memory computation execution unit uses at most ceil(j / 512) second general-purpose vector registers, where ceil indicates the round-up function.

7. The method of claim 5, wherein the step of performing the multiplication calculation of the input vector and the unaligned portion comprises: The result of the multiplication and accumulation is cached in the second general-purpose vector register; And write the multiplication and accumulation result cached in the second general-purpose vector register into the bank. The method further includes: The column cells in the bank are reduced and summed according to the number of second general-purpose vector registers used by each in-memory computation execution unit to obtain the multiplication result of the input vector and the unaligned portion.

8. A computational device for general vector-matrix multiplication, comprising: The first distribution unit is configured to distribute the aligned portion of the weight matrix to each in-memory compute execution unit of the in-memory compute device according to a first granularity; A first control unit is configured to control the in-memory computing device to perform a multiplication calculation of the input vector and the aligned portion; The second distribution unit is configured to distribute the unaligned portion of the weight matrix to each PIM execution unit of the in-memory computing device according to a second granularity. as well as The second control unit is configured to control the in-memory computing device to perform a multiplication calculation between the input vector and the unaligned portion. The input vector has a size of [1, K], and the weight matrix has a size of [K, N], where K = k m, where m is a positive integer. , where, and column interval The corresponding part is the unaligned portion of the weight matrix, and the remaining part of the weight matrix is ​​the aligned portion of the weight matrix, where k and n are values ​​corresponding to the type of the in-memory computing device. The size of the first particle size is larger than the size of the second particle size.

9. The apparatus of claim 8, wherein, The in-memory computing device is an HBM2-PIM device, where k=256 and n=4096.

10. A computer-readable storage medium storing a computer program, wherein, When the computer program is executed by a processor, the processor causes the processor to implement the method as described in any one of claims 1-7.

Images