Computing method and apparatus for general matrix multiplication

Abstract

A computing method and apparatus for general matrix multiplication (GEMM) are provided. The method comprises: in response to determining that a host needs to obtain a first calculation result of multiplication of first weight data and an input vector, performing a multiplication operation of second weight data and the input vector to obtain a second calculation result, wherein the first weight data satisfies a first arrangement rule for the host, and the second weight data is obtained by arranging the first weight data based on a second arrangement rule for a PIM apparatus different from the first arrangement rule; and obtaining the first calculation result based on the second calculation result.

Description

Technical Field

[0001] This application relates to the field of matrix computation technology, and more specifically, to a computational method and apparatus for General Matrix Multiplication (GEMM). Background Technology

[0002] When an AI model infers on a heterogeneous computing device (e.g., GPU-PIM (Processing In Memory)), the weight data of some layers of the model may be used by the GPU and may also be used by the PIM device.

[0003] When performing calculations on weight data, the PIM device has specific data layout requirements for the model's weight data, but existing host (e.g., CPU or GPU) kernel functions (e.g., GEMM kernel functions) do not support the data layout requirements of the PIM device.

[0004] In existing GPU-PIM heterogeneous computing device solutions, to meet the usage requirements of weight data by both the GPU and the PIM device, in addition to storing weight data that satisfies the data layout requirements of the host or host kernel functions, weight data that also satisfies the data layout requirements of the PIM device is also stored. This is equivalent to storing an extra copy of redundant weights, resulting in double the memory consumption.

[0005] Therefore, it is urgent to solve the problem of high memory consumption when performing general matrix multiplication calculations. Summary of the Invention

[0006] The purpose of this invention is to provide a calculation method and apparatus for general matrix multiplication, so as to at least solve the problems in the above-mentioned related technologies, or at least not solve any of the above-mentioned problems.

[0007] According to one aspect of the embodiments of this disclosure, a calculation method for General Matrix Multiplication (GEMM) is provided, the method comprising: in response to determining that a host needs to obtain a first calculation result of multiplying first weight data by an input vector, performing a multiplication operation of second weight data by the input vector to obtain a second calculation result, wherein the first weight data has a first arrangement rule satisfying the host, and the second weight data is obtained by arranging the first weight data according to a second arrangement rule for a PIM device that is different from the first arrangement rule; and obtaining the first calculation result based on the second calculation result.

[0008] Optionally, the step of performing the multiplication operation of the second weight data with the input vector to obtain the second calculation result includes: obtaining the second weight data based on obtaining a second index by modifying the first index in the GEMM kernel function of the host for reading the first weight data, wherein when multiple thread blocks read the second weight data based on the second index, the weight data read by each of the multiple thread blocks in the second weight data is continuous; and performing the multiplication operation of the obtained second weight data with the input vector to obtain the second calculation result.

[0009] Optionally, the step of obtaining the second weight data based on a second index obtained by modifying a first index in the GEMM kernel function of the host for reading the first weight data includes: reading the second weight data from the global memory into the shared memory based on a fourth index obtained by modifying a third index in the GEMM kernel function for reading the first weight data from global memory into shared memory; reading the second weight data read into the shared memory from the shared memory into the local memory based on a sixth index obtained by modifying a fifth index in the GEMM kernel function for reading the first weight data from the shared memory into local memory; and wherein the step of performing a multiplication operation between the obtained second weight data and the input vector to obtain a second calculation result includes: performing a multiplication operation between the second weight data read into the local memory and the input vector to obtain a second calculation result.

[0010] Optionally, the second index is determined based on the first index and the parameters of the PIM device.

[0011] Optionally, the first weight data includes a first part of weight data, and the second part of weight data, which corresponds to the second index of the first part of weight data, is different from the first part of weight data.

[0012] Optionally, the step of obtaining the first calculation result based on the second calculation result includes: determining the index of the first weight data corresponding to the index of the second weight data based on the mapping relationship between the index of the first weight data and the index of the second weight data, and writing the second calculation result into global memory as the first calculation result based on the index of the first weight data corresponding to the index of the second weight data.

[0013] Optionally, the mapping relationship is determined based on the index of the first weight data and the parameters of the PIM device.

[0014] Optionally, the host is a CPU or a GPU.

[0015] According to another aspect of the embodiments of this disclosure, a computing apparatus for a General Matrix Multiplication (GEMM) is provided, comprising: a first obtaining unit configured to, in response to determining that a host needs to obtain a first calculation result by multiplying first weight data by an input vector, perform a multiplication operation of second weight data by the input vector to obtain a second calculation result, wherein the first weight data has a first arrangement rule satisfying a host, and the second weight data is obtained by arranging the first weight data according to a second arrangement rule for a PIM device that is different from the first arrangement rule; and a second obtaining unit configured to obtain the first calculation result based on the second calculation result.

[0016] Optionally, the first obtaining unit is configured to: obtain second weight data based on a second index obtained by modifying a first index in the GEMM kernel function of the host for reading the first weight data, wherein when multiple thread blocks read the second weight data based on the second index, the weight data read by each of the multiple thread blocks in the second weight data is continuous; and perform a multiplication operation between the obtained second weight data and the input vector to obtain a second calculation result.

[0017] Optionally, the first obtaining unit is configured to: read second weight data from the global memory into the shared memory based on a fourth index obtained by modifying a third index in the GEMM kernel function for reading first weight data from global memory into shared memory; read the second weight data read into the shared memory into the local memory based on a sixth index obtained by modifying a fifth index in the GEMM kernel function for reading first weight data from the shared memory into local memory; and perform a multiplication operation between the second weight data read into the local memory and the input vector to obtain a second calculation result.

[0018] Optionally, the second index is determined based on the first index and the parameters of the PIM device.

[0019] Optionally, the first weight data includes a first part of weight data, and the second part of weight data, which corresponds to the second index of the first part of weight data, is different from the first part of weight data.

[0020] Optionally, the second obtaining unit is configured to: determine the index of the first weight data corresponding to the index of the second weight data based on the mapping relationship between the index of the first weight data and the index of the second weight data, and write the second calculation result into the global memory as the first calculation result based on the index of the first weight data corresponding to the index of the second weight data.

[0021] Optionally, the mapping relationship is determined based on the index of the first weight data and the parameters of the PIM device.

[0022] Optionally, the host is a CPU or a GPU.

[0023] According to another aspect of the embodiments of this disclosure, an electronic device is provided, comprising: a memory storing computer-readable instructions; and a processor configured to execute the computer-readable instructions to implement the method as described above.

[0024] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, the processor causes the processor to implement the method as described above. Attached Figure Description

[0025] The above and other objects and features of the present invention will become clearer from the following description taken in conjunction with the accompanying drawings, which illustrate exemplary embodiments, wherein: Figure 1 This is a flowchart illustrating a calculation method for GEMM according to an embodiment of the present disclosure; Figure 2 A schematic diagram of memory space from the perspective of kernel functions is shown; and Figure 3 This is a block diagram illustrating the structure of a computing device for GEMM according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0026] Hereinafter, various embodiments of the present disclosure are described with reference to the accompanying drawings, wherein the same reference numerals are used to denote the same or similar elements, features, and structures. However, it is not intended that the present disclosure be limited to the specific embodiments described herein, and it is intended that the present disclosure cover all modifications, equivalents, and / or substitutions of the present disclosure, provided they fall within the scope of the appended claims and their equivalents. The terms and words used in the following description and claims are not limited to their dictionary meanings, but are used only to enable a clear and consistent understanding of the present disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is for illustrative purposes only and is not intended to limit the purpose of the present disclosure as defined by the appended claims and their equivalents.

[0027] It should be understood that, unless the context clearly indicates otherwise, the singular form includes the plural form. The terms “comprising,” “including,” and “having” as used herein indicate the presence of a disclosed function, operation, or element, but do not exclude other functions, operations, or elements.

[0028] For example, the expression “A or B” or “at least one of A and / or B” can indicate A and B, or A or B. For example, the expression “A or B” or “at least one of A and / or B” can indicate (1) A, (2) B or (3) both A and B.

[0029] In various embodiments of this disclosure, it is intended that when a component (e.g., a first component) is referred to as being "coupled" or "connected" to, or being "coupled" or "connected" to, another component (e.g., a second component), the component may be directly connected to, or may be connected via, another component (e.g., a third component). Conversely, when a component (e.g., a first component) is referred to as being "directly coupled" or "directly connected" to, or being directly coupled to or directly connected to, another component (e.g., a second component), there is no other component (e.g., a third component) between the component and the other component.

[0030] The expression “configured as” used in describing the various embodiments of this disclosure may be used interchangeably, for example, with expressions such as “suitable for,” “capable of,” “designed to,” “suitable for,” “manufactured as,” and “capable,” depending on the context. The term “configured as” may not necessarily indicate that the hardware is “specifically designed for.” Rather, in some cases, the expression “a device configured as…” may indicate that the device and another device or part of it are “capable of…”. For example, the expression “a processor configured to perform A, B, and C” may indicate a dedicated processor (e.g., an embedded processor) for performing the respective operations or a general-purpose processor (e.g., a central processing unit CPU or application processor (AP)) for performing the respective operations by executing at least one software program stored in a memory device.

[0031] The terminology used herein is intended to describe certain embodiments of this disclosure but is not intended to limit the scope of other embodiments. Unless otherwise stated herein, all terms used herein (including technical or scientific terms) are to have the same meaning as commonly understood by one of ordinary skill in the art. Generally, terms as defined in dictionaries should be considered to have the same meaning as in the context of the relevant field and should not be interpreted differently or as having an overly formal meaning unless expressly defined herein. In no event should the terminology defined in this disclosure be construed as excluding embodiments of this disclosure.

[0032] As understood by those skilled in the art, since the PIM device has specific data arrangement requirements for weight data, when using the PIM device to perform the calculation of weight data with a first arrangement rule (hereinafter referred to as the first weight data) and the input vector, it is necessary to first arrange the first weight data based on the specific data arrangement rule of the PIM device (hereinafter referred to as the second irregular arrangement), and then perform the multiplication of the arranged weight data (hereinafter referred to as the second weight data) and the input vector to obtain the multiplication calculation result.

[0033] Because the first and second layout rules differ, the host cannot directly use the second weight data when calculating the first weight data, and similarly, the PIM device cannot directly use the first weight data when calculating the second weight data. Therefore, to satisfy the needs of both the host and the PIM device for weight data, in addition to storing the first weight data that meets the data layout requirements of the host or host kernel function, it is also necessary to store the second weight data that meets the data layout requirements of the PIM device. This is equivalent to storing an extra redundant weight, resulting in double the memory usage. The calculation method and apparatus for GEMM proposed in this paper can obtain the multiplication result of the first weight data and the input vector based on the second weight data, while only storing the second weight data, thereby reducing memory usage.

[0034] As understood by those skilled in the art, the host described herein may be a CPU or a GPU, the PIM device refers to a device capable of performing PIM, and the weight data may refer to the weight data of the corresponding layer (e.g., a convolutional layer) of an artificial intelligence model.

[0035] Figure 1 A flowchart of a calculation method for GEMM according to an embodiment of the present disclosure is shown.

[0036] Reference Figure 1 In step S110, in response to determining that the host needs to obtain a first calculation result of multiplying the first weight data with the input vector, a second weight data multiplication operation with the input vector is performed to obtain a second calculation result. The first weight data has a first arrangement rule that satisfies the host, and the second weight data is obtained by arranging the first weight data according to a second arrangement rule for the PIM device that is different from the first arrangement rule.

[0037] In step S120, the first calculation result is obtained based on the second calculation result.

[0038] According to embodiments of this disclosure, when it is necessary to obtain the first calculation result of multiplying the first weight data with the input vector, the calculation result of multiplying the first weight data with the input vector can be obtained based on the second weight data. This allows only the second weight data to be stored in global memory without storing the first weight data, avoiding redundant storage of weight data and thus reducing global memory usage.

[0039] As an example, the step of performing a multiplication operation of the second weight data with the input vector to obtain a second calculation result of the multiplication of the second weight data with the input vector includes: obtaining the second weight data based on a second index obtained by modifying a first index in the GEMM kernel function of the host for reading the first weight data, wherein when multiple thread blocks read the second weight data based on the second index, the weight data read by each of the multiple thread blocks in the second weight data is continuous; and performing a multiplication operation of the obtained second weight data with the input vector to obtain a first calculation result.

[0040] As those skilled in the art will understand, when the host performs the multiplication of the first weight data with the input vector, it reads the first weight data based on the index in the host's GEMM kernel function and performs the multiplication of the read first weight data with the input vector. In other words, the host's GEMM kernel function is only suitable for reading weight data that conforms to the first arrangement rule, i.e., the first weight data. Therefore, in order to perform the multiplication of the second weight data with the input vector by the host, the index in the host's GEMM kernel function used to read the first weight data can be modified to an index suitable for reading the second weight data. Furthermore, through index transformation, each thread can read consecutive weight data when reading the second weight data.

[0041] Those skilled in the art will understand that the term "continuous" indicates that the weight data is stored in a continuous location in memory (e.g., global memory).

[0042] As an example, the first weight data may include a first part of weight data, and the second part of weight data, which corresponds to the second index of the first part of weight data, is different from the first part of weight data.

[0043] As an example, the block order of the original weight data (e.g., the first weight data) can be block 1, block 2, block 3, block 4, block 5, block 6, block 7, and block 8. The block order of the rearranged weight data (e.g., the second weight data) can be block 1, block 4, block 2, block 6, block 3, block 7, block 4, and block 8. In the original GEMM kernel function, the identifier (ID) of the thread block is the identifier of the accessed data block (e.g., block 1 to block 8). However, after rearrangement, in order to make the order of thread block accessing data consistent with the memory order, the ID of the thread block can be mapped to the ID of the data block as shown in Table 1 below.

[0044] Table 1

[0045] For example, before modifying the kernel function, the second thread block reads the weight data in block 2; after modifying the kernel function, the second thread block reads the weight data in block 4. Therefore, before and after modifying the kernel function, the weight data read by the same thread block can be different (e.g., thread block 2, thread block 3, etc.) or the same (e.g., thread block 1).

[0046] Those skilled in the art should understand that Table 1 and the values ​​in Table 1 are merely examples and do not limit this disclosure. Figure 2 A schematic diagram of memory space from the perspective of kernel functions is shown.

[0047] Reference Figure 2 The memory space within a kernel function is highly dependent on its thread model. Different levels of thread groups correspond to different memory spaces. Each thread has local memory within its System in Package (SIP), each thread block can have shared memory within its Simplified Instructional Computer (SIC), and shared memory allows all threads within a thread block to share data; while all threads in the entire thread grid can access global memory / constant memory. The inputs and outputs of each kernel function must reside in global memory.

[0048] As an example, the step of obtaining second weight data based on a second index obtained by modifying a first index in the GEMM kernel function of the host for reading first weight data includes: reading the second weight data from the global memory into the shared memory based on a fourth index obtained by modifying a third index in the GEMM kernel function for reading the first weight data from global memory into shared memory; reading the second weight data read into the shared memory from the shared memory into the local memory based on a sixth index obtained by modifying a fifth index in the GEMM kernel function for reading the first weight data from the shared memory into local memory; and wherein the step of performing a multiplication operation between the obtained second weight data and the input vector to obtain a first calculation result includes: performing a multiplication operation between the second weight data read into the local memory and the input vector to obtain a first calculation result. As an example, the third index indicates the index in the code segment of the GEMM kernel function used to read the first weight data from global memory into shared memory, and the fifth index indicates the index in the code segment of the GEMM kernel function used to read the first weight data in shared memory from shared memory into local memory.

[0049] As an example, to read or copy weight data from global memory to shared memory, the index of the weight data read from global memory by the thread block can be modified for the GEMM kernel function to ensure that the data read by each thread block is continuous, and the thread block saves the current data to shared memory. According to embodiments of this disclosure, since the access speed of shared memory is higher than that of global memory, weight data can be read into shared memory. When performing multiplication of weight data with input vector, weight data can be obtained more quickly, thereby improving the efficiency of multiplication calculation.

[0050] As an example, to facilitate the copying or reading of weight data from shared memory to local memory, the weight index for each thread reading from shared memory within the thread block of the GEMM kernel function can be modified. This ensures that each thread reads the correct data, and then each thread saves the read data to local memory. By modifying the index for retrieving weight data from shared memory, the data arrangement in local memory is guaranteed to meet the computational requirements of the kernel function. As an example, the index used in the kernel function to read weights with the original data layout in shared memory can be converted into an index suitable for reading weighted data that meets the PIM layout rules, thereby copying the appropriate data into local memory.

[0051] After writing the weight data to local memory based on the modified index used to read the weights from shared memory, it ensures that the data arrangement in local memory meets the data arrangement requirements of the GPU or GPU kernel functions.

[0052] As an example, the indices used to read shared memory (i.e., the indices used to read weight data from shared memory into local memory) and the indices used to write to local memory (i.e., the indices used to write the result of multiplying the weight data with the input vector into local memory) in the code segment of the GEMM kernel function can be modified to indices suitable for PIM layout rules.

[0053] As an example, the second, fourth, and sixth indices are determined based on the first index and the parameters of the PIM device.

[0054] As an example, the parameters of a PIM device may include at least one of the PIM device's specifications, model, and H / W characteristics.

[0055] As an example, the steps of obtaining the first calculation result based on the second calculation result include: determining the index of the first weight data corresponding to the index of the second weight data based on the mapping relationship between the index of the first weight data and the index of the second weight data; and writing the second calculation result into global memory as the first calculation result based on the index of the first weight data corresponding to the index of the second weight data.

[0056] As an example, the mapping relationship is determined based on the index of the first weight data and the parameters of the PIM device.

[0057] As an example, the conversion between the first and second indices, the third and fourth indices, and / or the fifth and sixth indices can be achieved based on the index mapping equation (1): map(origin_index)=rearrange_index (1), where origin_index and rearrange_index indicate the index of the original weights (e.g., the first weight data) and the index of the weights rearranged according to the layout rules of the PIM device (e.g., the second weight data), respectively.

[0058] Those skilled in the art should understand that when the first index of the first weight element of the first weight data is obtained based on equation (1) to obtain the second index, the weight element in the second weight data corresponding to the second index is the same as or different from the first weight element (as described in Table 2 above).

[0059] As an example, the index mapping relationship between the first weight data and the second weight data can be expressed as the following index remap equation (2): remap(rearrange_index)=origin_index(2).

[0060] Those skilled in the art should understand that if the first index of the first weight element in the first weight data corresponds to the index of the second weight element in the second weight data based on the mapping relationship, then the first weight element and the second weight element are the same weight element.

[0061] As an example, for the GEMM kernel function, the output write index of the thread block (i.e., the index used to write the calculation result from local memory to global memory) can be modified to write the calculation result to the correct index position, thereby ensuring that the output result of the kernel function still satisfies the data arrangement of the GEMM kernel function before the modification.

[0062] Equation (3) below shows an example of equation (2) for a PIM device.

[0063] origin_index = origin_row_index weight_width + origin_col_index (3), where origin_row_index = (rearrange_index / ((num_vrf_output 16384) weight_width)) (num_vrf_output 16384) + (rearrange_index % (2097152 num_vrf_output) / 2097152) 16384+ (rearrange_index%65536 / 32768) 8192+ (rearrange_index%32768 / 8192) 1024 + (rearrange_index%8192 / 4096) 4096 + (rearrange_index%4096 / 1024) 256 + (rearrange_index%1024 / 128) 32 + (rearrange_index%64 / 32) 16 + rearrange_index%16; origin_col_index = (rearrange_index % ((num_vrf_output 16384) weight_width) / (2097152 num_vrf_output)) 128 +(rearrange_index%2097152 / 65536) 4 + (rearrange_index%128 / 64) 2 + rearrange_index%32 / 16, rearrange_index = rearrange_row weight_width + rearrange_col, where weight_width indicates the width of the original weight.

[0064] Table 2 below shows the meaning of some of the parameters in equation (3).

[0065] Table 2

[0066]

[0067] Wherein, num_pim_chan represents the number of channels in the PIM device (e.g., 64); num_pim_blocks_per_chan represents the number of PIM processing units included in a single PIM channel (e.g., 16); num_element_per_vrf represents the number of elements included in each vector register in the PIM (e.g., 16); num_vrf_output=M / 16384 represents the number of vector registers required in the PIM device to store the calculation results, where M represents the M-dimensional weights; input_tile_size=SRF_size SRF_element=8 16=128, SRF_size and SRF_element represent the number of SRF registers in the PIM device and the amount of data that a single SRF can store (e.g., 16 scalars), respectively; num_element_per_column=16 represents the number of FP16 elements included in each column in the PIM device (e.g., 16). It should be understood that equation (3) and Table 1 are index transformations for the weights of the [16384,4096] dimension (i.e., weights of M=16384, K=4096).

[0068] It is important to understand that M and K in [M,K] indicate the dimensions of the original weights. After rearrangement, the dimensions of the weights are the same as those of the original weights. M corresponds to the weight height and H corresponds to the weight width.

[0069] As an example, for the GEMM kernel function, the output write index of the thread block (i.e., the index used to write the calculation result from local memory to global memory) can be modified to write the calculation result to the correct index position, thereby ensuring that the output result of the kernel function still satisfies the data arrangement of the GEMM kernel function before the modification.

[0070] For example, before modifying the kernel function, if the first thread block reads the original weight data, it will read the first part of the weight data, P1. After modifying the kernel function, when the first thread block reads the second weight data based on the modified kernel function, it will read the second part of the weight data, P2. The first part and the second part can be different data, but the amount of data (or the number of elements in the weight data) is the same.

[0071] When the host machine completes the multiplication of input vectors I and P2, if the result is written to global memory using the original kernel function, the result of the multiplication of I and P2 will be stored in the location where the result of the multiplication of I and P1 should be stored. Therefore, to ensure the result of the multiplication of I and P2 is written to the correct location instead of the location where the result of the multiplication of I and P1 should be stored, the index in the GEMM kernel function that writes the first result from the host's local memory to global memory needs to be modified. This modified index will ensure the result is written to the correct location. Equation (4) below shows an example of equation (1) for a PIM device.

[0072] rearrange_index = ((index % 16)+((index % 256 / 128) 16)+((index %1024 / 512) 4)+((index % 16384 / 8192) 64)+((index % 131072 / 16384) 256)+((index / 131072) 1024 ))+row_num(4) Where row_num = 4096 (origin_index / (weight_width / 4096)); index =origin_index%(weight_width 4096).

[0073] Table 3 below shows the meaning of some of the parameters in equation (4).

[0074] Table 3

[0075] Those skilled in the art should understand that the parameters of the PIM device mentioned above can be changed depending on the PIM device.

[0076] As an example, the computation of the GEMM kernel can be mainly divided into the following four parts: data preparation, data loading, GEMM computation, and result output.

[0077] As an example, assuming the address (or starting address) of the weight is A and the address (or starting address) of the output result is C, each thread can operate as follows: During the data preparation phase, the thread determines the original weight position to be read (which can be marked as ori_pointer) based on information such as the thread block and thread ID (thread_id). Then, the actual weight position to be accessed can be calculated according to the mapping equation (1) (which can be marked as actual_pointer=map(ori_pointer)). During the data loading phase, the thread reads the weight data from the global memory according to A+actual_pointer to load the data into the shared memory.

[0078] During the GEMM calculation phase, the thread calculates the original weight position to be read based on the thread block information, and marks it as ori_pointer. Then, it calculates the actual weight position to be accessed based on the mapping function, and marks it as actual_pointer=map(ori_pointer). Then, the thread reads data directly from shared memory into local memory based on A+actual_pointer for calculation.

[0079] In the result output stage, the thread can calculate the original output position to be written (which can be marked as ori_pointer_output) based on information such as the thread block, and then calculate the actual output position to be written (which can be marked as actual_pointer_output=map(ori_pointer_output)) based on the remap equation (2). Then the thread writes the calculation result to the position pointed to by C+actual_pointer_output, thereby obtaining the multiplication result of the first weight data and the input vector.

[0080] According to embodiments of this disclosure, by modifying the host GEMM kernel function to be suitable for reading rearranged weight data (i.e., second weight data), the second weight data can be obtained, thereby performing a multiplication of the second weight data with the input vector. This allows the result of the multiplication of the second weight data with the input vector to be subsequently used to obtain the result of the multiplication of the first weight data with the input vector. Furthermore, by modifying the write index for global memory, the multiplication result in global memory (i.e., the first calculation result) can be consistent with the calculation result obtained by performing the multiplication of the first weight data with the input vector.

[0081] Reference above Figures 1 to 2 A calculation method for GEMM according to exemplary embodiments of the present disclosure has been described below, with reference to... Figure 3 A computing device for GEMM according to exemplary embodiments of the present disclosure will be described.

[0082] Figure 3 This is a block diagram illustrating the structure of a computing device for GEMM according to an exemplary embodiment of the present disclosure.

[0083] Reference Figure 3 The computing device 300 for GEMM may include a first obtaining unit 310 and a second obtaining unit 320. Those skilled in the art will understand that the computing device 300 for GEMM may additionally include other components, and at least one of the components included in the computing device 300 for GEMM may be split or combined.

[0084] As an example, the first obtaining unit 310 may be configured to perform a multiplication operation of the second weight data and the input vector to obtain a second calculation result in response to determining that the host needs to obtain a first calculation result of multiplying the first weight data by the input vector, wherein the first weight data has a first arrangement rule for the host, and the second weight data is obtained by arranging the first weight data according to a second arrangement rule for the PIM device that is different from the first arrangement rule.

[0085] As an example, the second obtaining unit 320 can be configured to obtain the first calculation result based on the second calculation result.

[0086] As an example, the first obtaining unit 310 may be configured to: obtain second weight data based on a second index obtained by modifying a first index in the GEMM kernel function of the host for reading first weight data, wherein when multiple thread blocks read the second weight data based on the second index, the weight data read by each of the multiple thread blocks in the second weight data is continuous; and perform a multiplication operation between the obtained second weight data and the input vector to obtain a second calculation result.

[0087] As an example, the first obtaining unit 310 may be configured to: read second weight data from the global memory into the shared memory based on a fourth index obtained by modifying a third index in the GEMM kernel function for reading first weight data from global memory into shared memory; read the second weight data read into the shared memory from the shared memory into the local memory based on a sixth index obtained by modifying a fifth index in the GEMM kernel function for reading first weight data from the shared memory into local memory; and perform a multiplication operation between the second weight data read into the local memory and the input vector to obtain a second calculation result.

[0088] As an example, the second index is determined based on the first index and the parameters of the PIM device.

[0089] As an example, the first weight data includes a first part of weight data, and the second part of weight data, which corresponds to the second index of the first part of weight data, is different from the first part of weight data.

[0090] As an example, the second obtaining unit 320 can be configured to: determine the index of the first weight data corresponding to the index of the second weight data based on the mapping relationship between the index of the first weight data and the index of the second weight data, and write the second calculation result into global memory as the first calculation result based on the index of the first weight data corresponding to the index of the second weight data.

[0091] As an example, the mapping relationship is determined based on the index of the first weight data and the parameters of the PIM device.

[0092] As an example, the host is a CPU or a GPU.

[0093] According to embodiments of this disclosure, an electronic device may also be provided, comprising: a memory storing computer-readable instructions; and a processor configured to execute the computer-readable instructions to implement the methods described herein.

[0094] According to embodiments of this disclosure, a computer-readable storage medium storing instructions is also provided, wherein when the instructions are executed by at least one processor, the at least one processor causes the at least one processor to perform a computation method for general matrix multiplication according to embodiments of this disclosure. Examples of computer-readable storage media herein include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0095] According to embodiments of this disclosure, a computer program product may also be provided, wherein the instructions in the computer program product are executable by a processor of a computer device to perform the computational method for general matrix multiplication described herein.

[0096] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

Claims

1. A method for calculating the General Matrix Multiplication Method (GEMM), the method comprising: In response to determining that the host needs to obtain a first calculation result of multiplying a first weight data with an input vector, a second calculation result is obtained by multiplying the second weight data with the input vector. The first weight data satisfies a first arrangement rule for the host, and the second weight data is obtained by arranging the first weight data according to a second arrangement rule for the PIM device, which differs from the first arrangement rule. The first calculation result is obtained based on the second calculation result.

2. The method as described in claim 1, wherein, The step of performing the multiplication operation between the second weight data and the input vector to obtain the second calculation result includes: The second weight data is obtained by modifying the first index in the GEMM kernel function of the host to obtain the second index, wherein when multiple thread blocks read the second weight data based on the second index, the weight data read by each of the multiple thread blocks in the second weight data is continuous; and Perform a multiplication operation between the acquired second weight data and the input vector to obtain a second calculation result.

3. The method as described in claim 2, wherein, The step of obtaining the second weight data based on obtaining the second index by modifying the first index used to read the first weight data in the GEMM kernel function of the host includes: The second weight data is read from the global memory into the shared memory based on the fourth index obtained by modifying the third index in the GEMM kernel function used to read the first weight data from the global memory into the shared memory; The second weight data, which is read from the shared memory, is read into the local memory based on the sixth index obtained by modifying the fifth index in the GEMM kernel function used to read the first weight data from the shared memory into the local memory; and The step of performing the multiplication operation between the acquired second weight data and the input vector to obtain the second calculation result includes: Perform a multiplication operation between the second weight data read from the local memory and the input vector to obtain a second calculation result.

4. The method of claim 2, wherein, The second index is determined based on the first index and the parameters of the PIM device.

5. The method of claim 2, wherein, The first weight data includes a first part of weight data, and the second part of weight data, which corresponds to the second index of the first part of weight data, is different from the first part of weight data.

6. The method of claim 1, wherein the step of obtaining the first calculation result based on the second calculation result includes: Based on the mapping relationship between the index of the first weight data and the index of the second weight data, the index of the first weight data corresponding to the index of the second weight data is determined. Based on the index of the first weight data corresponding to the index of the second weight data, the second calculation result is written into the global memory as the first calculation result.

7. The method of claim 6, wherein, The mapping relationship is determined based on the index of the first weight data and the parameters of the PIM device.

8. The method of claim 1, wherein, The host is either a CPU or a GPU.

9. A computing device for General Matrix Multiplication (GEMM), comprising: A first obtaining unit is configured to, in response to determining that the host needs to obtain a first calculation result by multiplying first weight data by an input vector, perform a multiplication operation of second weight data by the input vector to obtain a second calculation result, wherein the first weight data has a first arrangement rule for the host, and the second weight data is obtained by arranging the first weight data according to a second arrangement rule for a PIM device, which is different from the first arrangement rule; and The second obtaining unit is configured to obtain the first calculation result based on the second calculation result.

10. An electronic device comprising: Memory, which stores computer-readable instructions; as well as A processor is configured to execute the computer-readable instructions to implement the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium storing a computer program, wherein, When the computer program is executed by a processor, it causes the processor to implement the method as described in any one of claims 1-8.

Images