Data processing method and device, electronic equipment and storage medium

Abstract

The application relates to a data processing method and device, electronic equipment and a storage medium. The method comprises the following steps: reading an i-th data block from global data based on an i-th data read-write instruction, and writing the i-th data block into a shared memory, wherein the data read-write instruction is an assembly instruction used for executing data reading and writing processing, the global data is divided into N data blocks, N is an integer greater than 0, and i is an integer less than or equal to N; reading the i-th data block from the shared memory into a vector register for multiply-accumulate processing to obtain a multiply-accumulate operation result of the i-th data block; obtaining a general matrix multiplication (GEMM) operation result for the global data according to multiply-accumulate operation results corresponding to the N data blocks; and writing the GEMM operation result into a global memory. The method provided in the embodiment of the application can improve the GEMM optimization effect.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of science and technology, GEMM (General Matrix to Matrix Multiplication) usually plays a very important role in computing applications. It is a computationally intensive and memory-intensive operation that places very high demands on the processor's computing power, memory bandwidth, and latency.

[0003] The optimized implementation method of GEMM can be found by referring to Figure 1 As shown. Figure 1 As shown, GEMM can typically divide global data into many data blocks and continuously move these data blocks to shared memory.

[0004] When global data is moved to shared memory, it needs to be read into a register first, and then the data in the register needs to be written back to shared memory. Due to the limited number of registers and bandwidth in the processor, GEMM operation has high latency and low efficiency, and the optimization effect of GEMM is not significant. Summary of the Invention

[0005] Based on this, this application provides a data processing method, apparatus, electronic device, and storage medium that can improve GEMM operation efficiency and reduce GEMM operation latency.

[0006] In a first aspect, this application provides a data processing method, the method comprising:

[0007] Based on the i-th data read / write instruction, the i-th data block is read from the global data and written into the shared memory. The data read / write instruction is an assembly instruction used to perform data read and write processing. The global data is divided into N data blocks, where N is an integer greater than 0 and i is an integer less than or equal to N.

[0008] The i-th data block is read from the shared memory into the vector register for multiplication and accumulation processing to obtain the result of the multiplication and accumulation operation of the i-th data block;

[0009] Based on the multiplication and accumulation results corresponding to the N data blocks, the general multiplication matrix (GEMM) operation result for the global data is obtained;

[0010] Write the GEMM calculation result into global memory.

[0011] Based on the data processing method provided in this disclosure, global data in global memory can be read directly into shared memory through assembly instructions without going through registers, which speeds up the data read and write speed, thereby reducing the operation latency and improving the GEMM operation efficiency. The optimization effect of GEMM is significant.

[0012] In one embodiment, the method further includes:

[0013] In response to a shared memory setup operation for each of the data blocks, the write address corresponding to each of the data blocks in the shared memory is written into the target register.

[0014] The data processing method provided in this disclosure can write the write addresses of each data block in the shared memory to the target register. Then, during data processing, the write addresses of each data block in the shared memory can be read from the target register through data read and write instructions, and the global data in the global memory can be directly read into the shared memory based on the read write addresses. This speeds up the data read and write rate, thereby reducing the computation latency and improving the GEMM computation efficiency. The optimization effect of GEMM is significant.

[0015] In one embodiment, the i-th data read / write instruction includes the read address of the i-th data block in the global memory, and the step of reading the i-th data block from the global data and writing the i-th data block into the shared memory based on the i-th data read / write instruction includes:

[0016] According to the read address of the i-th data block in the global memory in the i-th data read / write instruction, the i-th data block is read from the global data;

[0017] According to the i-th data read / write instruction, the write address corresponding to the i-th data block in the shared memory is read from the target register, and the i-th data block is written into the shared memory based on the write address corresponding to the i-th data block in the shared memory.

[0018] The data processing method provided in this disclosure can read the write address of each data block in the shared memory from the target register through data read and write instructions, and directly read the global data in the global memory into the shared memory based on the read write address, thereby accelerating the data read and write speed, reducing the operation latency, improving the GEMM operation efficiency, and the optimization effect of GEMM is significant.

[0019] In one embodiment, reading the i-th data block from the global data and writing the i-th data block into the shared memory includes:

[0020] Read the i-th data block from the global data and write the i-th data block into the first cache area of ​​the shared memory;

[0021] Read the (i+1)th data block from the global data and write the (i+1)th data block into the second cache area of ​​the shared memory;

[0022] The step of reading the i-th data block from the shared memory into the vector register for multiplication and accumulation processing to obtain the result of the multiplication and accumulation operation of the i-th data block includes:

[0023] The i-th data block is read from the first cache region of the shared memory into the vector register for multiplication and accumulation processing to obtain the result of the multiplication and accumulation operation of the i-th data block.

[0024] The data processing method provided in this embodiment can perform data transfer in the first cache area and the second cache area. That is, before the multiplication and accumulation operation is performed on the i-th data block, the execution of moving the i+1-th data block to the shared memory is performed first, thereby masking part of the instruction delay and improving the GEMM operation efficiency. The optimization effect of GEMM is significant.

[0025] In one embodiment, the data block comprises m rows and k columns of elements, the vector register comprises a first vector register and a second vector register, and the step of reading the i-th data block from the first cache region of the shared memory into the vector register for multiplication and accumulation processing to obtain the multiplication and accumulation result of the i-th data block includes:

[0026] From the first cache region of the shared memory, the j-th channel data of the i-th data block is read into the first vector register, where j is an integer less than or equal to k;

[0027] When j is less than k, the (j+1)th channel data of the i-th data block is read from the first cache region of the shared memory and entered into the second vector register;

[0028] Perform a multiplication and accumulation operation on the j-th channel data in the first vector register;

[0029] When j+1 is less than k, the j+2 channel data of the i-th data block is read from the first cache area of ​​the shared memory into the first vector register;

[0030] Perform a multiply-accumulate operation on the (j+1)th channel data in the second vector register.

[0031] If j+2 is less than k, j+2 is taken as the new j, and the steps of performing multiplication and accumulation operation on the j-th channel data in the first vector register are repeated until j+2 equals k. Then, the multiplication and accumulation operation on the j+2-th channel data in the second vector register is performed to obtain the multiplication and accumulation operation result of the i-th data block.

[0032] The data processing method provided in this disclosure can mask part of the instruction delay by performing data prefetching operations outside and inside the loop of the multiply-accumulate operation, thereby reducing the delay in the GEMM operation process and improving the GEMM operation efficiency.

[0033] In one embodiment, the size of the data block is positively correlated with the scale of the multiply-accumulate operation.

[0034] The data processing method provided in this disclosure can reduce the number of data reads by reasonably dividing the data into blocks, thereby improving the efficiency of GEMM operations and enhancing the optimization effect of GEMM operations.

[0035] Secondly, this application also provides a data processing apparatus, the apparatus comprising:

[0036] The first reading module is used to read the i-th data block from the global data based on the i-th data read / write instruction, and write the i-th data block into the shared memory. The data read / write instruction is an assembly instruction used to perform data reading and writing processing. The global data is divided into N data blocks, where N is an integer greater than 0 and i is an integer less than or equal to N.

[0037] The second reading module is used to read the i-th data block from the shared memory into the vector register for multiplication and accumulation processing to obtain the multiplication and accumulation result of the i-th data block;

[0038] The operation module is used to obtain the general multiplication matrix (GEMM) operation result for the global data based on the multiplication and accumulation operation results corresponding to the N data blocks;

[0039] The first writing module is used to write the GEMM operation results into global memory.

[0040] In one embodiment, the device further includes:

[0041] The second write module is configured to, in response to a shared memory setting operation for each of the data blocks, write the write address corresponding to each of the data blocks in the shared memory into a target register.

[0042] In one embodiment, the i-th data read / write instruction includes the read address of the i-th data block in the global memory, and the first read module is further configured to:

[0043] According to the read address of the i-th data block in the global memory in the i-th data read / write instruction, the i-th data block is read from the global data;

[0044] According to the i-th data read / write instruction, the write address corresponding to the i-th data block in the shared memory is read from the target register, and the i-th data block is written into the shared memory based on the write address corresponding to the i-th data block in the shared memory.

[0045] In one embodiment, the first reading module is further configured to:

[0046] Read the i-th data block from the global data and write the i-th data block into the first cache area of ​​the shared memory;

[0047] Read the (i+1)th data block from the global data and write the (i+1)th data block into the second cache area of ​​the shared memory;

[0048] The i-th data block is read from the first cache region of the shared memory into the vector register for multiplication and accumulation processing to obtain the result of the multiplication and accumulation operation of the i-th data block.

[0049] In one embodiment, the data block comprises m rows and k columns of elements, the vector register comprises a first vector register and a second vector register, and the first read module is further configured to:

[0050] From the first cache region of the shared memory, the j-th channel data of the i-th data block is read into the first vector register, where j is an integer less than or equal to k;

[0051] When j is less than k, the (j+1)th channel data of the i-th data block is read from the first cache region of the shared memory and entered into the second vector register;

[0052] Perform a multiplication and accumulation operation on the j-th channel data in the first vector register;

[0053] When j+1 is less than k, the j+2 channel data of the i-th data block is read from the first cache area of ​​the shared memory into the first vector register;

[0054] Perform a multiply-accumulate operation on the (j+1)th channel data in the second vector register.

[0055] If j+2 is less than k, j+2 is taken as the new j, and the steps of performing multiplication and accumulation operation on the j-th channel data in the first vector register are repeated until j+2 equals k. Then, the multiplication and accumulation operation on the j+2-th channel data in the second vector register is performed to obtain the multiplication and accumulation operation result of the i-th data block.

[0056] In one embodiment, the size of the data block is positively correlated with the scale of the multiply-accumulate operation.

[0057] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement any of the aforementioned data processing configuration methods.

[0058] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements any of the above-described data processing methods.

[0059] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements any of the aforementioned data processing methods.

[0060] The aforementioned data processing method, apparatus, electronic device, and storage medium, based on the i-th data read / write instruction, read the i-th data block from global data and write the i-th data block into shared memory. The data read / write instruction is an assembly instruction used to perform data read and write processing. The global data is divided into N data blocks, where N is an integer greater than 0 and i is an integer less than or equal to N. Then, the i-th data block is read from shared memory into a vector register for multiplication and accumulation processing to obtain the multiplication and accumulation result of the i-th data block. Based on the multiplication and accumulation results corresponding to the N data blocks, the general multiplication matrix (GEMM) operation result for the global data is obtained, and the GEMM operation result is written into global memory. Based on the data processing method, apparatus, electronic device, and storage medium provided in this disclosure, global data in global memory can be read directly into shared memory through assembly instructions without going through registers, accelerating the data read / write speed, thereby reducing computational latency, improving GEMM operation efficiency, and demonstrating significant optimization effects of GEMM. Attached Figure Description

[0061] Figure 1 This is a flowchart of an optimized implementation method of GEMM in one embodiment;

[0062] Figure 2This is a flowchart of a data processing method in one embodiment;

[0063] Figure 3 This is a flowchart of a data processing method in one embodiment;

[0064] Figure 4 This is a flowchart of a data processing method in one embodiment;

[0065] Figure 5 This is a flowchart of a data processing method in one embodiment;

[0066] Figure 6 This is a schematic diagram of a data processing method in one embodiment;

[0067] Figure 7 This is a structural block diagram of a data processing device in one embodiment;

[0068] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0070] Figure 2 This is a flowchart illustrating a data processing method according to an exemplary embodiment. This embodiment uses the application of the method to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and can be implemented through the interaction between the terminal and the server.

[0071] In this embodiment, the method includes the following steps:

[0072] In step 202, based on the i-th data read / write instruction, the i-th data block is read from the global data and written to the shared memory. The data read / write instruction is an assembly instruction used to perform data read and write processing. The global data is divided into N data blocks, where N is an integer greater than 0 and i is an integer less than or equal to N.

[0073] In this embodiment of the disclosure, the global data stored in the global memory can be pre-divided into N data blocks, and the corresponding assembly instructions (data read / write instructions) can be pre-written based on the read address of each data block in the global memory. For any data block, the assembly instruction corresponding to the data block is used to read the data block from the global memory and write it into the shared memory. That is, based on the i-th data read / write instruction, the i-th data block corresponding to the i-th data read / write instruction can be read from the global data and written into the shared memory.

[0074] In step 204, the i-th data block is read from the shared memory into the vector register for multiplication and accumulation processing to obtain the result of the multiplication and accumulation operation of the i-th data block.

[0075] In this embodiment of the disclosure, when the i-th data block is written into the shared memory, data can be read from the shared memory into the vector register to perform the corresponding multiply-accumulate processing in the vector register. After completing the multiply-accumulate processing of all data in the i-th data block, the corresponding multiply-accumulate operation result is obtained.

[0076] In step 206, the GEMM operation result for the global data is obtained based on the multiplication and accumulation operation results corresponding to the N data blocks;

[0077] In step 208, the GEMM calculation result is written to global memory.

[0078] In this embodiment of the disclosure, based on the data read and write instructions corresponding to each data block, the process of reading and writing the data block from global memory to shared memory, and reading it from shared memory to a vector register for multiplication and accumulation processing can be executed. After completing the multiplication and accumulation processing of N data blocks, the GEMM operation result of global data is obtained, and the GEMM operation result is written into global memory.

[0079] Based on the data processing method provided in this disclosure, the i-th data block is read from global data and written to shared memory using the i-th data read / write instruction. The data read / write instruction is an assembly instruction used to perform data read and write processing. The global data is divided into N data blocks, where N is an integer greater than 0 and i is an integer less than or equal to N. Then, the i-th data block is read from shared memory into a vector register for multiplication and accumulation processing to obtain the result of the multiplication and accumulation operation of the i-th data block. Based on the multiplication and accumulation operation results corresponding to the N data blocks, the general multiplication matrix (GEMM) operation result for the global data is obtained, and the GEMM operation result is written to global memory. Based on the data processing method provided in this disclosure, global data in global memory can be read directly into shared memory using assembly instructions without going through registers, thus accelerating the data read / write speed, reducing computational latency, and improving GEMM operation efficiency. The optimization effect of GEMM is significant.

[0080] In one exemplary embodiment, the above data processing method further includes:

[0081] In response to the shared memory setup operation for each data block, the write address corresponding to each data block in shared memory is written to the destination register.

[0082] In this embodiment, taking a 4×4 multiplication-accumulation operation, a 16×16 workgroup, a 64×64 block size, and a loop depth of 8 as an example, the data block selected for global data matrix A is 64×8 in size, and the data block selected for global data matrix B is also 64×8 in size. Each thread reads 64×8 / 256 = 2 data items. Therefore, the write address of the data block in shared memory is (thread_num&31)*2+(thread_num / 32)*64, where thread_num is the thread number, and the write address is the address where data is written to shared memory.

[0083] After obtaining the write addresses of each data block in shared memory, these addresses can be written into the destination register. This allows the read / write operation of the i-th data block based on the i-th data read / write instruction to retrieve the write address of the i-th data block in shared memory from the destination register, and then directly write the i-th data block read from global memory into shared memory based on this write address. The destination register is set before each global data read and is used to store the write data address in shared memory; for example, the destination register can be set to register m0.

[0084] The data processing method provided in this disclosure can write the write addresses of each data block in the shared memory to the target register. Then, during data processing, the write addresses of each data block in the shared memory can be read from the target register through data read and write instructions, and the global data in the global memory can be directly read into the shared memory based on the read write addresses. This speeds up the data read and write rate, thereby reducing the computation latency and improving the GEMM computation efficiency. The optimization effect of GEMM is significant.

[0085] In an exemplary embodiment, the i-th data read / write instruction includes the read address of the i-th data block in global memory, as shown below. Figure 3 As shown, in step 202, based on the i-th data read / write instruction, the i-th data block is read from the global data and written to the shared memory, which may include:

[0086] In step 302, the i-th data block is read from the global data according to the read address of the i-th data block in the global memory in the i-th data read / write instruction;

[0087] In step 304, according to the i-th data read / write instruction, the write address corresponding to the i-th data block in the shared memory is read from the target register, and the i-th data block is written into the shared memory based on the write address corresponding to the i-th data block in the shared memory.

[0088] In this embodiment, taking a 4×4 multiplication-accumulation operation, a 16×16 workgroup, a 64×64 block size, and a loop depth of 8 as an example, the data blocks selected for global data matrix A are both 64×8 in size, and the data blocks selected for global data matrix B are also 64×8 in size. Each thread reads 64×8 / 256 = 2 data items. The read address corresponding to the data block is = blockx*64 + (thread_num&31)*2 + (thread_num / 32)*width1, where blockx is the total number of blocks in global data matrix A, and width1 is the span between each channel in global data matrix A. Similarly, for global data matrix B, the read address corresponding to the data block is = blocky*64 + (thread_num&31)*2 + (thread_num / 32)*width2, where blocky is the total number of blocks in global data matrix B, and width2 is the span between each channel in global data matrix B.

[0089] The write instruction for the i-th data includes a read instruction, which contains the read address of the i-th data block in global memory. After the read instruction, the configuration of the shared memory write address is added. The hardware will automatically read the address (write address) of the shared memory write data from the target register. Based on the read address of the i-th data block in global memory in the i-th data read / write instruction, the i-th data block is read from the global data and written directly to the shared memory according to the read write address.

[0090] The data processing method provided in this disclosure can read the write address of each data block in the shared memory from the target register through data read and write instructions, and directly read the global data in the global memory into the shared memory based on the read write address, thereby accelerating the data read and write speed, reducing the operation latency, improving the GEMM operation efficiency, and the optimization effect of GEMM is significant.

[0091] In one exemplary embodiment, reference is made to Figure 4 As shown, in step 202, the i-th data block is read from the global data and written into the shared memory, which can be achieved through the following steps:

[0092] In step 402, the i-th data block is read from the global data and written to the first cache area of ​​the shared memory;

[0093] In step 404, the (i+1)th data block is read from the global data and written to the second cache area of ​​the shared memory;

[0094] In this embodiment, in step 204, the i-th data block is read from the shared memory into the vector register for multiplication and accumulation processing to obtain the result of the multiplication and accumulation operation of the i-th data block. This can be achieved through the following steps:

[0095] In step 406, the i-th data block is read from the first cache region of the shared memory into the vector register for multiplication and accumulation processing to obtain the result of the multiplication and accumulation operation of the i-th data block.

[0096] In this embodiment of the disclosure, the i-th data block can be read from the global data and written to the first cache area of ​​the shared memory. Then, the (i+1)-th data block can be read from the global memory and written to the second cache area of ​​the shared memory. The i-th data block can be read from the first cache area of ​​the shared memory into the vector register for multiplication and accumulation processing to obtain the multiplication and accumulation result of the i-th data block.

[0097] In this embodiment, the shared memory is configured with two cache areas. The i-th data block is placed in the first cache area through the first data read / write operation. Before the multiplication and accumulation operation of the i-th data block is executed, the next i+1 data block is moved to the second cache area through the i+1 data read / write instruction. The global data movement takes a relatively large number of cycles, which means that if the data is used immediately after movement, it will cause a relatively large delay. Therefore, the instruction to move the data is placed before the loop corresponding to the multiplication and accumulation operation. The data movement is ensured to be completed at the end of the loop. The intermediate operation steps can cover up some instruction delay.

[0098] The data processing method provided in this embodiment can perform data transfer through a first cache area and a second cache area. That is, before the multiplication and accumulation operation is performed on the i-th data block, the (i+1)-th data block is first transferred to the shared memory, thereby masking part of the instruction delay and improving the GEMM operation efficiency. The optimization effect of GEMM is significant.

[0099] In one exemplary implementation, the data block comprises m rows and k columns of elements, i.e., the data block is m×k, and the vector registers include a first vector register and a second vector register, as shown below. Figure 5 As shown, in step 406, the i-th data block is read from the first cache region of shared memory into the vector register for multiplication and accumulation processing to obtain the result of the multiplication and accumulation operation of the i-th data block, including:

[0100] In step 502, the j-th channel data of the i-th data block is read from the first cache region of the shared memory into the first vector register, where j is an integer less than or equal to k;

[0101] In step 504, if j is less than k, the (j+1)th channel data of the i-th data block is read from the first cache region of the shared memory into the second vector register;

[0102] In step 506, the j-th channel data in the first vector register is subjected to a multiplication-accumulation operation;

[0103] In step 508, if j+1 is less than k, the j+2 channel data of the i-th data block is read from the first cache region of the shared memory into the first vector register;

[0104] In step 510, a multiply-accumulate operation is performed on the data of the (j+1)th channel in the second vector register.

[0105] In step 512, if j+2 is less than k, j+2 is taken as the new j, and the multiplication and accumulation operation of the j-th channel data in the first vector register is repeated until j+2 equals k. Then, the multiplication and accumulation operation of the j+2-th channel data in the second vector register is performed to obtain the multiplication and accumulation operation result of the i-th data block.

[0106] In this embodiment of the present disclosure, the j-th channel data of the i-th data block can be pre-read from the first cache area into the first vector register. If the j-th channel is not the last channel of the i-th data block, before calculating the multiplication and accumulation operation of the j-th channel data, an instruction is executed to read the (j+1)-th channel data of the i-th data block into the second vector register from the first cache area in the shared memory, and then the multiplication and accumulation operation is performed on the j-th channel data in the first vector register.

[0107] Since there is a delay in reading data, and the data can only be placed in the register after some instruction cycles, the instruction to read some data (read the data of the (j+1)th channel) is executed before the multiplication and accumulation operation of the j-th channel data is performed. The calculation is not performed immediately, which can mask some of the instruction delay.

[0108] In this embodiment of the disclosure, the aforementioned step 404 can be executed after step 502. Inside the loop of the multiply-accumulate operation, a data read is performed first. Considering the data read latency, the multiply-accumulate calculation is not performed on the data just read (the (i+1)th data block). Instead, the multiply-accumulate calculation is performed on the data of the jth channel that was prefetched earlier. The multiply-accumulate operation is only performed on the data in the first cache area of ​​the shared memory until all the data in the first cache area of ​​the shared memory is calculated. Then, the data is retrieved from the second cache area of ​​the shared memory to continue the multiply-accumulate operation. This can reduce the global data read latency.

[0109] If j+1 is less than k, it can be determined that there is still data that has not been read into the vector register. Therefore, the data of the (j+2)th channel can be read into the first vector register, and the (j+1)th channel data in the second vector register can be multiplied and accumulated. Then, j+2 is used as the new j, and the process returns to step 506.

[0110] Until j+2 equals k, all channels of data in the i-th data block have been read into the vector register. Therefore, by performing a multiplication and accumulation operation on the k-th channel data in the second vector register, the result of the multiplication and accumulation operation of the i-th data block can be obtained, and the loop ends.

[0111] The data processing method provided in this disclosure can mask part of the instruction delay by performing data prefetching operations outside and inside the loop of the multiply-accumulate operation, thereby reducing the delay in the GEMM operation process and improving the GEMM operation efficiency.

[0112] In one exemplary embodiment, the size of the data block is positively correlated with the scale of the multiply-accumulate operation.

[0113] In this embodiment of the disclosure, the data block size can be selected based on the size of the global data. The data block size is positively correlated with the scale of the multiply-accumulate operation; that is, the larger the scale of the multiply-accumulate operation, the larger the data block size can be selected. The data block size is related to the number of threads used and the scale of the multiply-accumulate operation calculated by each thread, i.e., data block size = number of threads * scale of the multiply-accumulate operation.

[0114] For example, if the size of matrix A in the GEMM operation is M×K and the size of matrix B is K×N, then the size of the data block can be the product of M×N and the size of the multiply-accumulate operation. For instance, the size of the multiply-accumulate operation can be various, such as 2×2, 4×4, 4×6, 6×4, 4×8, 8×4, 8×8, etc., and the workgroup (thread group) can be 8×8, 8×16, 16×8, 16×16, 16×32, etc. Taking a 4×4 multiply-accumulate operation as an example, and a workgroup of 16×16, the size of the data block would be 64×64.

[0115] Multiplication and accumulation can be 2×2, 4×4, 4×6, 6×4, 4×8, 8×4, 8x8, etc., and workgroup can be 8×8, 8×16, 16×8, 16×16, 16×32, etc. Therefore, there are many sizes of corresponding data blocks (multiplication and accumulation * workgroup = data block size).

[0116] The specific size of the data block depends on the input global data. When M×N is relatively small, the multiplication and accumulation can be smaller, and vice versa.

[0117] The data processing method provided in this disclosure can reduce the number of data reads by reasonably dividing the data into blocks, thereby improving the efficiency of GEMM operations and enhancing the optimization effect of GEMM operations.

[0118] The domestically produced heterogeneous GPU platform has 32 banks, each occupying 4 bytes. Assuming each thread needs to read 64 bytes of data, if read continuously, two threads will fill the bank. This means that threads 0 and 1 do not conflict, but threads 0 and 2 do, as do threads 1 and 3, and so on. In this case, 16 threads only use 1 / 8 of the bank, resulting in severe conflict. In this embodiment, the data reading method can be changed. Considering that the largest data read instruction can read 16 bytes at a time, the 64 bytes of data are divided into four parts for reading, with 16 bytes read every 32 banks. This means that threads 0, 1...7 do not conflict, greatly reducing bank conflict.

[0119] In this embodiment of the disclosure, when performing GEMM calculations, the total number of grids = blockx * blocky; where blockx is the number of data blocks corresponding to matrix A, and blocky is the number of data blocks corresponding to matrix B. Each card has 64 CUs (Control Units). If there are exactly 64 grids, then one grid corresponds to one CU. If there are many grids, the correspondence between grids and CUs can be modified by setting blockx and blocky. Specifically, it can be calculated as follows:

[0120]

[0121]

[0122] In this embodiment, the domestically produced heterogeneous GPU platform has 64 Computation Units (CUs). If the input data is small, the number of data blocks is relatively small, and each data block can correspond to one CU, thus fully utilizing the CUs. If the data is large and there are many data blocks, multiple blocks of data need to be computed on one CU. This can be addressed by configuring the platform to place data with similar memory addresses onto the same CU for computation, thereby increasing the cache hit rate.

[0123] In this embodiment of the disclosure, the multiply-accumulate calculation can be implemented directly using instructions provided by domestic heterogeneous GPU platforms, such as v_mac_f32 or v_mac_f64 instructions. The continuous use of multiply-accumulate instructions can reduce the instruction issuance cycle and implement multiply-accumulate together as soon as possible.

[0124] To enable those skilled in the art to better understand the embodiments of this disclosure, the embodiments of this disclosure are described below through specific examples.

[0125] Reference Figure 6As shown, the partitioning of global data is related to the number of threads used and the size of the multiply-accumulate operation calculated by each thread. That is, the partition size = number of threads * size of the multiply-accumulate operation. The number of threads can be 16×16, 16×8, 32×16, etc., and the multiply-accumulate operation can be 2×2, 4×4, 6×6, 4×6, etc. Therefore, there are many different partitioning options. For example, if the number of threads is 16×16 and the size of the multiply-accumulate operation is 4×4, then the data block size = number of threads * size of the multiply-accumulate operation = 64x64. The choice of partitioning is also related to the size of the input data. For example, when the input data is relatively small, if the input data is 100×100, then a partitioning with a multiply-accumulate operation size of 64×64 will have better performance than a partitioning with a multiply-accumulate operation size of 128×128. If the input data is 1024×1024, then a partitioning with a multiply-accumulate operation size of 128×128 will have better performance than a partitioning with a multiply-accumulate operation size of 64×64.

[0126] Offset address calculation includes reading address calculation for global data matrices A and B, reading and writing address calculation for global data matrix C, and writing and reading address offsets for data in shared memory. The domestic heterogeneous GPU platform has 64 Computation Units (CUs). If the input data is small, it is divided into fewer data blocks, and each data block can correspond to one CU, fully utilizing the CUs. If the data is large and there are many data blocks, multiple data points need to be calculated on a single CU. This can be configured to place data with similar memory addresses on the same CU for computation, increasing cache hit rate.

[0127] Using data read / write instruction technology, global data can be moved directly to shared memory without going through vector registers. When performing this calculation, it is important to note that the instructions are used in units of 4 bytes. Therefore, when using GEMM, a double-precision data type needs to be divided into two single-precision data types for movement. In addition, the address for writing data to shared memory needs to be placed in the m0 register.

[0128] During data transfer, there is a delay in reading data, and the data can only be placed in the register after some instruction cycles. Therefore, by using a prefetch operation to read some data outside the loop and not immediately perform calculations, some instruction delays can be masked.

[0129] Inside the GEMM operation loop, a data block can be moved to shared memory once first. Considering the delay, instead of performing multiplication and accumulation calculations on the data block that was just read, multiplication and accumulation calculations are performed on the data that was previously prefetched. In this way, the delay of the data just read can be masked.

[0130] The shared memory uses two cache areas. The data moved the first time using data read / write instructions is placed in the first cache area, shared1. At the beginning of the loop, the data for the next data block is moved to the second cache area, shared2, using the same data read / write instructions. Global data movement takes a relatively long number of cycles, meaning that if the data is used immediately after movement, there will be a significant delay. Therefore, the instructions for moving data are placed at the beginning of the loop, and the data movement is ensured to be completed at the end of the loop. The intermediate operation steps can mask some of the instruction delay.

[0131] Multiply-accumulate calculations can be implemented directly using instructions provided by domestic heterogeneous GPU platforms, such as v_mac_f32 or v_mac_f64 instructions. The v_mac_f32 instruction requires both instruction issue time and instruction execution time. When multiple MAC instructions are implemented together, instruction issue and execution can mask each other, reducing time consumption. The continuous use of multiply-accumulate instructions can reduce the instruction issue cycle, allowing multiply-accumulate to be implemented together as quickly as possible.

[0132] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0133] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the memory frequency configuration method described above, and will not be repeated here.

[0134] In one embodiment, such as Figure 7 As shown, a data processing device is provided, including: a first reading module 702, a second reading module 704, a calculation module 706, and a first writing module 708, wherein...

[0135] The first reading module 702 is used to read the i-th data block from the global data based on the i-th data read / write instruction, and write the i-th data block into the shared memory. The data read / write instruction is an assembly instruction used to perform data reading and writing processing. The global data is divided into N data blocks, where N is an integer greater than 0 and i is an integer less than or equal to N.

[0136] The second reading module 704 is used to read the i-th data block from the shared memory into the vector register for multiplication and accumulation processing to obtain the multiplication and accumulation result of the i-th data block;

[0137] The operation module 706 is used to obtain the general multiplication matrix (GEMM) operation result for the global data based on the multiplication and accumulation operation results corresponding to the N data blocks.

[0138] The first writing module 708 is used to write the GEMM operation result into global memory.

[0139] The aforementioned data processing device, based on the i-th data read / write instruction, reads the i-th data block from the global data and writes the i-th data block into shared memory. The data read / write instruction is an assembly instruction used to perform data read and write processing. The global data is divided into N data blocks, where N is an integer greater than 0 and i is an integer less than or equal to N. Then, the i-th data block is read from shared memory into a vector register for multiplication and accumulation processing to obtain the result of the multiplication and accumulation operation of the i-th data block. Based on the multiplication and accumulation operation results corresponding to the N data blocks, the general multiplication matrix (GEMM) operation result for the global data is obtained, and the GEMM operation result is written into global memory. Based on the data processing device provided by this embodiment, global data in global memory can be read directly into shared memory through assembly instructions without going through registers, accelerating the data read / write speed, thereby reducing computational latency, improving GEMM operation efficiency, and demonstrating a significant optimization effect of GEMM.

[0140] In one embodiment, the device further includes:

[0141] The second write module is configured to, in response to a shared memory setting operation for each of the data blocks, write the write address corresponding to each of the data blocks in the shared memory into a target register.

[0142] In one embodiment, the i-th data read / write instruction includes the read address of the i-th data block in the global memory, and the first read module 702 is further configured to:

[0143] According to the read address of the i-th data block in the global memory in the i-th data read / write instruction, the i-th data block is read from the global data;

[0144] According to the i-th data read / write instruction, the write address corresponding to the i-th data block in the shared memory is read from the target register, and the i-th data block is written into the shared memory based on the write address corresponding to the i-th data block in the shared memory.

[0145] In one embodiment, the first reading module 702 is further configured to:

[0146] Read the i-th data block from the global data and write the i-th data block into the first cache area of ​​the shared memory;

[0147] Read the (i+1)th data block from the global data and write the (i+1)th data block into the second cache area of ​​the shared memory;

[0148] The i-th data block is read from the first cache region of the shared memory into the vector register for multiplication and accumulation processing to obtain the result of the multiplication and accumulation operation of the i-th data block.

[0149] In one embodiment, the data block comprises m rows and k columns of elements, the vector register comprises a first vector register and a second vector register, and the first read module is further configured to:

[0150] From the first cache region of the shared memory, the j-th channel data of the i-th data block is read into the first vector register, where j is an integer less than or equal to k;

[0151] When j is less than k, the (j+1)th channel data of the i-th data block is read from the first cache region of the shared memory into the second vector register;

[0152] Perform a multiplication and accumulation operation on the j-th channel data in the first vector register;

[0153] When j+1 is less than k, the j+2 channel data of the i-th data block is read from the first cache area of ​​the shared memory into the first vector register;

[0154] Perform a multiply-accumulate operation on the (j+1)th channel data in the second vector register.

[0155] If j+2 is less than k, j+2 is taken as the new j, and the steps of performing multiplication and accumulation operation on the j-th channel data in the first vector register are repeated until j+2 equals k. Then, the multiplication and accumulation operation on the j+2-th channel data in the second vector register is performed to obtain the multiplication and accumulation operation result of the i-th data block.

[0156] In one embodiment, the size of the data block is positively correlated with the scale of the multiply-accumulate operation.

[0157] Each module in the aforementioned memory frequency configuration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0158] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a data processing method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0159] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0160] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0161] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0162] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0163] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0164] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0165] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0166] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: Based on the i-th data read / write instruction, the i-th data block is read from the global data and written into the shared memory. The data read / write instruction is an assembly instruction used to perform data read and write processing. The global data is divided into N data blocks, where N is an integer greater than 0 and i is an integer less than or equal to N. The global data includes a first matrix and a second matrix, which are the input matrices for GEMM operations. When reading the i-th data block based on the i-th data read / write instruction, a memory conflict avoidance operation is performed, including: dividing the data of the i-th data block into multiple data sub-blocks according to the maximum number of bytes read by the assembly instruction; controlling multiple parallel threads to read one data sub-block every preset number of memory banks; and writing the read data sub-blocks into the shared memory; if the global data is of double-precision type, before executing the i-th data read / write instruction, the double-precision type data is split into two single-precision type data, and the shared memory address is written into the m0 register; the i-th data read / write instruction reads the shared memory address from the m0 register in units of 4 bytes. The i-th data block is read from the shared memory into the vector register for multiply-accumulate processing to obtain the result of the multiply-accumulate operation of the i-th data block. The multiply-accumulate processing uses at least one of the v_mac_f32 or v_mac_f64 instructions, and multiple multiply-accumulate instructions are arranged consecutively to mask the instruction issue and execution cycle. Based on the multiplication and accumulation results corresponding to the N data blocks, the general multiplication matrix (GEMM) operation result for the global data is obtained; Write the GEMM calculation result into global memory; Specifically, a correspondence between grids and computing units is established based on the number of first data blocks corresponding to the first matrix and the number of second data blocks corresponding to the second matrix. This correspondence enables data blocks with similar memory addresses to be mapped to the same computing unit.

2. The method according to claim 1, characterized in that, The method further includes: In response to a shared memory setup operation for each of the data blocks, the write address corresponding to each of the data blocks in the shared memory is written into the target register.

3. The method according to claim 2, characterized in that, The i-th data read / write instruction includes the read address of the i-th data block in the global memory. The step of reading the i-th data block from the global data and writing the i-th data block into the shared memory based on the i-th data read / write instruction includes: According to the read address of the i-th data block in the global memory in the i-th data read / write instruction, the i-th data block is read from the global data; According to the i-th data read / write instruction, the write address corresponding to the i-th data block in the shared memory is read from the target register, and the i-th data block is written into the shared memory based on the write address corresponding to the i-th data block in the shared memory.

4. The method according to claim 1, characterized in that, The step of reading the i-th data block from the global data and writing the i-th data block into the shared memory includes: Read the i-th data block from the global data and write the i-th data block into the first cache area of ​​the shared memory; Read the (i+1)th data block from the global data and write the (i+1)th data block into the second cache area of ​​the shared memory; The step of reading the i-th data block from the shared memory into the vector register for multiplication and accumulation processing to obtain the result of the multiplication and accumulation operation of the i-th data block includes: The i-th data block is read from the first cache region of the shared memory into the vector register for multiplication and accumulation processing to obtain the result of the multiplication and accumulation operation of the i-th data block.

5. The method according to claim 4, characterized in that, The data block comprises m rows and k columns of elements. The vector register includes a first vector register and a second vector register. The step of reading the i-th data block from the first cache region of the shared memory into the vector register for multiplication and accumulation processing to obtain the result of the multiplication and accumulation operation of the i-th data block includes: From the first cache region of the shared memory, the j-th channel data of the i-th data block is read into the first vector register, where j is an integer less than or equal to k; When j is less than k, the (j+1)th channel data of the i-th data block is read from the first cache region of the shared memory and entered into the second vector register; Perform a multiplication and accumulation operation on the j-th channel data in the first vector register; When j+1 is less than k, the j+2 channel data of the i-th data block is read from the first cache area of ​​the shared memory into the first vector register; Perform a multiply-accumulate operation on the (j+1)th channel data in the second vector register. If j+2 is less than k, j+2 is taken as the new j, and the steps of performing multiplication and accumulation operation on the j-th channel data in the first vector register are repeated until j+2 equals k. Then, the multiplication and accumulation operation on the j+2-th channel data in the second vector register is performed to obtain the multiplication and accumulation operation result of the i-th data block.

6. The method according to any one of claims 1 to 5, characterized in that, The size of the data block is positively correlated with the scale of the multiply-accumulate operation.

7. A data processing apparatus, characterized in that, The device includes: The first reading module is used to read the i-th data block from global data based on the i-th data read / write instruction, and write the i-th data block into shared memory. The data read / write instruction is an assembly instruction for performing data read and write processing. The global data is divided into N data blocks, where N is an integer greater than 0 and i is an integer less than or equal to N. The global data includes a first matrix and a second matrix, which are input matrices for GEMM operations. When reading the i-th data block based on the i-th data read / write instruction, a memory conflict avoidance operation is performed, including: dividing the data of the i-th data block into multiple data sub-blocks according to the maximum number of bytes read by the assembly instruction; controlling multiple parallel threads to read one data sub-block every preset number of memory banks; and writing the read data sub-block into the shared memory. If the global data is of double-precision type, before executing the i-th data read / write instruction, the double-precision type data is split into two single-precision type data, and the shared memory address is written to the m0 register. The i-th data read / write instruction reads the shared memory address from the m0 register in units of 4 bytes. The second read module is used to read the i-th data block from the shared memory into the vector register for multiply-accumulate processing to obtain the multiply-accumulate operation result of the i-th data block. The multiply-accumulate processing uses at least one of v_mac_f32 or v_mac_f64 instructions and arranges multiple multiply-accumulate instructions consecutively to mask the instruction issue and execution cycle. The operation module is used to obtain the general multiplication matrix (GEMM) operation result for the global data based on the multiplication and accumulation operation results corresponding to the N data blocks; The first writing module is used to write the GEMM operation result into global memory; Specifically, a correspondence between grids and computing units is established based on the number of first data blocks corresponding to the first matrix and the number of second data blocks corresponding to the second matrix. This correspondence enables data blocks with similar memory addresses to be mapped to the same computing unit.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images