Code generation method, device, electronic device and storage medium

Abstract

The present application discloses a code generation method, device, electronic device and storage medium, belonging to the field of software development technology. In this method, in response to an instruction to perform an intermediate representation of a tensor using MMA, the operation granularity information of the MMA, the tensor information of the input tensor and the output tensor are determined, the input tensor includes a left tensor and a right tensor, the output tensor is partitioned according to the tensor information and operation granularity information of each tensor, and a code segment is generated based on the left data block corresponding to each region in the output tensor in the left tensor and the right data block corresponding to each region in the right tensor, so that the matrix multiplication result of the left data block and the right data block is written into this region, and the code segments corresponding to each region are fused to obtain the intermediate representation of the MMA. In this way, regardless of the dimension of each tensor, the intermediate representation of the MMA can be automatically generated according to the target information, without the developer having to develop loop code or understand the memory layout of the tensor, so the development difficulty is relatively low.

Description

Technical Field

[0001] The present application relates to the field of software development technology, and in particular to a code generation method, device, electronic device and storage medium. Background Art

[0002] To efficiently develop operators, assembly generators or compilers typically encapsulate some commonly used instructions in operators into intermediate representations (IRs), such as matrix multiply accumulate (MMA) into an intermediate representation (IR), and provide an interface for developers to call.

[0003] In existing technologies, the intermediate representation interface of MMA imposes some restrictions on input and output tensors, such as requiring both input and output tensors to be two-dimensional. Consequently, if developers need to perform matrix multiplication on higher-dimensional tensors, they must develop their own loop code to traverse the loop, which is prone to errors. Summary of the Invention

[0004] Embodiments of the present application provide a code generation method, apparatus, electronic device, and storage medium to reduce the restrictions of the intermediate representation of MMA on input tensors and output tensors, thereby reducing the difficulty of developing the intermediate representation of MMA.

[0005] In a first aspect, an embodiment of the present application provides a code generation method, comprising:

[0006] In response to an instruction to perform matrix multiplication and accumulation (MMA) of an intermediate representation on a tensor, determine target information, where the target information includes MMA operation granularity information supported by the chip, and tensor information of an input tensor and an output tensor, where the input tensor includes a left tensor and a right tensor, and the tensor information of each tensor includes at least shape information;

[0007] Partitioning the output tensor according to the tensor information of the input tensor and the output tensor, and the operation granularity information;

[0008] Based on a left data block corresponding to each region in the left tensor and a right data block corresponding to the right tensor, generating a code segment corresponding to the region, the code segment being used to write a matrix multiplication result of the left data block and the right data block into the region, wherein each data block in the left data block and the right data block is determined based on position information of the region and tensor information of the tensor to which the data block belongs;

[0009] The code segments corresponding to each region are fused to obtain the intermediate representation of the tensor for MMA.

[0010] In some embodiments, the input tensor further includes a bias tensor, and further includes:

[0011] After generating a code segment corresponding to the region, a target code is added to the code segment, wherein the target code is used to accumulate a bias data block corresponding to the region in the bias tensor to the matrix multiplication result of the left data block and the right data block, wherein the bias data block is determined based on the position information of the region and the tensor information of the bias tensor.

[0012] In some embodiments, generating a code segment corresponding to each region based on a left data block corresponding to the left tensor and a right data block corresponding to the right tensor includes:

[0013] Dividing the left data block and the right data block into blocks respectively according to the operation granularity information;

[0014] generating, based on each group of sub-blocks having the same block order in the left data block and the right data block, a set of MMA instructions, the set of MMA instructions being used to perform a matrix multiplication operation on the group of sub-blocks, wherein, when the group of sub-blocks is not the first group of sub-blocks corresponding to the region, the set of MMA instructions is further used to accumulate a result of the matrix multiplication operation of the group of sub-blocks onto a result of the matrix multiplication operation of a previous group of sub-blocks corresponding to the region;

[0015] Each group of MMA instructions is fused to obtain the code segment.

[0016] In some embodiments, based on each group of sub-blocks in the left data block and the right data block having the same block order, a set of MMA instructions is generated, including:

[0017] generating an address calculation instruction for each sub-block according to the address representation information of each sub-block in the group of sub-blocks;

[0018] generating an MMA master instruction based on the sub-block addresses obtained by the address calculation instructions of each sub-block and the region address obtained by the address calculation instruction of the region, wherein the address calculation instruction of the region is generated based on the address representation information of the region;

[0019] A plurality of configuration instructions of the MMA main instruction are generated according to the configuration information required for performing matrix multiplication operations on the group of sub-blocks in the chip.

[0020] In some embodiments, the tensor information of each tensor also includes stride information and address information;

[0021] When the group of sub-blocks is a first group of sub-blocks corresponding to a first region, the address representation information of each sub-block includes position information of the sub-block in a corresponding tensor, stride information of the corresponding tensor, and address information; and the address representation information of the region includes position information of the region in the output tensor, stride information of the output tensor, and address information;

[0022] When the group of sub-blocks is not the first group of sub-blocks corresponding to the first region, the address representation information of each sub-block is the offset address of the sub-block compared to the previous sub-block in the corresponding tensor, and the address representation information of the region is the offset address of the region compared to the previous region in the output tensor.

[0023] In some embodiments, when the group of sub-blocks is not the first group of sub-blocks corresponding to the first region, the method further includes:

[0024] When the configuration information corresponding to the group of sub-blocks is repeated with the configuration information corresponding to the previous group of sub-blocks, the configuration instructions corresponding to the repeated configuration information are skipped when generating the multiple configuration instructions.

[0025] In some embodiments, the input tensor further includes a bias tensor, the target information further includes input synchronization information, and further includes:

[0026] A first synchronization instruction is inserted after each address calculation instruction in the first code segment, where the first synchronization instruction is used to indicate waiting for data of at least one of the left tensor, the right tensor, and the bias tensor to be ready, and / or to indicate waiting for memory corresponding to the output tensor to be unlocked and available.

[0027] In some embodiments, the input tensor further includes a bias tensor, the target information further includes output synchronization information, and further includes:

[0028] Inserting a non-tail portion of the last code segment into a second synchronization instruction, where the second synchronization instruction is used to indicate that the memory occupied by at least one of the left tensor, the right tensor, and the bias tensor has been unlocked and is available, and is also used to indicate that data of the output tensor is ready; or,

[0029] In the last code segment, a synchronization code is configured. The synchronization code is used to indicate that the memory occupied by at least one of the left tensor, the right tensor, and the bias tensor has been unlocked and is available, and is also used to indicate that data of the output tensor is ready.

[0030] In a second aspect, an embodiment of the present application provides a code generation device, comprising:

[0031] a determination module, configured to determine target information in response to an instruction to perform matrix multiplication and accumulation (MMA) of an intermediate representation on a tensor, the target information including operation granularity information of the MMA supported by the chip, and tensor information of an input tensor and an output tensor, the input tensor including a left tensor and a right tensor, the tensor information of each tensor including at least shape information;

[0032] A partitioning module, configured to partition the output tensor according to the tensor information of the input tensor and the output tensor, and the operation granularity information;

[0033] a first generating module, configured to generate, based on a left data block corresponding to each region in the left tensor and a right data block corresponding to the right tensor, a code segment corresponding to the region, the code segment being configured to write a matrix multiplication result of the left data block and the right data block into the region, wherein each data block in the left data block and the right data block is determined based on position information of the region and tensor information of the tensor to which the data block belongs;

[0034] The second generation module is used to fuse the code segments corresponding to each region to obtain an intermediate representation of the tensor for MMA.

[0035] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein:

[0036] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform any of the above-mentioned code generation methods.

[0037] In a fourth aspect, an embodiment of the present application provides a storage medium. When a computer program in the storage medium is executed by a processor of an electronic device, the electronic device can execute any of the above-mentioned code generation methods.

[0038] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any of the above-mentioned code generation methods when executed by a processor.

[0039] In an embodiment of the present application, in response to an instruction to perform an intermediate representation of MMA on a tensor, target information is determined, the target information including the operation granularity information of MMA supported by the chip, and the tensor information of the input tensor and the output tensor, the input tensor including the left tensor and the right tensor. Then, based on the tensor information of the input tensor and the output tensor, as well as the operation granularity information, the output tensor is partitioned, and based on the left data block corresponding to each region in the left tensor and the right data block corresponding to the right tensor, a code segment corresponding to this region is generated, the code segment is used to write the matrix multiplication result of the left data block and the right data block into this region, and then the code segments corresponding to each region are fused to obtain the intermediate representation of MMA on the tensor. In this way, the output tensor is partitioned, and matrix multiplication operations are performed on the data blocks corresponding to each region in each input tensor. Regardless of the dimensions of each tensor, the intermediate representation of MMA can be automatically generated based on the target information required for the intermediate representation of MMA for the tensor. Developers do not need to develop loop code for traversal themselves, nor do they need to understand the memory layout of the tensor. Therefore, the difficulty of developing the intermediate representation of MMA can also be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0041] Figure 1 An application scenario diagram of a code generation method provided in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of an intermediate representation of an MMA provided in an embodiment of the present application;

[0043] Figure 3 A flowchart of a code generation method provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of performing a matrix multiplication operation on a tensor provided in an embodiment of the present application;

[0045] Figure 5 A flowchart of another code generation method provided in an embodiment of the present application;

[0046] Figure 6 A schematic diagram of another embodiment of the present application for performing matrix multiplication on a tensor;

[0047] Figure 7 A schematic diagram of the structure of a code generation device provided in an embodiment of the present application;

[0048] Figure 8A schematic diagram of the hardware structure of an electronic device for implementing a code generation method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to reduce the restrictions of the intermediate representation of MMA on input tensors and output tensors and reduce the difficulty of developing the intermediate representation of MMA, the embodiments of the present application provide a code generation method, device, electronic device and storage medium.

[0050] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application can be combined with each other if there is no conflict.

[0051] To facilitate understanding of this application, the technical terms involved in this application are:

[0052] 1. Tensor, a fundamental concept in deep learning models, is generally designed for graphics processing units (GPUs) and can run on GPUs to accelerate computational efficiency. In AI models, tensors can be viewed as multidimensional arrays. Generally, a tensor includes multiple dimensions (or logical axis dimensions). Tensor information may include shape information, stride information, and address information. Shape information is used to describe the number of elements in each dimension of the tensor. Stride information is used to describe the spacing between elements in each dimension of the tensor and can represent the memory layout of the tensor (that is, how the data in the tensor is stored in memory). Address information, such as the base address, is used to describe the physical storage address of the tensor in memory.

[0053] Assume that the tensor information of a tensor X is:

[0054] shape X = (shapeN, shapeH, shapeW) = (2, 128, 448);

[0055] stride X = (strideN, strideH, strideW) = (448×128=57344, 448, 1);

[0056] address= address X .

[0057] Then, tensor X has three dimensions: N, H, and W. The size of the N dimension is 2, the size of the H dimension is 128, and the size of the W dimension is 448. In addition, the element spacing of tensor X in the W dimension is 1, the element spacing in the H dimension is 448, and the element spacing in the N dimension is 57344. The starting address of tensor X is address X .

[0058] Generally, stride information is sorted from largest to smallest. For example, the shape and stride of an output tensor can be expressed as follows (for convenience, they are pre-sorted from largest to smallest by stride, and can also support unsorted tensors):

[0059] (shape): (stride) = (sh1, sh2, sh3, sh4): (st1, st2, st3, st4)

[0060] =(2, 2, 128, 64): (16384, 8192, 64, 1);

[0061] In the above case, the traversal order of the output tensor is st4 =>st3 =>st2 =>st1. Combined with the instruction granularity information of the MMA instruction supported by the chip, it is traversed from coordinates (0, 0, 0, 0) to (sh1, sh2, sh3, sh4).

[0062] 2. MMA operation granularity information refers to the data size that an MMA instruction can process, such as 64×32×32, 64×128×32, etc. Generally, different chips support different MMA operation granularities.

[0063] Suppose we calculate D = A × B, where D is the output tensor, A is the left tensor, and B is the right tensor. If we use an operation granularity of 64 × 128 × 32, we can process 64 × 128 data in the left tensor A and 128 × 32 data in the right tensor B at a time, resulting in 64 × 32 data in the output tensor D.

[0064] 3. Intermediate Representation. During operator development, to improve efficiency and reduce development costs, developers typically avoid writing low-level assembly code directly. Instead, they use higher-level abstractions such as intermediate representations to construct their operators. Intermediate representation is an abstract code representation between source code and target machine code. It allows code generation tools to analyze, optimize, and convert the instructions contained within it without having to worry about specific hardware details. By combining different intermediate representations, developers can implement complex computational logic.

[0065] In existing technologies, the intermediate representation interface of MMA imposes some restrictions on input and output tensors, such as requiring both input and output tensors to be two-dimensional. Consequently, if developers need to perform matrix multiplication on higher-dimensional tensors, they must develop their own loop code to traverse the loop, which is prone to errors.

[0066] To this end, an embodiment of the present application provides a new solution for performing MMA intermediate representation on tensors, partitions the output tensor, and performs matrix multiplication operations on the data blocks corresponding to each region in the output tensor in each input tensor. Regardless of the dimensions of each tensor, the intermediate representation of MMA can be automatically generated based on the target information required for performing MMA intermediate representation on the tensor. Developers do not have to develop loop code for traversal themselves, nor do they need to understand the memory layout of the tensor. Therefore, the difficulty of developing the intermediate representation of MMA can also be reduced.

[0067] In order to more clearly introduce the method of the embodiment of the present application, the application scenario of the embodiment of the present application is first introduced below.

[0068] See also Figure 1 , Figure 1 An application scenario provided for an embodiment of the present application includes an electronic device 130 and a chip 200, wherein the electronic device 130, such as a desktop computer, laptop computer, server, etc., can be installed with an assembly generation program or compiler. The assembly generation program or compiler provides an interface for performing an intermediate representation of the tensor MMA in the embodiment of the present application. The developer passes in the target information required for the intermediate representation of the tensor MMA through the interface, such as the tensor information of each tensor, the operation granularity information of the MMA supported by the chip, etc. The assembly generation program or compiler can automatically generate an intermediate representation of the tensor MMA based on the target information. The developer does not need to worry about the dimension of the tensor or understand the memory layout of the tensor, and the development difficulty of the intermediate representation of the MMA is relatively low.

[0069] Subsequently, developers or other technical personnel can import the intermediate representation of MMA into chip 200, parse and execute the intermediate representation in chip 200, and perform matrix multiplication operations on each tensor in chip 200.

[0070] The following first introduces the principle of the intermediate representation of MMA in the embodiment of the present application.

[0071] Suppose we want to calculate D=A×B+C, where A is the left tensor, B is the right tensor, C is the bias tensor, C is optional (that is, it can be present or not), A, B, and C are all input tensors, and D is the output tensor.

[0072] See also Figure 2, which is a schematic diagram of an intermediate representation of an MMA provided in an embodiment of the present application. When implementing the intermediate representation of the MMA, it is necessary to determine the target information. The target information includes at least the tensor information of each tensor and the operation granularity information of the MMA supported by the chip. It can also include input synchronization information and output synchronization information. The input synchronization information can be used to configure waiting for data to be ready for at least one tensor among A, B, and C, and can also be used to configure waiting for the memory corresponding to D to be unlocked and available; the output synchronization information is used to configure the sending of information that the data of D is ready, and can also be used to configure the sending of information that the memory corresponding to at least one tensor among A, B, and C is unlocked and available.

[0073] A first synchronization instruction, such as at least one wait instruction, may be generated for the input synchronization information. The first synchronization instruction is used to instruct to wait for data of at least one tensor among A, B, and C to be ready, and may also be used to instruct to wait for the memory corresponding to D to be unlocked and available.

[0074] Furthermore, based on the tensor information of each tensor and the chip's corresponding operation granularity information, D can be divided into regions 1, 2, ..., and n. Based on region information (such as size and location) of region i and the tensor information of each input tensor, the data block corresponding to region i in A, B, and C is determined. Based on the data blocks corresponding to region i in A, B, and C, code segment i is generated. Code segments 1, 2, ..., and n are sequentially concatenated and placed after the first synchronization instruction. i ranges from 1 to n. Code segment i can include address calculation instructions, MMA main instructions, and configuration instructions. The address calculation instructions typically include multiple instructions for calculating the physical address of region i and the physical address of the data blocks corresponding to region i in A, B, and C. There are also typically multiple MMA main instructions for performing matrix multiplication and accumulation calculations on the data blocks corresponding to region i in A, B, and C. There are also typically multiple configuration instructions for updating data in the chip's status registers. The data in the status registers describes various parameters required for executing the MMA main instructions, such as the data size and memory layout for a single process.

[0075] A second synchronization instruction, such as at least one arrive instruction, can be generated for the input synchronization information. The second synchronization instruction is used to indicate that the data waiting for D is ready and can also be used to indicate that the memory corresponding to at least one tensor among A, B, and C is unlocked and available. The second synchronization instruction can be placed last.

[0076] It should be noted that when the number of elements in D is relatively small, only one region may be created. Furthermore, when D is divided into multiple regions, the sizes of the regions can be the same or different. For the data blocks corresponding to region i in D in A, B, and C, the data blocks may be directly multiplied, or they may need to be partitioned and then subjected to MMA.

[0077] Next, the code generation method proposed in the embodiment of the present application is described with reference to a flowchart.

[0078] See also Figure 3 , Figure 3 A flowchart of a code generation method provided in an embodiment of the present application, which is applied to Figure 1 In the electronic device 130, the method includes the following steps.

[0079] In step 301 , in response to an instruction to perform an intermediate representation of an MMA on a tensor, target information is determined. The target information includes operation granularity information of the MMA supported by the chip, and tensor information of an input tensor and an output tensor. The input tensor includes a left tensor and a right tensor.

[0080] In actual applications, developers can pass in target information through the interface or enter target information through the configuration page.

[0081] The chip may support only one or more MMA operation granularity information. Assume that the chip supports the following MMA operation granularity information: 64×32×32, 64×128×32.

[0082] See also Figure 4 , assuming the left tensor is A, the right tensor is B, and the output tensor is D, it is necessary to perform MMA intermediate representation of D=A×B, and

[0083] The tensor information of the left tensor A is:

[0084] Shape information: shape A = (shapeL, shapeM, shapeK)=(2, 128, 448),

[0085] Step information: stride A =(strideL, strideM, strideK)=(448×128=57344, 448, 1),

[0086] Base address: addressA;

[0087] The tensor information of the right tensor B is:

[0088] Shape information: shape B = (shapeL, shapeK, shapeN)=(2, 448, 64),

[0089] Step information: stride B=(strideL, strideK, strideN)=(512×64 = 32768, 1, 512), where strideN=512 means that the data of the right tensor B is stored non-continuously on the N axis. After storing 448 K-axis data, 64 gaps are left and then 448 K-axis data are stored.

[0090] Base address: addressB;

[0091] The tensor information of the output tensor D is:

[0092] Shape information: shape D = (shapeL, shapeM, shapeN)=(2, 128, 64),

[0093] Step information: stride D =(strideL, strideM, strideN)=(64×128=8192, 64, 1),

[0094] Base address: addressD.

[0095] In some embodiments, a cross-logical check can be performed on the tensor information of each tensor in advance, such as whether the matrix multiplication conditions are met, whether the actual tensor shape is consistent with the tensor information, etc., to prompt developers of possible errors.

[0096] In step 302 , the output tensor is partitioned according to the tensor information of the input tensor and the output tensor, and the operation granularity information.

[0097] During specific implementation, one operation granularity information may be selected based on the tensor information of the input tensor and the output tensor, as well as the operation granularity information, and then the output tensor may be partitioned according to the operation granularity information.

[0098] Continuing with the above example, the chip supports MMA operation granularity information of 64×32×32 and 64×128×32. Both operation granularity information uses 64×32 size to partition the output tensor D. The shape and stride of the output tensor D also support this size. The left tensor A and the right tensor B both have 448 elements in the K dimension. In order to reduce the number of blocks, the larger granularity of 64×128×32 can be selected. Then, based on the selected 64×128×32, the output tensor D is partitioned by 64×32. The partitioned areas can be seen in Figure 4 , there are 8 regions in total, and the gray area of the output tensor D represents the first region.

[0099] It's important to note that the number of partitions in the output tensor is positively correlated with the number of instructions in the resulting intermediate representation, while the number of instructions in the intermediate representation is negatively correlated with the speed at which the chip can parse and execute the intermediate representation. To increase chip computing speed, the number of partitions in the output tensor needs to be reduced. Therefore, when multiple operation granularity options are available, the maximum operation granularity can be selected. This maximizes each region, thus reducing the number of partitions.

[0100] In step 303, based on the left data block corresponding to each region in the output tensor in the left tensor and the right data block corresponding to the right tensor, a code segment corresponding to this region is generated, and the code segment is used to write the matrix multiplication result of the left data block and the right data block into this region, wherein the left data block is determined according to the position information of this region and the tensor information of the left tensor, and the right data block is determined according to the position information of this region and the tensor information of the right tensor.

[0101] by Figure 4 Taking the first region of the output tensor D as an example, according to the position information of the first region in the output tensor D and the tensor information of the left tensor A, it can be determined that the left data block corresponding to the first region in the left tensor A is the first 64×448 elements ( Figure 4 The gray area of the left tensor A in the middle), according to the position information of the first area in the output tensor D and the tensor information of the right tensor B, it can be determined that the right data block corresponding to the first area in the right tensor B is the first 448×32 elements ( Figure 4 Gray area of tensor B in the middle right).

[0102] In some embodiments, the left data block corresponding to each region in the left tensor and the right data block corresponding to the right tensor just match the selected operation granularity information, and matrix multiplication operations can be performed directly on the left data block and the right data block.

[0103] Taking the left data block as 64×128 and the right data block as 128×32 as an example, when the operation granularity information is 64×128×32, the left data block and the right data block just match the selected operation granularity information. A set of MMA instructions can be directly generated based on the left data block and the right data block. This set of MMA instructions is used to perform matrix multiplication operations on the left data block and the right data block. This set of MMA instructions is also the code segment corresponding to this area.

[0104] In some embodiments, the left data block and right data block corresponding to each region in the left tensor do not match the selected operation granularity information, and MMA requires partitioning before it can be performed. In this case, the code segment corresponding to this region can be generated according to the following steps.

[0105] Step 1: Divide the left data block and the right data block corresponding to each region into blocks according to the operation granularity information.

[0106] by Figure 4 Take the left and right data blocks corresponding to the first region of the output tensor D as an example. When the operation granularity is 64×128×32, the left data block can be divided into 64×128 blocks to obtain four sub-blocks: sub-block 1, sub-block 2, sub-block 3, and sub-block 4. The right data block can be divided into 128×32 blocks to obtain four sub-blocks: sub-block 1′, sub-block 2′, sub-block 3′, and sub-block 4′.

[0107] It should be noted that for the left data block, after the 64×128 block division is completed, only 64×64 data remains. Therefore, the size of the last sub-block is 64×64. For the right data block, after the 128×32 block division is completed, only 64×32 data remains. Therefore, the size of the last sub-block is 64×32.

[0108] Step 2: Based on each group of sub-blocks with the same block order in the left data block and the right data block corresponding to each region, generate a group of MMA instructions corresponding to this group of sub-blocks, and use this group of MMA instructions to perform matrix multiplication operations on this group of sub-blocks. When this group of sub-blocks is not the first group of sub-blocks corresponding to this region, this group of MMA instructions is also used to accumulate the matrix multiplication results of this group of sub-blocks onto the matrix multiplication results of the previous group of sub-blocks corresponding to this region.

[0109] Still Figure 4 Taking the left data block and the right data block corresponding to the first region of the output tensor D as an example, the left data block and the right data block have a total of 4 groups of sub-blocks (indicated by the dotted lines): the first group of sub-blocks: sub-block 1 and sub-block 1′, the second group of sub-blocks: sub-block 2 and sub-block 2′, the third group of sub-blocks: sub-block 3 and sub-block 3′, and the fourth group of sub-blocks: sub-block 4 and sub-block 4′.

[0110] By performing MMA on these four groups of sub-blocks in sequence, the matrix multiplication results of the left data block and the right data block can be obtained. Specifically, for the first group of sub-blocks, a first group of MMA instructions can be generated, and the first group of MMA instructions are used to perform matrix multiplication operations on the first group of sub-blocks. For the second group of sub-blocks, a second group of MMA instructions are generated, and the second group of MMA instructions are used to perform matrix multiplication operations on the second group of sub-blocks, and the matrix multiplication operations are accumulated to the matrix multiplication results of the first group of sub-blocks. For the third group of sub-blocks, a third group of MMA instructions are generated, and the third group of MMA instructions are used to perform matrix multiplication operations on the third group of sub-blocks, and the matrix multiplication operations are accumulated to the matrix multiplication results of the second group of sub-blocks. For the fourth group of sub-blocks, a fourth group of MMA instructions are generated, and the fourth group of MMA instructions are used to perform matrix multiplication operations on the fourth group of sub-blocks, and the matrix multiplication operations are accumulated to the matrix multiplication results of the third group of sub-blocks.

[0111] The following describes the process of generating a set of MMA instructions.

[0112] 1. Generate address calculation instructions.

[0113] During actual calculation, the chip needs to know the address of each area and the address of each sub-block in each group of sub-blocks corresponding to the area. Therefore, the address calculation instruction of this area can be generated based on the address representation information of each area, and the address calculation instruction of this sub-block can be generated based on the address representation information of each sub-block in each group of sub-blocks.

[0114] The following first introduces the method of generating address calculation instructions for each area.

[0115] Method 1: The address representation information of each region includes the position information of the region in the output tensor, the step information of the output tensor, and address information such as the base address.

[0116] When the output tensor is stored in global shared memory (GSM), the address calculation instruction for each region is: region address = region position information in the output tensor × output tensor step information + output tensor base address.

[0117] When the output tensor is stored in thread local register (TLR) memory, the calculation is relatively complex. It is usually done by calling a library function function(), inputting the position information of each region in the output tensor, and then combining the step information of the output tensor and the thread arrangement information to calculate the offset of this region relative to the base address of the output tensor. Then, this offset is added to the base address of the output tensor to get the address of this region.

[0118] That is, when the output tensor is stored in the thread-local register memory, the address calculation instruction for each region is: region address = function (region position information in the output tensor, output tensor step information, thread arrangement information) + output tensor base address.

[0119] Method 2: The address representation of the first region includes its position in the output tensor, its stride, and its address. The address representation of each region after the first region includes its offset from the previous region in the output tensor.

[0120] The address calculation instruction of the first area is the same as method 1 and will not be repeated here.

[0121] For each region except the first region, regardless of whether the output tensor is stored in shared memory or thread-local register memory, the address calculation instruction is: region address = previous region address + offset address.

[0122] Since the calculation method of the offset address is relatively simple, the second method can reduce the execution complexity of the address calculation instruction, thereby improving the calculation speed of the chip.

[0123] In practical applications, the stride information of the output tensor can represent the memory layout of the output tensor. Regardless of the memory layout of the output tensor, the address of the region can be calculated based on the stride information of the output tensor. This makes the representation of the data storage layout of the output tensor more flexible.

[0124] Next, a method of generating an address calculation instruction for each sub-block in each group of sub-blocks is introduced.

[0125] Method 1: The address representation information of each sub-block in each group of sub-blocks includes the position information of the sub-block in the corresponding tensor, the step information of the corresponding tensor, and the address information such as the base address.

[0126] For each sub-block in each group of sub-blocks, when the tensor to which the sub-block belongs is stored in the shared memory, the address calculation instruction of the sub-block is: sub-block address = sub-block position information in the corresponding tensor × step information of the corresponding tensor + base address of the corresponding tensor.

[0127] Take the calculation of the address of sub-block 1′ in the right tensor B as an example,

[0128] The address of sub-block 1′ = the coordinate of sub-block 1′ in the right tensor B × stride B T +addressB

[0129] =(0,0,0)×(32768,1,512) T +addressB

[0130] = 0 × 32768 + 0 × 1 + 0 × 512 +addressB.

[0131] For each sub-block in each group of sub-blocks, when the tensor to which the sub-block belongs is stored in the thread-local register memory, the calculation is relatively complex. It is usually done by calling a library function, inputting the position information of the sub-block in the corresponding tensor, and then combining the step information of the corresponding tensor and the thread arrangement information to calculate the offset of the sub-block relative to the base address of the corresponding tensor. Then, this offset is added to the base address of the corresponding tensor to obtain the address of the sub-block.

[0132] That is, when the tensor to which a sub-block belongs is stored in the thread-local register memory, the address calculation instruction of the sub-block is: sub-block address = function (position information of the sub-block in the corresponding tensor, step information of the corresponding tensor, thread arrangement information) + base address of the corresponding tensor.

[0133] Take the calculation of the address of sub-block 1 of the left tensor A as an example,

[0134] Sub-block 1 address = function(sub-block 1 coordinates in left tensor A, stride A , thread arrangement information)+addressA.

[0135] Method 2: The address representation information for each sub-block in the first group of sub-blocks corresponding to the first region includes the sub-block's position information in the corresponding tensor, the stride information of the corresponding tensor, and the address information. For each group of sub-blocks other than the first group of sub-blocks corresponding to the first region (such as the non-first group of sub-blocks corresponding to the first region and the groups of sub-blocks corresponding to non-first regions), the address representation information for each sub-block in this group includes the offset address of the sub-block compared to the previous sub-block in the corresponding tensor.

[0136] The address calculation instruction for each sub-block in the first group of sub-blocks corresponding to the first region is the same as that in method 1, and will not be repeated here.

[0137] For each group of sub-blocks except the first group of sub-blocks corresponding to the first region, regardless of whether the tensors belonging to this group of sub-blocks are stored in shared memory or thread-local register memory, the address calculation instruction for each sub-block in this group of sub-blocks is: sub-block address = previous sub-block address + offset address.

[0138] Since the calculation method of the offset address is relatively simple, the second method can reduce the execution complexity of the address calculation instruction and improve the calculation speed of the chip.

[0139] In practical applications, for any input tensor in the left tensor and the right tensor, the stride information of the input tensor can represent the memory layout of the input tensor. Regardless of the memory layout of the input tensor, the address of the corresponding sub-block can be calculated based on the stride information of the input tensor. In this way, the representation of the data storage layout of the input tensor is more flexible.

[0140] 2. Generate MMA main instructions.

[0141] For example, the MMA main instruction is generated according to the sub-block address obtained by the address calculation instruction of each sub-block in each group of sub-blocks and the region address obtained by the address calculation instruction of the corresponding region.

[0142] 3. Generate multiple configuration instructions for MMA main instructions.

[0143] In order to ensure that the chip can operate normally, multiple configuration instructions of the MMA main instructions corresponding to this group of sub-blocks can be generated according to the configuration information required for matrix multiplication operations on each group of sub-blocks in the chip. These multiple configuration instructions are used to update the data in the status register in the chip. The data in the status register is used to describe various parameters required to execute the MMA main instructions, such as the size of the sub-block (that is, the size of the data processed in a single time), offset address, memory layout, etc.

[0144] Considering that each sub-block requires configuration of information such as the sub-block size and offset address, the chip copies the relevant data to the status register based on these configurations when performing calculations for each sub-block. The chip then performs calculations based on the data in the status register. If the data in the status register has not changed, the chip does not need to copy the relevant data to the register again and can continue to use the data in the status register.

[0145] To this end, for non-first sub-blocks, if the configuration information corresponding to this sub-block duplicates the configuration information corresponding to the previous sub-block (corresponding to the same or different regions as this sub-block), the generation of multiple configuration instructions for this sub-block can skip the generation of the duplicate configuration information. The duplicate configuration information can include the size and offset address of the configuration sub-block.

[0146] In this way, deduplication is performed on the configuration instructions corresponding to the current group of sub-blocks and the configuration instructions corresponding to the previous group of sub-blocks, which can reduce the total number of instructions and the hardware operations that the chip needs to perform, thereby improving the chip's computing speed.

[0147] Step 3: Fuse the MMA instructions corresponding to each group of sub-blocks in each region to obtain the code segment corresponding to this region.

[0148] For example, according to the block division order of each group of sub-blocks, the MMA instructions corresponding to each group of sub-blocks are spliced to obtain the code segment corresponding to this area.

[0149] In step 304 , the code segments corresponding to the respective regions are fused to obtain an intermediate representation of the tensor subjected to MMA.

[0150] For example, according to the division order of each area, the code segments corresponding to each area are spliced to obtain the intermediate representation of MMA for the tensor. This intermediate representation is used to write the matrix multiplication result of the left tensor and the right tensor into the output tensor.

[0151] In some embodiments, the input tensor may also include a bias tensor. After generating a code segment corresponding to each region, a target code may be added to this code segment. The target code is used to accumulate the bias data block corresponding to this region in the bias tensor to the matrix multiplication result of the left data block and the right data block corresponding to this region, wherein the bias data block is determined based on the position information of this region and the tensor information of the bias tensor.

[0152] In some embodiments, developers can also be supported to configure input synchronization information (generally information that the current intermediate representation needs to know), such as configuring to wait for data to be ready for at least one of the left tensor, right tensor, and bias tensor, and configuring to wait for the memory corresponding to the output tensor to be unlocked and available. In this case, the target information will also include input synchronization information, and a first synchronization instruction such as at least one wait instruction can be inserted at the beginning of the first code segment, or the first synchronization instruction can be inserted after each address calculation instruction in the first code segment. The first synchronization instruction is used to indicate waiting for data to be ready for at least one of the left tensor, right tensor, and bias tensor, and / or to indicate waiting for memory corresponding to the output tensor to be unlocked and available.

[0153] In actual applications, when the chip waits for the data of at least one of the left tensor, right tensor, and bias tensor to be ready, it is actually executing a copy instruction (that is, copying a tensor to the corresponding address). The copy instruction and the address calculation instructions can be executed in parallel in the chip. However, when the first synchronization instruction is placed before the address calculation instructions in the first code segment (that is, at the very beginning), since the first synchronization instruction itself will block the execution of subsequent instructions, the chip needs to wait for the data of the corresponding tensor to be ready before executing the address calculation instructions when actually executing the instructions. However, in fact, there is no dependency between these two types of instructions. For this reason, the first synchronization instruction can no longer be placed at the very beginning of the first code segment, but can be appropriately postponed to after the address calculation instructions that have no dependency on it. In this way, the number of instructions blocked by the first synchronization instruction can be reduced, and the first synchronization instruction can be executed in parallel with the address calculation instructions in the chip, which can further improve the chip's computing speed.

[0154] In some embodiments, developers can also be supported to configure output synchronization information (generally, the information that the current intermediate representation wants to tell other intermediate representations), such as configuring to wait for the memory occupied by at least one of the left tensor, right tensor, and bias tensor to be unlocked and available, and configuring to wait for the data of the output tensor to be ready. At this time, the target information also includes output synchronization information, so a second synchronization instruction such as an arrive instruction can be inserted at the non-tail part of the last code segment. The second synchronization instruction is used to send a message that the memory occupied by at least one of the left tensor, right tensor, and bias tensor is unlocked and available, and is also used to send a message that the data of the output tensor is ready. Alternatively, synchronization code can be configured in the last code segment. The synchronization code is used to send a message that the memory occupied by at least one of the left tensor, right tensor, and bias tensor is unlocked and available, and is also used to send a message that the data of the output tensor is ready.

[0155] In other words, the second synchronization instruction, which is placed in the middle to represent the end, can be placed not at the end but before any instructions that have no dependencies on it. This allows the corresponding completion information to be sent in advance, allowing subsequent computations in the chip to begin earlier. Furthermore, for certain hardware architectures that support the placement of the arrive instruction on an MMA main instruction or its configuration instructions, it can also be placed directly on the last MMA main instruction or its configuration instruction, reducing the number of instructions generated and further improving chip computation speed.

[0156] See also Figure 5 , Figure 5 A flowchart of another code generation method provided in an embodiment of the present application, which is applied to Figure 1 In the electronic device 130, the method includes the following steps.

[0157] In step 501 , in response to an instruction to add an intermediate representation of an MMA to a tensor, the operation granularity information of the MMA supported by the chip, and tensor information of the input tensor and the output tensor are determined, where the input tensor includes a left tensor, a right tensor, and a bias tensor.

[0158] For example, perform MMA on D=A×B+C, where D is the input tensor, A is the left tensor, B is the right tensor, and C is the bias tensor.

[0159] In step 502 , the output tensor is divided into regions according to the tensor information of the input tensor and the output tensor, and the operation granularity information.

[0160] In step 503, for region i in the output tensor, the left data block i corresponding to region i is determined in the left tensor, the right data block i corresponding to region i is determined in the right tensor, and the bias data block i corresponding to region i is determined in the bias tensor.

[0161] Initially, region i represents the first region in the output tensor, and the value of i is a preset value such as 0 or 1.

[0162] In step 504, the left data block i and the right data block i are divided into blocks according to the operation granularity information corresponding to the region i, and two sub-blocks with the same division order in the left data block i and the right data block i are taken as a group of sub-blocks.

[0163] In step 505 , a j-th group of MMA instructions is generated for the j-th group of sub-blocks in the left data block i and the right data block i.

[0164] Initially, the jth group of sub-blocks is the first group of sub-blocks in the left data block i and the right data block i, and j is a set value such as 0 or 1. In addition, the jth group of MMA instructions includes an address calculation instruction, an MMA main instruction, and multiple configuration instructions.

[0165] In a specific implementation, when the jth group of sub-blocks is the first group of sub-blocks corresponding to region i, the jth group of MMA instructions is used to perform a matrix multiplication operation on the jth group of sub-blocks in the left data block i and the right data block i. When the jth group of sub-blocks is not the first group of sub-blocks corresponding to region i, the jth group of MMA instructions is used to perform a matrix multiplication operation on the jth group of sub-blocks in the left data block i and the right data block i, and the matrix multiplication operation result is accumulated to the matrix multiplication operation result of the j-1th group of sub-blocks in the left data block i and the right data block i.

[0166] In practical applications, in order to improve the computing speed of subsequent chips, the jth group of MMA instructions can also be optimized.

[0167] For example, if input synchronization information is also determined in response to an instruction to add an intermediate representation of MMA to a tensor, then when the jth group of sub-blocks is the first group of sub-blocks corresponding to the first region in the output tensor, a first synchronization instruction such as at least one wait instruction can be inserted after each address calculation instruction. This set of synchronization instructions is used to indicate waiting for data of at least one tensor among the left tensor, the right tensor, and the bias tensor to be ready, and / or to indicate waiting for the memory corresponding to the output tensor to be unlocked and available.

[0168] For another example, when the jth group of sub-blocks is not the first group of sub-blocks corresponding to the first region, and if the configuration information corresponding to the jth group of sub-blocks duplicates the configuration information corresponding to the previous group of sub-blocks (which may correspond to the same region as the jth group of sub-blocks or may correspond to a different region), then when generating multiple configuration instructions for the jth group of sub-blocks, the configuration instructions corresponding to the duplicate configuration information are skipped. In other words, the configuration instructions for the jth group of sub-blocks and the previous group of sub-blocks can be deduplicated.

[0169] In step 506 , it is determined whether the jth group of sub-blocks is the last group of sub-blocks corresponding to region i. If not, the process proceeds to step 507 . If yes, the process proceeds to step 508 .

[0170] In step 507 , j is updated to j+1, and the process returns to step 505 .

[0171] In step 508, the groups of MMA instructions corresponding to the left data block i and the right data block i are merged to obtain the code segment corresponding to the region i. The code segment corresponding to the region i is used to write the matrix multiplication result of the left data block i and the right data block i into the region i.

[0172] In specific implementation, if in response to an instruction to add an intermediate representation of MMA to a tensor, input synchronization information is also determined, then when region i is the last region, a synchronization code can also be configured at the end of the code segment corresponding to region i. The synchronization code is used to indicate that the memory occupied by at least one of the left tensor, right tensor, and bias tensor has been unlocked and available, and is also used to indicate that the data of the output tensor is ready.

[0173] In step 509, target code is added to the code segment corresponding to region i, where the target code is used to accumulate the offset data block i to the matrix multiplication result of the left data block i and the right data block i.

[0174] In step 510 , it is determined whether region i is the last region. If not, the process proceeds to step 511 . If yes, the process proceeds to step 512 .

[0175] In step 511 , i is updated to i+1, and the process returns to step 503 .

[0176] In step 512, the code segments corresponding to each region are fused to obtain an intermediate representation of the tensor for MMA.

[0177] The following describes the solutions of the embodiments of the present application with reference to specific examples.

[0178] See also Figure 6 , suppose that we want to perform MMA calculation on the left tensor A, right tensor B, and bias tensor C to obtain the output tensor D, that is, D = A × B + C. And suppose that the tensor information of each tensor is as follows.

[0179] The tensor information of the left tensor A is:

[0180] Shape information: shape A = (shapeL, shapeM, shapeK)=(2, 128, 448),

[0181] Step information: stride A =(strideL, strideM, strideK)=(448×128=57344, 448, 1),

[0182] Base address: addressA;

[0183] The tensor information of the right tensor B is:

[0184] Shape information: shape B = (shapeL, shapeK, shapeN)=(2, 448, 64),

[0185] Step information: stride B =(strideL, strideK, strideN)=(512×64 = 32768, 1, 512),

[0186] Base address: addressB;

[0187] The tensor information of the bias tensor C is:

[0188] Shape information: shape C = (shapeL, shapeM, shapeN)=(2, 128, 64),

[0189] Step information: stride C =( strideL, strideM, strideN)=(64×128=8192, 64, 1),

[0190] Base address: addressC;

[0191] The tensor information of the output tensor D is:

[0192] Shape information: shape D = (shapeL, shapeM, shapeN)=(2, 128, 64),

[0193] Stride information: Stride D =( strideL, strideM, strideN)=(64×128=8192, 64, 1),

[0194] Base address: addressD.

[0195] After obtaining the tensor information of each tensor passed in by the user, tensor traversal and instruction generation begin. The steps are as follows.

[0196] Step 1. Refer to the shape information and step information of each tensor, as well as the operation granularity information supported by the chip, and traverse the output tensor D (the first outer traversal, that is, traversing to the first area of the output tensor D), starting from the coordinate (0, 0, 0) of the output tensor D.

[0197] Step 2. For either left tensor A or right tensor B, referencing the shape and stride information, traverse tensor B (first pass in the outer layer, first pass in the inner layer). Logically, the coordinate (0, 0, 0) in output tensor D corresponds to the coordinate (0, 0, X) in left tensor A and to the coordinate (0, X, 0) in right tensor B, where X represents the number of elements in the K-dimensional space to be accumulated. This is the first pass in the inner layer, so the actual coordinates used are (0, 0, 0) for left tensor A and (0, 0, 0) for right tensor B.

[0198] Step 3. Assume that based on current hardware limitations, such as the MMA operation granularity information supported by the chip, the granularity of a single traversal of the output tensor D is determined to be M×N=64×32. Combined with the current hardware limitations and the number of elements of the left tensor A and the right tensor B on the K-axis, the granularity of a single traversal of the left tensor A is determined to be M×K=64×128, and the granularity of a single traversal of the right tensor B is determined to be K×N=128×32. Therefore, the granularity of a single MMA instruction is M×N×K=64×32×128.

[0199] Step 4. Generate a single MMA and its related instructions.

[0200] Step 4.1 First calculate the addresses of the small blocks of the left tensor A, right tensor B, and output tensor D.

[0201] For GSM memory, taking the right tensor B as an example, the address calculation formula of the small blocks it divides is:

[0202] The address of the sub-block in the right tensor B = the coordinate of the sub-block in the right tensor B × stride B T + The base address of the right tensor B.

[0203] Taking the first sub-block in the right tensor B as an example,

[0204] The address of the first sub-block = (0, 0, 0) × (32768, 1, 512) T +addressB;

[0205] = 0 × 32768 + 0 × 1 + 0 × 512 +addressB.

[0206] For TLR memory, the calculation is relatively complicated. It can be done by calling an assembly generation program or the library function function() in the compiler, inputting the coordinates of the sub-block currently traversed, and then combining the step information and thread arrangement information of the tensor to which the sub-block belongs to calculate the offset of the sub-block relative to the base address of the corresponding tensor. Then, this offset is added to the base address to get the address of the small block.

[0207] Taking the left tensor A as an example, the addresses of the small blocks it divides into are:

[0208] The address of the sub-block in the left tensor A = function(the coordinates of the sub-block in the left tensor A, stride A , thread arrangement information) + the base address of the left tensor A.

[0209] In step 4.2, in order to reduce the complexity of address calculation, for a sub-block that is not the first sub-block, its offset address relative to the previous sub-block can be used for address calculation.

[0210] Taking the offset address of right tensor B as an example, after calculating the address and before generating specific MMA main instructions and configuration instructions, the next traversal is attempted. At this time, the traversal reaches the first outer layer and the second inner layer (that is, the traversal reaches sub-block 2′ of the right tensor). Combined with the coordinates of the right tensor B (0, 0, 0) in the previous traversal and the granularity K=128, the coordinates of the right tensor B traversed at this time are (0, 128, 0). Therefore, using the previous calculation formula, it can be obtained that in the next MMA calculation, the address of the second sub-block in right tensor B = the coordinates of the sub-block in right tensor B × stride B T =(0, 128, 0) × (32768, 1, 512) T = 0 × 32768 + 128 × 1 + 0 × 512 + addressB, the memory offset compared to the current traversal is 128.

[0211] The left tensor A is similar and will not be described here.

[0212] In step 4.3, assuming the user has configured input synchronization information, such as waiting for the data of left tensor A to be ready, a waitA instruction is inserted into the intermediate representation. Since the current traversal corresponds to the first MMA instruction, waitA needs to be inserted within its instruction block. Furthermore, since only the MMA instruction itself depends on the matrix A being ready, while the preceding address calculation does not, the waitA instruction is inserted after the instruction corresponding to the address calculation. Subsequently, during chip execution, the "wait" and "address calculation" processes are actually processed in parallel, which improves computation speed and optimizes chip performance.

[0213] Step 4.4 combines the granularity information in step 3, the address information calculated in step 4.1, the bias information in step 4.2, the input synchronization information in step 4.3, and some other information (such as the data type of the tensor), to generate a set of instructions including address calculation instructions, synchronization instructions, MMA main instructions, and configuration instructions.

[0214] Step 5. After completing the first outer and inner traversals (i.e., for the first region in the output tensor D, traversing sub-block 1 in the left tensor A and sub-block 1′ in the right tensor B), combined with the size of each tensor and the traversal granularity, it is found that for the first region in the output tensor D, only 128 elements in the K dimension have been traversed, and the calculation has not been completed. Therefore, the traversal is continued. Similar to step 3, in the current traversal (the first outer layer and the second inner layer, i.e., for the first region in the output tensor D, traversing sub-block 2 in the left tensor A and sub-block 2′ in the right tensor B), the execution granularity of the MMA instruction is still M×N×K=64× 32×128, this is still the first traversal of the outer layer, and the coordinates of the first area are still (0, 0, 0), while the inner layer is the second traversal, the coordinates of sub-block 2 in the left tensor A are (0, 0, 128), and the coordinates of sub-block 2′ in the right tensor B are (0, 128, 0). Then, similar to step 4 above, a group of instructions including address calculation instructions, MMA instructions, and configuration instructions are generated (since this traversal does not correspond to the first or last MMA instruction, there is no need to insert the first synchronization instruction or the second synchronization instruction), and this group of instructions is spliced after the group of instructions generated in step 4.

[0215] Step 5.1 Before generating specific MMA main instructions and configuration instructions, the configuration instructions to be generated and the configuration instructions generated last time can be deduplicated to reduce the number of instructions executed by the chip and improve computing performance.

[0216] Step 6. Continue to determine whether the K-dimensional traversal is complete. If it is not yet complete, repeat step 5 until the K-dimensional traversal is complete.

[0217] Note that in step 6.1, after traversing the outer layer for the first time and the inner layer for the fourth time (i.e., after traversing the first region in the output tensor D, after traversing sub-block 4 in the left tensor A and sub-block 4′ in the right tensor B), there are still 64 numbers left in the K dimension. Therefore, the calculation granularity of the last MMA instruction is M×N×K=64×32×64. The address calculation method is the same as before and will not be repeated here.

[0218] In step 6.2, the user also provided the bias tensor C. Therefore, during the fourth and final pass through the inner layer, the address of the first bias data block 1′ in the bias tensor C can be assigned to the relevant instructions to add the bias tensor C to the MMA calculation. The coordinates of the first bias data block in the bias tensor C and the first region in the output tensor D are the same, and the address calculation method is also the same, so I will not repeat it here.

[0219] Step 7. After completing the first outer layer traversal, perform a second outer layer traversal from small to large according to the stepping order of the D matrix. At this time, the coordinates of the second region in the output tensor D are (0, 0, 32). In this traversal, except for not inserting the first synchronization instruction, the rest of the address calculation, MMA and its configuration instruction generation logic are the same as steps 3-6 and are not repeated here.

[0220] Step 8. Similar to step 7, continue the outer traversal and output the coordinates (l, m, n) of the region in tensor D as follows:

[0221] (0, 64, 0) => (0, 64, 32) => (1, 0, 0) => (1, 0, 32) => (1, 64, 0) => (0, 64, 32);

[0222] In each inner traversal of the outer traversal, the coordinates of the sub-block in the left tensor A are (l, m, k), and the coordinates of the sub-block in the right tensor B are (l, k, n). The l, m, and n in these two coordinates are the same as the coordinates (l, m, n) of the area in the output tensor D. The k in these two coordinates starts from 0 and increases until the accumulation is completed.

[0223] Step 9. When the user configures output synchronization information, after traversing the outer layer for the last time and the inner layer for the last time (that is, after all calculations are completed), a second synchronization instruction can be inserted at the non-tail end of the last group of MMA instructions. Alternatively, synchronization code can be directly added to the last group of MMAs without inserting additional synchronization instructions to reduce the total number of instructions.

[0224] Based on the same technical concept, the embodiment of the present application also provides a code generation device. The principle of solving the problem by the code generation device is similar to that of the above-mentioned code generation method. Therefore, the implementation of the code generation device can refer to the implementation of the code generation method, and the repeated parts will not be repeated.

[0225] Figure 7 A schematic diagram of the structure of a code generation device provided in an embodiment of the present application includes:

[0226] A determination module 701 is configured to determine target information in response to an instruction to perform matrix multiplication and accumulation (MMA) of an intermediate representation on a tensor, where the target information includes MMA operation granularity information supported by the chip, and tensor information of input tensors and output tensors, where the input tensors include a left tensor and a right tensor, and the tensor information of each tensor includes at least shape information.

[0227] A partitioning module 702 is configured to partition the output tensor according to the tensor information of the input tensor and the output tensor, and the operation granularity information;

[0228] A first generating module 703 is configured to generate a code segment corresponding to each region based on a left data block corresponding to the left tensor and a right data block corresponding to the right tensor, the code segment being configured to write a matrix multiplication result of the left data block and the right data block into the region, wherein each data block in the left data block and the right data block is determined based on position information of the region and tensor information of the tensor to which the data block belongs;

[0229] The second generation module 704 is used to fuse the code segments corresponding to each region to obtain an intermediate representation of the tensor for performing MMA.

[0230] In some embodiments, the input tensor further includes a bias tensor, and the first generation module 703 is further configured to:

[0231] After generating a code segment corresponding to the region, a target code is added to the code segment, wherein the target code is used to accumulate a bias data block corresponding to the region in the bias tensor to the matrix multiplication result of the left data block and the right data block, wherein the bias data block is determined based on the position information of the region and the tensor information of the bias tensor.

[0232] In some embodiments, the first generating module 703 is specifically configured to:

[0233] Dividing the left data block and the right data block into blocks respectively according to the operation granularity information;

[0234] Based on each group of sub-blocks in the left data block and the right data block having the same block order, a group of MMA instructions is generated, wherein the group of MMA instructions is used to perform a matrix multiplication operation on the group of sub-blocks, wherein:

[0235] When the group of sub-blocks is not the first group of sub-blocks corresponding to the region, the group of MMA instructions is further used to accumulate the matrix multiplication operation result of the group of sub-blocks to the matrix multiplication operation result of the previous group of sub-blocks corresponding to the region;

[0236] Each group of MMA instructions is fused to obtain the code segment.

[0237] In some embodiments, the first generating module 703 is specifically configured to:

[0238] generating an address calculation instruction for each sub-block according to the address representation information of each sub-block in the group of sub-blocks;

[0239] generating an MMA master instruction based on the sub-block addresses obtained by the address calculation instructions of each sub-block and the region address obtained by the address calculation instruction of the region, wherein the address calculation instruction of the region is generated based on the address representation information of the region;

[0240] A plurality of configuration instructions of the MMA main instruction are generated according to the configuration information required for performing matrix multiplication operations on the group of sub-blocks in the chip.

[0241] In some embodiments, the tensor information of each tensor also includes stride information and address information;

[0242] When the group of sub-blocks is a first group of sub-blocks corresponding to a first region, the address representation information of each sub-block includes position information of the sub-block in a corresponding tensor, stride information of the corresponding tensor, and address information; and the address representation information of the region includes position information of the region in the output tensor, stride information of the output tensor, and address information;

[0243] When the group of sub-blocks is not the first group of sub-blocks corresponding to the first region, the address representation information of each sub-block is the offset address of the sub-block compared to the previous sub-block in the corresponding tensor, and the address representation information of the region is the offset address of the region compared to the previous region in the output tensor.

[0244] In some embodiments, when the group of sub-blocks is not the first group of sub-blocks corresponding to the first region, the first generating module 703 is further configured to:

[0245] When the configuration information corresponding to the group of sub-blocks is repeated with the configuration information corresponding to the previous group of sub-blocks, the configuration instructions corresponding to the repeated configuration information are skipped when generating the multiple configuration instructions.

[0246] In some embodiments, the input tensor further includes a bias tensor, the target information further includes input synchronization information, and an optimization module 705 is further included for:

[0247] A first synchronization instruction is inserted after each address calculation instruction in the first code segment, where the first synchronization instruction is used to indicate waiting for data of at least one of the left tensor, the right tensor, and the bias tensor to be ready, and / or to indicate waiting for memory corresponding to the output tensor to be unlocked and available.

[0248] In some embodiments, the input tensor further includes a bias tensor, the target information further includes output synchronization information, and an optimization module 705 is further included for:

[0249] Inserting a non-tail portion of the last code segment into a second synchronization instruction, where the second synchronization instruction is used to indicate that the memory occupied by at least one of the left tensor, the right tensor, and the bias tensor has been unlocked and is available, and is also used to indicate that data of the output tensor is ready; or,

[0250] In the last code segment, a synchronization code is configured. The synchronization code is used to indicate that the memory occupied by at least one of the left tensor, the right tensor, and the bias tensor has been unlocked and is available, and is also used to indicate that data of the output tensor is ready.

[0251] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, the functional modules in the embodiments of the present application may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The coupling between the modules can be achieved through some interfaces, which are usually electrical communication interfaces, but it is not ruled out that they may be mechanical interfaces or other forms of interfaces. Therefore, the modules described as separate components may or may not be physically separated, and may be located in one place or distributed to different locations of the same or different devices. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0252] After introducing the code generation method and apparatus according to an exemplary embodiment of the present application, an electronic device according to another exemplary embodiment of the present application is introduced next.

[0253] Refer to the following Figure 8 The electronic device 130 implemented according to this embodiment of the present application is described. Figure 8 The electronic device 130 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0254] like Figure 8 As shown, the electronic device 130 is a general electronic device. Components of the electronic device 130 may include, but are not limited to, the at least one processor 131 , the at least one memory 132 , and a bus 133 connecting different system components (including the memory 132 and the processor 131 ).

[0255] Bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a processor or local bus using any of a variety of bus architectures.

[0256] The memory 132 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322 , and may further include a read-only memory (ROM) 1323 .

[0257] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0258] The electronic device 130 may also communicate with one or more external devices 134 (e.g., a keyboard, pointing device, etc.), one or more devices that enable a user to interact with the electronic device 130, and / or any device that enables the electronic device 130 to communicate with one or more other electronic devices (e.g., a router, modem, etc.). This communication may occur via an input / output (I / O) interface 135. Furthermore, the electronic device 130 may communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 136. As shown, the network adapter 136 communicates with other modules of the electronic device 130 via a bus 133. It should be understood that, although not shown, other hardware and / or software modules may be used in conjunction with the electronic device 130, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0259] In an exemplary embodiment, the electronic device of the present application may include at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores a computer program that can be executed by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor can perform the steps of any code generation method provided in the embodiments of the present application.

[0260] In an exemplary embodiment, a storage medium is also provided. When a computer program stored in the storage medium is executed by a processor of an electronic device, the electronic device can perform any of the above-described code generation methods. Optionally, the storage medium can be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0261] In an exemplary embodiment, a computer program product is also provided. When the computer program is executed by a processor, any exemplary method provided in the present application is implemented.

[0262] It should be noted that although several modules or submodules of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the application, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.

[0263] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0264] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0265] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0266] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application also includes these modifications and variations.

Claims

1. A code generation method, characterized in that: include: In response to an instruction for performing matrix multiplication and accumulation (MMA) of an intermediate representation on a tensor, target information is determined, where the target information includes MMA operation granularity information supported by the chip, and tensor information of input tensors and output tensors, where the input tensors include a left tensor and a right tensor, and the tensor information of each tensor includes at least shape information. The MMA operation granularity information refers to a data size that can be processed by one MMA instruction. Partitioning the output tensor according to the tensor information of the input tensor and the output tensor, and the operation granularity information; Based on a left data block corresponding to each region in the left tensor and a right data block corresponding to the right tensor, a code segment corresponding to the region is generated, the code segment being used to write a matrix multiplication result of the left data block and the right data block into the region, wherein each data block in the left data block and the right data block is determined based on position information of the region and tensor information of the tensor to which the data block belongs, and when the left data block and the right data block match the operation granularity information, a group of MMA instructions is generated based on the left data block and the right data block, and the group of MMA instructions is used as the code segment; The code segments corresponding to each region are fused to obtain the intermediate representation of the tensor for MMA.

2. The method according to claim 1, wherein The input tensor also includes a bias tensor and: After generating a code segment corresponding to the region, a target code is added to the code segment, wherein the target code is used to accumulate a bias data block corresponding to the region in the bias tensor to the matrix multiplication result of the left data block and the right data block, wherein the bias data block is determined based on the position information of the region and the tensor information of the bias tensor.

3. The method according to claim 1, wherein Based on a left data block corresponding to each region in the left tensor and a right data block corresponding to each region in the right tensor, a code segment corresponding to the region is generated, including: When the left data block and the right data block do not match the operation granularity information, dividing the left data block and the right data block into blocks according to the operation granularity information; generating, based on each group of sub-blocks having the same block order in the left data block and the right data block, a set of MMA instructions, the set of MMA instructions being used to perform a matrix multiplication operation on the group of sub-blocks, wherein, when the group of sub-blocks is not the first group of sub-blocks corresponding to the region, the set of MMA instructions is further used to accumulate a result of the matrix multiplication operation of the group of sub-blocks onto a result of the matrix multiplication operation of a previous group of sub-blocks corresponding to the region; Each group of MMA instructions is fused to obtain the code segment.

4. The method according to claim 3, wherein Based on each group of sub-blocks in the left data block and the right data block having the same block order, a set of MMA instructions is generated, including: generating an address calculation instruction for each sub-block according to the address representation information of each sub-block in the group of sub-blocks; generating an MMA master instruction based on the sub-block addresses obtained by the address calculation instructions of each sub-block and the region address obtained by the address calculation instruction of the region, wherein the address calculation instruction of the region is generated based on the address representation information of the region; A plurality of configuration instructions of the MMA main instruction are generated according to the configuration information required for performing matrix multiplication operations on the group of sub-blocks in the chip.

5. The method according to claim 4, wherein The tensor information of each tensor also includes step information and address information; When the group of sub-blocks is a first group of sub-blocks corresponding to a first region, the address representation information of each sub-block includes position information of the sub-block in a corresponding tensor, stride information of the corresponding tensor, and address information; and the address representation information of the region includes position information of the region in the output tensor, stride information of the output tensor, and address information; When the group of sub-blocks is not the first group of sub-blocks corresponding to the first region, the address representation information of each sub-block is the offset address of the sub-block compared to the previous sub-block in the corresponding tensor, and the address representation information of the region is the offset address of the region compared to the previous region in the output tensor.

6. The method according to claim 4, wherein When the group of sub-blocks is not the first group of sub-blocks corresponding to the first region, the method further includes: When the configuration information corresponding to the group of sub-blocks is repeated with the configuration information corresponding to the previous group of sub-blocks, the configuration instructions corresponding to the repeated configuration information are skipped when generating the multiple configuration instructions.

7. The method according to any one of claims 1 to 6, characterized in that: The input tensor also includes a bias tensor, the target information also includes input synchronization information, and further includes: A first synchronization instruction is inserted after each address calculation instruction in the first code segment, where the first synchronization instruction is used to indicate waiting for data of at least one of the left tensor, the right tensor, and the bias tensor to be ready, and / or to indicate waiting for memory corresponding to the output tensor to be unlocked and available.

8. The method according to any one of claims 1 to 6, characterized in that: The input tensor also includes a bias tensor, the target information also includes output synchronization information, and further includes: Inserting a non-tail portion of the last code segment into a second synchronization instruction, where the second synchronization instruction is used to indicate that the memory occupied by at least one of the left tensor, the right tensor, and the bias tensor has been unlocked and is available, and is also used to indicate that data of the output tensor is ready; or, In the last code segment, a synchronization code is configured. The synchronization code is used to indicate that the memory occupied by at least one of the left tensor, the right tensor, and the bias tensor has been unlocked and is available, and is also used to indicate that data of the output tensor is ready.

9. A code generating device, characterized in that: include: a determination module configured to determine target information in response to an instruction for performing matrix multiplication and accumulation (MMA) of an intermediate representation on a tensor, the target information including MMA operation granularity information supported by the chip, and tensor information of input tensors and output tensors, the input tensors including a left tensor and a right tensor, the tensor information of each tensor including at least shape information, and the MMA operation granularity information referring to a data size that can be processed by one MMA instruction; A partitioning module, configured to partition the output tensor according to the tensor information of the input tensor and the output tensor, and the operation granularity information; a first generation module, configured to generate a code segment corresponding to each region based on a left data block corresponding to the left tensor and a right data block corresponding to the right tensor, the code segment being configured to write a matrix multiplication result of the left data block and the right data block into the region, wherein each data block in the left data block and the right data block is determined based on position information of the region and tensor information of the tensor to which the data block belongs, and when the left data block and the right data block match the operation granularity information, generating a set of MMA instructions based on the left data block and the right data block, and using the set of MMA instructions as the code segment; The second generation module is used to fuse the code segments corresponding to each region to obtain an intermediate representation of the tensor for MMA.

10. An electronic device, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor, wherein: The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 8.

11. A storage medium, characterized in that: When the computer program in the storage medium is executed by a processor of an electronic device, the electronic device can perform the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 when executed by a processor.

Images