Matrix multiplier execution method and device, equipment and storage medium

Abstract

The invention provides an execution method and device of a matrix multiplier, equipment and a storage medium, relates to the technical field of artificial intelligence, and is suitable for executing the matrix multiplier in parallel through a plurality of calculation thread bundle groups, and the method comprises the following steps: through the same carrying thread bundle group, matrix data required by the plurality of calculation thread bundle groups to execute the matrix multiplier is converted into matrix data required by the plurality of calculation thread bundle groups to execute the matrix multiplier; carrying to an on-chip cache from a video memory according to a loading sequence; wherein the loading sequence is that the preorder data required by the plurality of calculation thread beam groups are loaded in series, and the subsequent data required by each calculation thread beam group are continuously loaded; and respectively scheduling the calculation units through the plurality of calculation thread bundle groups, and carrying out matrix multiplication calculation on the respective required matrix data acquired from the on-chip cache. By continuously loading the subsequent data required by the same calculation thread bundle group, one calculation thread bundle group starts to calculate the subsequent steps by using the calculation unit more quickly, and the result of the calculation thread bundle group is more obviously staggered from the result of the other calculation thread bundle group, so that the calculation unit is prevented from being scrambled.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a matrix multiplication operator execution method and device, equipment and a storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, the multi-head attention mechanism (Multi-Head Latent Attention, MLA for short) is widely used in deep learning. At present, the MLA operator is implemented in software design using two compute warp groups (such as Consume warp group) and one Tma producer warp group.

[0003] In order to avoid the competition for resources by the computing units (such as Tcore and Vcore), the Tma producer warp group loads the data in the order of first loading a Consume warp group and then loading another Consume warp group, which is staggered in time. However, this can cause a long wait for the Consume warp group due to the busy computing units, resulting in slow data calculation. SUMMARY

[0004] The embodiments of the present application provide a matrix multiplication operator execution method, device, equipment and storage medium, which are used to improve the execution efficiency of the matrix multiplication operator.

[0005] In a first aspect, the embodiments of the present application provide a matrix multiplication operator execution method, which is suitable for parallel execution of a matrix multiplication operator by multiple compute warp groups, and includes: loading the matrix data required by the multiple compute warp groups for executing the matrix multiplication operator from the video memory to the on-chip cache in a loading order by the same Tma producer warp group; wherein the loading order is that the preceding data required by the multiple compute warp groups is loaded in series and the subsequent data required by each compute warp group is loaded continuously. performing matrix multiplication calculation on the matrix data required by each compute warp group obtained from the on-chip cache by scheduling the computing units by the multiple compute warp groups respectively.

[0006] Optionally, at least one of the multiple matrix data corresponding to the matrix multiplication operator is a long-dimension matrix, and any matrix data is a two-dimensional matrix corresponding to the length dimension and the feature dimension; the long-dimension matrix has more length dimensions than feature dimensions.

[0007] Optionally, the subsequent data is the last two data in the matrix data required by the compute warp group.

[0008] Optionally, the matrix multiplication operator is at least one of a fusion operator; the computing unit comprises a tensor computing unit and a vector computing unit; The matrix multiplication calculation is completed by scheduling the tensor computing unit by a computing thread bundle group; After the matrix multiplication calculation, the method further comprises: performing vector calculation on the calculation result of the matrix multiplication operator by scheduling the vector computing unit by a computing thread bundle group.

[0009] Optionally, the intermediate calculation result obtained by the fusion operator through the computing unit is stored in the on-chip cache or the register.

[0010] Optionally, the matrix multiplication operator is a matrix multiplication accumulation (MMA) operator; and the fusion operator is a multi-head latent attention (MLA) operator.

[0011] In the embodiments of the present application, the matrix data required by a continuous loading of the same computing thread bundle group can make the calculation result of one computing thread bundle group output faster. Since the same computing unit is scheduled by multiple computing thread bundle groups, the computing thread bundle group with faster calculation result output can start to use the computing unit for subsequent calculation faster, and the result flow can be more obviously staggered with other computing thread bundle groups. In this way, multiple computing thread bundle groups can avoid competing for the same computing unit at the same time, the execution speed of the matrix multiplication operator is improved, and the delay is reduced.

[0012] In a second aspect, the embodiments of the present application provide an execution device of a matrix multiplication operator. The device is suitable for executing a matrix multiplication operator in parallel by multiple computing thread bundle groups, and comprises: A loading module is configured to load matrix data required by the multiple computing thread bundle groups for executing the matrix multiplication operator from a video memory to an on-chip cache in a loading order by the same computing thread bundle group; wherein the loading order is that the multiple computing thread bundle groups are loaded in series with the required preceding data and the subsequent data required by each computing thread bundle group is loaded continuously. A scheduling module is configured to schedule a computing unit by the multiple computing thread bundle groups respectively, and perform matrix multiplication calculation on the matrix data required by each computing thread bundle group obtained from the on-chip cache.

[0013] Optionally, at least one of the multiple matrix data corresponding to the matrix multiplication operator is a long-dimension matrix, and any matrix data is a two-dimensional matrix corresponding to a length dimension and a feature dimension; the long-dimension matrix has a length dimension greater than a feature dimension.

[0014] Optionally, the subsequent data is the last two data in the matrix data required by the computing thread bundle group.

[0015] Optionally, the matrix multiplication operator is at least one of a fusion operator; the computing unit comprises a tensor computing unit and a vector computing unit; and the matrix multiplication computation is completed by scheduling the tensor computing unit by a computing thread bundle group. Optionally, the scheduling module is specifically configured to: perform vector computation on the computation result of the matrix multiplication operator by scheduling the vector computing unit by a computing thread bundle group.

[0016] Optionally, the intermediate computation result of the fusion operator obtained by the computing unit is stored in the on-chip cache or a register.

[0017] Optionally, the matrix multiplication operator is a matrix multiplication accumulation (MMA) operator; and the fusion operator is a multi-head latent attention (MLA) operator.

[0018] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and capable of running on the artificial intelligence chip, and the artificial intelligence chip implements the steps of the execution method of the matrix multiplication operator when executing the computer program.

[0019] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program capable of being executed by a computer device, and when the computer program runs on the computer device, the computer device executes the steps of the execution method of the matrix multiplication operator.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product, which comprises a computer program stored in a computer readable storage medium, and the computer program comprises program instructions, and when the program instructions are executed by a computer device, the computer device executes the steps of the execution method of the matrix multiplication operator.

[0021] In the embodiment of the present application, by continuously loading the matrix data required by the same computing thread bundle group, the computation result output of one computing thread bundle group can be faster, and since the same computing unit is scheduled by multiple computing thread bundle groups, the computing thread bundle group with faster computation result output can start to use the computing unit for subsequent computation faster, and the result flow can be more obviously staggered with other computing thread bundle groups, thereby avoiding multiple computing threads competing for the same computing unit at the same time, improving the execution speed of the matrix multiplication operator, and reducing the delay. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0023] Figure 1 A structural schematic diagram of an artificial intelligence chip provided by an embodiment of the present application is provided. Figure 2 A structural schematic diagram of a calculation unit in an artificial intelligence chip provided by an embodiment of the present application is provided. Figure 3 A flowchart of an Attention operator performing calculation provided by an embodiment of the present application is provided. Figure 4 A loading sequence of a TMA warp group loading Query data from a GLM to a GSM provided by an embodiment of the present application is provided. Figure 5 A loading sequence of a TMA warp group loading Query data from a GLM to a GSM provided by an embodiment of the present application is provided. Figure 6 An execution method of a matrix multiplication operator provided by an embodiment of the present application is provided. Figure 7 A loading sequence of a TMA warp group loading Query data from a GLM to a GSM provided by an embodiment of the present application is provided. Figure 8 A loading sequence of a TMA warp group loading Query data from a GLM to a GSM provided by an embodiment of the present application is provided. Figure 9 A structural schematic diagram of an execution device of a matrix multiplication operator provided by an embodiment of the present application is provided. Figure 10 A structural schematic diagram of a computer device provided by an embodiment of the present application is provided. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and beneficial effects of the present application more clear, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0025] In order to facilitate those skilled in the art to understand, the following will explain and describe some terms in the embodiments of the present application.

[0026] Matrix Multiply-Accumulate (MMA) is a core computing operation in the Tensor Core of a GPU, which performs matrix operations of D = A x B + C, and is widely used in linear transformation and convolution operations in deep learning, and is used for AI training and inference, etc. D, A, B and C are all matrices.

[0027] Multi-head Latent Attention (MLA) greatly reduces the memory occupation and computational overhead of large language model inference through low-rank compression technology, while maintaining or even improving model performance.

[0028] Attention operator: Large language models generally use the Transformer structure as the core basic architecture, and the core computing part of the Transformer structure is the Attention operator.

[0029] On-chip cache is a temporary memory with smaller capacity than video memory, but faster data exchange speed than video memory.

[0030] Video memory can be High Bandwidth Memory (HBM) or other types of memory.

[0031] Warp group is composed of a group of warps, and warp is the scheduling unit of artificial intelligence chips when executing programs. The threads in a warp can execute the same instructions with different data resources.

[0032] Tensor is the basic data structure of operator calculation, which can be a scalar, a one-dimensional or multi-dimensional array, and different tensors can have different data types and shapes.

[0033] As shown in Figure 1 The structure of an artificial intelligence chip provided by an embodiment of the present application is shown in FIG. 1. The artificial intelligence chip 1001 includes at least a plurality of compute units (CU) 101, an on-chip cache 102 and a video memory 103. Each compute unit 101 includes a tensor computing unit 1011, a vector computing unit 1012 and a register 1013, wherein the tensor computing unit 1011 and the vector computing unit 1012 are the core of the compute unit 101.

[0034] The tensor computing unit 1011 and the vector computing unit 1012 are heterogeneous cores, the tensor computing unit 1011 acts as an accelerator in an artificial intelligence chip, and usually performs convolution calculation or matrix multiplication calculation, that is, the tensor computing unit 1011 is usually used to perform matrix multiplication operators, convolution operators, etc.

[0035] The tensor computing unit 1011 and the vector computing unit 1012 are heterogeneous cores, the tensor computing unit 1011 acts as an accelerator in an artificial intelligence chip, and usually performs convolution calculation or matrix multiplication calculation, that is, the tensor computing unit 1011 is usually used to perform matrix multiplication operators, convolution operators, etc.

[0036] The vector computing unit 1012 usually has a plurality of single instruction multiple data (SIMD) arithmetic logic units and / or a plurality of single instruction multiple thread (SIMT) arithmetic logic units, which are used to perform scalar and vector calculations. Therefore, the vector computing unit 1012 is usually used to perform normalization (softmax) operators, random dropout (dropout) operators, etc.

[0037] The cache resources in the on-chip cache 102 include group shared memory (GSM) and thread local register (TLR). The display memory 103 can be a high bandwidth memory (HBM), or other types of memory such as GLM. In addition to the GSM and TLR cache resources, the on-chip cache 102 also includes other types of cache resources, which are not limited by the present application.

[0038] The plurality of computing units 101 share the GSM, and each computing unit 101 corresponds to a unique TLR. The number of TLRs that each computing unit 101 uniquely owns can be set according to actual conditions. Compared with the GSM, the capacity of the TLR is smaller than that of the GSM, but the data exchange speed is faster than that of the GSM.

[0039] If the capacity of the register 1013 can meet the data transmission between the tensor computing unit 1011 and the vector computing unit 1012, the data between the tensor computing unit 1011 and the vector computing unit 1012 is transmitted through the register 1013.

[0040] If the capacity of register 1013 is insufficient to meet the data transmission between tensor calculation unit 1011 and vector calculation unit 1012, then the data between tensor calculation unit 1011 and vector calculation unit 1012 is transmitted through on-chip cache 102.

[0041] If the capacity of the on-chip cache 102 is insufficient to meet the data transfer requirements between the tensor calculation unit 1011 and the vector calculation unit 1012, then the data between the tensor calculation unit 1011 and the vector calculation unit 1012 is transferred through the video memory 103.

[0042] In addition to the structures described above, the artificial intelligence chip 1001 in this application may also include other structures, which are not specifically limited in this application. The artificial intelligence chip 1001 may be: a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), a domain-specific architecture (DSA), etc.

[0043] against Figure 1 The calculation unit 101 in the middle, Figure 2 An exemplary structure of a computation unit 101 is provided. The tensor computation unit 1011 within the computation unit 101 includes four tensor computation cores (T-Cores), all of which are used for tensor computation. The vector computation unit 1012 within the computation unit 101 includes sixteen vector computation cores (V-Cores), all of which are used for vector computation.

[0044] The core of the MLA operator is "low-rank KV joint compression + latent vector caching + decompression + attention computation". Its operation requires deep collaboration between the T-Core (high-throughput matrix operations) and the V-Core (high-precision vector / scalar operations). The T-Core leads all matrix multiplication-accumulation (MMA) stages, while the V-Core is responsible for all element-wise / scalar operations. The two complement each other to maximize the efficiency and accuracy of the MLA operator.

[0045] The core logic of the MLA operator is to perform low-rank compression on the KV cache of the Transformer attention, which is divided into 5 core stages: 1. Input preprocessing: Location encoding (RoPE) and format conversion of raw Q / K / V features; 2. Low-rank compression: compress KV jointly into low-dimensional latent vectors c^KV (Q optional compression); 3. Latent vector caching: store c^KV into GPU cache (replace original KV cache, reduce memory footprint); 4. Decompression: restore c^KV to K / V matrices for each head during inference; 5. Attention computation: Q K^T→softmax→ V, output attention results.

[0046] Take the MLA operator in the large model inference scenario as an example, the two types of core collaboration logic are as follows: Data loading: TMA Warp Group loads the original Q / K / V in GLM (global memory) to GSM (shared memory), and Warp Group schedules V-Core to complete RoPE encoding and format conversion; KV compression: V-Core preprocessing X → T-Core executes X W^DKV generates c^KV → V-Core stores c^KV in GSM (replaces the original KV cache, reducing memory footprint by 80%+); KV decompression: TMA loads c^KV → T-Core executes c^KV in parallel W^UK / W^UV generates K / V → V-Core corrects K / V features; Attention computation: T-Core: Q K^T / √D_h → score matrix; V-Core: softmax generates attention weights; T-Core: weights V → single-head output; aggregated output: T-Core concatenates multiple heads and performs output projection → V-Core completes the final normalization → results are written to GSM / GLM.

[0047] In MLA operator processing, the core actions of the TMA Warp Group focus on "critical nodes in data flow." Preloading: Batch loading the raw Q / K / V and compressed matrices (W^DKV / W^UK / W^UV) from the GLM to the GSM; Cache scheduling: Precisely sending the latent vector c^KV (KV compression result) from the GSM to the T-Core's dedicated registers (rather than general-purpose registers); Result write-back: Writing the final MLA result calculated by the Consumer Warp Group from the GSM back to the GLM (for use by downstream operators); Redundancy cleanup: Timely cleaning up used temporary data in the GSM (such as decompressed K / V intermediate values) to free up on-chip memory.

[0048] Within the same Consumer Warp Group, binding to either T-Core or V-Core can be dynamically switched to adapt to the "matrix operation → vector operation" switching of MLA operators; long sequences of MLA (such as 8K / 16K tokens) are split into 32 / 64 token fragments, with each Warp processing one fragment to maximize parallelism; the TMA Warp Group moves the data to GSM / registers in advance, so the Consumer Warp Group does not need to wait for global memory, and the computing power utilization rate can reach over 80% (compared to only around 50% for ordinary Warp); during the low-rank compression / decompression stage of MLA, numerical precision verification (such as preventing rank collapse) is performed simultaneously to ensure the stability of the calculation results.

[0049] The collaboration between TMA Warp Group and Consumer Warp Group is a closed loop of "asynchronous pipeline + granularity matching + resource linkage". Taking the core process of MLA operator processing QKV as an example, the collaboration sequence is as follows: Initialization: TMA Warp Group: Configure transport parameters (starting address of Q / K / V in GLM, destination address of GSM, data format, fragment size); Consumer Warp Group: Initialize computing context (bind T / V-Core, set operation precision (FP16 / BF16), split attention head fragments).

[0050] Asynchronous data transfer + preprocessing (spatiotemporal overlap). TMA Warp Group's data transfer and V-Core preprocessing are executed in parallel, with no waiting time; on-chip data is processed "as soon as it arrives".

[0051] KV compression. TMA Warp Group carries the compressed matrix granularity (per attention head) to match the T-Core computation granularity completely, avoiding data sharding redundancy.

[0052] KV decompression. TMA Warp Group directly sends data to T-Core registers (not GSM), reducing one memory access, and 90% latency reduction.

[0053] Attention computation. TMA preloads the Q matrix action in parallel with the V-Core Softmax execution action, pipelining without interruption.

[0054] Tail, resource recycling.

[0055] In the embodiments of the present application, the processes of training or reasoning of any batch task using the Attention operator are the same. In order to facilitate explanation, the following will take the reasoning of any batch task using the Attention operator as an example for detailed explanation.

[0056] In actual application, the Attention operator in large language model reasoning generates characters in each round in an autoregressive manner. When generating characters in each round, not only the query (Query) representation, key (Key) representation, and value (Value) representation calculated in the current round are needed, but also the key (Key) representation and value (Value) representation calculated in each previous round. In order to avoid repeated calculation of the key (Key) representation and value (Value) representation calculated in each previous round when generating characters in each round, the related technology uses the key-value cache (KVCache) technology to save the key (Key) representation and value (Value) representation calculated in each round in the video memory, so that the key (Key) representation and value (Value) representation calculated in each previous round can be directly read from the video memory when generating characters in each round.

[0057] The query representation refers to the specific content of the query (Query) in the attention mechanism, and the query representation can be a scalar, a vector, a multidimensional array, etc. Similarly, the key representation refers to the specific content of the key (Key) in the attention mechanism, and the key representation can be a scalar, a vector, a multidimensional array, etc. The value representation refers to the specific content of the value (Value) in the attention mechanism, and the value representation can be a scalar, a vector, a multidimensional array, etc.

[0058] The specific calculation process of the Attention operator can refer to the steps shown in Figure 3 The specific calculation process of the Attention operator can refer to the steps shown in K, and the calculation result Q K is sent to the V-core for softmax calculation, and finally the calculation result is multiplied by the Value to obtain Q K V. As Figure 2 shown, assuming there are 4 T-cores, the Query data is stored in the idle T-core according to the loading order, such as storing the first loaded Query data in T-core0, then storing the second loaded Query data in T-core1, storing the third loaded Query data in T-core2, and storing the fourth loaded Query data in T-core3. When the fifth loaded Query data is received, it is stored in T-core0. Because the T-core receives the Query data at the same time as participating in the calculation, when the fifth loaded Query data is received, it is most likely that the calculation in T-core0 has been completed and is in an idle state.

[0059] In a large language model, the Attention operator includes a raw key linear (KLinear) layer, a raw value linear (VLinear) layer, and a raw query linear (QLinear) layer, wherein the raw key linear layer, the raw value linear layer, and the raw query linear layer each correspond to a raw weight (Weight) which is obtained through model training and learning and saved in the video memory.

[0060] When using the Attention operator to infer any batch task, the software design generally uses 1 TMAwarp group and 2 Consume warp groups for programming. Among them, 1 TMA warp group is responsible for loading Query / Key / Value from GLM to GSM, and 2 Consume warp groups process different Head Query data, sharing Key / Value data. The TMA warp group loads Query data from GLM to GSM, as Figure 4 shown.

[0061] Consume warp group processes Query data by row. For two Consume warp groups, the first row is processed by Consume0 warp group, and the second row is processed by Consume1 warp group. Assuming that there are 10 Query data, which are Q00, Q01, Q02, Q03, Q04, Q10, Q11, Q12, Q13, and Q14. Consume0 warp group uses Q00, Q01, Q02, Q03, and Q04 for subsequent MMA / softmax calculation, and Consume1 warp group uses Q10, Q11, Q12, Q13, and Q14 for subsequent MMA / softmax calculation.

[0062] The TMA warp group first loads Q00 of Consume0 warp group, then loads Q10 of Consume1 warp group, then loads Q01 of Consume0 warp group, and then loads Q11 of Consume1 warp group. That is, the TMA warp group loads Query data required by one Consume0 warp group, and then loads Query data required by one Consume1 warp group.

[0063] As can be seen, there is a time interval between the time when the two Consume warp groups receive Query data required by themselves. This time offset is beneficial to avoid the problem of resource contention for the same calculation unit at the same time. However, when the data volume is too large, the problem of contention for the same T-core and V-core still exists.

[0064] In view of this, when the Attention operator loads multiple Query data, the time offset can be further increased to avoid the problem of contention for the same calculation unit at the same time. The specific operation steps of the embodiments of the present application are as follows: The embodiments of the present application change the order of loading Query data. The TMA warp group no longer loads Query data required by one Consume0 warp group, and then loads Query data required by one Consume1 warp group. Instead, multiple Query data required by Consume0 warp group are loaded in succession, and then multiple Query data required by Consume1 warp group are loaded in succession.

[0065] As shown in Figure 5 the loading sequence of the TMA warp group loading Query data provided by the embodiment of the application, the loading sequence of the preceding data such as Q00, Q10, Q01, Q11, Q02, Q12, and Q03 is unchanged. In the subsequent data, after Q03 is loaded, Q04 is loaded first, i.e., the Query data required by the Consume0 warp group is loaded continuously, and then Q13 and Q14 are loaded, i.e., the Query data required by the Consume1 warp group is loaded continuously. Therefore, when the MMA QK calculation is completed, the softmax calculation is performed, the result (S) of the mmaQK is used as the input of the softmax calculation, and the continuous loading of Q03 / Q04 can make the Q K calculation is earlier than before, the result (S) can be output faster, and the softmax calculation can be started faster; then Q13 / Q14 is loaded for the Consume1 warp group, and the result flow water between the Consume0 warp group and the Consume1 warp group is staggered more obviously, and the two Consume warp groups can better avoid competing for the same calculation unit at the same time.

[0066] As shown in Figure 6 the execution method of the matrix multiplication operator provided by the embodiment of the application is suitable for parallel execution of the matrix multiplication operator by a plurality of compute thread bundle groups, and includes the following steps: Step 601, by the same transfer thread bundle group, the matrix data required by the plurality of compute thread bundle groups for executing the matrix multiplication operator is transferred from the video memory to the on-chip cache according to the loading sequence; wherein the loading sequence is that the preceding data required by the plurality of compute thread bundle groups is loaded in series and the subsequent data required by each compute thread bundle group is loaded continuously.

[0067] An optional implementation, the transfer thread bundle group can be a TMA warp group. The query representation, the key representation, and the value representation are loaded from the video memory into the GSM or the TLR using the TMA warp group. Compared with the GSM, the capacity of the TLR is smaller than that of the GSM, but the data exchange speed is faster than that of the GSM; therefore, one or more of the query representation, the key representation, and the value representation can be loaded from the GSM to the TLR.

[0068] For example, the query representation is loaded from the GSM to the TLR assigned for the query representation. In this way, during the attention mechanism calculation, the key representation is read from the GSM, the query representation is read from the TLR, and then the read query representation and the key representation are subjected to matrix multiplication calculation to obtain the calculation result Q K. Compared with reading the query representation from the GSM, reading the query representation from the TLR is faster, and thus the data reading speed can be further improved.

[0069] It should be noted that the embodiments of the present application can also use only the GSM to save the query representation, the key representation and the value representation. In this way, during the attention mechanism calculation, the key representation and the query representation are read from the GSM, and then the read query representation and the key representation are input into a matrix multiplication operator for matrix multiplication calculation to obtain the calculation result Q K. In this regard, the present application is not specifically limited.

[0070] When the query representation, the key representation and the value representation are loaded from the video memory to the GSM or loaded from the video memory to the TLR using the TMA warp group, the loading sequence is as shown in Figure 5 .

[0071] In another alternative embodiment, the loading sequence is as shown in Figure 7 , that is, the Query data in the Consume0 warp group is completely loaded first, and then the Query data in the Consume1 warp group is loaded.

[0072] In yet another alternative embodiment, the loading sequence is as shown in Figure 8 , that is, Q02, Q03, Q04 in the Consume0 warp group are continuously loaded, and then Q12, Q13, Q14 in the Consume1 warp group are continuously loaded.

[0073] It should be understood that the loading sequence provided by the embodiments of the present application is only for reference and is not used to limit the present application, and there can be multiple loading sequences.

[0074] In step 602, the matrix data required by each of the plurality of computing thread bundle groups is subjected to matrix multiplication calculation by scheduling the computing unit through the plurality of computing thread bundle groups.

[0075] In an alternative embodiment, the plurality of computing thread bundle groups such as the Consume0 warp group and the Consume1 warp group schedule the T-core and the V-core according to the operation steps as shown in Figure 3 .

[0076] MMAQ of Consume0 warp group is calculated using Q00 on T-core0 and Key K intermediate result; MMAQ of Consume1 warp group is calculated using Q10 on T-core1 and Key K intermediate result; MMAQ of Consume0 warp group is calculated using Q01 on T-core2 and Key K intermediate result; MMAQ of Consume1 warp group is calculated using Q11 on T-core3 and Key K intermediate result; MMAQ of Consume0 warp group is calculated using Q02 on T-core0 and Key K intermediate result; MMAQ of Consume1 warp group is calculated using Q12 on T-core1 and Key K intermediate result; MMAQ of Consume0 warp group is calculated using Q03 on T-core2 and Key K intermediate result; MMAQ of Consume0 warp group is calculated using Q04 on T-core3 and Key K intermediate result; MMAQ of Consume1 warp group is calculated using Q13 on T-core0 and Key K intermediate result; MMAQ of Consume1 warp group is calculated using Q14 on T-core1 and Key K intermediate result.

[0077] The intermediate calculation results of the Consume0 warp group are integrated to obtain the calculation results of the Consume0 warp group, and the intermediate calculation results of the Consume1 warp group are integrated to obtain the calculation results of the Consume1 warp group. Since Q03 and Q04 are loaded continuously, and then Q13 and Q14 are loaded continuously, the calculation results of the Consume0 warp group are earlier than before, so that the calculation results can be output faster, and the calculation of the softmax can be started faster, avoiding the contention of the Consume0 warp group and the Consume1 warp group for the V-core at the same time. When the contention of the Consume0 warp group and the Consume1 warp group for the same calculation unit at the same time is avoided to a certain extent, the waiting of the hardware caused by the busy calculation unit can be reduced. For a single Consume warp group view, the calculation execution will be faster, and the overall delay will be shortened.

[0078] In the embodiments of the application, the matrix data required by one of the plurality of compute thread bundle groups is loaded continuously, so that the calculation results of the one compute thread bundle group can be output faster. Since the plurality of compute thread bundle groups are scheduled to the same calculation unit, the compute thread bundle group with faster calculation result output can use the calculation unit to perform subsequent calculation faster, and the result flow of the compute thread bundle group can be staggered more obviously from the results of the other compute thread bundle groups. In this way, the plurality of compute threads can avoid contention for the same calculation unit at the same time, the execution speed of the matrix multiplication operator is improved, and the delay is reduced.

[0079] In some embodiments, at least one of the plurality of matrix data corresponding to the matrix multiplication operator is a long-dimension matrix, and any matrix data is a two-dimensional matrix corresponding to the length dimension and the feature dimension; the long-dimension matrix has a length dimension greater than a feature dimension.

[0080] In an optional implementation, the MMA is a core operation of a GPU T-Core, and a standard form is: D = A x B + C (A / B / C / D are matrices). In an Attention operator, A = Query (Q), B = Key (K) T, and the MMA operation is Score = Q x K T / sqrt (D h) (the core is the MMA of Q x K T); in a fully connected layer, A = input feature (X), B = weight matrix (W), and the MMA operation is Y = X x W + b; in a convolution layer, after the convolution kernel is expanded into a matrix, the essence is also the MMA operation.

[0081] The length dimension of the K matrix is much larger than the feature dimension. The K matrix is a tensor (which can be understood as the "second input matrix of the MMA") that is the "multiplication matrix (B / B^T)" in the MMA operation. The core feature of the "length" is that the shape of the K matrix is [L, D] (L=length dimension, D=feature dimension), and L>>D (length dimension is much larger than feature dimension).

[0082] For example, in the Attention operator, the K matrix is [S, D_h] (S=sequence length, D_h=single-head feature dimension), when S=8K / 16K / 32K and D_h=64 / 128, S is much larger than D_h, which is a typical long-dimension matrix. In a popular understanding, the K matrix is a "long and narrow" matrix (the number of rows is much larger than the number of columns), rather than a "square" or "short and wide" matrix.

[0083] The long-dimension matrix occupies a lot of cache. The long-dimension matrix and the matching V matrix occupy a large amount of global memory (GLM), for example, when S=32K and D_h=128, a single-head K matrix occupies 32Kx128x2B (FP16)=8MB, and the cache occupation increases dramatically after the multi-head superposition; moreover, the data transfer efficiency is extremely low. The MMA of the long-dimension matrix needs to read a large amount of data from the GLM, the GLM bandwidth is limited, which leads to the data transfer time being greater than the calculation time; moreover, the long-dimension matrix sequence dimension splitting granularity is large, and the ordinary Consume warp group scheduling is difficult to maximize the parallel computing power of T / V-Core.

[0084] Therefore, the embodiments of the present application improve the loading order of the Tma warp group for the long-dimension matrix, avoiding the influence of the long-dimension matrix on the cache and the calculation efficiency.

[0085] In some embodiments, the post-sequential data is the last two data in the matrix data required by the compute thread bundle group.

[0086] An alternative implementation, the loading order is Figure 5 The loading order in the above formula. The embodiments of the present application take 10 Query data as an example to explain the loading order. Among them, the pre-sequential data is Q00, Q01, Q02, Q10, Q11, Q12, and the post-sequential data is Q03, Q04, Q13, Q14.

[0087] Another alternative implementation, the loading order is Figure 7 The loading order in the above formula. Among them, the pre-sequential data is Q00, Q01, Q02, Q03, Q04. The post-sequential data is Q10, Q11, Q12, Q13, Q14.

[0088] Yet another alternative embodiment, the loading order is Figure 8

[0089] In some embodiments, the matrix multiplication operator is at least one of the fusion operators; the computing unit includes a tensor computing unit and a vector computing unit; the matrix multiplication calculation is completed by scheduling the tensor computing unit by the computing thread bundle group; after the matrix multiplication calculation, the method further includes: performing vector calculation on the calculation result of the matrix multiplication operator by scheduling the vector computing unit by the computing thread bundle group.

[0090] An alternative embodiment, T-Core is a special hardware unit designed for matrix multiplication accumulation (MMA) operator, the core goal is to accelerate the tensor operation related to deep learning. Complete the matrix operation in the form of D = A × B + C (that is, the MMA operator), support low precision mixed calculation such as FP16 / BF16 / INT8 / FP8, which is the core algorithm source of deep learning training / inference. Suitable for fully connected layers, convolutional layers, attention mechanisms and other deep learning operators that rely on matrix operations.

[0091] V-Core is a general-purpose computing unit designed for scalar / vector operators, which is the core of traditional parallel computing of GPU, focusing on scientific computing, graphics rendering, and general-purpose parallel computing scenarios. Perform vector-level arithmetic logic operations (such as addition, subtraction, multiplication, division, trigonometric functions, floating point operations, etc.), support FP32 / FP64 high-precision calculation, based on SIMT (Single Instruction Multiple Thread) architecture, process element-wise vector / scalar operators. Suitable for general linear algebra (BLAS) operators, physical simulation (such as fluid mechanics), rasterization / texture sampling operators for graphics rendering, and scalar / vector operations for scientific computing; In actual GPU computing, the two types of cores often work together, for example, in the inference process of a deep learning model: V-Core processes vector operators for data preprocessing (such as normalization, data format conversion); T-Core executes the core MMA operator (such as attention matrix operation, fully connected layer); V-Core processes the output post-processing vector operator (such as element-wise calculation of softmax).

[0092] T-Core = Matrix (MMA) operator dedicated accelerator, focusing on low-precision matrix operations for deep learning; V-Core = vector / scalar operator general-purpose computing unit, focusing on high-precision general-purpose parallel computing; both are GPU hardware-level operator execution cores, not software-level operator logic.

[0093] ​In some embodiments, the intermediate calculation results obtained by the fusion operator through the calculation unit are stored in an on-chip cache or a register.

[0094] In an alternative embodiment, the data types of the query representation, the key representation and the value representation are all set as BF16 (i.e., 16-bit floating point numbers), the key representation and the value representation are stored in the GSM, and the query representation is stored in a designated cache area (32 TLRs).

[0095] The query representation and the key representation are subjected to matrix multiplication calculation to obtain a first calculation result in FP32 (32-bit floating point numbers), and the first calculation result in FP32 is stored in the cache area (64 TLRs).

[0096] In some embodiments, the matrix multiplication operator is a matrix multiplication accumulation (MMA) operator, and the fusion operator is a multi-head latent attention (MLA) operator.

[0097] In an alternative embodiment, the core innovation of the MLA lies in low-rank joint compression of key-value pairs (KVs) of the multi-head attention mechanism, which converts the complete KV matrix originally requiring a large amount of memory storage into a compact latent vector representation.

[0098] In the embodiments of the present application, the matrix data required by one of the computing thread bundle groups can be loaded continuously to make the output of the calculation result of the computing thread bundle group faster. Since the same calculation units are scheduled by the multiple computing thread bundle groups, the computing thread bundle group with faster calculation result output can start using the calculation units for subsequent calculation faster, and the result flow can be staggered more obviously with other computing thread bundle groups. In this way, the multiple computing threads do not compete for the same calculation unit at the same time, the execution speed of the matrix multiplication operator is improved, and the delay is reduced.

[0099] Based on the same technical concept, the embodiments of the present application provide an execution device 900 of a matrix multiplication operator, which is suitable for parallel execution of the matrix multiplication operator by multiple computing thread bundle groups, as shown in Figure 9 The device comprises: A loading module 901 is configured to load, by the same computing thread bundle group, matrix data required by the multiple computing thread bundle groups for executing the matrix multiplication operator from the video memory to the on-chip cache in a loading order, wherein the loading order is that the multiple computing thread bundle groups are loaded in series with the preceding data required by each computing thread bundle group and the subsequent data required by each computing thread bundle group is loaded continuously. A scheduling module 902 is configured to schedule the calculation units by the multiple computing thread bundle groups respectively to perform matrix multiplication calculation on the matrix data required by each of the multiple computing thread bundle groups and obtained from the on-chip cache.

[0100] Optionally, among the multiple matrix data corresponding to the matrix multiplication operator, at least one matrix data is a long-dimensional matrix, and any matrix data is a two-dimensional matrix corresponding to the length dimension and the feature dimension; the long-dimensional matrix has more length dimensions than feature dimensions.

[0101] Optionally, the subsequent data is the last two data points in the matrix data required to calculate the thread bundle.

[0102] Optionally, the matrix multiplication operator is at least one of the fusion operators; the computation unit includes a tensor computation unit and a vector computation unit; the matrix multiplication computation is performed by scheduling the tensor computation unit through a computation thread bundle. Optionally, the scheduling module 902 is specifically used for: The vector computation unit is scheduled by a computation thread bundle to perform vector computation on the computation results of the matrix multiplication operator.

[0103] Optionally, the intermediate calculation results obtained by the fusion operator through the computing unit are stored in the on-chip cache or register.

[0104] Optionally, the matrix multiplication operator is a matrix multiplication accumulation MMA operator; the fusion operator is a multi-head latent attention MLA operator.

[0105] Based on the same technical concept, embodiments of this application provide a computer device, such as... Figure 10 As shown, it includes at least one artificial intelligence chip 1001 and a memory 1002 connected to at least one artificial intelligence chip 1001. In this embodiment, the specific connection medium between the artificial intelligence chip 1001 and the memory 1002 is not limited. Figure 10 Taking the connection between the AI ​​chip 1001 and the memory 1002 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.

[0106] In this embodiment of the application, the memory 1002 stores instructions that can be executed by at least one artificial intelligence chip 1001. By executing the instructions stored in the memory 1002, the at least one artificial intelligence chip 1001 can execute the steps of the above-mentioned matrix multiplication operator execution method.

[0107] The artificial intelligence chip 1001 is a control center of the computer device, can connect various parts of the computer device through various interfaces and lines, and can implement execution of the matrix multiplication operator by running or executing instructions stored in the memory 1002 and calling data stored in the memory 1002. Optionally, the artificial intelligence chip 1001 can include one or more processing units, and the artificial intelligence chip 1001 can integrate an application processor and a modem processor. The application processor mainly processes an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication. It can be understood that the modem processor can also not be integrated into the artificial intelligence chip 1001. In some embodiments, the artificial intelligence chip 1001 and the memory 1002 can be implemented on the same chip, and in some embodiments, they can also be implemented on separate chips respectively.

[0108] The artificial intelligence chip 1001 can be a general processor, for example, a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware processor execution or executed by a combination of hardware and software modules in the processor.

[0109] The memory 1002, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 1002 can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. The memory 1002 is any other medium capable of carrying or storing desired program codes in the form of instructions or data structures and capable of being accessed by a computer device, but is not limited thereto. The memory 1002 in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function, used to store program instructions and / or data.

[0110] Based on the same inventive concept, the embodiments of the present application provide a computer readable storage medium storing a computer program executable by a computer device, which, when executed on the computer device, causes the computer device to perform the steps of the execution method of the matrix multiplication operator.

[0111] Based on the same inventive concept, the embodiments of the present application provide a computer program product including a computer program stored on a computer readable storage medium, the computer program including program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the execution method of the matrix multiplication operator.

[0112] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer-usable program code.

[0113] The present application is described in reference to the flowchart and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0114] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0115] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0116] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments described and shown, and it is therefore intended that the application cover any and all variations of the preferred embodiments which fall within the scope of the present application. Accordingly, the appended claims are intended to cover all such modifications and variations as falling within the scope of the application.

[0117] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method of performing a matrix multiplication operator, the method comprising: The method is suitable for performing a matrix multiplication operator in parallel through a plurality of compute thread bundle groups, comprising: loading matrix data required by the plurality of compute thread bundle groups for performing the matrix multiplication operator from a video memory to an on-chip cache in a loading order through the same transfer thread bundle group; wherein the loading order is that the preceding data required by the plurality of compute thread bundle groups is loaded in series and the subsequent data required by each compute thread bundle group is loaded continuously; performing matrix multiplication on the matrix data required by each compute thread bundle group respectively acquired from the on-chip cache through the plurality of compute thread bundle groups respectively scheduling compute units.

2. The method of claim 1, wherein, The method comprises: at least one of the plurality of matrix data corresponding to the matrix multiplication operator is a long-dimension matrix, and any matrix data is a two-dimensional matrix corresponding to a length dimension and a feature dimension; the long-dimension matrix has a length dimension greater than a feature dimension.

3. The method of claim 1, wherein, The method comprises: the subsequent data is the last two data in the matrix data required by the compute thread bundle group.

4. The method according to any one of claims 1 to 3, characterized in that, The matrix multiplication operator is at least one of fusion operators; the compute unit comprises a tensor compute unit and a vector compute unit; the matrix multiplication is completed by scheduling the tensor compute unit through the compute thread bundle group; after the matrix multiplication, the method further comprises: performing vector computation on the calculation result of the matrix multiplication operator by scheduling the vector compute unit through the compute thread bundle group.

5. The method of claim 4, wherein, The method comprises: the intermediate calculation result obtained by the fusion operator through the compute unit is stored in the on-chip cache or a register.

6. The method of claim 4, wherein, The method comprises: the matrix multiplication operator is a matrix multiplication accumulation (MMA) operator; and the fusion operator is a multi-head latent attention (MLA) operator.

7. An apparatus for performing a matrix multiplication operator, the apparatus comprising: The device is suitable for performing a matrix multiplication operator in parallel through a plurality of compute thread bundle groups, comprising: a loading module configured to load matrix data required by the plurality of compute thread bundle groups for performing the matrix multiplication operator from a video memory to an on-chip cache in a loading order through the same transfer thread bundle group; wherein the loading order is that the preceding data required by the plurality of compute thread bundle groups is loaded in series and the subsequent data required by each compute thread bundle group is loaded continuously; a scheduling module configured to perform matrix multiplication on the matrix data required by each compute thread bundle group respectively acquired from the on-chip cache through the plurality of compute thread bundle groups respectively scheduling compute units.

8. A computer device comprising a memory, the artificial intelligence chip and a computer program stored on the memory and running on the artificial intelligence chip, characterized in that, The artificial intelligence chip implements the steps of the method of any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer program product comprises a computer program stored on a computer readable storage medium, and the computer program comprises program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the method of any one of claims 1-6.

10. A computer program product, characterised in that, The computer program product comprises a computer program stored on a computer readable storage medium, and the computer program comprises program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the method of any one of claims 1-6.

Images