Methods, computing devices, media, and program products for performing matrix multiplication.

Abstract

This invention relates to a method, computing apparatus, computer-readable storage medium, and computer program product for performing matrix multiplication. The method includes, in response to determining a control variable regarding a configuration of computing units, instructing a computing unit pair mode such that data concerning an input right matrix stored in shared memory of a thread group is shared by the computing unit pair, which includes a first computing unit and a second computing unit; after the data concerning the input left matrix and the data concerning the input right matrix of the first computing unit are both ready, generating first ready information indicating that the input data of the first computing unit is ready and sending the first ready information to the second computing unit; in response to the first computing unit acknowledging receipt of the first ready information and the second ready information indicating that the input data of the second computing unit in the computing unit pair is ready, the first computing unit initiates a matrix multiplication operation. This improves the utilization of hardware resources and computational throughput.

Description

Technical Field

[0001] The embodiments of the present invention generally relate to the field of artificial intelligence, and more specifically to a method, computing device, computer-readable storage medium, and computer program product for performing matrix multiplication. Background Technology

[0002] In traditional methods for performing matrix multiplication, when a single computation unit (CU, e.g., a computation core) in an image processor performs matrix multiplication, the left matrix data (or simply "input data A") comes from the CU's register, and the right matrix data (or simply "input data B") comes from the CU's group shared memory (GSM). Before computation, the CU needs to wait for barrier information regarding input data A and input data B to be notified that the input data is ready before performing matrix multiplication. In the traditional matrix multiplication architecture, the image processor is configured with multiple CUs, but only the GSM resources of a single CU can be utilized. Input data B in the GSM of another CU needs to be reused by other CUs in the matrix multiplication, but this requires other CUs to repeatedly load data B. The lack of an efficient coordination mechanism between CUs leads to low hardware resource utilization, making it difficult to improve computational throughput through multi-CU parallelism, thus limiting overall computational performance.

[0003] In summary, the shortcomings of existing methods for performing matrix multiplication are limited hardware resource utilization and computational throughput. Summary of the Invention

[0004] This invention provides a method, computing device, computer-readable storage medium, and computer program product for performing matrix multiplication, which can improve the utilization of hardware resources and adapt to larger-scale operations.

[0005] According to a first aspect of the present invention, a method for performing matrix multiplication is provided. The method includes: in response to determining a configuration mode of computational units by a control variable indicating a computational unit pair mode, such that data about an input right matrix stored in shared memory of a thread group is shared by the computational unit pair, the computational unit pair including a first computational unit and a second computational unit, the first computational unit and the second computational unit being different computational cores for executing the same matrix multiplication instruction; after the data about the input left matrix and the data about the input right matrix of the first computational unit are both ready, generating first ready information indicating that the input data of the first computational unit is ready and sending the first ready information to the second computational unit; and in response to the first computational unit acknowledging receipt of the first ready information and the second ready information indicating that the input data of the second computational unit in the computational unit pair is ready, the first computational unit initiates a matrix multiplication operation.

[0006] According to a second aspect of the present invention, a computing device is also provided. The computing device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the computing device to perform the method of the first aspect of the present invention.

[0007] According to a third aspect of the present invention, a computer-readable storage medium is also provided. The computer-readable storage medium stores a computer program that, when executed by a machine, performs the method of the first aspect of the present invention.

[0008] According to a fourth aspect of the present invention, a computer program product is also provided, comprising a computer program that, when executed by a machine, performs the method of the first aspect of the present invention.

[0009] In some embodiments, making the data about the input right matrix stored in the thread group shared memory shared by the computation unit pair includes: making the thread group shared memory of the first computation unit and the second computation unit respectively store the data about the input right matrix and share the stored data about the input right matrix with each other in the computation unit pair.

[0010] In some embodiments, the method for performing matrix multiplication further includes: in response to a control variable indicating a single computation unit mode that determines the configuration of the computation unit, data about the input right matrix stored in the thread group shared memory of the current computation unit is accessible and readable only by the current computation unit and cannot be shared by other computation units.

[0011] In some embodiments, the method for performing matrix multiplication further includes: in response to determining a control variable indicating the configuration mode of the computing unit to a mode, after the input data of the second computing unit is ready, simultaneously sending second ready information to the second computing unit and the first computing unit; and in response to the second computing unit acknowledging receipt of the second ready information and the first ready signal, the second computing unit initiates matrix multiplication operation.

[0012] In some embodiments, the method for performing matrix multiplication further includes: in response to a control variable indicating the configuration mode of the computing unit to a mode, after the data of the second computing unit regarding the input left matrix and the data regarding the input right matrix are both ready, generating second ready information and sending the second ready information to the synchronization unit of the first computing unit via the synchronization unit of the second computing unit; and in response to the second computing unit acknowledging receipt of the second ready information and the first ready information, the second computing unit initiates matrix multiplication operation.

[0013] In some embodiments, the method for performing matrix multiplication further includes: in response to a control variable indicating a single computation unit mode for determining the configuration of the computation unit, after the data of the first computation unit regarding the input left matrix and the data regarding the input right matrix are both ready, the first computation unit performs matrix multiplication using a local general matrix multiplication operation unit.

[0014] In some embodiments, generating first ready information to indicate that the input data of the first computing unit is ready and sending the first ready information to the second computing unit includes: confirming that the data about the input left matrix from the registers local to the first computing unit is ready, and that the data about the input right matrix stored in the thread group shared memory local to the first computing unit is ready, generating the first ready information; and sending the first ready information to the synchronization unit of the second computing unit via the synchronization unit of the first computing unit.

[0015] In some embodiments, the method for performing matrix multiplication further includes: determining a first time when a plurality of computing units configured by a graphics processor perform input data loading and a second time when perform matrix multiplication operations; and configuring at least a portion of the computing units in a computation pair mode so that the first time matches the second time, thereby realizing pipelined execution of input data loading and matrix multiplication operations.

[0016] This invention, when the two computing units are in a computing unit pair mode, can reuse the data about the input right matrix in the shared memory of the thread groups of both units, reducing the repeated loading of this data. Furthermore, in computing unit pair mode, the two computing units can perform MMA calculations on the two input left matrices and the shared input right matrix data respectively, while minimizing the repeated loading of the input right matrix data. Therefore, it can improve the utilization of hardware resources and adapt to larger-scale operations. It also supports collaborative access to data in shared memory by the two computing units, thereby significantly improving computational throughput. In addition, the computing unit mode can be switched by controlling variables, thus adapting to diverse application scenarios and improving the compatibility and flexibility of data processing. Moreover, two readiness messages ensure synchronous startup of the two computing units, avoiding computational errors caused by data loss and achieving efficient synchronization between the two computing units.

[0017] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0018] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements.

[0019] Figure 1 The diagram illustrates a computing apparatus for performing a method of matrix multiplication according to an embodiment of the present invention.

[0020] Figure 2 The flowchart of a method for performing matrix multiplication according to some embodiments of the present invention is shown.

[0021] Figure 3 A schematic diagram of the architecture of a graphics processor according to some embodiments of the present invention is shown.

[0022] Figure 4 A schematic diagram of the interaction flow between a computing unit and a first computing unit and a second computing unit in a mode, according to some embodiments of the present invention, is shown.

[0023] Figure 5 The diagram illustrates flowcharts of some embodiments of the present invention for implementing a method of pipelined input data loading and matrix multiplication operations.

[0024] Figure 6 The illustration shows the effect of some embodiments of the present invention for implementing a method of pipelined input data loading and matrix multiplication operations.

[0025] In the various figures, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0026] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0027] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects.

[0028] As described above, the shortcomings of existing methods for performing matrix multiplication are limited hardware resource utilization and computational throughput.

[0029] To at least partially address one or more of the aforementioned problems and other potential issues, exemplary embodiments of the present invention provide at least one method for performing matrix multiplication. The method includes: in response to determining a configuration mode of computational units, indicating a computational unit pair mode such that data about the input right matrix stored in shared memory of a thread group is shared by the computational unit pair, the computational unit pair including a first computational unit and a second computational unit, the first and second computational units being different computational cores for executing the same matrix multiplication instruction; after the data about the input left matrix and the data about the input right matrix of the first computational unit are both ready, generating first ready information indicating that the input data of the first computational unit is ready and sending the first ready information to the second computational unit; and in response to the first computational unit acknowledging receipt of the first ready information and the second ready information indicating that the input data of the second computational unit in the computational unit pair is ready, the first computational unit initiates a matrix multiplication operation.

[0030] In this way, when the two computing units are in compute unit pair mode, the data about the input right matrix in the shared memory of the thread groups of both units can be reused, reducing the repeated loading of the input right matrix data. Furthermore, in compute unit pair mode, the two computing units can perform MMA calculations on the two input left matrices and the shared input right matrix data respectively, while reducing the repeated loading of the input right matrix data. Therefore, it can improve the utilization of hardware resources and adapt to larger-scale operations. Simultaneously, it supports collaborative access to data in shared memory by the two computing units, which can significantly improve computational throughput. In addition, the computing unit mode can be switched by controlling variables, thereby adapting to diverse application scenarios and improving the compatibility and flexibility of data processing. Moreover, two readiness messages ensure that the two computing units start synchronously, avoiding computational errors caused by data loss and achieving efficient synchronization between the two computing units.

[0031] Figure 1 A schematic diagram illustrates a computing apparatus 100 implementing a method for performing matrix multiplication according to an embodiment of the present invention. Figure 1 As shown, the computing device 100 may have one or more processors and a memory communicatively connected to the one or more processors; wherein the memory stores instructions executable by at least one processor. The processors include dedicated processing units such as graphics processing units (GPUs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), general-purpose computing on graphics processing units (GPGPUs), and general-purpose processors such as CPUs. In some embodiments, the computing device 100 further includes at least: an input right matrix data calculation unit sharing module 102, a first ready information sending module 104, and a first calculation unit initiation matrix multiplication operation module 106.

[0032] Regarding the input right matrix data calculation unit pair sharing module 102, it is used to respond to the control variable indicating the calculation unit pair mode in response to determining the configuration mode of the calculation unit, such that the data about the input right matrix stored in the shared memory of the thread group is shared by the calculation unit pair. The calculation unit pair includes a first calculation unit and a second calculation unit, which are different calculation cores for executing the same matrix multiplication instruction.

[0033] Regarding the first ready information sending module 104, it is used to generate first ready information to indicate that the input data of the first calculation unit is ready and send the first ready information to the second calculation unit after the data of the first calculation unit regarding the input left matrix and the data of the first calculation unit regarding the input right matrix are both ready.

[0034] Regarding the first computing unit initiating matrix multiplication operation module 106, it is used to initiate matrix multiplication operation in response to the first computing unit confirming receipt of first ready information and second ready information indicating that the input data of the second computing unit in the computing unit pair is ready.

[0035] The following will combine Figure 2 and Figure 3 A method 200 for performing matrix multiplication, according to an embodiment of the present invention, is described. It should be understood that method 200 can, for example, be performed in... Figure 1 The described computing device 100 performs the operation. Method 200 may also include additional actions not shown and / or the actions shown may be omitted; the scope of the invention is not limited in this respect.

[0036] In step 202, in response to the computing device 100 determining a control variable indicating a computing unit pair mode regarding the configuration of computing units, the data about the input right matrix stored in the shared memory of the thread group is shared by the computing unit pair, which includes a first computing unit and a second computing unit, which are different computing cores for executing the same matrix multiplication instruction.

[0037] Regarding the computing device 100, it includes, for example, an artificial intelligence chip for performing matrix multiplication. It should be understood that the computing device 100 may also include other components besides the artificial intelligence chip. The artificial intelligence chip is, for example, but not limited to, […]. Figure 3 The graphics processing unit (or GPU) 300 is shown. The graphics processing unit 300 includes, for example, multiple computing units (such as...) Figure 3 The computing units include a first computing unit 310, high-bandwidth memory 304 (HBM), and a second-level cache (e.g., L2 cache 306). In some embodiments, at least some of the computing units are configured, for example, in a computing unit pair mode. Figure 3 As indicated by 308. In a multi-computation unit configuration, the remaining computation units are configured, for example, as indicated by 336.

[0038] Each computational unit includes, for example, two computational cores for executing the same matrix multiplication instruction. Each computational unit is, for example, a computational core. Each computational unit further includes, for example, a level 1 cache (e.g., ...). Figure 3 L1 cache 312), thread group shared memory (such as...) Figure 3 Shared memory 314), memory access units (such as...) Figure 3 The memory access unit 318 in the middle), the control unit (such as Figure 3 Control unit 322), execution unit (such as...) Figure 3 Execution unit 320) and tensor kernel (such as Figure 3 Tensor kernel 316 in the tensor kernel). The tensor kernel includes synchronization units (such as... Figure 3 Synchronization unit 328), cache (such as...) Figure 3 (Cache 330 in the middle) and general matrix multiplication unit (such as Figure 3 The general matrix multiplication unit 332 in the execution unit may include registers (such as...). Figure 3 Register 326 in the middle) and arithmetic logic unit (such as Figure 3 (Arithmetic logic unit 324 in the middle).

[0039] If the current computing unit is configured in single computing unit mode, the data about the input right matrix stored in the shared memory of the thread group of the current computing unit can only be read by the current computing unit and cannot be shared by other computing units.

[0040] If two computational units (e.g., a first computational unit and a second computational unit) are configured in a computational unit pair mode, the first and second computational units configured in this mode each store data about the input right matrix in their respective thread group shared memory, and share the stored data about the input right matrix with each other in the computational unit pair. For example, the data about the input right matrix stored in the thread group shared memory of the first computational unit can be shared by the first and second computational units, and the data about the input right matrix stored in the thread group shared memory of the second computational unit can be shared by the first and second computational units. It should be understood that the first and second computational units configured in the computational unit pair mode are different computational cores for executing the same matrix multiplication instruction.

[0041] For example, the left matrix block A1 of the first computing unit 310 is stored in a register in the execution unit 320 included in the first computing unit (e.g., ...). Figure 3 (As indicated by marker 326 in the text). The right matrix block B1 of the first computing unit 310 is stored in the thread group shared memory 314 local to the first computing unit 310. The second computing unit The left matrix block A2 is stored in the registers of the execution unit included in the second computation unit. The right matrix block B2 is stored in the thread group shared memory of the second computing unit. In computing unit pair mode, the right matrix block B1 stored in the thread group shared memory 314 of the first computing unit and the right matrix block B2 stored in the thread group shared memory of the second computing unit can be read by the first computing unit and the second computing unit respectively. In some embodiments, the right matrix block B1 stored in the thread group shared memory 314 of the first computing unit 310 and the right matrix block B2 stored in the thread group shared memory 314 of the second computing unit 310 are respectively read by the first computing unit and the second computing unit. The right matrix block B2 stored in the shared memory of the thread group can form a larger right matrix block B.

[0042] In step 204, after the data about the input left matrix and the data about the input right matrix of the first computing unit are both ready, the computing device 100 generates first ready information to indicate that the input data of the first computing unit is ready and sends the first ready information to the second computing unit.

[0043] Regarding the method for generating first ready information to indicate that input data of the first computing unit is ready and sending the first ready information to the second computing unit, in some embodiments, it includes, for example, the following: the computing device 100 confirms that data about the input left matrix (i.e., left matrix block A1) from a local register 326 of the first computing unit 310 is ready, and data about the input right matrix (i.e., right matrix block B1) stored in a thread group shared memory 314 local to the first computing unit is ready, and generates the first ready information, for example... Figure 3 As indicated by marker 334; and via synchronization unit 328 of the first computing unit 310 to the second computing unit. The synchronization unit sends the first ready information.

[0044] In step 206, in response to the first computing unit of computing device 100 confirming receipt of the first ready information and the second ready information for indicating that the input data of the second computing unit in the computing unit pair is ready, the first computing unit initiates matrix multiplication operation.

[0045] For example, the first computing unit 310 confirms whether it has received first ready information indicating that its own data is ready and a second computing unit indicating that the computing unit is in the pair. The input data is ready for the second readiness information (as indicated by marker 338); if the first and second readiness information are confirmed to be received, the first computing unit 310 initiates matrix multiplication to obtain the output matrix (e.g., ...). Figure 3 The corresponding data D). If no ready information is received from either the first ready information or the second ready information, the first computing unit 310 waits and does not start the matrix multiplication operation.

[0046] Similarly, the second computing unit The system checks whether it has received a second ready information indicating that its own data is ready and a first ready information indicating that the input data of the first computing unit in the computing unit pair is ready; if it confirms that it has received the first ready information and the second ready information, the second computing unit... Initiating matrix multiplication yields the output matrix (e.g.) Figure 3 The corresponding data D). If no ready information is received from either the first or second ready information, the second computing unit... Wait, matrix multiplication will not be started for now.

[0047] In this way, when the two computing units are in a computing unit pair mode, the present invention can reuse the data about the input right matrix in the shared memory of the thread groups of both units, reducing the repeated loading of the input right matrix data. Furthermore, in computing unit pair mode, the two computing units can perform MMA calculations on the two input left matrices and the shared input right matrix data respectively, while reducing the repeated loading of the input right matrix data. Therefore, it can improve the utilization of hardware resources and adapt to larger-scale operations. It also supports collaborative access to data in shared memory by the two computing units, thereby significantly improving computational throughput. In addition, the computing unit mode can be switched by controlling variables, thus adapting to diverse application scenarios and improving the compatibility and flexibility of data processing. Moreover, two readiness messages ensure that the two computing units start synchronously, avoiding computational errors caused by data loss and achieving efficient synchronization of the two computing units.

[0048] The following combination Figure 3 The architecture of a graphics processor 300 according to some embodiments of the present invention is described. For example... Figure 3 As shown, the graphics processor 300 includes, for example, multiple computing units, high-bandwidth memory 304 (HBM), and a secondary cache (e.g., L2 cache 306).

[0049] Regarding the computing unit, it is used to perform matrix multiplication operations. In some embodiments, the computing unit is, for example, a computing core. The computing unit is configured, for example, such that, when configured to a computing unit pair mode via a control variable, data concerning the input right matrix stored in the shared memory of the local thread groups of the first and second computing units configured in computing unit pair mode is shared by the computing unit pair; and when configured to a single computing unit mode via a control variable, data concerning the input right matrix stored in the shared memory of the thread group of the current computing unit can only be accessed and read by the current computing unit and cannot be shared by other computing units. Figure 3 As shown, label 308 indicates a pair of computing units configured in a computing unit pair mode. Label 336 indicates a computing unit configured in a single computing unit mode.

[0050] like Figure 3 As shown, in some embodiments, each computing unit includes, for example, a level 1 cache (e.g., an L1 cache, such as...). Figure 3 (as indicated by marker 312), shared memory (e.g., thread group shared memory, such as...) Figure 3 (as indicated by marker 314), tensor kernel (such as...) Figure 3 As indicated by mark 316), execution unit (such as...) Figure 3 (as indicated by mark 320) and control unit (such as Figure 3 (As indicated by marker 322). The tensor kernel further includes, for example, a synchronization unit (such as...). Figure 3 As indicated by marker 328), cache (such as Figure 3 (as indicated by mark 330) and general matrix multiplication unit (such as...) Figure 3 (As indicated by mark 332).

[0051] Regarding the synchronization unit, it is used to synchronize the ready signals between the two computing units in a pair of computing units. For example, the synchronization unit of the first computing unit is configured to: after the data of the first computing unit regarding the input left matrix and the data regarding the input right matrix are both ready, generate first ready information indicating that the input data of the first computing unit is ready, and simultaneously send the first ready information to the second computing unit. The synchronization unit of the second computing unit is configured to: after the data of the second computing unit regarding the input left matrix and the data regarding the input right matrix are both ready, generate second ready information indicating that the input data of the second computing unit is ready, and simultaneously send the first ready information to the first computing unit.

[0052] Regarding caching (e.g.) Figure 3 As indicated by 330 in the middle, it is configured at least to: facilitate the sharing of data (e.g., data B) about the input right matrix between the first and second computational units in computational unit pair mode. Figure 3 As shown, the right matrix block B1 in the local shared memory of the first computing unit is provided to the second computing unit via the cache of the first computing unit. Similarly, the right matrix block B2 in the local shared memory of the second computing unit is provided to the first computing unit via the cache of the second computing unit.

[0053] Regarding the general matrix multiplication operation unit (such as...) Figure 3As indicated by 332 in the diagram, the matrix multiplication unit performs matrix multiplication, for example, based on data about the input left matrix (e.g., left matrix block A1) from the local register of the current computation unit and data about the input right matrix from the computation unit pair (e.g., right matrix block B1 in the shared memory of the current computation unit, or right matrix block B2 in the shared memory of another computation unit in the computation unit pair), to generate a result matrix, and provides the result matrix to a local register (e.g., a thread register). In some embodiments, the general matrix multiplication unit also reads intermediate calculation results from the local register for accumulation.

[0054] Regarding the execution unit, it includes, for example, registers (such as...) Figure 3 (as indicated by mark 326), arithmetic logic unit (such as...) Figure 3 (As indicated by mark 324). The register is used to store the data of the input left matrix used for matrix multiplication (e.g., left matrix blocks) and intermediate calculation results. Additionally, the register can also be used to store the calculation result matrix (e.g., data D).

[0055] Regarding the high bandwidth memory 304 (HBM), it is used to store data, descriptors, and instructions for matrix multiplication. In some embodiments, before performing a computational task (e.g., a computational task related to matrix multiplication), each computing unit needs to retrieve the data required to perform the matrix multiplication and the instructions for performing the computational task from the HBM. The path for retrieving the aforementioned data and / or instructions includes, for example, via the compute core bus interface (or "CBI"). Figure 3 (Not shown in the image) goes to the second-level cache (e.g., L2 cache 306), and then via a memory link (link, Figure 3 (Not shown in the image) retrieves relevant data from high-bandwidth memory 304; then the retrieved data is returned to the computing unit via memory link, L2 cache 306 and computing core bus interface (for example, data about the input right matrix is ​​provided to the thread group shared memory of the computing unit, and data about the input left matrix is ​​provided to the registers of the execution unit).

[0056] In some embodiments, a secondary cache (e.g., L2 cache 306) is used by the computing unit to control the system (e.g., a control unit) before performing a computational task. Figure 3 (As indicated by marker 322) will load data from HBM into the second-level cache in order to enable data preloading.

[0057] Figure 4 A schematic diagram of the interaction flow between a computing unit and a first computing unit and a second computing unit in a mode, according to some embodiments of the present invention, is shown.

[0058] like Figure 4 As shown, in step 402, the first computing unit 310 confirms that the data regarding the input left matrix from its local register is ready. For example, after loading the data regarding the input left matrix (e.g., left matrix block A1) into the local register of the first computing unit, the first computing unit generates a signal indicating that the data regarding the input left matrix is ​​ready.

[0059] In step 404, the second calculation unit The system confirms that the data concerning the input left matrix from the local registers of the second computing unit is ready. For example, this is done after loading the data concerning the input left matrix (e.g., left matrix block A2) into the local registers of the second computing unit. The second computing unit then generates a signal indicating that the data concerning the input left matrix is ​​ready. It should be understood that the present invention does not limit the execution order of steps 402 and 404; the execution order of step 402 can be before or after step 404, depending on the data loading status.

[0060] In step 406, it is determined that the data regarding the input right matrix stored in the thread group shared memory local to the first computing unit is ready. For example, after the right matrix block B1 is loaded into the GSM of the first computing unit, a signal indicating that the data regarding the input right matrix of the first unit is ready is generated.

[0061] In step 408, the synchronization unit of the first computing unit 310 generates first ready information. This first ready information indicates that the input data for the first computing unit is ready.

[0062] In step 410, the data is transmitted to the second computing unit via the synchronization unit of the first computing unit 310. The synchronization unit sends the first ready information. It can be understood that the first ready information can be sent simultaneously to the synchronization unit of the first computing unit 310 and the second computing unit. Synchronization unit.

[0063] In step 412, the second calculation unit The system determines that the data about the input right matrix stored in the thread group shared memory (GSM) local to the second computing unit is ready. For example, the right matrix block B2 is loaded into the second computing unit. After GSM connection, a signal indicating that the input right matrix of the second computing unit is ready is generated. In some embodiments, the second computing unit... The right matrix block B2 stored in the GSM of the first computing unit 310 and the right matrix block B1 stored in the GSM of the first computing unit 310 together form data B.

[0064] In step 414, the second calculation unit The synchronization unit generates second readiness information to indicate that the input data of the second computing unit in the computing unit pair is ready.

[0065] In step 416, via the second computing unit The synchronization unit sends the second ready information to the synchronization unit of the first computing unit.

[0066] In step 418, if the first computing unit 310 confirms receipt of the first ready information and the second ready information indicating that the input data of the second computing unit in the computing unit pair is ready, the first computing unit initiates matrix multiplication; and if the second computing unit... The second computing unit confirms receipt of both the second and first readiness messages. Initiate matrix multiplication. This ensures the operation of the first computational unit 310 and the second computational unit. Matrix multiplication operations are initiated only when the input data is ready.

[0067] In the above scheme, by using the computing unit pair mode, each computing unit starts matrix multiplication only after obtaining the readiness information of the input data from the two computing units. This ensures that the two computing units start synchronously, avoids calculation errors caused by missing data, and achieves efficient synchronization of the two computing units.

[0068] The following combination Figure 5 This describes method 500 for implementing pipelined input data loading and matrix multiplication operations. Method 500 can be implemented by... Figure 1 The computing device 100 executes.

[0069] In step 502, the computing device 100 determines the first time when the plurality of computing units configured by the graphics processor perform input data loading and the second time when they perform matrix multiplication operations.

[0070] For example, determine the time (T1) required for the multiple computing units configured in the graphics processor to perform input data loading and the time (T2) required to perform matrix multiplication operations.

[0071] In some embodiments, the time taken for the computing unit to load input data is related to the size of the input data, memory bandwidth, or data transmission path. The time taken for the computing unit to perform matrix multiplication on the loaded matrix data can be related to factors such as the number of multiplications and additions in the matrix operation and the parallelism of the multiplication accumulator.

[0072] In step 504, the computing device 100 configures at least a portion of the computing units among the plurality of computing units into a computing pair mode so that the first time and the second time are matched, thereby realizing the pipelined execution of input data loading and matrix multiplication operations.

[0073] For example, computing device 100 compares the size or ratio of T1 and T2. It should be understood that if T1 and T2 do not match, the data loading process and the matrix multiplication operation need to be performed linearly. For example, the matrix multiplication operation might be performed only after thread group 1 has completed loading the data required for all iterations of the loop calculations. If T1 and T2 match, it is easier to pipeline the loading of input data and the matrix multiplication operation alternately.

[0074] As mentioned earlier, in the unit-to-unit mode, the data about the input right matrix stored in the shared memory of the thread groups of the first and second computing units can be shared by both units. For example, the shared memory of the thread group of the first computing unit can store the data of the right matrix (such as right matrix block B1). The shared memory of the thread group of the second computing unit can store the data of the right matrix (such as right matrix block B2). Both shared memory groups allow access to each other (i.e., the first computing unit can read right matrix block B2, and the second computing unit can read right matrix block B1). The first and second computing units can load data simultaneously, making full use of the bandwidth for data transport and avoiding repeated transport of right matrix blocks (no need to load from global memory multiple times), thus improving data access efficiency. It should be understood that, since the second computing unit does not need to load data from the second-level cache or global memory (e.g., HBM) based on the data block address required for the corresponding loop calculation round when performing MMA calculations, the time required to load input data in the unit-to-unit mode is less than the total time required to load input data in the two single-computing-unit modes.

[0075] Therefore, the computing device 100 can configure at least a portion of the computing units among the plurality of computing units in the graphics processor into a computing pair mode based on the comparison results regarding T1 and T2, so that the first time and the second time are matched. For example, if it is determined that the time required to load input data (T1) is greater than the time required to perform matrix multiplication (T2), the time required to load input data (T1) is reduced by configuring the number of computing unit pairs until the time required to load input data (T1) matches the time required to perform matrix multiplication (T2), thereby realizing the pipelined execution of input data loading and matrix multiplication.

[0076] like Figure 6 As shown in the left half, thread group 1 is used to execute multiple groups (such as...). Figure 6The diagram shows eight sets of matrix multiplication operations (i.e., MMA operations). During this process, thread group 2 continuously loads (or moves) data, thus covering a certain scale of matrix multiplication operations performed by thread group 1. In the traditional approach, thread group 1, for example, executes four sets of MMA operations before loading the input data required for the next four sets of MMA operations; similarly, thread group 2 executes another four sets of MMA operations before loading the input data required for the next four sets of MMA operations. Therefore, in the traditional method, input data loading and MMA operations can only be performed linearly. However, this invention can match the time required for input data loading (T1) to the time required for performing matrix multiplication operations (T2) by configuring the number of computational unit pairs, thereby conveniently achieving parallel processing of input data loading and MMA operations.

[0077] Figure 6 The right half illustrates another pipelined approach to input data loading and matrix multiplication. For example... Figure 6 As shown, the inner loop executes two sets of operations (loading two sets of matrix data), while the outer loop executes an additional eight sets (storing the results of eight matrix operations). The inner loop performs multiplication and addition calculations on the block combinations of the matrix. The outer loop can be an iterative control loop for large-scale matrices, responsible for traversing all combinations of matrix blocks. Iterative looping can cover the block processing of large-scale matrices. In other words, by alternately executing matrix multiplication tasks from multiple thread groups, parallel coverage of data loading and matrix multiplication operations can be achieved, maximizing hardware utilization.

[0078] Through the above methods, the present invention can conveniently realize the execution of matrix multiplication in a pipeline manner.

[0079] The various processes and procedures described above, such as methods 200 and 500, can be executed at a computing device. This computing device may include, for example, at least one processor (at least one graphics processor and at least one central processing unit); and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor. In some embodiments, methods 200 and 500 may be implemented as a computer software program or program product tangibly contained in a machine-readable medium. In some embodiments, part or all of the computer program may be loaded and / or installed on the computing device via read-only memory (ROM) and / or a communication unit. When the computer program is loaded into random-access memory (RAM) and executed by the GPU and CPU, one or more actions of methods 200 and 500 described above can be performed.

[0080] This invention can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention. The computer-readable storage medium may be a tangible device capable of holding and storing instructions used by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof.

[0081] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network, to an external computer or external storage device. Various aspects of the invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0082] These computer-readable program instructions can be provided to the central processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the central processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0084] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps loaded in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0085] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors.

Claims

1. A method for performing matrix multiplication, characterized by, The method comprises: in response to determining that the control variable indicative of the configuration mode of the computing units indicates the pair mode, causing the data about the input right matrix stored in the thread group shared memory to be shared by the pair of computing units, the pair of computing units comprising a first computing unit and a second computing unit, the first computing unit and the second computing unit being different computing cores for executing the same matrix multiplication instruction; after the data about the input left matrix and the data about the input right matrix at the first computing unit are both ready, generating first ready information for indicating that the input data at the first computing unit is ready and sending the first ready information to the second computing unit; and in response to the first computing unit confirming that the first ready information and second ready information for indicating that the input data at the second computing unit in the pair of computing units is ready are received, the first computing unit initiates the matrix multiplication operation.

2. The method of claim 1, wherein, causing the data about the input right matrix stored in the thread group shared memory to be shared by the pair of computing units comprises: causing the thread group shared memory of the first computing unit and the second computing unit to each store the data about the input right matrix and share the stored data about the input right matrix to the other one in the pair of computing units.

3. The method of claim 1, wherein, further comprising: in response to determining that the control variable indicative of the configuration mode of the computing units indicates the single-computing-unit mode, the data about the input right matrix stored in the thread group shared memory of the current computing unit is only accessible for reading by the current computing unit and cannot be shared by other computing units.

4. The method of claim 1, wherein, further comprising: in response to determining that the control variable indicative of the configuration mode of the computing units indicates the pair mode, after the data about the input left matrix and the data about the input right matrix at the second computing unit are both ready, generating second ready information and sending the second ready information to the synchronization unit of the first computing unit via the synchronization unit of the second computing unit; and in response to the second computing unit confirming that the second ready information and the first ready information are received, the second computing unit initiates the matrix multiplication operation.

5. The method of claim 3, wherein, further comprising: in response to determining that the control variable indicative of the configuration mode of the computing units indicates the single-computing-unit mode, after the data about the input left matrix and the data about the input right matrix at the first computing unit are both ready, the first computing unit executes the matrix multiplication operation by using a local general matrix multiplication operation unit.

6. The method of claim 1, wherein, generating the first ready information for indicating that the input data at the first computing unit is ready and sending the first ready information to the second computing unit comprises: confirming that the data about the input left matrix from a register local to the first computing unit is ready and the data about the input right matrix stored in the thread group shared memory local to the first computing unit is ready, generating the first ready information; and sending the first ready information to the synchronization unit of the second computing unit via the synchronization unit of the first computing unit.

7. The method of claim 1, wherein, further comprising: determining a first time for a plurality of computing units configured by the graphics processor to execute input data loading and a second time for executing a matrix multiplication operation; and ​ At least some of the plurality of computing units are configured to compute a pattern so that a first time matches a second time, thereby enabling performing input data loading and matrix multiplication operations in a pipelined manner.

8. A computing device, comprising: Comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer readable storage medium, the computer program being executed by a machine to perform the method of any one of claims 1-7.

10. A computer program product, characterised in that, A computer program is stored on the computer readable storage medium, the computer program being executed by a machine to perform the method of any one of claims 1-7.

Images