Method and device for executing operation task on multi-core system and related product

Abstract

The invention discloses a method and device for executing an operation task on a multi-core system and a related product. The device may be included as a processing device in a combined processing device, which may further include an interface device and a computing device. The computing device interacts with the processing device to jointly complete computing operation specified by a user. The combined processing device can further comprise a storage device, and the storage device is connected with the computing device and the processing device and used for storing data of the computing device and the processing device. According to the scheme, the method for executing the operation task on the multi-core system is provided, data memory access can be effectively reduced, and the processing efficiency is improved.

Description

Technical Field

[0001] The present disclosure generally relates to the field of data processing. More specifically, the present disclosure relates to a method, device, computer-readable storage medium, computer program product, processing device, chip and board for performing computing tasks on a multi-core system. Background Art

[0002] The Mixture of Experts (MoE) model is a machine learning architecture that divides an AI model into different sub-networks (or "experts"), each of which focuses on a subset of the input data and works together to complete a task. This architecture enables even large models with billions of parameters to significantly reduce computational costs during pre-training and inference, improving efficiency by selectively activating the experts required for a specific task rather than activating the entire neural network for each task.

[0003] Group General Matrix Multiplication (Group GEMM) is the core operator in MoE, which deals with batch matrix multiplication problems. However, unlike ordinary batch matrix multiplication, the matrix dimension of each expert in GroupGEMM is variable. This design makes Group GEMM flexible when dealing with different experts, but it also brings computational challenges. The division of computing tasks among multiple cores has a great impact on the performance of Group GEMM. Good task division can enable multiple cores to work together and greatly improve the performance of Group GEMM.

[0004] In view of this, there is an urgent need to provide a solution for executing computing tasks on a multi-core system to support multi-core efficient parallel processing of computing tasks such as Group GEMM. Summary of the invention

[0005] In order to at least solve one or more of the technical problems mentioned above, the present disclosure proposes a solution for executing computing tasks on a multi-core system in multiple aspects.

[0006] In a first aspect, the present disclosure provides a method for executing a computing task on a multi-core system, wherein the task includes a group matrix multiplication computing task, wherein the group matrix multiplication computing task includes multiple groups of matrix multiplications, and the matrix size of each group of matrix multiplications is variable. The multi-core system includes multiple computing units and a global cache, wherein each computing unit includes one or more processor cores, and the global cache is used to cache data, so that the computing unit preferentially loads data that needs to be cached from the global cache; the method includes:

[0007] According to predetermined task splitting information, each group of matrix multiplications in the group matrix multiplication operation task is split into a plurality of task blocks;

[0008] The multi-core system adjusts the grid order of the task blocks of the output matrix in the group matrix multiplication type operation task in the corresponding dimension;

[0009] The multi-core system uniformly manages the task blocks of each group of matrix multiplications, and each operation unit performs operations in the grid order of M dimensions and N dimensions according to the adjusted task blocks of the output matrix, so that after the multi-core system calculates the task blocks of the input matrix in each cycle, it obtains output matrix blocks of Sm×Sn grid size, where M is the row dimension of the output matrix, N is the column dimension of the output matrix, Sm is the number of output matrix blocks in the M dimension, and Sn is the number of output matrix blocks in the N dimension.

[0010] In a second aspect, the present disclosure provides a device for executing computing tasks on a multi-core system, comprising a processor and a memory, wherein: the processor is configured to execute program instructions; the memory is configured to store the program instructions; when the program instructions are loaded and executed by the processor, the processor executes the method for executing computing tasks on a multi-core system according to the first aspect.

[0011] In a third aspect, the present disclosure provides a computer-readable storage medium having program instructions stored therein, which, when loaded and executed by a processor, causes the processor to execute the method for executing computing tasks on a multi-core system according to the first aspect.

[0012] In a fourth aspect, the present disclosure provides a computer program product, comprising a computer program or instructions, which, when executed by a processor, implements the method for performing computing tasks on a multi-core system according to the first aspect.

[0013] In a fifth aspect, the present disclosure provides a chip, comprising the device for executing computing tasks on a multi-core system according to the fifth aspect.

[0014] In a sixth aspect, the present disclosure provides a board comprising the chip according to the fifth aspect.

[0015] Through the method, device and related products for executing computing tasks on a multi-core system as provided above, the disclosed embodiments provide an optimized solution for implementing computing tasks such as group matrix multiplication on a multi-core system, which can effectively reduce frequent data exchange and loading, avoid input / output (I / O) bottleneck problems, and fully utilize the parallel operation characteristics of the multi-core architecture to improve processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0017] Figure 1 An exemplary structural diagram of a board in some embodiments of the present disclosure is shown;

[0018] Figure 2 An exemplary structural diagram of a combined processing device in some embodiments of the present disclosure is shown;

[0019] Figure 3 An exemplary internal structure diagram of a computing device in some embodiments of the present disclosure is shown;

[0020] Figure 4 An exemplary structural diagram of a processor core in some embodiments of the present disclosure is shown;

[0021] Figure 5 An exemplary schematic diagram showing a processor core writing data to a processor core of another cluster in some embodiments of the present disclosure;

[0022] Figure 6 An exemplary schematic diagram of a two-layer three-stage pipeline in some embodiments of the present disclosure is shown;

[0023] Figure 7 A schematic diagram of a storage (memory) architecture of a multi-core system to which the disclosed embodiments can be applied is shown;

[0024] Figure 8 An exemplary flow chart of a task scheduling method based on a multi-core system according to an embodiment of the present disclosure is shown;

[0025] Fig. 9 An abstract schematic diagram of grid partitioning of a matrix multiplication task according to some embodiments of the present disclosure is shown;

[0026] Fig.10 The task block processing method of the group matrix multiplication task according to some embodiments of the present disclosure is shown;

[0027] Fig.11 A logic flow chart of a task block loop execution process according to some embodiments of the present disclosure is shown;

[0028] Fig.12 The schematic diagram of data storage on different storage levels is shown schematically;

[0029] Fig.13 The four-stage pipeline processing process of a matrix multiplication task according to some embodiments of the present disclosure is shown;

[0030] Fig.14 A schematic diagram showing the order rearrangement of output matrix blocks according to some embodiments of the present disclosure;

[0031] Fig.15 Schematic diagram showing the order rearrangement of output matrix blocks according to other embodiments of the present disclosure

[0032] Fig.16 The schematic diagram shows the load imbalance situation;

[0033] Fig.17 A schematic logic flow chart of adaptively scheduling and executing group matrix multiplication tasks according to some embodiments of the present disclosure is shown;

[0034] Fig.18 A flow chart of a method for executing a computing task on a multi-core system according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0036] It should be understood that the terms "include" and "comprising" used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0037] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0038] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0039] The specific implementation of the present disclosure is described in detail below with reference to the accompanying drawings.

[0040] Exemplary Hardware Environment

[0041] Figure 1 FIG. 1 is a schematic diagram showing the structure of a board 10 according to an embodiment of the present disclosure. Figure 1 As shown, the board 10 includes a chip 101, which is a system-on-chip, or a system on chip (SoC), which integrates one or more combined processing devices. The combined processing device is an artificial intelligence computing unit that supports various deep learning and machine learning algorithms to meet the intelligent processing needs in complex scenarios in the fields of computer vision, speech, natural language processing, data mining, etc. In particular, deep learning technology is widely used in the field of cloud intelligence. A notable feature of cloud-based intelligent applications is the large amount of input data, which places high demands on the storage capacity and computing power of the platform. The board 10 of this embodiment is suitable for cloud-based intelligent applications and has huge off-chip storage, on-chip storage, and a large amount of computing power.

[0042] The chip 101 is connected to an external device 103 via an external interface device 102. The external device 103 is, for example, a server, a computer, a camera, a display, a mouse, a keyboard, a network card, or a wifi interface. The data to be processed can be transmitted from the external device 103 to the chip 101 via the external interface device 102. The calculation result of the chip 101 can be transmitted back to the external device 103 via the external interface device 102. According to different application scenarios, the external interface device 102 can have different interface forms, such as a PCIe interface.

[0043] The board 10 also includes a storage device 104 for storing data, which includes one or more storage units 105. The storage device 104 is connected to the control device 106 and the chip 101 through a bus and transmits data. The control device 106 in the board 10 is configured to control the state of the chip 101. To this end, in an application scenario, the control device 106 may include a microcontroller (MCU).

[0044] Figure 21 is a block diagram showing the combined processing device in the chip 101 of this embodiment. Figure 2 As shown in , the combined processing device 20 includes a computing device 201 , an interface device 202 , a processing device 203 and a storage device (DRAM) 204 .

[0045] The computing device 201 is configured to execute user-specified operations, and is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0046] The interface device 202 is used to transmit data and control instructions between the computing device 201 and the processing device 203. For example, the computing device 201 can obtain input data from the processing device 203 via the interface device 202 and write it into the storage device on the computing device 201 chip. Further, the computing device 201 can obtain control instructions from the processing device 203 via the interface device 202 and write them into the control cache on the computing device 201 chip. Alternatively or optionally, the interface device 202 can also read data in the storage device of the computing device 201 and transmit it to the processing device 203.

[0047] The processing device 203, as a general processing device, performs basic controls including but not limited to data handling, starting and / or stopping the computing device 201. Depending on the implementation, the processing device 203 can be a central processing unit (CPU), a graphics processing unit (GPU) or one or more types of processors in other general and / or special processors, these processors include but are not limited to digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, only with respect to the computing device 201 disclosed in the present invention, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are integrated and considered together, the two are regarded as forming a heterogeneous multi-core structure.

[0048] The storage device 204 may be DRAM, which is an off-chip memory used to store data to be processed. It is DDR and is usually 16G or larger in size. It is used to save data of the computing device 201 and / or the processing device 203.

[0049] Figure 3 The internal structure diagram of the computing device 201 is shown. The computing device 201 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 201 in the figure adopts a multi-core hierarchical structure design. The computing device 201 is a system on a chip, which includes multiple clusters, and each cluster includes multiple processor cores. In other words, the computing device 201 is composed of a hierarchy of system on a chip-cluster-processor core.

[0050] At the system-on-chip level, Figure 3 As shown, the computing device 201 includes an external storage controller 301 , a peripheral communication module 302 , an on-chip interconnect module 303 , a synchronization module 304 , and multiple clusters 305 .

[0051] There may be multiple external storage controllers 301, two of which are shown in the figure as an example, and are used to respond to access requests issued by the processor core and access external storage devices, such as Figure 2 DRAM 204 in the chip, thereby reading data from outside the chip or writing data to outside the chip. The peripheral communication module 302 is used to receive control signals from the processing device 203 through the interface device 202 to start the computing device 201 to perform tasks. The on-chip interconnect module 303 connects the external storage controller 301, the peripheral communication module 302 and multiple clusters 305 to transmit data and control signals between each module. The synchronization module 304 is a global synchronization barrier controller (GBC) used to coordinate the work progress of each cluster and ensure information synchronization. Multiple clusters 305 are the computing cores of the computing device 201. Four are shown as examples in the figure. With the development of hardware, the computing device 201 disclosed in the present invention can also include 8, 16, 64, or even more clusters 305. Clusters 305 are used to efficiently execute deep learning algorithms.

[0052] At the cluster level, Figure 3 As shown, each cluster 305 includes a plurality of processor cores (IPU cores) 306 and a memory core (MEM core) 307 .

[0053] The figure shows four processor cores 306 as an example, but the present disclosure does not limit the number of processor cores 306. Figure 4Each processor core 306 includes three modules: a control module 41 , a computing module 42 and a storage module 43 .

[0054] The control module 41 is used to coordinate and control the operation of the computing module 42 and the storage module 43 to complete the deep learning task, and includes an instruction fetch unit (IFU) 411 and an instruction decode unit (IDU) 412. The instruction fetch unit 411 is used to obtain instructions from the processing device 203, and the instruction decode unit 412 decodes the obtained instructions and sends the decoding results to the computing module 42 and the storage module 43 as control information.

[0055] The operation module 42 includes a vector operation unit 421 and a matrix operation unit 422. The vector operation unit 421 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformation; the matrix operation unit 422 is responsible for the core calculation of the deep learning algorithm, such as matrix multiplication and convolution.

[0056] The storage module 43 is used to store or move related data, including a neuron RAM (NRAM) 431, a weight RAM (WRAM) 432, an input / output direct memory access module (IODMA) 433, and a move direct memory access module (MVDMA) 434. NRAM 431 is used to store feature maps and intermediate results calculated by the processor core 306; WRAM 432 is used to store the weights of the deep learning network; IODMA 433 controls the memory access between NRAM 431 / WRAM 432 and DRAM 204 through the broadcast bus 309; MVDMA 434 is used to control the memory access between NRAM 431 / WRAM 432 and SRAM 308.

[0057] Back to Figure 3 The storage core 307 is mainly used for storage and communication, that is, to store shared data or intermediate results between the processor cores 306, and to perform communication between the cluster 305 and the DRAM 204, communication between the clusters 305, and communication between the processor cores 306. In other embodiments, the storage core 307 has the capability of scalar operation and is used to perform scalar operation.

[0058] The storage core 307 includes a shared memory unit (SRAM) 308, a broadcast bus 309, a cluster direct memory access module (cluster direct memory access, CDMA) 310 and a global direct memory access module (global direct memory access, GDMA) 311. The SRAM 308 plays the role of a high-performance data transfer station. The data reused between different processor cores 306 in the same cluster 305 does not need to be obtained from the DRAM 204 by each processor core 306, but is transferred between the processor cores 306 via the SRAM 308. The storage core 307 only needs to quickly distribute the reused data from the SRAM 308 to multiple processor cores 306, so as to improve the efficiency of inter-core communication and greatly reduce on-chip and off-chip input / output access.

[0059] The broadcast bus 309, CDMA 310 and GDMA 311 are respectively used to perform communication between processor cores 306, communication between clusters 305 and data transmission between clusters 305 and DRAM 204. They will be described below.

[0060] The broadcast bus 309 is used to complete high-speed communication between the processor cores 306 in the cluster 305. The broadcast bus 309 of this embodiment supports inter-core communication modes including unicast, multicast and broadcast. Unicast refers to point-to-point (i.e., single processor core to single processor core) data transmission, multicast is a communication mode of transmitting a copy of data from SRAM 308 to specific processor cores 306, and broadcast is a communication mode of transmitting a copy of data from SRAM 308 to all processor cores 306, which is a special case of multicast.

[0061] The CDMA 310 is used to control the memory access of the SRAM 308 between different clusters 305 in the same computing device 201 . Figure 5 A schematic diagram is shown when a processor core wants to write data to a processor core of another cluster to illustrate the working principle of CDMA310. In this application scenario, the same computing device includes multiple clusters. For the convenience of explanation, only cluster 0 and cluster 1 are shown in the figure. Cluster 0 and cluster 1 each include multiple processor cores. For the convenience of explanation, cluster 0 in the figure only shows processor core 0, and cluster 1 only shows processor core 1. Processor core 0 wants to write data to processor core 1.

[0062] First, processor core 0 sends a unicast write request to write data into the local SRAM 0. CDMA 0 acts as the master and CDMA 1 acts as the slave. The master pushes the write request to the slave, that is, the master sends the write address AW and write data W to transfer the data to SRAM 1 of cluster 1. Then the slave sends a write response B in response. Finally, processor core 1 of cluster 1 sends a unicast read request to read the data from SRAM 1.

[0063] Back to Figure 3 , GDMA 311 cooperates with external memory controller 301 to control the access of SRAM 308 of cluster 305 to DRAM 204, or reads data from DRAM 204 to SRAM 308. As can be seen from the above, the communication between DRAM 204 and NRAM 431 or WRAM 432 can be realized through two channels. The first channel is to directly connect DRAM 204 and NRAM 431 or WRAM 432 through IODMA 433; the second channel is to first transfer data between DRAM 204 and SRAM 308 through GDMA 311, and then transfer data between SRAM 308 and NRAM 431 or WRAM 432 through MVDMA 434. Although it seems that the second channel requires more components to participate and the data flow is longer, in fact, in some embodiments, the bandwidth of the second channel is much greater than that of the first channel, so the communication between DRAM 204 and NRAM 431 or WRAM 432 may be more efficient through the second channel. The embodiments of the present disclosure can select a data transmission channel according to the hardware conditions thereof.

[0064] In other embodiments, the functions of GDMA 311 and IODMA 433 can be integrated into the same component. For the convenience of description, this disclosure regards GDMA 311 and IODMA 433 as different components. For those skilled in the art, as long as the functions they implement and the technical effects they achieve are similar to those of this disclosure, they fall within the protection scope of this disclosure. Furthermore, the functions of GDMA 311, IODMA 433, CDMA 310, and MVDMA 434 can also be implemented by the same component. Similarly, as long as the functions they implement and the technical effects they achieve are similar to those of this disclosure, they fall within the protection scope of this disclosure.

[0065] An important reason why the computing device 201 has strong computing power is the three-level computing hierarchy of system on chip - cluster - processor core, combined with a three-layer memory design such as DRAM-SRAM-NRAM / WRAM, which enables data to be cached and calculated at the appropriate level to form a sufficient pipeline.

[0066] When the computing device 201 performs computing, the process can be divided into the following three phases: loading phase (load): loading data; computing phase (compute): moving data, computing, and moving intermediate results; and storing phase (store): storing results.

[0067] In more detail, in some embodiments, a two-layer three-stage pipeline may be used, such as Figure 6 As shown, the first-level loading stage 601, calculation stage 602, and storage stage 603 occur in the cluster level. The first-level loading stage 601 is that the GDMA 330 loads data from the DRAM 204 to the SRAM 308, the first-level calculation stage 602 is that the cluster 305 calculates the loaded on-chip unit map and generates calculation results, and the first-level storage stage 603 is that the GDMA 330 stores the calculation results from the SRAM 308 back to the DRAM 204.

[0068] Since the cluster 305 includes multiple processor cores 306, the first layer calculation stage 602 actually divides the on-chip unit map into corresponding sub-maps through the storage core 307, and broadcasts them to at least one processor core 306 for calculation. Therefore, the second layer three-level waterline occurs in the processor core 306. In more detail, the second layer loading stage 604 is to load the sub-map from the SRAM 308 to the NRAM 431 by the MVDMA 434, the second layer calculation stage 605 is to move the sub-map and the sub-weight to the operation module 42 for calculation, and then move the intermediate result back to the NRAM 431, and the second layer storage stage 606 is to store the intermediate result from the NRAM 431 back to the SRAM 308 by the MVDMA 434.

[0069] The first-level pipeline means that the first-level loading stage 601, the first-level calculation stage 602 and the first-level storage stage 603 can be performed in parallel. Now, taking the case where the same cluster 305 wants to process the j-th on-chip unit diagram, the j+1-th on-chip unit diagram and the j+2-th on-chip unit diagram as an example, first, the j-th on-chip unit diagram is loaded into the SRAM 308 in the first-level loading stage 601. Then, the j-th on-chip unit diagram is calculated in the first-level calculation stage 602, and the first calculation result is transferred back to the SRAM 308. While the j-th on-chip unit diagram is being calculated, the j+1-th on-chip unit diagram is loaded into the SRAM 308 in the first-level loading stage 607. When the first calculation result is stored back to DRAM 204 in the first level storage-back stage 603, the j+1th on-chip unit map is calculated in the first level calculation stage 608, and the second calculation result is transferred back to SRAM 308, and the j+2th on-chip unit map is loaded into SRAM 308 in the first level loading stage 610. The first level pipeline is rolled in this way.

[0070] In order to cooperate with the operation of the aforementioned pipeline, the SRAM 308 of this embodiment includes two storage spaces: a ping storage unit and a pong storage unit. The data flow is divided into three types according to the ping-pong attribute of the SRAM 308: input / output ping-pong (IOparity), input ping-pong (input parity), and no ping-pong (no parity). Input / output ping-pong can support parallel loading, calculation, and storage. To realize input / output ping-pong, the ping storage unit and the pong storage unit need to be completely equal, and are used for loading and storage respectively. Input ping-pong only supports storage and calculation in parallel, which will increase the time of transportation on the SRAM 308. Compared with input / output ping-pong, the ping storage unit and the pong storage unit do not need to be completely equal, but an additional cache (cache) of the same size as the storage space needs to be allocated. No ping-pong means that loading / storage and calculation are serial, and space does not need to be additionally allocated.

[0071] In order to implement the aforementioned first-level pipeline, the SRAM 308 of some embodiments has ping memory cells and pong memory cells of the same size to achieve the input / output ping-pong effect. Figure 6 For illustration, the storage area involved in the first-level loading stage 601, the first-level calculation stage 602 and the first-level storage-back stage 603 of the j-th on-chip unit diagram is limited to the ping storage unit, while the storage area involved in the first-level loading stage 607, the first-level calculation stage 608 and the first-level storage-back stage 609 of the j+1-th on-chip unit diagram is limited to the pong storage unit, and the storage area involved in the first-level loading stage 610, the first-level calculation stage 611 and the first-level storage-back stage 612 of the j+2-th on-chip unit diagram is again limited to the ping storage unit, and in this way, the ping storage unit and the pong storage unit are used alternately for storage.

[0072] The second-level pipeline means that the second-level loading stage 604, the second-level calculation stage 605, and the second-level storage stage 606 can be processed in parallel. Take the case where the same processor core 306 wants to process the i-th sub-graph, the i+1-th sub-graph, and the i+2-th sub-graph in the j-th on-chip unit graph as an example. First, the i-th sub-graph is broadcast to the NRAM 431 in the second-level loading stage 604. Then, the i-th sub-graph is calculated in the second-level calculation stage 605 to generate the i-th intermediate result, and the i-th intermediate result is transferred back to the NRAM 431. At the same time, the i+1-th sub-graph is broadcast to the NRAM 431 in the second-level loading stage 613. The i-th intermediate result is stored back to the SRAM 308 in the second-level storage-back stage 606, and at the same time, the i+1-th sub-graph is calculated in the second-level calculation stage 614 to generate the i+1-th intermediate result, and the i+1-th intermediate result is moved back to the NRAM 431, and the i+2-th sub-graph is loaded into the NRAM 431 in the second-level loading stage 615.

[0073] Considering that the tasks of each cluster 305 are different and the completion times are naturally inconsistent, the synchronization module 304 of some embodiments may use synchronization barrier instructions to synchronize the task completion times to avoid timing errors.

[0074] Although an exemplary multi-core hierarchical structure is described above, considering that group matrix multiplication type computing tasks may be implemented on various multi-core architectures, the task scheduling scheme needs to consider the compatibility and universality of different multi-core architectures. The inventors note that the goals of task splitting optimization on a multi-core architecture are mainly in two aspects: IO access and processor core utilization.

[0075] On the one hand, since the processor core is subject to the capacity limit of internal storage resources, the huge amount of data calculation may cause a large amount of data interaction between the processor core and the external storage device. The bandwidth of the input / output (I / O) bus between the processor core and the external memory is limited, which is likely to cause a serious I / O bottleneck problem, and then cause data transmission delays and greatly reduce the computing efficiency during parallel computing. Furthermore, not only will the bandwidth limit of the I / O bus become a bottleneck for system performance, but the large amount of I / O memory access between the processor core and the external storage device will also have an adverse impact on computing and power consumption.

[0076] On the other hand, current computer hardware uses tensor core technology to accelerate large-scale matrix multiplication, that is, to implement a relatively large-scale matrix multiplication on a single computing unit. For example, in some hardware architectures, a computing unit can calculate a 16*8*16 (M*K*N) matrix multiplication in one beat. This requires that the M and N dimensions be aligned to 16. However, the matrix dimensions of each expert in MoE's Group GEMM are variable, which results in a loss of computing power due to the alignment of the M and N dimensions. Therefore, it is also necessary to optimize the utilization of processor cores.

[0077] In view of this, the disclosed embodiments abstract a storage (memory) architecture for various multi-core system structures to be compatible with different existing multi-core architectures. On this basis, considering the amount of data interaction between the processor core and the external circuit (such as an external storage device) and / or the utilization rate of the processor core, a task scheduling scheme is provided to efficiently schedule group matrix multiplication type computing tasks to the multi-core system for parallel execution, thereby improving the computing efficiency of parallel computing of the multi-core system.

[0078] Figure 7 A schematic diagram of a storage (memory) architecture of a multi-core system to which the disclosed embodiment can be applied is shown. The storage architecture comprehensively considers a variety of multi-core systems and is compatible with different existing multi-core architectures.

[0079] As shown in the figure, the bottom layer DRAM 710 is an off-chip memory used to store input and output data, such as input data to be processed by the processor core and processing results obtained by the multi-core system. DRAM 710 can be DDR memory, and the DDR memory types are: LPDDR, GDDR, HBM, etc. The processor core can access the off-chip storage based on the bus protocol through the on-chip storage control unit (such as DDR controller). DRAM 710 can be regarded as global storage or global memory at this level of the chip.

[0080] The next layer up in the figure is the L2 cache 720. This layer of cache also belongs to the off-chip memory. The L2 cache 720 may include LLC (Last Level Cache). As mentioned above, LLC can have multiple operation modes, including data latching, streaming access, clearing and writing back to system memory, etc. When the data latching mode of LLC is enabled, the characteristics of LLC latching access data can be fully utilized to maximize the utilization of data resident in LLC and achieve the effect of bandwidth expansion.

[0081] The next layer in the figure is the shared memory 730. The shared memory 730 can share on-chip storage between multiple processor cores in a computing cluster. It is understood that there can be multiple shared memories, each of which serves multiple processor cores respectively, and only one shared memory is shown in the figure as an example.

[0082] The top layer in the figure is the local memory 740 of each processor core 750, and the shared memory 730 is the on-chip storage of the corresponding processor core 750. Depending on the specific hardware architecture, the local memory 740 can have different implementations, such as a data buffer or a register page, and the disclosed embodiment is not limited in this respect.

[0083] although Figure 7 The memory architecture abstracted based on various multi-core architectures is shown, but those skilled in the art should understand that in different hardware implementations, Figure 7 Some storage levels in the system may be omitted. For example, in one implementation, there may be no shared memory 730; in another implementation, there may be no L2 cache 720, or even if LLC is configured on the hardware, its data latching mode is not enabled, which is equivalent to no L2 cache; in another implementation, there may be neither shared memory 730 nor L2 cache 720, or the data latching mode of LLC is not enabled. In the task scheduling scheme of the disclosed embodiment, based on Figure 7 The storage architecture takes into account the above-mentioned possible implementations and provides an optimized task splitting solution to support efficient parallel execution of group matrix multiplication tasks in a multi-core architecture.

[0084] In different multi-core systems or different working modes of the same multi-core system, multiple processor cores may be scheduled to perform parallel operations in different units or granularities. For example, in NVIDIA's multi-core system, one SM is generally used as the basic unit, while in some Cambrian multi-core systems, one computing cluster is generally used as the basic unit. In the disclosed embodiment, the scheduling granularity of the processor core is referred to as an "operation unit", and each operation unit includes one or more processor cores. Depending on the different multi-core systems, the number of processor cores contained in an operation unit is also different. For example, in some implementations, a computing cluster can be used as an operation unit, which contains 4 processor cores; in other implementations, four computing clusters can be used as an operation unit, which contains 16 processor cores; in some other implementations, one processor core can be used as an operation unit.

[0085] It should be noted that the "group matrix multiplication type operation tasks" mentioned in this article include not only direct group matrix multiplication operation tasks, but also other operation tasks that can be converted into group matrix multiplication operations, such as multiple convolution operations, multiple Einstein sums, etc.

[0086] A single matrix multiplication operation can usually be expressed as: C = op(A)*op(B) (1)

[0087] Where op indicates whether to transpose the matrix. The left multiplication matrix op(A) and the right multiplication matrix op(B) are matrices of size M×K and K×N respectively, and the output matrix C is a matrix of size M×N. In most cases, matrices A and B do not need to be transposed, so op(A)=A, op(B)=B. For simplicity, matrices A and B are directly mentioned below, omitting op. When there are multiple sets of matrix multiplication operations, the matrix size of each set of matrix multiplication is variable. The input and output data of the convolution operation usually involve dimensions such as batch (Batch), height (H), width (W), input channel (Ci), output channel (Co), convolution kernel height (Kh) and width (Kw), which can be converted into matrix multiplication operations by merging dimensions. For example: for the convolution operation, its M, N and K dimensions can be M=Batch×H×W, N=Co, K=Ci×Kh×Kw respectively. Therefore, the solution described in the embodiment of the present disclosure based on the matrix multiplication operation task can also be similarly applied to these operation tasks that can be converted into matrix multiplication operations.

[0088] In intelligent computing systems, high-performance multi-core systems are usually used as server accelerator cards, which exchange data with the host CPU via the PCIe bus, forming a host-device working mode. Figure 2 In the combined processing device 20 shown, the computing device 201 is a high-performance multi-core system, which serves as the device side. The processing device 203 can serve as the host side. The scheme of the disclosed embodiment includes two aspects: on the one hand, it is a task scheduling scheme on the host side, and on the other hand, it is a task execution scheme on the device side. The host side determines the optimization parameters of the matrix multiplication splitting scheme and clarifies the matrix multiplication splitting strategy based on the optimization goal of the task splitting scheduling. Afterwards, the host side passes these optimization parameters from the host side to the device side by launching the kernel, that is, a high-performance multi-core system. Each computing unit in the multi-core system serving as the device side determines and executes the Group GEMM tasks that need to be processed by itself based on these optimization parameters.

[0089] Group matrix multiplication task scheduling scheme

[0090] In an embodiment of the present disclosure, a task scheduling scheme based on a multi-core system is provided to distribute group matrix multiplication type operation tasks to be executed on multiple processor cores of the multi-core system.

[0091] The biggest problem with existing group matrix multiplication splitting is that the M, K, and N dimensions of a single group matrix multiplication are not large enough to fully utilize the computing power of a multi-core system, so its splitting is a very big challenge. If M and N are large enough, the existing splitting method of GEMM can be reused, for example, refer to the chessboard splitting solution provided in Chinese patent application publication CN118733206A. However, when M and N are moderate and it is a computing bottleneck, using the thinking of a single matrix multiplication to consider the problem of group matrix multiplication cannot fully utilize the performance of group matrix multiplication.

[0092] Therefore, the disclosed embodiment specifically designs a splitting optimization method for group matrix multiplication based on the characteristics of group matrix multiplication, that is, splitting is performed from the perspective of one operation unit. In determining the size of the split block, the different situations of IO bottleneck and computing bottleneck are distinguished to obtain the optimal splitting solution. The split block here refers to the size of the data block that needs to be calculated in each round of each operation unit, which includes three dimensions M, N, and K, and the sizes of each dimension are block_m, block_n, and block_k respectively.

[0093] Figure 8 An exemplary flow chart of a task scheduling method 800 based on a multi-core system according to an embodiment of the present disclosure is shown. The tasks to be scheduled include group matrix multiplication type operation tasks, and the group matrix multiplication type operation tasks include multiple groups of matrix multiplications, and the matrix size of each group of matrix multiplications is variable. The multi-core system includes multiple operation units, each operation unit includes one or more processor cores, and each processor core has a local memory (e.g., reference Figure 7 storage architecture).

[0094] like Figure 8 As shown, in step 810, the layout information of the group matrix multiplication type operation tasks is obtained.

[0095] Before splitting the task, you need to have some understanding of the computational scale of the task. The layout information may include the matrix size of each group of matrix multiplications, such as the size of the M, K, and N dimensions, and whether transposition is required. Considering that the matrix size of each group of matrix multiplications in group matrix multiplication type operation tasks is variable, the splitting scheme is designed based on the maximum size of the dimension in each group of matrix multiplications in the disclosed embodiment. In some embodiments, obtaining the above-mentioned layout information includes: obtaining the maximum values ​​m_max, k_max, and n_max of the M, K, and N dimensions in the multiple groups of matrix multiplications. Depending on the location of the matrix multiplication data, there may be different ways to obtain the maximum values ​​of the above-mentioned dimensions.

[0096] In some embodiments, in response to the dimensional data being on the host side, the maximum value of each dimension can be directly searched from the dimensional data. It can be understood that for Group GEMM, since the M, K, and N of each group of matrix multiplications are variable, the dimensional data provided to the host side is an M, K, and N array (e.g., m_array[], k_array[], n_array[]), and the maximum value needs to be searched from each array.

[0097] In other embodiments, in response to the dimensional data on the device side, the maximum value of each dimension can be determined based on prior knowledge. For example, in the reasoning scenario of a general MoE network, M is variable, and K and N are immutable, so the maximum values ​​of the K and N dimensions are themselves, and the maximum value of the M dimension m_max can be taken as m_max=batch*sequence, where batch is the number of batches in the reasoning scenario, and sequence is the sequence length. For another example, in the training scenario of a general MoE network, the scale of the forward data and the reverse data is the same as that of the reasoning scenario, and can be set similarly; and in the backward filter (backward filter, also known as the reverse convolution kernel), K is variable, and M and N are immutable, so the maximum value of K can be set to the maximum value of the M dimension m_array[] in the reasoning stage.

[0098] In some other embodiments, in response to the dimensional data being on the device side, it is necessary to obtain the dimensional data from the device side and search for the maximum value of each dimension therefrom. This approach is general. For example, when M, K, and N are all variable on the device side, the memcpy (memory copy) instruction can be used to copy from the device side to the host side, and then the maximum value can be retrieved. Of course, based on the previous implementation, the maximum estimate can be given directly using the information from the device malloc (device memory allocation) stage or other prior information. These estimates do not need to be very precise, as long as they can help define whether the task is a computing bottleneck type or an IO bottleneck type, as described below.

[0099] Next, in step 820, based on the acquired layout information, the split size block_k of the split block in the K dimension and the split strategy within a single operation unit are determined, where the K dimension is the column dimension of the left multiplication matrix and the row dimension of the right multiplication matrix.

[0100] Since the K dimension involves both the column dimension of the left multiplication matrix and the row dimension of the right multiplication matrix, pre-defining the split size block_k of the K dimension is more conducive to reducing the search amount in the subsequent search for the optimized split sizes block_m and block_n of the M and N dimensions, thereby reducing the overhead on the host side.

[0101] In some embodiments, determining the split size block_k of the split block in the K dimension may include: in response to the group matrix multiplication type operation task being a computing bottleneck type task, setting block_k to meet the efficiency requirements of the instructions for implementing the matrix multiplication operation supported by the multi-core system; or in response to the group matrix multiplication type operation task being a memory access bottleneck type task, setting block_k to prioritize improving the memory access efficiency of the largest matrix; wherein the largest matrix is ​​determined based on the size of the left multiplication matrix and the size of the right multiplication matrix. In these embodiments, by distinguishing whether the task is a computing bottleneck or a memory access bottleneck type, the requirements of different types of tasks can be considered more carefully to better match the optimization goals.

[0102] Generally, according to the storage dimension where the K dimension is located, block_k can be set as follows: when the storage dimension where the K dimension is located is the lowest dimension, block_k is set to a multiple of the cache line size; where the multiple is a natural number; or when the storage dimension where the K dimension is located is not the lowest dimension, block_k is set to the alignment granularity align_k number of the instruction that implements the matrix multiplication operation; where the multiple is a natural number.

[0103] Specifically, for computing bottleneck tasks, it is necessary to improve the computing efficiency as much as possible, that is, to meet the efficiency requirements of the instructions for implementing matrix multiplication operations supported by the multi-core system as much as possible. The instruction for implementing the matrix multiplication operation can be a single matrix multiplication instruction or an instruction composed of multiple matrix multiplication and accumulation (MMA) instructions. The disclosed embodiment has no restrictions in this regard. When the storage dimension where the K dimension is located is the lowest dimension, block_k is set to a cache line size according to the transposition information. For example, for a GPU, in the scenario where the transposition information is transB=true or transA=false, the size of a memory access sector (sector) can be selected, such as 64B. This setting method can not only ensure the basic memory access function and efficiency, but also reserve space for the splitting of the M and N dimensions to improve the computing efficiency. When the storage dimension where the K dimension is located is not the lowest dimension, block_k is set to a multiple of the alignment granularity align_k of the instruction for implementing the matrix multiplication operation according to the transposition information. For example, in the scenario where the transposition information is transB=false and transA=true, the multiple of the alignment granularity align_k is 2 times, 3 times, etc. This setting can not only meet the minimum alignment granularity requirement of matrix multiplication operation instructions, but also, in order to meet the efficiency of on-chip matrix multiplication operation instructions and improve the depth of K-dimensional loops, you can select several times the size of align_k to fully utilize the computing power of the operation unit and improve the computing efficiency of the matrix multiplication operation instructions.

[0104] For tasks with memory access bottlenecks, the memory access performance of the largest matrix needs to be guaranteed to avoid the inability to read the largest matrix in time to participate in the operation due to the memory access bottleneck. Here, the largest matrix is ​​determined based on the size of the left multiplication matrix and the size of the right multiplication matrix, that is, it is determined based on the layout information obtained previously, specifically, it is determined based on the maximum value of N and M. When the storage dimension where the K dimension is located is the lowest dimension, the size of block_k can be appropriately enlarged. For example, in the transformer decoder scenario, when the transposition information is transB=true, block_k is set to a multiple of the cache line, such as 2 times, 3 times, etc. In the multi-core system described above, 2 times the cache line size is taken, that is, 1024B. When the storage dimension where the K dimension is located is not the lowest dimension, block_k can be set smaller to ensure the minimum alignment granularity requirement of the matrix multiplication operation instruction, for example, block_k is set to the alignment granularity align_k of the instruction that implements the matrix multiplication operation.

[0105] In some embodiments, based on the layout information, the task type of the group matrix multiplication type operation task is determined. In further detail, the computing power limit is determined based on the layout information of the group matrix multiplication type operation task, the data bit width of the left multiplication matrix, and the data bit width of the right multiplication matrix; the computational memory access ratio of the multi-core system is determined based on the bandwidth of the off-chip memory DRAM of the multi-core system and the number of floating-point operations per second of the multi-core system to one trillion times; the computational memory access ratio of the multi-core system is compared with the computing power limit, and the task type of the group matrix multiplication type operation task is determined based on the comparison result. Specifically, the following conditions can be used to determine whether the task is a computational bottleneck type or a memory access bottleneck type. When the following conditions are met, the group matrix multiplication type operation task is determined to be a computational bottleneck type task, otherwise it is a memory access bottleneck type task:

[0106]

[0107] in, CGMA DEV The compute-to-memory ratio of the multi-core system itself is the ratio of the number of compute operations to the number of memory operations, which can reflect the efficiency of compute operations and memory operations. TFLOPS is the number of floating-point operations per second; BandWidth DRAM For off-chip memory devices DRAM (e.g. Figure 2204), in GB / s. group is the number of matrix multiplication groups in the group matrix multiplication task, sizeof() represents the data bit width, A is the left multiplication matrix, B is the right multiplication matrix, M, K, and N are the maximum values ​​of the M, K, and N dimensions obtained in the previous steps, respectively. Unless otherwise specified, for the sake of brevity, M, K, and N used in the task scheduling scheme of the disclosed embodiment refer to the maximum values ​​of their respective dimensions.

[0108] As described above, the K-dimensional split size block_k is determined for computing bottlenecks and memory access bottlenecks. The split strategy within a single computing unit can also be determined in step 820. As mentioned above, depending on different multi-core systems, the number of processor cores included in a computing unit in different multi-core systems is also different.

[0109] In some embodiments, determining the splitting strategy within a single computing unit may include: determining the number of processor cores Ncore within a single computing unit; splitting the split block into pm*pn parts, where pm*pn=Ncore, pm is the number of splits in the M dimension, and pn is the number of splits in the N dimension.

[0110] Specifically, any of the following splitting methods may be adopted: evenly distributing pm and pn in the M dimension and the N dimension; or in response to any one of the M dimension and the N dimension in the split block being smaller than a corresponding threshold, splitting the other dimension.

[0111] For example, for an architecture with 4 processor cores inside a computing unit, a 2×2 split is generally adopted inside a split block, that is, it is split into two parts along block_m and block_n respectively, and distributed to the 4 processor cores. However, for some special cases, such as when one dimension is small and the other dimension is large, the larger dimension can be split and the smaller dimension can be kept intact. For example, when block_n is very small, less than 2 times align_n (align_n is the alignment granularity of the instruction to implement the matrix multiplication operation in the N dimension), and block_m is very large, only block_m can be split into 4 parts, and the complete block_n can be kept. For another example, when block_m is very small, less than align_m (align_m is the alignment granularity of the instruction to implement the matrix multiplication operation in the M dimension) or 2*align_m, and block_n is very large, only block_n can be split into 4 parts, and the complete block_m can be kept.

[0112] Next, in step 830, based on the block_k determined above, the split size block_n in the N dimension and the split size block_m in the M dimension are determined to meet the optimization goal; wherein M is the row dimension of the left multiplication matrix, and N is the column dimension of the right multiplication matrix, and the above optimization goal includes any of the following: memory access, processor core utilization.

[0113] In some embodiments, determining the split size block_n of the split block in N dimensions and the split size block_m of the split block in M ​​dimensions may include: searching for block_n that meets the above optimization goal and determining the corresponding block_m in the search space of N dimensions; or searching for block_m that meets the above optimization goal and determining the corresponding block_n in the search space of M dimensions. As mentioned above, the search may be conducted in N dimensions or in M ​​dimensions, and the disclosed embodiments are not limited thereto.

[0114] In some embodiments, the maximum search space is constructed as follows: according to the split size block_k of the split block in the K dimension, the capacity of the local memory of the processor core in the multi-core system, and the splitting strategy within a single computing unit, the maximum search space of the N dimension or M dimension is determined. By establishing the maximum search space of the N dimension or M dimension according to the above parameters, it is possible to avoid performing a full domain search on the N dimension or M dimension, thereby reducing the amount of search.

[0115] It can also be seen from the construction of the search space above that when searching for the split size of the block, a brute force search in the three dimensions of M, K, and N is not used. Instead, a cyclic branch reduction method is used to reduce the search volume, thereby reducing the host-side overhead. This optimized search scheme can significantly shorten the search time. For example, in the case of M=K=N=8192, a three-dimensional brute force search scheme is used, and the host-side search time can reach about 3ms; while in the same case, the cyclic pruning method of the disclosed embodiment is used, and the host-side search time only takes 20us to 30us to complete the search, and the performance of matrix multiplication can also be guaranteed.

[0116] The following is a detailed description using the search in N dimensions as an example. Those skilled in the art will appreciate that when a search in M ​​dimensions is adopted, the specific search process can be implemented similarly and will not be described in detail here.

[0117] First, based on block_k, the capacity of the local memory, and the splitting strategy (pm*pn) within a single computing unit, determine the maximum search space in N dimensions. In other words, based on these known information, determine the maximum block_n value that can be accommodated on the chip. Figure 7The memory architecture of the multi-core system shown in FIG. 1 may include slightly different storage hierarchies in different hardware implementations of multi-core systems. For example, in one implementation, the on-chip storage hierarchy may only include local memory, in which case only the capacity of the local memory needs to be considered. In another implementation, the on-chip storage hierarchy includes local memory and shared memory, so the capacity of the two needs to be considered, and the smaller value is taken. Taking the latter implementation as an example, the maximum search space search_max_n in N dimensions can be set as follows:

[0118] search_max_n=min(max_n_l1*pn,max_n_smem)(3)

[0119] In the above formula (3), max_n_l1*pn represents the maximum size of block_n that can be accommodated due to the limitation of L1_cache storage space; max_n_smem represents the maximum size of block_n that can be accommodated due to the limitation of shared memory SMEM storage space. Then the smaller value of the two is taken as the maximum search space search_max_n of N dimension. l1_cache indicates the capacity of the local memory, and SMEM indicates the capacity of the shared memory. Figure 4 As for the internal structure of the processor core shown, it can be regarded as a collection of NRAM 431 and WRAM 432, for example. align_m represents the alignment granularity of the instruction implementing the matrix multiplication operation in the M dimension, pk represents the number of blocks_k split when the software pipeline is implemented during the data transfer process between the local memory l1_cache and the shared memory SMEM; sizeof() represents the data bit width, A is the left multiplication matrix, B is the right multiplication matrix, and C is the output matrix. align_m*(block_k / pk)*sizeof(A)*2 represents the storage space occupied by the ping-pong buffer of the left multiplication matrix A; (align_m*sizeof(C))*2 represents the storage space required for the ping-pong buffer of the output matrix C; (block_k / pk)*sizeof(B)*2 represents the storage space required for the ping-pong buffer of the right multiplication matrix B. “pm*align_m*block_k*sizeof(A)*2” indicates the size of the storage space occupied by the left multiplication matrix A on the shared memory SMEM; “block_k*sizeof(B)*2” indicates the size of the storage space occupied by the right multiplication matrix B on the shared memory SMEM.

[0120] Based on the maximum search space, the search steps of the above-mentioned loop pruning method include: determining the corresponding block_m or block_n according to the capacity of the local memory, block_k, and block_n or block_m in the current search loop; calculating the corresponding optimization target according to the block_m or block_n corresponding to the current search loop; and determining the block_n and block_m corresponding to the optimal optimization target according to the optimization target corresponding to each search loop in the search space.

[0121] The following is a detailed description using the search in N dimensions as an example. Those skilled in the art will appreciate that when a search in M ​​dimensions is adopted, the specific search process can be implemented similarly and will not be described in detail here.

[0122] In this embodiment, the search step can be performed in the above-mentioned N-dimensional maximum search space in the following loop, including: determining the corresponding block_m according to the capacity of the local memory, block_k, and block_n in the current search loop; calculating the corresponding optimization target according to the block_m corresponding to the current search loop; and determining the block_n and block_m corresponding to the optimal optimization target according to the optimization target corresponding to each search loop in the search space. The optimization target may be different according to different task types.

[0123] When the group matrix multiplication task is a memory access bottleneck task, the optimization goal is the memory access amount, that is, the smaller the memory access amount, the better. In some embodiments, the memory access amount io can be calculated as follows:

[0124]

[0125] Among them, M, N, K are the maximum values ​​of M, N, K dimensions respectively, and sizeof() indicates the data bit width.

[0126] When the group matrix multiplication task is a computing bottleneck task, the optimization target is the processor core utilization, that is, the higher the processor core utilization, the better. In some embodiments, the processor core utilization DEV utils It can be calculated as follows:

[0127]

[0128] in, CGMA DEV It is the computing memory access ratio of the multi-core system itself. TFLOPS is one trillion floating-point operations per second. BandWidth DRAM is the bandwidth of the off-chip memory DRAM, in GB / s; group is the number of groups of matrix multiplication in the group matrix multiplication task, num cluster is the number of operation units, total block Indicates the total number of split blocks, PAD_UP (total block ,num cluster ) is equal to total block Divide by num cluster The result is rounded up and multiplied by num cluster In the above formula (5), the first term in the multiplication indicates whether the computing memory access performance corresponding to the split blocks can meet the requirement that the memory access of the multi-core system is masked by the computing; the second term in the multiplication indicates the utilization rate of the processor core of the multi-core system.

[0129] The above loop search process can be expressed in pseudo code as follows:

[0130]

[0131] It can be seen from the above pseudo code that in the maximum search space of N dimensions, in each search process, the method of determining the corresponding temp_block_m based on temp_block_n in the current search loop is similar to the method of determining the maximum search space previously described. That is, the size of the largest block_m that can be accommodated is determined based on the limitation of the local memory l1_cache storage space and the size of the largest block_m that can be accommodated is determined based on the limitation of the shared memory SMEM storage space, and then the smaller value of the two is taken as the temp_block_m of the current search loop. Afterwards, according to whether the task type is a computing bottleneck or a memory access bottleneck, the corresponding optimization targets are calculated using formulas (4) and (5) respectively, and the combination of block_n and block_m that optimizes the optimization target is selected.

[0132] Thus, the split sizes block_k, block_n, and block_m that optimize the optimization target can be determined.

[0133] continue Figure 8 In step 840, according to the determined split sizes block_k, block_n, and block_m, the group matrix multiplication operation tasks are scheduled to be executed in parallel on the operation units in the multi-core system. Specifically, it can be instructed to split each group of matrix multiplications in the group matrix multiplication operation tasks into multiple split blocks according to the determined split sizes block_k, block_n, and block_m, and then distribute the corresponding split blocks cyclically to each operation unit to perform operations according to the grid order of the M and N dimensions of the output matrix block.

[0134] It can be seen from the task scheduling scheme on the host side described above that the scheme splits the group matrix multiplication task from the perspective of an operation unit, which can be more in line with the implementation of the device side and facilitate the task distribution on the device side. When searching for the split size, by predetermining block_k, the search volume of the subsequent optimization parameter search can be greatly reduced, the overhead of the host side can be reduced, and the processing efficiency can be improved. Further, when predetermining block_k, the task types of memory access bottlenecks and computing bottlenecks are distinguished, so that the optimization target can be better matched. Further, in the design of the optimization target, different optimization targets are considered for different task types. For example, for tasks of the computing bottleneck type, the processor core utilization is taken into consideration, so as to make up for the limitations of splitting from the perspective of a single operation unit. In short, the disclosed embodiment provides an optimized task scheduling scheme for the implementation of group matrix multiplication type operation tasks on a multi-core system, which can effectively reduce frequent data exchange and loading, avoid input / output (I / O) bottleneck problems, and make full use of the parallel operation characteristics of the multi-core architecture to improve the processor core utilization.

[0135] Accordingly, the present disclosure also provides a task scheduling device, including a processor and a memory, wherein: the processor is configured to execute program instructions; the memory is configured to store the program instructions; when the program instructions are loaded and executed by the processor, the processor executes the task scheduling method described in any of the foregoing embodiments. The present disclosure also provides a computer-readable storage medium, in which program instructions are stored, and when the program instructions are loaded and executed by the processor, the processor executes the task scheduling method described in any of the foregoing embodiments. The present disclosure also provides a computer program product, including a computer program or instructions, which implements the task scheduling method described in any of the foregoing embodiments when executed by the processor. The present disclosure also provides a processing device, including the task scheduling device described above. The present disclosure also provides a chip, including the processing device described above. The present disclosure also provides a board, including the chip described above.

[0136] Group matrix multiplication task execution scheme

[0137] As mentioned above, in intelligent computing systems, high-performance multi-core systems are usually used as accelerator cards for servers, which exchange data with the host host through the PCIe bus. The group matrix multiplication task scheduling implemented on the host host side of the disclosed embodiment is described above, that is, the splitting process of the group matrix multiplication task. The host side determines the optimization parameters of the splitting scheme according to the optimization goal of the task splitting scheduling, including the splitting size block_k, block_n, block_m and the splitting strategy (pm, pn) within a single computing unit. Afterwards, the host side passes these optimization parameters from the host side to the high-performance multi-core system on the device side by launching the kernel (launch kernel). Each computing unit in the multi-core system determines and executes the Group GEMM tasks that need to be processed by itself according to these optimization parameters.

[0138] However, in some scenarios, such as when the group matrix multiplication task itself is a computing bottleneck type, if the matrix multiplication is executed one by one in a loop on the entire multi-core system, the efficiency is very low. For example, if the group is 6, M and N are relatively small, such as 512, and K is relatively large, if each group (that is, a matrix multiplication) is distributed to 12 computing units for processing, and then the 6 groups are run in a loop, the efficiency will be very low. For example, the computing efficiency on a certain multi-core system is only a little more than 60%, but in fact this scenario is a computing bottleneck scenario on this multi-core system. If the entire group matrix multiplication task is considered together, M is split into two parts, N is split into one part, 6 groups have a total of 12 block outputs, and then distributed together to 12 computing units, the computing efficiency can reach more than 90%. Moreover, in the group matrix multiplication task, M, K and N of each group are dynamically variable, and it is difficult to effectively implement the group matrix multiplication task using the previous general matrix multiplication solution. Therefore, in the execution of group matrix multiplication tasks on the device side, further adjustments need to be made to improve processing efficiency.

[0139] Fig. 9 FIG. 2 shows an abstract schematic diagram of a grid block for a matrix multiplication task according to some embodiments of the present disclosure. Fig. 9As shown, a matrix multiplication task can be abstracted as a three-dimensional cuboid, where the horizontal coordinate X represents the dimension N (column dimension of the right-hand matrix), the vertical coordinate Y represents the dimension M (row dimension of the left-hand matrix), and the depth coordinate Z represents the dimension K (column dimension of the left-hand matrix and row dimension of the right-hand matrix). The three-dimensional cuboid is divided into grids, each grid represents a task corresponding to a split block, and its sizes on the X, Y, and Z coordinates are block_n, block_m, and block_k, respectively. Each grid can be identified by the index on the X, Y, and Z coordinates. For example, gridId.x represents the block grid index of N dimensions, gridId.y represents the block grid index of M dimensions, and gridId.z represents the block grid index of K dimensions. It can be understood that since the matrix multiplication operation is accumulated in the K dimension, the size of the output matrix is ​​M×N, that is, corresponding to the XY plane. At this time, the output matrix can be divided into multiple output matrix blocks according to the grid on the XY plane, and each output matrix block corresponds to a task block.

[0140] Since the size of M, K, and N of each group in the group matrix multiplication task is variable, these task blocks can be uniformly numbered so that the operation task can be effectively executed from the perspective of the entire group matrix multiplication task. For example, a counter of excuted_blocks can be maintained to indicate that the current loop has processed the excuted_blocks task block grid, and the task block grid is distributed sequentially to each operation unit of the multi-core system.

[0141] like Fig.18 As shown, it is a flow chart of a method for executing a computing task on a multi-core system according to an embodiment of the present disclosure. The task includes a group matrix multiplication computing task, the group matrix multiplication computing task includes multiple groups of matrix multiplications, the matrix size of each group of matrix multiplications is variable, the multi-core system includes multiple computing units, each computing unit includes one or more processor cores, each processor core has a local memory, and the method includes:

[0142] Step 1801): according to predetermined task splitting information, split each group of matrix multiplications in the group matrix multiplication type operation task into a plurality of task blocks; and

[0143] Step 1802): The task blocks of each group of matrix multiplication are managed uniformly, and each operation unit executes the task block in a loop according to the grid order of the M dimension and N dimension of the output matrix block corresponding to the task block to generate the corresponding output matrix block, where M is the row dimension of the output matrix and N is the column dimension of the output matrix.

[0144] In some embodiments, a method for executing a group matrix multiplication type operation task on a multi-core system may include: splitting each group of matrix multiplications in the group matrix multiplication type operation task into multiple task blocks according to predetermined task splitting information; and managing the task blocks of each group of matrix multiplications in a unified manner, and each operation unit executes the task block cyclically according to the network order of the M and N dimensions of the output matrix block corresponding to the task block to generate the corresponding output matrix block, wherein M is the row dimension of the output matrix, and N is the column dimension of the output matrix. The above-mentioned predetermined task splitting information may be determined and transmitted by the host side according to the aforementioned group matrix multiplication task scheduling scheme, and of course it may also be determined in other ways, and the disclosed embodiments are not limited in this respect.

[0145] Fig.10 FIG. 2 shows a task block processing method for a group matrix multiplication task according to some embodiments of the present disclosure. Fig.10 As shown, the output matrix block of the group matrix multiplication task of group=2 is shown. The horizontal dimension represents the N dimension and the vertical dimension represents the M dimension. Each grid represents the output matrix block corresponding to a task block block, and its size is block_n×block_m. The N dimension size of the first group is N1=10×block_n, and the M dimension size is M1=3×block_m; the N dimension size of the second group is N2=9×block_n, and the M dimension size is M2=2×block_m. Assuming that there are 8 computing units involved in the calculation, deviceId represents the number of each computing unit in the hardware, and deviceId=0~7 corresponds to these 8 computing units respectively. The group matrix multiplication task is implemented through multiple rounds of calculation cycles. The same round of cycles is represented by the same shading.

[0146] As can be seen from the figure, in each task cycle, 8 output matrix blocks are sequentially assigned to 8 computing units for parallel processing. In the fourth task cycle, 8 output matrix blocks are assigned to 8 computing units across groups. Since the task blocks of each group of matrix multiplication are managed in a unified manner, cross-group allocation can be achieved. This cross-group allocation method can fully utilize the computing power of the available computing units in each cycle and improve the execution efficiency of the entire group matrix multiplication task.

[0147] Furthermore, since matrix multiplication is accumulated in the K dimension of the input data, in some embodiments, when processing the distributed task blocks, the operation unit may perform iterative processing in a loop in the K dimension until the corresponding output matrix block is obtained.

[0148] Fig.11 FIG. 1 shows a logic flow chart of a task block loop execution process according to some embodiments of the present disclosure. Fig.11As shown, in step 1101, parameters are first initialized. These parameters include, for example, the number of executed task blocks excuted_blocks, the matrix multiplication group identifier group_id currently being processed, the task block identifier block_id currently being processed, and the like.

[0149] Next, in step 1102, it is determined whether there are any unprocessed matrix multiplication groups. For example, it can be determined based on whether the current group number is less than the number of groups. If so, proceed to step 1103, otherwise directly jump to step 1108 and end.

[0150] In step 1103, the total number of task blocks BLKS split by the current matrix multiplication group is calculated, and the upper limit of the current total task blocks is updated to upper_bound=excuted_blocks+BLKS.

[0151] In step 1104, it is determined whether block_id is valid. For example, it can be determined based on whether block_id is less than the upper limit of the total task block upper_bound and greater than or equal to the number of executed task blocks excuted_blocks. If yes, proceed to step 1105, otherwise jump to step 1107, it is necessary to cross to the next matrix multiplication group and update the number of executed task blocks excuted_blocks. It can be seen from the update of block_id in the following step 1106 that the determination of whether block_id is valid here is essentially to determine whether there are enough task blocks in the current matrix multiplication group to be allocated to deviceDim computing units for processing in a single task cycle.

[0152] In step 1105, the corresponding task block in the current matrix multiplication group is located and executed. In the execution of a single task block, the product results are accumulated in a loop in the K dimension to obtain the final output matrix block.

[0153] Then in step 1106, the task cycle parameter block_id=block_id+deviceDim is updated, where deviceDim represents the number of computing units. Here, by increasing block_id by the deviceDim step each time, it can be ensured that there are enough task blocks in each task cycle to be allocated to deviceDim computing units, and task blocks can be processed across groups in a single task cycle.

[0154] Then, return to step 1104 to continue the next task loop processing.

[0155] The above task block loop execution process can be expressed in pseudo code as follows:

[0156]

[0157]

[0158] The above describes the execution process of uniformly considering the task blocks of each group of matrix multiplication in the group matrix multiplication task on a multi-core system and distributing them to the operation units for processing in sequence. It can be seen from the above process that since the task blocks of each group of matrix multiplication are uniformly considered, the problem of low performance of group processing due to the variability of M, N, and K of different groups can be avoided. In the above execution process, the disclosed embodiment also provides many optimization measures to further improve the processing efficiency.

[0159] In some embodiments, based on the above combination Figure 7 The memory architecture of the multi-core system described above can use a pipeline method to implement the processing of task blocks, thereby improving the parallelism and efficiency of task block processing. For example, for a multi-core system with a three-level cache architecture of global memory (first-level storage unit), shared memory (second-level storage unit), and local memory (L1 cache, third-level storage unit), a four-level pipeline can be used to perform task block matrix multiplication processing.

[0160] Fig.12 The schematic diagram of data storage at different storage levels is shown. As shown in the figure, the input matrices A and B that need to perform matrix multiplication operations are stored in the global memory 1210. The global memory can be, for example, Figure 7 The off-chip memory DRAM 710 in the FIG. 12 is used to implement the loading pipeline of the input matrices A and B on the shared memory (SMEM) 1220. In some embodiments, the output matrix C is not cached in the shared memory, but directly resides in the local memory of the operation unit, and then cyclically accumulated in the K dimension as needed. The local memory 1230 of the operation unit also stores the relatively small A matrix and B matrix. The figure also shows the matrix data used and the calculated matrix data at each processor core of the operation unit 1240.

[0161] Data exchange or data communication can be implemented between the global memory (first level storage unit) 1210 and the shared memory (second level storage unit) 1220 through a first direct memory access (DMA) interface. Figure 3 The first DMA interface may be a GDMA interface. The shared memory (second level storage unit) 1220 and the local memory (third level storage unit) 1230 may exchange data or communicate data via the second DMA interface. Figure 4, the second DMA interface may be an MVDMA interface. In some embodiments, a third DMA interface may exist between the local memory (third level storage unit) 1230 and the global memory (first level storage unit) 1210. For example, referring to Figure 4 , the third DMA interface can be an IODMA interface.

[0162] From the above description, it can be seen that the second-level storage unit 1220 and the third-level storage unit 1230 support parallel processing, and data communication between different functional units can be achieved through different DMA interfaces, thereby making full use of parallelism to realize data-level pipeline.

[0163] In some embodiments, the aforementioned four-stage pipeline can be implemented through global memory (off-chip memory, first-level storage unit), shared memory (second-level storage unit), local memory (third-level storage unit) and processor core (specifically the computing circuit therein), thereby forming a four-stage pipeline of data transfer-data transfer-calculation-restore. In this pipeline, off-chip memory, shared memory, and local memory can all be processed in parallel, and the processor core and off-chip memory, shared memory and / or local memory can also be processed in parallel.

[0164] Specifically, in some implementations, the shared memory may be configured with at least two shared memory areas to support data access between one of the shared memory areas and the off-chip memory through a first DMA interface, while data access between another shared memory area and the local memory through a second DMA interface different from the first DMA interface. Each of the configured shared memory areas may be used for time-sharing storage of input data blocks and / or corresponding output data blocks.

[0165] In some implementations, the local memory may be configured with multiple local storage areas to support the processor core performing computational processing on data in another local storage area while data is accessed between one local storage area and the shared memory through the second DMA interface. Each local storage area configured in the local memory is used to store input data blocks and output data blocks as computational processing results in a time-sharing manner.

[0166] Furthermore, in some computational processing that requires caching, the local memory may also be configured with a computational cache area to temporarily store data for computational processing of the processor core, such as caching intermediate results accumulated in dimension K. In some embodiments, the final computational result of the computational cache area may be directly stored in the off-chip memory via the local memory via the third DMA interface, without passing through the shared memory.

[0167] Fig.13The four-stage pipeline processing of the matrix multiplication task according to some embodiments of the present disclosure is more intuitively shown. Fig.13 As shown, the first level 1310 represents the transfer of data from the global memory (global memory, off-chip memory) to the shared memory SMEM, the second level 1320 represents the transfer of data from the shared memory SMEM to the local memory, the third level 1330 represents the processor core using the data in the local memory to perform matrix multiplication operations, and the fourth level 1340 represents the last loop iteration of the K dimension and storing the results back to the off-chip memory.

[0168] In the implementation, each level of the above pipeline can be uniformly encapsulated into an operator Op, for example, the first level is slowMemPromtOp, the second level is fastMemPromtOp, the third level is matmulOp, and the fourth level is tailOp. The pipeline execution process in the K-dimensional loop can be expressed in pseudo code as follows:

[0169]

[0170]

[0171]

[0172] The above pseudo code shows the pipeline in the K-dimensional loop. In the above pipeline, slowMemPromptOp, fastMemPromptOp and matmulOp (or tailOp) are all parallel. syncOp is an intra-core synchronization barrier within an operation unit, using the producer / consumer model.

[0173] Furthermore, it can be seen from the above pseudocode that the operation of loading data from the off-chip memory to the shared memory (slowMemPromptOp()) in the first loop of the K dimension of the current task block (gridId.z=0) and the operation of calculating the last loop of the K dimension of the previous task block and storing the result back to the off-chip memory (tailOp(currentMatmulInfo)) are executed in parallel, thereby making the loop processing of the K dimension connected from beginning to end to achieve seamless flow.

[0174] for Figure 3 For the described architecture, slowMemPromptOp is initiated by the storage core SMEM inside the cluster.

[0175] In some embodiments, when the number of split blocks in the K dimension is 1, a data residency operation may be triggered, thereby reducing data memory access processing. For example, when the number of split blocks in the K dimension is 1, if the data currently required to be loaded already exists in the shared memory, there is no need to load the data again, and the relevant processing components may be notified through a message passing mechanism.

[0176] Specifically, in some implementations, as shown in the pseudo code above, slowMemPromptOp records the information of the previous copied tensor data (copy tensor) previousSlowMemTensors. Before the Op executes the copy, it needs to compare the information of the tensor currently to be copied with the information of the previous copy (note that the A and B matrices here are compared separately). When the two are the same, the copy will not be triggered. At the same time, slowMemPromptOp passes this information to the next Op, fastMemPromptOp, through prepare_fastMemOp. When the Op receives this information, it will not trigger the copy from the shared memory SMEM to the local memory (L1 cache), thus achieving the purpose of data residency.

[0177] How to fully utilize the global cache feature for performance optimization

[0178] As previously combined Figure 7 The memory architecture described may also include the L2 cache 720 in the figure in some multi-core systems. The L2 cache 720 may include LLC (last level cache, also referred to as global cache in this article). LLC can have multiple operating modes, including data latching, streaming access, clearing and writing back to system memory, etc. When the data latching mode of LLC is enabled, the characteristics of LLC latching access data can be fully utilized to maximize the utilization of resident data in LLC and achieve the effect of bandwidth expansion. Therefore, in the case where the multi-core system also includes a global cache, the global cache can be used to cache data so that the computing unit can preferentially load the data that needs to be cached from the global cache.

[0179] Specifically, on a hardware architecture with LLC (global cache), the characteristics of LLC can be fully utilized to rearrange the order of output matrix blocks and improve the data reuse rate in LLC, thereby indirectly improving the computational memory access ratio or reducing the memory access amount.

[0180] like Fig.18The figure is a flow chart of a method for executing a computing task on a multi-core system according to an embodiment of the present disclosure. The task includes a group matrix multiplication computing task, the group matrix multiplication computing task includes multiple groups of matrix multiplications, and the matrix size of each group of matrix multiplications is variable. The multi-core system includes multiple computing units and a global cache, each computing unit includes one or more processor cores, and the global cache is used to cache data, so that the computing unit can preferentially load the data that needs to be cached from the global cache; the method includes:

[0181] Step 1801): according to predetermined task splitting information, each group of matrix multiplications in the group matrix multiplication type operation task is split into a plurality of task blocks;

[0182] Step 1802): the multi-core system adjusts the grid order of the task blocks of the output matrix in the group matrix multiplication type operation task in the corresponding dimension;

[0183] Step 1803): The multi-core system uniformly manages the task blocks of each group of matrix multiplications, and each operation unit performs operations in the grid order of M dimensions and N dimensions according to the adjusted task blocks of the output matrix, so that after the multi-core system calculates the task blocks of the input matrix in each cycle, it obtains output matrix blocks of Sm×Sn grid size, where M is the row dimension of the output matrix, N is the column dimension of the output matrix, Sm is the number of output matrix blocks in the M dimension, and Sn is the number of output matrix blocks in the N dimension.

[0184] Specifically, in the rearrangement scheme of the output matrix blocks of this embodiment, the grid order of the output matrix blocks in the M dimension and the N dimension is adjusted so that in each cycle calculation, the multi-core system obtains output matrix blocks of a grid size of Sm×Sn when performing one calculation, where S m ×S n =num cluster , num cluster The number of computing units in a multi-core system.

[0185] To determine Sm and Sn, num cluster Factoring to get x and y, that is: x*y=num cluster It is necessary to select a combination of x and y that optimizes the memory access ratio or the memory access amount.

[0186] Since an output matrix block may include the accumulation of multiple split blocks in K dimensions, depending on the splitting of K dimensions, the combination of x and y can be determined according to different principles, corresponding to the rearranged Sm and Sn, that is, the data that needs to be cached.

[0187] When the K dimension value of the task block is greater than block_k (that is, there are multiple split blocks in the K dimension), the overall computational memory access ratio of the multi-core system can be used as the optimization target, so that the computational memory access ratio of the corresponding multi-core system after adjusting the output matrix block reaches the maximum, thereby determining the data to be cached based on the maximum computational memory access ratio of the multi-core system. The expression for computing the memory access ratio is: That is, a combination of x and y that maximizes the computational memory access ratio is selected as the values ​​of Sm and Sn, thereby determining the data that needs to be cached.

[0188] When the K dimension value of the task block is less than or equal to block_k (that is, the K dimension is not split and there is only one split block), it is necessary to consider the cases of rearranging the output matrix C along the M dimension and along the N dimension respectively, as well as the corresponding rearrangement sizes in different cases, and take the minimum memory access (IO amount) of the input data (such as A matrix and B matrix) of the group matrix multiplication task as the optimization goal.

[0189] Specifically, after adjusting the grid order of the output matrix block in N dimensions, in num cluster From the combinations of x and y obtained by factorization, determine the minimum IO amount of input data corresponding to the A matrix. The IO amount of input data corresponding to the A matrix can be calculated as follows:

[0190]

[0191] According to formula (11), the combination of x and y that minimizes IOA is selected.

[0192] Similarly, after adjusting the grid order of the output matrix block in the M dimension, in num cluster From the combination of x and y obtained by factorization, determine the minimum IO amount of input data corresponding to the B matrix. The IO amount of input data corresponding to the B matrix can be calculated as follows:

[0193]

[0194] According to formula (12), select the combination of x and y that minimizes IOB.

[0195] Then, the minimum IO amount is selected from the minimum IO amount of the input data corresponding to the A matrix and the minimum IO amount of the input data corresponding to the B matrix, and the data to be cached is determined based on the minimum IO amount. That is, the combination of x and y that minimizes the memory access amount is selected as the value of Sm and Sn, thereby determining the data to be cached. It can be seen that in a multi-core system with LLC, in order to fully utilize the characteristics of LLC, the output matrix C is rearranged, that is, the calculation order of the output block is changed to improve the data reuse rate in LLC. In the above processing, the output IO amount is not considered, because the IO amount stored on the chip is constant and will not increase innocently, so the output IO amount is a public item and does not require additional consideration.

[0196] In the specific implementation, we can first traverse all combinations of x and y to find the smallest IOA and the corresponding x and y, assuming the record is (x1, y1, IO1); then traverse all combinations of x and y to find the smallest IOB and the corresponding x and y, assuming the record is (x2, y2, IO2); then compare the two IO quantities IO1 and IO2, select the smallest IO quantity and the corresponding combination of x and y, and thus determine the rearrangement plan, including the rearrangement direction and the rearrangement pattern Sm×Sn.

[0197] Fig.14 A schematic diagram of rearranging the order of output matrix blocks according to some embodiments of the present disclosure is shown, wherein the output matrix blocks processed in the same round of loops use the same background color or shadow. In these embodiments, for simplicity, only the matrix multiplication of one group is shown, but those skilled in the art will appreciate that the above rearrangement method can be applied to the entire group matrix multiplication task. Assume that this embodiment is a case where the K dimension value is greater than block_k.

[0198] like Fig.14 As shown, the grid diagram above represents the processing order of the output matrix blocks before rearrangement. In this example, it is shown that the M dimension size is M=4×block_m, the N dimension size is N=10×block_n, and there are 8 operation units involved in the calculation. DeviceId=0~7 corresponds to these 8 operation units respectively. In the order of rows first and columns later, in each round of loop, the 8 adjacent output matrix blocks are assigned to 8 operation units for parallel processing. For example, in the first round of loop, 1×8 output matrix blocks will be calculated. It can be seen that in this round of loop, if reuse is implemented based on LLC, only one block of the left multiplication matrix A can be reused in LLC, and there is no reuse of the B matrix. The calculation memory access ratio at this time is

[0199] Fig.14The grid diagram in the middle and lower part represents the processing order of the output matrix blocks after rearrangement. Assume that in this embodiment, according to the previous rearrangement principle, the combination of x and y that maximizes the computational memory access ratio is 2×4. Then after rearrangement, 2×4 output matrix blocks will be calculated in the first cycle. It can be seen that in this cycle, if the reuse is based on LLC, the A and B matrices can be reused with each other, and the 2 blocks of A and the 4 blocks of B can be reused. The computational memory access ratio at this time is Compared with before the rearrangement, the computation-to-memory ratio can be improved. Similarly, there are three loops that will calculate 2×4 output matrix blocks each time, and one loop that will calculate 4×2 output matrix blocks. The A and B matrices can also be reused with each other, and the 4 blocks of A and the 2 blocks of B can be reused.

[0200] Fig.15 A schematic diagram of rearranging the order of output matrix blocks according to other embodiments of the present disclosure is shown, wherein the output matrix blocks processed in the same round of loop use the same background color or shadow. In these embodiments, for simplicity, only the matrix multiplication of one group is shown. Assuming that the K dimension value is less than or equal to block_k in this embodiment, that is, there is only one split block in the K dimension, this can be implemented in combination with the resident technology.

[0201] like Fig.15 As shown in the figure, the grid diagram above represents the output matrix block processing order before rearrangement. Fig.14 Similar, no further description will be given.

[0202] Fig.15 The grid diagram in the lower middle represents the processing order of the output matrix blocks after rearrangement. Assume that in this embodiment, according to the previous rearrangement principle, the rearrangement scheme that minimizes the IO amount is selected as rearrangement along the N dimension, and the rearrangement mode is 4×2. After rearrangement, in the first round of loops, 4×2 output matrix blocks will be calculated. It can be seen that in this round of loops, if reuse is implemented based on LLC, the A and B matrices can be reused with each other, and the 4 blocks of A and the 2 blocks of B can be reused. Combined with the resident technology, the A matrix used by each computing unit only needs to be loaded once, and then it will reside on the chip without reloading. Before the rearrangement, based on the first computing unit deviceId=0, the A matrix was reused only once in the first and second rounds of loops, and other loops would trigger another copy.

[0203] The above rearrangement process can be expressed in pseudo code as follows:

[0204]

[0205]

[0206] In the implementation schemes of various group matrix multiplication tasks described above, they are mainly aimed at scenarios where the M dimension is variable. However, in future training scenarios, scenarios where the K dimension is variable will be encountered, which will result in a very unbalanced load situation.

[0207] Fig.16 The schematic diagram shows the situation of load imbalance. As shown in the figure, assuming that according to the task distribution, the i-th computing unit is assigned to calculate Group0, Group2 and Group4, the K values ​​of these groups are relatively small, while the j-th computing unit is assigned to calculate Group1 and Group3, the K values ​​of these two groups are very large. At this time, the i-th computing unit will be idle for a long time, resulting in very low overall hardware utilization.

[0208] More generally, during the MoE network reasoning process, at the encoder stage, when a Group GEMM task encounters a mixed scenario of computational bottlenecks and memory access bottlenecks, for example, in a scenario where M is variable, some experts are assigned very large tokens, reaching thousands or even tens of thousands of tokens, but some experts are assigned only a few or dozens of tokens. This will also face the above-mentioned problem encountered when K is variable, that is, some computing units complete the computing tasks very quickly, so that they are always idle in the later stage, while other computing units are always performing matrix multiplication calculations at full capacity. These situations can be classified as load imbalance.

[0209] High-performance multi-core systems are usually designed to handle highly parallel computing tasks. In order to simplify the design of the processor, it is generally not necessary to implement complex task scheduling logic. Therefore, in order to solve the above-mentioned uneven load problem, according to conventional design ideas, before assigning tasks to computing units, the impact of various factors will be analyzed in detail, and tasks will be split with this impact in mind to distribute them as evenly as possible. However, for this scenario where the K dimension is variable, the K value varies with different group tasks, so it is necessary to analyze and design task splitting solutions for different groups separately, which increases the workload. Furthermore, if the K value cannot be known in the first place, effective task splitting cannot be performed in advance.

[0210] In view of this, the inventors have jumped out of the conventional design ideas and did not try to build a complex task balancing algorithm to implement the task splitting scheme, which is then distributed to the computing units, and the computing units passively accept the assigned computing tasks and execute them. Instead, the computing units can actively request the next computing task based on their own completion status, that is, to achieve adaptive scheduling. In this way, the parallel computing power of multiple computing units can be fully utilized, so as to complete the group matrix multiplication task at a faster running speed.

[0211] In some embodiments, until the group matrix multiplication operation task is completed, each operation unit in the multi-core system cyclically executes: completing the current task block; and requesting the next task block.

[0212] Specifically, a counter can be maintained in the global memory to record the number of executed task blocks. After each computing unit completes the calculation task of the current task block (for example, the loading of the last K-dimensional block is completed), the computing unit immediately applies for the next task block. In this way, each computing unit can always have a computing task, which can maximize the overall performance of Group GEMM.

[0213] Fig.17 A schematic logic flow chart of adaptively scheduling and executing group matrix multiplication tasks according to some embodiments of the present disclosure is shown.

[0214] like Fig.17 As shown, in step 1700, first, a space can be applied for in the global memory to store the task block counter counter, and the space can be initialized. The task block counter is used to record the number of executed task blocks. In some implementations, the "application" process of the operation unit applying for the task block is implemented using the atomic operation atomicAdd. The function of atomicAdd is to add 1 to counter and return the result before adding counter to the current block identifier block_id, so counter needs to be initialized to 0. For example, memset can be used or a kernel can be started to perform the initialization operation counter = 0.

[0215] Then the kernel of the group matrix multiplication task is started, and the process enters the matrix multiplication task block loop execution process.

[0216] In step 1701, the relevant parameters are initialized. These parameters include, for example, the number of executed task blocks excuted_blocks described above, the matrix multiplication group identifier group_id currently being processed, the task block identifier block_id currently being processed, etc. In addition, in the case of a shared memory, in order to allow all processor cores in an operation unit to share the requested task blocks, a counter smem_task_counter on the shared memory can be set to count the tasks on the shared memory.

[0217] After that, loop each group. For example, in step 1702, determine whether there are unprocessed matrix multiplication groups. If yes, proceed to step 1703, otherwise directly jump to step 1708 and end.

[0218] In step 1703, the information of the group is obtained, including M, K, N and the main dimension size, the starting address of the A, B, and C matrices, and the upper limit of the total task blocks upper_bound, specifically including calculating the total number of task blocks BLKS split by the current matrix multiplication group, and updating the current total task block upper limit to upper_bound = excuted_blocks + BLKS.

[0219] In step 1704, when the currently processed task block identifier block_id is valid (block_id is greater than the executed excuted_block and less than upper_bound), a task loop is entered and the process proceeds to step 1705; otherwise, the process jumps to step 1707 to prepare for processing the next matrix multiplication group and update the number of executed task blocks excuted_blocks.

[0220] In step 1705, at the beginning of the loop, the corresponding task block in the current matrix multiplication group is located and executed. Specifically, the position of the current block to be calculated in the grid space of the group output matrix block is calculated, and then the product results are accumulated in a loop in the K dimension to obtain the final output matrix block, completing the calculation of the task block.

[0221] After a certain operation unit completes the task block calculation task of the current block matrix multiplication, a processor core within the operation unit can use the atomicAdd instruction to apply for the next calculation task. Specifically, in step 1706, apply for the next task block, update the task loop parameter block_id = atomicAdd (& counter, 1), and then return to step 1704 to continue the next task loop processing. The process of updating the task loop parameters may include accessing the counter counter maintained on the global memory, updating the task block counter, specifically adding 1 to its result, thereby updating the number of executed task blocks. However, based on the update result of the task block counter, the information of the next task block is determined. For example, the update result of the task block counter is used as the block_id of the next task block.

[0222] The process of applying for a task block can only be initiated by one processor core in the computing unit, and the application result needs to be written into the shared memory, and the application result needs to be notified to other processor cores in the computing unit.

[0223] In step 1704, when the requested task block block_id is less than upper_bound, it indicates that the current group has not completed the calculation, and the process proceeds to step 1705 to execute the loop. When the requested task block block_id exceeds upper_bound, the process exits the loop and proceeds to step 1707 to update the number of executed task blocks excuted_blocks, and then enters the calculation of the next group.

[0224] A processor core in the computing unit uses the atomicAdd instruction to apply for a task, that is, block_id = atomicAdd(&counter, 1), where block_id is the blockid of the output matrix block corresponding to the applied task block. Then the processor core writes the task to smem_task_counter[] on the shared memory SMEM and inserts a synchronization instruction (depending on different multi-core systems, the synchronization range may be slightly different, for example, in Figure 3 In a multi-core system, all processor cores in the cluster are synchronized), and then each processor core obtains the current task id from smem_task_counter.

[0225] In some embodiments, considering that pipeline can be used to realize the calculation of task block, the timing of applying for task block does not need to wait until the current task block is calculated, but can be applied immediately by the processor core after the last split block is loaded in the K-dimensional loop (that is, in response to the execution of loading data from the off-chip memory in the last loop of the K dimension of the current task block, a request can be initiated immediately), so that the pipeline can be seamlessly connected and continue to flow. The above process of applying for task block can be expressed in pseudo code as follows:

[0226]

[0227]

[0228]

[0229] From the above-described group matrix multiplication task execution scheme of the device side, it can be seen that since the task blocks split from each group are considered uniformly, the task blocks processed in a single round of task cycle can cross groups, so the computing power of the available computing units is fully utilized in each round of cycle, the computing units are avoided to be idle, and the execution efficiency of the entire group matrix multiplication task is improved. Further, in some embodiments, a four-stage pipeline can be used to process the block matrix multiplication task, and the pipeline method can improve the processing parallelism and improve the processing efficiency. In the pipeline process, the head and tail of the K-dimensional loop can be identified, so as to achieve the connection between the head and the tail and seamless flow. In the pipeline process, the matrix resident technology can also be adopted in some scenarios to reduce the amount of memory access. In some embodiments, the size of the split block block can also be adjusted in real time according to the actual dimension size of the group currently being processed to better adapt to the current group and improve the processing performance. In some embodiments, the characteristics of LLC can be fully utilized to rearrange the output matrix blocks, thereby improving the reuse rate in LLC, improving the calculation memory access ratio or reducing the amount of memory access. In some embodiments, for the case of unbalanced load, an adaptive scheduling scheme can be provided to fully utilize the computing power of the computing unit, avoid idleness, and provide utilization.

[0230] Accordingly, the present disclosure also provides a multi-core device, including a processor and a memory, wherein: the processor is configured to execute program instructions; the memory is configured to store the program instructions; when the program instructions are loaded and executed by the processor, the processor executes the method for performing a computing task on a multi-core system as described in any of the foregoing embodiments. The present disclosure also provides a computer-readable storage medium, in which program instructions are stored, and when the program instructions are loaded and executed by the processor, the processor executes the method for performing a computing task on a multi-core system as described in any of the foregoing embodiments. The present disclosure also provides a computer program product, including a computer program or instructions, which, when executed by the processor, implements the method for performing a computing task on a multi-core system as described in any of the foregoing embodiments. The present disclosure also provides a chip, including the multi-core system described above. The present disclosure also provides a board, including the chip described above.

[0231] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will appreciate that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teaching of the present disclosure, those skilled in the art will appreciate that some of the steps therein may be performed in other orders or simultaneously. Further, those skilled in the art will appreciate that the embodiments described in the present disclosure may be regarded as optional embodiments, i.e., the actions or modules involved therein may not necessarily be necessary for the implementation of one or some of the schemes of the present disclosure. In addition, depending on the different schemes, the present disclosure also has different focuses on the description of some embodiments. In view of this, those skilled in the art may appreciate the parts that are not described in detail in a certain embodiment of the present disclosure, and may also refer to the relevant descriptions of other embodiments.

[0232] In terms of specific implementation, based on the disclosure and teaching of this disclosure, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented by other methods not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this article divides them on the basis of considering logical functions, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. In terms of the connection relationship between different units or components, the connection discussed in the above text in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0233] In the present disclosure, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed on multiple network units. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the scheme described in the embodiments of the present disclosure. In addition, in some scenarios, multiple units in the embodiments of the present disclosure may be integrated into one unit or each unit may exist physically separately.

[0234] In some implementation scenarios, the above-mentioned integrated unit can be implemented in the form of a software program module. If implemented in the form of a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable memory. Based on this, when the scheme of the present disclosure is embodied in the form of a software product (such as a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to enable a computer device (such as a personal computer, a server or a network device, etc.) to perform some or all of the steps of the method described in the embodiment of the present disclosure. The aforementioned memory may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0235] In some other implementation scenarios, the above-mentioned integrated unit can also be implemented in the form of hardware, that is, a specific hardware circuit, which may include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as CPU, GPU, FPGA, DSP and ASIC, etc. Further, the aforementioned storage unit or storage device can be any appropriate storage medium (including magnetic storage medium or magneto-optical storage medium, etc.), which can be, for example, a variable resistive memory (Resistive Random Access Memory, RRAM), a dynamic random access memory (Dynamic Random Access Memory, DRAM), a static random access memory (Static Random Access Memory, SRAM), an enhanced dynamic random access memory (Enhanced Dynamic Random Access Memory, EDRAM), a high bandwidth memory (High Bandwidth Memory, HBM), a hybrid memory cube (Hybrid Memory Cube, HMC), ROM and RAM, etc.

[0236] Although multiple embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art may think of many changes, modifications, and alternatives without departing from the thought and spirit of the present disclosure. It should be understood that in the process of practicing the present disclosure, various alternatives to the embodiments of the present disclosure described herein may be adopted. The attached claims are intended to define the scope of protection of the present disclosure, and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A method for executing a computing task on a multi-core system, wherein the task includes a group matrix multiplication computing task, wherein the group matrix multiplication computing task includes multiple groups of matrix multiplications, and the matrix size of each group of matrix multiplications is variable. The multi-core system includes multiple computing units and a global cache, wherein each computing unit includes one or more processor cores, and the global cache is used to cache data, so that the computing unit can preferentially load data that needs to be cached from the global cache; the method includes: According to predetermined task splitting information, each group of matrix multiplications in the group matrix multiplication operation task is split into a plurality of task blocks; The multi-core system adjusts the grid order of the task blocks of the output matrix in the group matrix multiplication type operation task in the corresponding dimension; The multi-core system uniformly manages the task blocks of each group of matrix multiplications, and each operation unit performs operations in the grid order of M dimensions and N dimensions according to the adjusted task blocks of the output matrix, so that after the multi-core system calculates the task blocks of the input matrix in each cycle, it obtains output matrix blocks of Sm×Sn grid size, where M is the row dimension of the output matrix, N is the column dimension of the output matrix, Sm is the number of output matrix blocks in the M dimension, and Sn is the number of output matrix blocks in the N dimension.

2. The method according to claim 1, wherein: The computing unit executes the task block as follows: The process is iterated in a loop in K dimensions to generate the corresponding output matrix block, where K is the column dimension of the left multiplication matrix and the row dimension of the right multiplication matrix.

3. The method according to claim 1, wherein: The step of adjusting the grid order of the task blocks of the output matrix in the group matrix multiplication operation task in the multi-core system in the corresponding dimension further includes: When the K dimension value of the task block is greater than block_k, the grid order of the task block of the output matrix in the group matrix multiplication operation task in the M dimension and N is adjusted so that the calculation memory access ratio of the multi-core system corresponding to the adjustment of the output matrix block reaches the maximum, and the grid order of the task block of the output matrix in the group matrix multiplication operation task in the corresponding dimension of the multi-core system adjustment is determined according to the maximum calculation memory access ratio of the multi-core system; wherein the expression of the calculation memory access ratio is: Where x*y=num cluster , num cluster is the number of the operation units, num cluster Factorize to get x and y; When the K dimension value of the task block is less than or equal to block_k, adjust the grid order of the output matrix block in N dimensions and then cluster In the combination of x and y obtained by factorization, the minimum memory access amount of the input data of the group matrix multiplication task corresponding to the resident left multiplication matrix A is determined; after adjusting the grid order of the output matrix block in the M dimension, in num cluster From the combination of x and y obtained by factorization, determine the minimum memory access amount of the input data of the group matrix multiplication type operation task corresponding to the right multiplication matrix B; select the minimum memory access amount of the multi-core system from the minimum memory access amount corresponding to the A matrix and the minimum memory access amount corresponding to the B matrix; and determine the grid order of the task blocks of the multi-core system adjusting the output matrix in the group matrix multiplication type operation task in the corresponding dimension according to the minimum memory access amount of the multi-core system.

4. The method according to claim 3, wherein: The method further comprises: When the K dimension value of the task block is greater than block_k, determining the data that the global cache needs to cache according to the maximum computation-to-memory ratio of the multi-core system; When the K dimension value of the task block is less than or equal to block_k, the data that needs to be cached in the global cache is determined according to the minimum memory access amount of the multi-core system.

5. The method according to claim 2, wherein: The operation unit includes a shared memory, and the execution of the task block is implemented by a four-stage pipeline, which is implemented through the off-chip memory of the multi-core system, the shared memory, the local memory and the processor core.

6. The method according to claim 5, wherein: The shared memory is configured with at least two shared memory areas, in the four-stage pipeline: While data access is performed between one of the shared memory areas and the off-chip memory through a first DMA interface, data access is performed between another shared memory area and the local memory through a second DMA interface different from the first DMA interface.

7. The method according to claim 6, wherein: The local memory is configured with a plurality of local storage areas, in the four-stage pipeline: While data is accessed between one of the local storage areas and the shared memory through the second DMA interface, the processor core performs a matrix multiplication operation on data in the other local storage area.

8. The method according to claim 7, wherein: In the four-stage pipeline: While the first loop of the K dimension of the current task block loads data from the off-chip memory to the shared memory, the last loop calculation of the K dimension of the previous task block is performed and the result is stored back to the off-chip memory.

9. The method according to any one of claims 1 to 8, wherein: The steps of each computing unit of the multi-core system performing operations in the grid sequence of M dimensions and N dimensions according to the adjusted task blocks of the output matrix include: The operation unit cyclically performs the following operations until the group matrix multiplication operation task is completed: Complete the current task block; and Request the next task block.

10. The method according to claim 9, wherein: The operation unit requests the next task block including: The request is initiated by a processor core in the operation unit through an atomic operation.

11. The method according to claim 10, wherein: The processor core initiating the request through an atomic operation includes: The request is initiated in response to the execution of loading data from the off-chip memory in the last cycle of the K dimension of the current task block.

12. The method according to any one of claims 10-11, wherein: The processor core initiating the request through an atomic operation includes: Updating a task block counter on a global memory of the multi-core system, wherein the task block counter is used to record the number of executed task blocks; and According to the update result of the task block counter, the information of the next task block is determined.

13. A device for executing a computing task on a multi-core system, comprising a processor and a memory, wherein: The processor is configured to execute program instructions; The memory is configured to store the program instructions; When the program instructions are loaded and executed by the processor, the processor executes the method for executing computing tasks on a multi-core system according to any one of claims 1-12.

14. A computer-readable storage medium storing program instructions, which, when loaded and executed by a processor, causes the processor to execute the method for executing a computing task on a multi-core system according to any one of claims 1 to 12.

15. A computer program product, comprising a computer program or instructions, wherein when the computer program or instructions are executed by a processor, the method for performing a computing task on a multi-core system according to any one of claims 1 to 12 is implemented.

16. A chip comprising the device for executing computing tasks on a multi-core system according to claim 13.

17. A board comprising the chip according to claim 16.

Images