Method and device for realizing task splitting based on multi-core processor and related product

Abstract

The invention discloses a method and a device for realizing task splitting based on a multi-core processor and a related product. A device may be included as a processing device in a combined processing device, which may further include an interface device and a computing device. The computing device interacts with the processing device to jointly complete computing operation specified by a user. The combined processing device can further comprise a storage device, and the storage device is connected with the computing device and the processing device and used for storing data of the computing device and the processing device. The scheme disclosed by the invention provides a method for scheduling group matrix multiplication class operation task splitting on the multi-core processor, which can effectively reduce data memory access and improve processing efficiency.

Description

Technical Field

[0001] The present disclosure generally relates to the field of data processing. More specifically, the present disclosure relates to a method for implementing task splitting based on a multi-core processor, a computer-readable storage medium, a computer program product, and an apparatus for implementing task splitting using a multi-core processor. Background Art

[0002] The Mixture of Experts (MoE) model is a machine learning architecture that divides an AI model into different sub-networks (or "experts"), each of which focuses on a subset of the input data and works together to complete a task. This architecture enables even large models with billions of parameters to significantly reduce computational costs during pre-training and inference, improving efficiency by selectively activating the experts required for a specific task rather than activating the entire neural network for each task.

[0003] Group General Matrix Multiplication (Group GEMM) is the core operator in MoE, which deals with batch matrix multiplication problems. However, unlike ordinary batch matrix multiplication, the matrix dimension of each expert in GroupGEMM is variable. This design makes Group GEMM flexible when dealing with different experts, but it also brings computational challenges. The division of computing tasks among multiple cores has a great impact on the performance of Group GEMM. Good task division can enable multiple cores to work together and greatly improve the performance of Group GEMM.

[0004] In view of this, there is an urgent need to provide a solution that can realize task splitting based on multi-core processors to support multi-core efficient parallel processing of computing tasks such as Group GEMM. Summary of the invention

[0005] In order to at least solve one or more of the technical problems mentioned above, the present disclosure proposes a solution for implementing task splitting based on a multi-core processor in multiple aspects.

[0006] In a first aspect, the present disclosure provides a method for implementing task splitting based on a multi-core processor, wherein the method is applicable to group matrix multiplication type operation tasks, the group matrix multiplication type operation tasks include multiple groups of matrix multiplications, and the matrix size of the matrix multiplication is variable; the method includes:

[0007] Obtaining layout information of the group matrix multiplication type operation task;

[0008] Based on the layout information, determining the task type of the group matrix multiplication task; wherein the task type includes a computing bottleneck type and a memory access bottleneck type;

[0009] According to the task type, determine the split size block_k of the split block in the K dimension; wherein the K dimension is the column dimension of the left multiplication matrix and the row dimension of the right multiplication matrix;

[0010] Based on block_k, obtain the split size block_n of the split block in N dimensions and the split size block_m of the split block in M ​​dimensions, so that the split block sizes block_k, block_n, and block_m meet the optimization target; wherein M is the row dimension of the left multiplication matrix, and N is the column dimension of the right multiplication matrix, and the optimization target includes any one of the following: memory access, processor core utilization; and

[0011] The group matrix multiplication operation task is split according to the determined split block sizes block_k, block_n, and block_m, so as to be allocated to the operation units in the multi-core processor for parallel execution.

[0012] In the second aspect, the present disclosure provides a device for implementing task splitting using a multi-core processor, comprising a processor and a memory, wherein: the processor is configured to execute program instructions; the memory is configured to store the program instructions; when the program instructions are loaded and executed by the processor, the processor executes the method for implementing task splitting based on a multi-core processor according to the first aspect.

[0013] In a third aspect, the present disclosure provides a computer-readable storage medium having program instructions stored therein. When the program instructions are loaded and executed by a processor, the processor executes the method for implementing task splitting based on a multi-core processor according to the first aspect.

[0014] In a fourth aspect, the present disclosure provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the method for implementing task splitting based on a multi-core processor according to the first aspect.

[0015] In a fifth aspect, the present disclosure provides a processing device, including a device for implementing task splitting based on a multi-core processor according to the second aspect.

[0016] In a sixth aspect, the present disclosure provides a chip, comprising the processing device according to the fifth aspect.

[0017] In a seventh aspect, the present disclosure provides a board comprising the chip according to the sixth aspect.

[0018] Through the method, device and related products for implementing task splitting based on a multi-core processor as provided above, the disclosed embodiments implement an optimized splitting scheme for group matrix multiplication type calculation tasks on a multi-core processor, which can effectively reduce frequent data exchange and loading, avoid input / output (I / O) bottleneck problems, and fully utilize the parallel operation characteristics of the multi-core architecture to improve processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0020] Figure 1 An exemplary structural diagram of a board in some embodiments of the present disclosure is shown;

[0021] Figure 2 An exemplary structural diagram of a combined processing device in some embodiments of the present disclosure is shown;

[0022] Figure 3 An exemplary internal structure diagram of a computing device in some embodiments of the present disclosure is shown;

[0023] Figure 4 An exemplary structural diagram of a processor core in some embodiments of the present disclosure is shown;

[0024] Figure 5 An exemplary schematic diagram showing a processor core writing data to a processor core of another cluster in some embodiments of the present disclosure;

[0025] Figure 6 An exemplary schematic diagram of a two-layer three-stage pipeline in some embodiments of the present disclosure is shown;

[0026] Figure 7 A schematic diagram of the storage (memory) architecture of a multi-core processor to which the disclosed embodiments can be applied is shown;

[0027] Figure 8 A flow chart of a method for implementing task splitting based on a multi-core processor according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0029] It should be understood that the terms "include" and "comprising" used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0030] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0031] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0032] The specific implementation of the present disclosure is described in detail below with reference to the accompanying drawings.

[0033] Exemplary Hardware Environment

[0034] Figure 1 FIG. 1 is a schematic diagram showing the structure of a board 10 according to an embodiment of the present disclosure. Figure 1 As shown, the board 10 includes a chip 101, which is a system-on-chip, or a system on chip (SoC), which integrates one or more combined processing devices. The combined processing device is an artificial intelligence computing unit that supports various deep learning and machine learning algorithms to meet the intelligent processing needs in complex scenarios in the fields of computer vision, speech, natural language processing, data mining, etc. In particular, deep learning technology is widely used in the field of cloud intelligence. A notable feature of cloud-based intelligent applications is the large amount of input data, which places high demands on the storage capacity and computing power of the platform. The board 10 of this embodiment is suitable for cloud-based intelligent applications and has huge off-chip storage, on-chip storage, and a large amount of computing power.

[0035] The chip 101 is connected to an external device 103 via an external interface device 102. The external device 103 is, for example, a server, a computer, a camera, a display, a mouse, a keyboard, a network card, or a wifi interface. The data to be processed can be transmitted from the external device 103 to the chip 101 via the external interface device 102. The calculation result of the chip 101 can be transmitted back to the external device 103 via the external interface device 102. According to different application scenarios, the external interface device 102 can have different interface forms, such as a PCIe interface.

[0036] The board 10 also includes a storage device 104 for storing data, which includes one or more storage units 105. The storage device 104 is connected to the control device 106 and the chip 101 through a bus and transmits data. The control device 106 in the board 10 is configured to control the state of the chip 101. To this end, in an application scenario, the control device 106 may include a microcontroller (MCU).

[0037] Figure 2 1 is a block diagram showing the combined processing device in the chip 101 of this embodiment. Figure 2 As shown in , the combined processing device 20 includes a computing device 201 , an interface device 202 , a processing device 203 and a storage device (DRAM) 204 .

[0038] The computing device 201 is configured to execute user-specified operations, and is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0039] The interface device 202 is used to transmit data and control instructions between the computing device 201 and the processing device 203. For example, the computing device 201 can obtain input data from the processing device 203 via the interface device 202 and write it into the storage device on the computing device 201 chip. Further, the computing device 201 can obtain control instructions from the processing device 203 via the interface device 202 and write them into the control cache on the computing device 201 chip. Alternatively or optionally, the interface device 202 can also read data in the storage device of the computing device 201 and transmit it to the processing device 203.

[0040] The processing device 203, as a general processing device, performs basic controls including but not limited to data handling, starting and / or stopping the computing device 201. Depending on the implementation, the processing device 203 can be a central processing unit (CPU), a graphics processing unit (GPU) or one or more types of processors in other general and / or special processors, these processors include but are not limited to digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, only with respect to the computing device 201 disclosed in the present invention, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are integrated and considered together, the two are regarded as forming a heterogeneous multi-core structure.

[0041] The storage device 204 may be DRAM, which is an off-chip memory used to store data to be processed. It is DDR and is usually 16G or larger in size. It is used to save data of the computing device 201 and / or the processing device 203.

[0042] Figure 3 The internal structure diagram of the computing device 201 is shown. The computing device 201 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 201 in the figure adopts a multi-core hierarchical structure design. The computing device 201 is a system on a chip, which includes multiple clusters, and each cluster includes multiple processor cores. In other words, the computing device 201 is composed of a hierarchy of system on a chip-cluster-processor core.

[0043] At the system-on-chip level, Figure 3 As shown, the computing device 201 includes an external storage controller 301 , a peripheral communication module 302 , an on-chip interconnect module 303 , a synchronization module 304 , and multiple clusters 305 .

[0044] There may be multiple external storage controllers 301, two of which are shown in the figure as an example, and are used to respond to access requests issued by the processor core and access external storage devices, such as Figure 2DRAM 204 in the chip, thereby reading data from outside the chip or writing data to outside the chip. The peripheral communication module 302 is used to receive control signals from the processing device 203 through the interface device 202 to start the computing device 201 to perform tasks. The on-chip interconnect module 303 connects the external storage controller 301, the peripheral communication module 302 and multiple clusters 305 to transmit data and control signals between each module. The synchronization module 304 is a global synchronization barrier controller (GBC) used to coordinate the work progress of each cluster and ensure information synchronization. Multiple clusters 305 are the computing cores of the computing device 201. Four are shown as examples in the figure. With the development of hardware, the computing device 201 disclosed in the present invention can also include 8, 16, 64, or even more clusters 305. Clusters 305 are used to efficiently execute deep learning algorithms.

[0045] At the cluster level, Figure 3 As shown, each cluster 305 includes multiple processor cores (IPU cores) 306 and a memory core (MEM core) 307 .

[0046] The figure shows four processor cores 306 as an example, but the present disclosure does not limit the number of processor cores 306. Figure 4 Each processor core 306 includes three modules: a control module 41 , a computing module 42 and a storage module 43 .

[0047] The control module 41 is used to coordinate and control the operation of the computing module 42 and the storage module 43 to complete the deep learning task, and includes an instruction fetch unit (IFU) 411 and an instruction decode unit (IDU) 412. The instruction fetch unit 411 is used to obtain instructions from the processing device 203, and the instruction decode unit 412 decodes the obtained instructions and sends the decoding results to the computing module 42 and the storage module 43 as control information.

[0048] The operation module 42 includes a vector operation unit 421 and a matrix operation unit 422. The vector operation unit 421 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformation; the matrix operation unit 422 is responsible for the core calculation of the deep learning algorithm, such as matrix multiplication and convolution.

[0049] The storage module 43 is used to store or move related data, including a neuron RAM (NRAM) 431, a weight RAM (WRAM) 432, an input / output direct memory access module (IODMA) 433, and a move direct memory access module (MVDMA) 434. NRAM 431 is used to store feature maps and intermediate results calculated by the processor core 306; WRAM 432 is used to store the weights of the deep learning network; IODMA 433 controls the memory access between NRAM 431 / WRAM 432 and DRAM 204 through the broadcast bus 309; MVDMA 434 is used to control the memory access between NRAM 431 / WRAM 432 and SRAM 308.

[0050] Back to Figure 3 The storage core 307 is mainly used for storage and communication, that is, to store shared data or intermediate results between the processor cores 306, and to perform communication between the cluster 305 and the DRAM 204, communication between the clusters 305, and communication between the processor cores 306. In other embodiments, the storage core 307 has the capability of scalar operation and is used to perform scalar operation.

[0051] The storage core 307 includes a shared memory (SRAM) 308, a broadcast bus 309, a cluster direct memory access module (CDMA) 310, and a global direct memory access module (GDMA) 311. The SRAM 308 plays the role of a high-performance data transfer station. The data reused between different processor cores 306 in the same cluster 305 does not need to be obtained from the DRAM 204 by each processor core 306, but is transferred between the processor cores 306 via the SRAM 308. The storage core 307 only needs to quickly distribute the reused data from the SRAM 308 to multiple processor cores 306, so as to improve the efficiency of inter-core communication and greatly reduce on-chip and off-chip input / output access.

[0052] The broadcast bus 309, CDMA 310 and GDMA 311 are respectively used to perform communication between processor cores 306, communication between clusters 305 and data transmission between clusters 305 and DRAM 204. They will be described below.

[0053] The broadcast bus 309 is used to complete high-speed communication between the processor cores 306 in the cluster 305. The broadcast bus 309 of this embodiment supports inter-core communication modes including unicast, multicast and broadcast. Unicast refers to point-to-point (i.e., single processor core to single processor core) data transmission, multicast is a communication mode of transmitting a copy of data from SRAM 308 to specific processor cores 306, and broadcast is a communication mode of transmitting a copy of data from SRAM 308 to all processor cores 306, which is a special case of multicast.

[0054] The CDMA 310 is used to control the memory access of the SRAM 308 between different clusters 305 in the same computing device 201 . Figure 5 A schematic diagram is shown when a processor core wants to write data to a processor core of another cluster to illustrate the working principle of CDMA310. In this application scenario, the same computing device includes multiple clusters. For the convenience of explanation, only cluster 0 and cluster 1 are shown in the figure. Cluster 0 and cluster 1 each include multiple processor cores. For the convenience of explanation, cluster 0 in the figure only shows processor core 0, and cluster 1 only shows processor core 1. Processor core 0 wants to write data to processor core 1.

[0055] First, processor core 0 sends a unicast write request to write data into the local SRAM 0. CDMA 0 acts as the master and CDMA 1 acts as the slave. The master pushes the write request to the slave, that is, the master sends the write address AW and write data W to transfer the data to SRAM 1 of cluster 1. Then the slave sends a write response B in response. Finally, processor core 1 of cluster 1 sends a unicast read request to read the data from SRAM 1.

[0056] Back to Figure 3, GDMA 311 cooperates with external memory controller 301 to control the access of SRAM 308 of cluster 305 to DRAM 204, or reads data from DRAM 204 to SRAM 308. As can be seen from the above, the communication between DRAM 204 and NRAM 431 or WRAM 432 can be realized through two channels. The first channel is to directly connect DRAM 204 and NRAM 431 or WRAM 432 through IODMA 433; the second channel is to first transfer data between DRAM 204 and SRAM 308 through GDMA 311, and then transfer data between SRAM 308 and NRAM 431 or WRAM 432 through MVDMA 434. Although it seems that the second channel requires more components to participate and the data flow is longer, in fact, in some embodiments, the bandwidth of the second channel is much greater than that of the first channel, so the communication between DRAM 204 and NRAM 431 or WRAM 432 may be more efficient through the second channel. The embodiments of the present disclosure can select a data transmission channel according to the hardware conditions thereof.

[0057] In other embodiments, the functions of GDMA 311 and IODMA 433 can be integrated into the same component. For the convenience of description, this disclosure regards GDMA 311 and IODMA 433 as different components. For those skilled in the art, as long as the functions they implement and the technical effects they achieve are similar to those of this disclosure, they fall within the protection scope of this disclosure. Furthermore, the functions of GDMA 311, IODMA 433, CDMA 310, and MVDMA 434 can also be implemented by the same component. Similarly, as long as the functions they implement and the technical effects they achieve are similar to those of this disclosure, they fall within the protection scope of this disclosure.

[0058] An important reason why the computing device 201 has strong computing power is the three-level computing hierarchy of system on chip - cluster - processor core, combined with a three-layer memory design such as DRAM-SRAM-NRAM / WRAM, which enables data to be cached and calculated at the appropriate level to form a sufficient pipeline.

[0059] When the computing device 201 performs computing, the process can be divided into the following three phases: loading phase (load): loading data; computing phase (compute): moving data, computing, and moving intermediate results; and storing phase (store): storing results.

[0060] In more detail, in some embodiments, a two-layer three-stage pipeline may be used, such as Figure 6As shown, the first-level loading stage 601, calculation stage 602, and storage stage 603 occur in the cluster level. The first-level loading stage 601 is that the GDMA 330 loads data from the DRAM 204 to the SRAM 308, the first-level calculation stage 602 is that the cluster 305 calculates the loaded on-chip unit map and generates calculation results, and the first-level storage stage 603 is that the GDMA 330 stores the calculation results from the SRAM 308 back to the DRAM 204.

[0061] Since the cluster 305 includes multiple processor cores 306, the first layer calculation stage 602 actually divides the on-chip unit map into corresponding sub-maps through the storage core 307, and broadcasts them to at least one processor core 306 for calculation. Therefore, the second layer three-level waterline occurs in the processor core 306. In more detail, the second layer loading stage 604 is to load the sub-map from the SRAM 308 to the NRAM 431 by the MVDMA 434, the second layer calculation stage 605 is to move the sub-map and the sub-weight to the operation module 42 for calculation, and then move the intermediate result back to the NRAM 431, and the second layer storage stage 606 is to store the intermediate result from the NRAM 431 back to the SRAM 308 by the MVDMA 434.

[0062] The first-level pipeline means that the first-level loading stage 601, the first-level calculation stage 602 and the first-level storage stage 603 can be performed in parallel. Now, taking the case where the same cluster 305 wants to process the j-th on-chip unit diagram, the j+1-th on-chip unit diagram and the j+2-th on-chip unit diagram as an example, first, the j-th on-chip unit diagram is loaded into the SRAM 308 in the first-level loading stage 601. Then, the j-th on-chip unit diagram is calculated in the first-level calculation stage 602, and the first calculation result is transferred back to the SRAM 308. While the j-th on-chip unit diagram is being calculated, the j+1-th on-chip unit diagram is loaded into the SRAM 308 in the first-level loading stage 607. When the first calculation result is stored back to DRAM 204 in the first level storage-back stage 603, the j+1th on-chip unit map is calculated in the first level calculation stage 608, and the second calculation result is transferred back to SRAM 308, and the j+2th on-chip unit map is loaded into SRAM 308 in the first level loading stage 610. The first level pipeline is rolled in this way.

[0063] In order to cooperate with the operation of the aforementioned pipeline, the SRAM 308 of this embodiment includes two storage spaces: a ping storage unit and a pong storage unit. The data flow is divided into three types according to the ping-pong attribute of the SRAM 308: input / output ping-pong (IOparity), input ping-pong (input parity), and no ping-pong (no parity). Input / output ping-pong can support parallel loading, calculation, and storage. To realize input / output ping-pong, the ping storage unit and the pong storage unit need to be completely equal, and are used for loading and storage respectively. Input ping-pong only supports storage and calculation in parallel, which will increase the time of transportation on the SRAM 308. Compared with input / output ping-pong, the ping storage unit and the pong storage unit do not need to be completely equal, but an additional cache (cache) of the same size as the storage space needs to be allocated. No ping-pong means that loading / storage and calculation are serial, and space does not need to be additionally allocated.

[0064] In order to implement the aforementioned first-level pipeline, the SRAM 308 of some embodiments has ping memory cells and pong memory cells of the same size to achieve the input / output ping-pong effect. Figure 6 For illustration, the storage area involved in the first-level loading stage 601, the first-level calculation stage 602 and the first-level storage-back stage 603 of the j-th on-chip unit diagram is limited to the ping storage unit, while the storage area involved in the first-level loading stage 607, the first-level calculation stage 608 and the first-level storage-back stage 609 of the j+1-th on-chip unit diagram is limited to the pong storage unit, and the storage area involved in the first-level loading stage 610, the first-level calculation stage 611 and the first-level storage-back stage 612 of the j+2-th on-chip unit diagram is again limited to the ping storage unit, and in this way, the ping storage unit and the pong storage unit are used alternately for storage.

[0065] The second-level pipeline means that the second-level loading stage 604, the second-level calculation stage 605, and the second-level storage stage 606 can be processed in parallel. Take the case where the same processor core 306 wants to process the i-th sub-graph, the i+1-th sub-graph, and the i+2-th sub-graph in the j-th on-chip unit graph as an example. First, the i-th sub-graph is broadcast to the NRAM 431 in the second-level loading stage 604. Then, the i-th sub-graph is calculated in the second-level calculation stage 605 to generate the i-th intermediate result, and the i-th intermediate result is transferred back to the NRAM 431. At the same time, the i+1-th sub-graph is broadcast to the NRAM 431 in the second-level loading stage 613. The i-th intermediate result is stored back to the SRAM 308 in the second-level storage-back stage 606, and at the same time, the i+1-th sub-graph is calculated in the second-level calculation stage 614 to generate the i+1-th intermediate result, and the i+1-th intermediate result is moved back to the NRAM 431, and the i+2-th sub-graph is loaded into the NRAM 431 in the second-level loading stage 615.

[0066] Considering that the tasks of each cluster 305 are different and the completion times are naturally inconsistent, the synchronization module 304 of some embodiments may use synchronization barrier instructions to synchronize the task completion times to avoid timing errors.

[0067] Although an exemplary multi-core hierarchical structure is described above, considering that group matrix multiplication type computing tasks may be implemented on various multi-core architectures, the task scheduling scheme needs to consider the compatibility and universality of different multi-core architectures. The inventors note that the goals of task splitting optimization on a multi-core architecture are mainly in two aspects: IO access and processor core utilization.

[0068] On the one hand, since the processor core is subject to the capacity limit of internal storage resources, the huge amount of data calculation may cause a large amount of data interaction between the processor core and the external storage device. The bandwidth of the input / output (I / O) bus between the processor core and the external memory is limited, which is likely to cause a serious I / O bottleneck problem, and then cause data transmission delays and greatly reduce the computing efficiency during parallel computing. Furthermore, not only will the bandwidth limit of the I / O bus become a bottleneck for system performance, but the large amount of I / O memory access between the processor core and the external storage device will also have an adverse impact on computing and power consumption.

[0069] On the other hand, current computer hardware uses tensor core technology to accelerate large-scale matrix multiplication, that is, to implement a relatively large-scale matrix multiplication on a single computing unit. For example, in some hardware architectures, a computing unit can calculate a 16*8*16 (M*K*N) matrix multiplication in one beat. This requires that the M and N dimensions be aligned to 16. However, the matrix dimensions of each expert in MoE's Group GEMM are variable, which results in a loss of computing power due to the alignment of the M and N dimensions. Therefore, it is also necessary to optimize the utilization of processor cores.

[0070] In view of this, the disclosed embodiments abstract a storage (memory) architecture for various multi-core processor structures to be compatible with different existing multi-core architectures. On this basis, considering the amount of data interaction between the processor core and the external circuit (such as an external storage device) and / or the utilization rate of the processor core, a task scheduling scheme is provided to efficiently schedule group matrix multiplication type computing tasks to the multi-core processor for parallel execution, thereby improving the computing efficiency of parallel computing of the multi-core processor.

[0071] Figure 7 A schematic diagram of a storage (memory) architecture of a multi-core processor to which the disclosed embodiment can be applied is shown. The storage architecture comprehensively considers a variety of multi-core processors and is compatible with different existing multi-core architectures.

[0072] As shown in the figure, the bottom layer DRAM 710 is an off-chip memory used to store input and output data, such as input data to be processed by the processor core and processing results obtained by the multi-core processor. DRAM 710 can be DDR memory, and the DDR memory types are: LPDDR, GDDR, HBM, etc. The processor core can access the off-chip storage based on the bus protocol through the on-chip storage control unit (such as DDR controller). DRAM 710 can be regarded as global storage or global memory at this level of the chip.

[0073] The next level up in the figure is the L2 cache 720. The L2 cache 720 may include LLC (Last Level Cache). As mentioned above, LLC may have multiple operation modes, including data latching, streaming access, clearing and writing back to system memory, etc. When the data latching mode of LLC is enabled, the characteristic of LLC latching access data can be fully utilized to maximize the utilization of data resident in LLC, thereby achieving the effect of bandwidth expansion.

[0074] The next layer in the figure is the shared memory 730. The shared memory 730 can share on-chip storage between multiple processor cores in a computing cluster. It is understood that there can be multiple shared memories, each of which serves multiple processor cores respectively, and only one shared memory is shown in the figure as an example.

[0075] The top layer in the figure is the local memory 740 of each processor core 750, and the shared memory 730 is the on-chip storage of the corresponding processor core 750. Depending on the specific hardware architecture, the local memory 740 can have different implementations, such as a data buffer or a register page, and the disclosed embodiment is not limited in this respect.

[0076] although Figure 7 The memory architecture abstracted based on various multi-core architectures is shown, but those skilled in the art should understand that in different hardware implementations, Figure 7 Some storage levels in the system may be omitted. For example, in one implementation, there may be no shared memory 730; in another implementation, there may be no L2 cache 720, or even if LLC is configured on the hardware, its data latching mode is not enabled, which is equivalent to no L2 cache; in another implementation, there may be neither shared memory 730 nor L2 cache 720, or the data latching mode of LLC is not enabled. In the task scheduling scheme of the disclosed embodiment, based on Figure 7 The storage architecture takes into account the above-mentioned possible implementations and provides an optimized task splitting solution to support efficient parallel execution of group matrix multiplication tasks in a multi-core architecture.

[0077] In different multi-core processors or different working modes of the same multi-core processor, multiple processor cores may be scheduled to perform parallel operations in different units or granularities. For example, in NVIDIA's multi-core processors, one SM is generally used as the basic unit, while in some Cambrian multi-core processors, one computing cluster is generally used as the basic unit. In the disclosed embodiment, the scheduling granularity of the processor core is referred to as an "operation unit", and each operation unit includes one or more processor cores. Depending on the different multi-core processors, the number of processor cores contained in an operation unit is also different. For example, in some implementations, a computing cluster can be used as an operation unit, which contains 4 processor cores; in other implementations, four computing clusters can be used as an operation unit, which contains 16 processor cores; in some other implementations, one processor core can be used as an operation unit.

[0078] It should be noted that the "group matrix multiplication type operation tasks" mentioned in this article include not only direct group matrix multiplication operation tasks, but also other operation tasks that can be converted into group matrix multiplication operations, such as multiple convolution operations, multiple Einstein sums, etc.

[0079] A single matrix multiplication operation can usually be expressed as: C = op(A)*op(B) (1)

[0080] Where op indicates whether to transpose the matrix. The left multiplication matrix op(A) and the right multiplication matrix op(B) are matrices of size M×K and K×N respectively, and the output matrix C is a matrix of size M×N. In most cases, matrices A and B do not need to be transposed, so op(A)=A, op(B)=B. For simplicity, matrices A and B are directly mentioned below, omitting op. When there are multiple sets of matrix multiplication operations, the matrix size of each set of matrix multiplication is variable. The input and output data of the convolution operation usually involve dimensions such as batch (Batch), height (H), width (W), input channel (Ci), output channel (Co), convolution kernel height (Kh) and width (Kw), which can be converted into matrix multiplication operations by merging dimensions. For example: for the convolution operation, its M, N and K dimensions can be M=Batch×H×W, N=Co, K=Ci×Kh×Kw respectively. Therefore, the solution described in the embodiment of the present disclosure based on the matrix multiplication operation task can also be similarly applied to these operation tasks that can be converted into matrix multiplication operations.

[0081] In intelligent computing systems, high-performance multi-core processors are usually used as server accelerator cards, which exchange data with the host CPU via the PCIe bus, forming a host-device working mode. Figure 2In the combined processing device 20 shown, the computing device 201 is a high-performance multi-core processor, which serves as the device side. The processing device 203 can serve as the host side. The scheme of the disclosed embodiment includes two aspects: on the one hand, it is a task scheduling scheme on the host side, and on the other hand, it is a task execution scheme on the device side. The host side determines the optimization parameters of the matrix multiplication splitting scheme and clarifies the matrix multiplication splitting strategy based on the optimization goal of the task splitting scheduling. Afterwards, the host side passes these optimization parameters from the host side to the device side, that is, the high-performance multi-core processor, by launching the kernel. Each computing unit in the multi-core processor serving as the device side determines and executes the Group GEMM tasks that need to be processed by itself based on these optimization parameters.

[0082] Splitting scheme for group matrix multiplication tasks

[0083] In an embodiment of the present disclosure, a task scheduling scheme based on a multi-core processor is provided to distribute group matrix multiplication type operation tasks to be executed on multiple processor cores of the multi-core processor.

[0084] The biggest problem with existing group matrix multiplication splitting is that the M, K, and N dimensions of a single group matrix multiplication are not large enough to fully utilize the computing power of a multi-core processor, so its splitting is a very big challenge. If M and N are large enough, the existing splitting method of GEMM can be reused, for example, refer to the chessboard splitting solution provided in Chinese patent application publication CN118733206A. However, when M and N are moderate and it is a computational bottleneck, using the thinking of a single matrix multiplication to consider the problem of group matrix multiplication cannot fully utilize the performance of group matrix multiplication.

[0085] Therefore, the disclosed embodiment specifically designs a splitting optimization method for group matrix multiplication based on the characteristics of group matrix multiplication, that is, splitting is performed from the perspective of one operation unit. In determining the size of the split block, the different situations of IO bottleneck and computing bottleneck are distinguished to obtain the optimal splitting solution. The split block here refers to the size of the data block that needs to be calculated in each round of each operation unit, which includes three dimensions M, N, and K, and the sizes of each dimension are block_m, block_n, and block_k respectively.

[0086] Figure 8 An exemplary flow chart of a method 800 for implementing task splitting based on a multi-core processor according to an embodiment of the present disclosure is shown. The tasks to be split include group matrix multiplication type operation tasks, and the group matrix multiplication type operation tasks include multiple groups of matrix multiplications, and the matrix size of each group of matrix multiplications is variable. The multi-core processor includes multiple operation units, each operation unit includes one or more processor cores, and each processor core has a local memory (e.g., reference Figure 7 storage architecture).

[0087] like Figure 8 As shown, in step 810, the layout information of the group matrix multiplication type operation tasks is obtained.

[0088] Before splitting the task, you need to have some understanding of the computational scale of the task. The layout information may include the matrix size of each group of matrix multiplications, such as the size of the M, K, and N dimensions, and whether transposition is required. Considering that the matrix size of each group of matrix multiplications in group matrix multiplication type operation tasks is variable, the splitting scheme is designed based on the maximum size of the dimension in each group of matrix multiplications in the disclosed embodiment. In some embodiments, obtaining the above-mentioned layout information includes: obtaining the maximum values ​​m_max, k_max, and n_max of the M, K, and N dimensions in the multiple groups of matrix multiplications. Depending on the location of the matrix multiplication data, there may be different ways to obtain the maximum values ​​of the above-mentioned dimensions.

[0089] In some embodiments, in response to the dimensional data being on the host side, the maximum value of each dimension can be directly searched from the dimensional data. It can be understood that for Group GEMM, since the M, K, and N of each group of matrix multiplications are variable, the dimensional data provided to the host side is an M, K, and N array (e.g., m_array[], k_array[], n_array[]), and the maximum value needs to be searched from each array.

[0090] In other embodiments, in response to the dimensional data on the device side, the maximum value of each dimension can be determined based on prior knowledge. For example, in the reasoning scenario of a general MoE network, M is variable, and K and N are immutable, so the maximum values ​​of the K and N dimensions are themselves, and the maximum value of the M dimension m_max can be taken as m_max=batch*sequence, where batch is the number of batches in the reasoning scenario, and sequence is the sequence length. For another example, in the training scenario of a general MoE network, the scale of the forward data and the reverse data is the same as that of the reasoning scenario, and can be set similarly; and in the backward filter (backward filter, also known as the reverse convolution kernel), K is variable, and M and N are immutable, so the maximum value of K can be set to the maximum value of the M dimension m_array[] in the reasoning stage.

[0091] In some other embodiments, in response to the dimensional data being on the device side, it is necessary to obtain the dimensional data from the device side and search for the maximum value of each dimension therefrom. This approach is general. For example, when M, K, and N are all variable on the device side, the memcpy (memory copy) instruction can be used to copy from the device side to the host side, and then the maximum value can be retrieved. Of course, based on the previous implementation, the maximum estimate can be given directly using the information from the device malloc (device memory allocation) stage or other prior information. These estimates do not need to be very precise, as long as they can help define whether the task is a computing bottleneck type or an IO bottleneck type, as described below.

[0092] Next, in step 820, the task type of the group matrix multiplication operation task is determined; wherein the task type includes a computing bottleneck type and a memory access bottleneck type.

[0093] In some embodiments, based on the layout information, the task type of the group matrix multiplication type operation task is determined. In further detail, the computing power limit is determined according to the layout information of the group matrix multiplication type operation task, the data bit width of the left multiplication matrix, and the data bit width of the right multiplication matrix; the computational memory access ratio of the multi-core processor is determined according to the bandwidth of the off-chip memory DRAM of the multi-core processor and the number of floating-point operations per second of the multi-core processor to one trillion times; the computational memory access ratio of the multi-core processor is compared with the computing power limit, and the task type of the group matrix multiplication type operation task is determined according to the comparison result. Specifically, the following conditions can be used to determine whether the task is a computational bottleneck type or a memory access bottleneck type. When the following conditions are met, the group matrix multiplication type operation task is determined to be a computational bottleneck type task, otherwise it is a memory access bottleneck type task:

[0094]

[0095] in, CGMA DEV The compute-to-memory ratio of the multi-core processor itself is the ratio of the number of compute operations to the number of memory operations, which can reflect the efficiency of compute operations and memory operations. TFLOPS is the number of floating-point operations per second; BandWidth DRAM For off-chip memory devices DRAM (e.g. Figure 2204), in GB / s. group is the number of matrix multiplication groups in the group matrix multiplication task, sizeof() represents the data bit width, A is the left multiplication matrix, B is the right multiplication matrix, M, K, and N are the maximum values ​​of the M, K, and N dimensions obtained in the previous steps, respectively. Unless otherwise specified, for the sake of brevity, M, K, and N used in the task scheduling scheme of the disclosed embodiment refer to the maximum values ​​of their respective dimensions.

[0096] In step 830, the split size block_k of the split block in the K dimension is determined according to the task type; wherein the K dimension is the column dimension of the left multiplication matrix and the row dimension of the right multiplication matrix.

[0097] Since the K dimension involves both the column dimension of the left multiplication matrix and the row dimension of the right multiplication matrix, pre-defining the split size block_k of the K dimension is more conducive to reducing the search amount in the subsequent search for the optimized split sizes block_m and block_n of the M and N dimensions, thereby reducing the overhead on the host side.

[0098] In some embodiments, determining the split size block_k of the split block in the K dimension may include: in response to the group matrix multiplication type operation task being a computing bottleneck type task, setting block_k to meet the efficiency requirements of the instructions for implementing the matrix multiplication operation supported by the multi-core processor; or in response to the group matrix multiplication type operation task being a memory access bottleneck type task, setting block_k to prioritize improving the memory access efficiency of the largest matrix; wherein the largest matrix is ​​determined based on the size of the left multiplication matrix and the size of the right multiplication matrix. In these embodiments, by distinguishing whether the task is a computing bottleneck or a memory access bottleneck type, the requirements of different types of tasks can be considered more carefully to better match the optimization goals.

[0099] Generally, according to the storage dimension where the K dimension is located, block_k can be set as follows: when the storage dimension where the K dimension is located is the lowest dimension, block_k is set to a multiple of the cache line size; where the multiple is a natural number; or when the storage dimension where the K dimension is located is not the lowest dimension, block_k is set to the alignment granularity align_k number of the instruction that implements the matrix multiplication operation; where the multiple is a natural number.

[0100] Specifically, for computing bottleneck tasks, it is necessary to improve the computing efficiency as much as possible, that is, to meet the efficiency requirements of the instructions for implementing matrix multiplication operations supported by multi-core processors as much as possible. The instructions for implementing matrix multiplication operations can be a single matrix multiplication instruction or an instruction composed of multiple matrix multiplication and accumulation (MMA) instructions. The disclosed embodiment has no restrictions in this regard. When the storage dimension where the K dimension is located is the lowest dimension, block_k is set to a cache line size according to the transposition information. For example, for a GPU, in the scenario where the transposition information is transB=true or transA=false, the size of a memory access sector (sector) can be selected, such as 64B. This setting method can not only ensure the basic memory access function and efficiency, but also reserve space for the splitting of M and N dimensions to improve computing efficiency. When the storage dimension where the K dimension is located is not the lowest dimension, block_k is set to a multiple of the alignment granularity align_k of the instruction for implementing the matrix multiplication operation according to the transposition information. For example, in the scenario where the transposition information is transB=false and transA=true, the multiple of the alignment granularity align_k is 2 times, 3 times, etc. This setting can not only meet the minimum alignment granularity requirement of matrix multiplication operation instructions, but also, in order to meet the efficiency of on-chip matrix multiplication operation instructions and improve the depth of K-dimensional loops, you can select several times the size of align_k to fully utilize the computing power of the operation unit and improve the computing efficiency of the matrix multiplication operation instructions.

[0101] For tasks with memory access bottlenecks, the memory access performance of the largest matrix needs to be guaranteed to avoid the inability to read the largest matrix in time to participate in the operation due to the memory access bottleneck. Here, the largest matrix is ​​determined according to the size of the left multiplication matrix and the size of the right multiplication matrix, that is, it is determined according to the layout information obtained previously, specifically, it is determined according to the maximum value of N and M. When the storage dimension where the K dimension is located is the lowest dimension, the size of block_k can be appropriately enlarged. For example, in the transformer decoder scenario, when the transposition information is transB=true, block_k is set to a multiple of the cache line, such as 2 times, 3 times, etc. In the multi-core processor described above, 2 times the cache line size is taken, that is, 1024B. When the storage dimension where the K dimension is located is not the lowest dimension, block_k can be set smaller to ensure the minimum alignment granularity requirement of the matrix multiplication operation instruction, for example, block_k is set to the alignment granularity align_k of the instruction that implements the matrix multiplication operation.

[0102] As described above, the K-dimensional split size block_k is determined for computing bottlenecks and memory access bottlenecks. The split strategy within a single computing unit can also be determined in step 830. As mentioned above, depending on different multi-core processors, the number of processor cores contained in a computing unit in different multi-core processors is also different.

[0103] In some embodiments, determining the splitting strategy within a single computing unit may include: determining the number of processor cores Ncore within a single computing unit; splitting the split block into pm*pn parts, where pm*pn=Ncore, pm is the number of splits in the M dimension, and pn is the number of splits in the N dimension.

[0104] Specifically, any of the following splitting methods may be adopted: evenly distributing pm and pn in the M dimension and the N dimension; or in response to any one of the M dimension and the N dimension in the split block being smaller than a corresponding threshold, splitting the other dimension.

[0105] For example, for an architecture with 4 processor cores inside a computing unit, a 2×2 split is generally adopted inside a split block, that is, it is split into two parts along block_m and block_n respectively, and distributed to the 4 processor cores. However, for some special cases, such as when one dimension is small and the other dimension is large, the larger dimension can be split and the smaller dimension can be kept intact. For example, when block_n is very small, less than 2 times align_n (align_n is the alignment granularity of the instruction to implement the matrix multiplication operation in the N dimension), and block_m is very large, only block_m can be split into 4 parts, and the complete block_n can be kept. For another example, when block_m is very small, less than align_m (align_m is the alignment granularity of the instruction to implement the matrix multiplication operation in the M dimension) or 2*align_m, and block_n is very large, only block_n can be split into 4 parts, and the complete block_m can be kept.

[0106] Next, in step 840, based on the block_k determined above, the split size block_n in the N dimension and the split size block_m in the M dimension are determined to meet the optimization goal; wherein M is the row dimension of the left multiplication matrix, and N is the column dimension of the right multiplication matrix, and the above optimization goal includes any of the following: memory access, processor core utilization.

[0107] In some embodiments, determining the split size block_n of the split block in N dimensions and the split size block_m of the split block in M ​​dimensions may include: determining the maximum search space of N dimensions; searching for block_n that meets the optimization goal and determining the corresponding block_m within the maximum search space of N dimensions; or determining the maximum search space of M dimensions; searching for block_m that meets the optimization goal and determining the corresponding block_n within the maximum search space of M dimensions. As mentioned above, the search may be conducted in N dimensions or in M ​​dimensions, and the disclosed embodiments are not limited thereto.

[0108] In some embodiments, the maximum search space is constructed as follows: the maximum search space in N dimensions is determined based on the size of the largest block_n that can be accommodated due to the storage space limitation of the local memory of the multi-core processor and the size of the largest block_n that can be accommodated due to the storage space limitation of the shared memory of the multi-core processor; or the maximum search space in M ​​dimensions is determined based on the size of the largest block_m that can be accommodated due to the storage space limitation of the local memory of the multi-core processor and the size of the largest block_m that can be accommodated due to the storage space limitation of the shared memory of the multi-core processor; wherein the size of the largest block_n or block_m that can be accommodated due to the storage space limitation of the local memory of the multi-core processor is based on the splitting of the split blocks in K dimensions. The size block_k, the number pk of blocks split when implementing software pipelining during the data transfer process between the local memory l1_cache and the shared memory SRAM, the left multiplication matrix A, the right multiplication matrix B, the output matrix C, the alignment granularity of the instructions for implementing the matrix multiplication operation in the corresponding dimension, the capacity of the local memory of the multi-core processor, and the splitting strategy within a single operation unit are determined; the size of the maximum block_n or block_m that can be accommodated due to the limitation of the storage space of the shared memory SRAM of the multi-core processor is based on the split size block_k of the split block in the K dimension, the left multiplication matrix A, the right multiplication matrix B, the alignment granularity of the instructions for implementing the matrix multiplication operation in the corresponding dimension, the capacity of the shared memory of the multi-core processor, and the splitting strategy within a single operation unit.

[0109] By establishing the maximum search space of N dimensions or M dimensions according to the above parameters, it is possible to avoid performing a full domain search on N dimensions or M dimensions, thereby reducing the amount of search.

[0110] It can also be seen from the construction of the search space above that when searching for the split size of the block, a brute force search in the three dimensions of M, K, and N is not used. Instead, a cyclic branch reduction method is used to reduce the search volume, thereby reducing the host-side overhead. This optimized search scheme can significantly shorten the search time. For example, in the case of M=K=N=8192, a three-dimensional brute force search scheme is used, and the host-side search time can reach about 3ms; while in the same case, the cyclic pruning method of the disclosed embodiment is used, and the host-side search time only takes 20us to 30us to complete the search, and the performance of matrix multiplication can also be guaranteed.

[0111] The following is a detailed description using the search in N dimensions as an example. Those skilled in the art will appreciate that when a search in M ​​dimensions is adopted, the specific search process can be implemented similarly and will not be described in detail here.

[0112] First, based on block_k, the capacity of the local memory, and the splitting strategy (pm*pn) within a single computing unit, determine the maximum search space in N dimensions. In other words, based on these known information, determine the maximum block_n value that can be accommodated on the chip. Figure 7 The memory architecture of the multi-core processor shown in the figure may include slightly different storage hierarchies in different hardware implementations of multi-core processors. For example, in one implementation, the on-chip storage hierarchy may only include local memory, in which case only the capacity of the local memory needs to be considered. In another implementation, the on-chip storage hierarchy includes local memory and shared memory, so the capacity of the two needs to be considered, and the smaller value is taken. Taking the latter implementation as an example, the maximum search space search_max_n in N dimensions can be set as follows:

[0113] search_max_n=min(max_n_l1*pn,max_n_smem)(3)

[0114] In the above formula (3), max_n_l1*pn represents the maximum size of block_n that can be accommodated due to the limitation of L1_cache storage space; max_n_smem represents the maximum size of block_n that can be accommodated due to the limitation of shared memory SRAM storage space. SRAM then takes the smaller value of the two as the maximum search space search_max_n of N dimension. l1_cache indicates the capacity of the local memory, and SRAM indicates the capacity of the shared memory. Figure 4As for the internal structure of the processor core shown, it can be regarded as a collection of NRAM 431 and WRAM 432, for example. align_m represents the alignment granularity of the instruction implementing the matrix multiplication operation in the M dimension, pk represents the number of blocks_k split when the software pipeline is implemented during the data transfer process between the local memory l1_cache and the shared memory SRAM; sizeof() represents the data bit width, A is the left multiplication matrix, B is the right multiplication matrix, and C is the output matrix. align_m*(block_k / pk)*sizeof(A)*2 represents the storage space occupied by the ping-pong buffer of the left multiplication matrix A; (align_m*sizeof(C))*2 represents the storage space required for the ping-pong buffer of the output matrix C; (block_k / pk)*sizeof(B)*2 represents the storage space required for the ping-pong buffer of the right multiplication matrix B. “pm*align_m*block_k*sizeof(A)*2” indicates the size of the storage space occupied by the left multiplication matrix A on the shared memory SRAM; “block_k*sizeof(B)*2” indicates the size of the storage space occupied by the right multiplication matrix B on the shared memory SRAM.

[0115] Based on the maximum search space, the search steps of the above-mentioned loop pruning method include: determining the corresponding block_m or block_n according to the capacity of the local memory, block_k, and block_n or block_m in the current search loop; calculating the corresponding optimization target according to the block_m or block_n corresponding to the current search loop; and determining the block_n and block_m corresponding to the optimal optimization target according to the optimization target corresponding to each search loop in the search space.

[0116] The following is a detailed description using the search in N dimensions as an example. Those skilled in the art will appreciate that when a search in M ​​dimensions is adopted, the specific search process can be implemented similarly and will not be described in detail here.

[0117] In this embodiment, the search step can be performed in the above-mentioned N-dimensional maximum search space in the following loop, including: determining the corresponding block_m according to the capacity of the local memory, block_k, and block_n in the current search loop; calculating the corresponding optimization target according to the block_m corresponding to the current search loop; and determining the block_n and block_m corresponding to the optimal optimization target according to the optimization target corresponding to each search loop in the search space. The optimization target may be different according to different task types.

[0118] When the group matrix multiplication task is a memory access bottleneck task, the optimization goal is the memory access amount, that is, the smaller the memory access amount, the better. In some embodiments, the memory access amount io can be calculated as follows:

[0119]

[0120] Among them, M, N, K are the maximum values ​​of M, N, K dimensions respectively, and sizeof() indicates the data bit width.

[0121] When the group matrix multiplication task is a computing bottleneck task, the optimization target is the processor core utilization, that is, the higher the processor core utilization, the better. In some embodiments, the processor core utilization DEV utils It can be calculated as follows:

[0122]

[0123] in, CGMA DEV It is the computing memory access ratio of the multi-core processor itself. TFLOPS is one trillion floating-point operations per second. BandWidth DRAM is the bandwidth of the off-chip memory DRAM, in GB / s; group is the number of groups of matrix multiplication in the group matrix multiplication task, num cluster is the number of operation units, total block Indicates the total number of split blocks, PAD_UP (total block ,num cluster ) is equal to total block Divide by num cluster The result is rounded up and multiplied by num cluster In the above formula (5), the first term in the multiplication indicates whether the computing memory access performance corresponding to the split blocks can meet the requirement that the memory access of the multi-core processor is masked by the computing; the second term in the multiplication indicates the utilization rate of the processor core of the multi-core processor.

[0124] In summary, the search steps include:

[0125] In the maximum search space, the corresponding block_m or block_n is determined according to the number of searches i in the current search loop, the splitting strategy within a single operation unit, and the alignment granularity of the instructions implementing the matrix multiplication operation in the corresponding dimension;

[0126] Based on the corresponding block_m, determine the size of the storage block_n of the local memory corresponding to the multi-core processor and the size of the storage block_n of the shared memory corresponding to the multi-core processor, and determine the N-dimensional split size block_n in the current search loop according to the size of the storage block_n of the local memory corresponding to the multi-core processor and the size of the storage block_n of the shared memory corresponding to the multi-core processor; or, based on the corresponding block_n, determine the size of the storage block_m of the local memory corresponding to the multi-core processor and the size of the storage block_m of the shared memory corresponding to the multi-core processor, and determine the M-dimensional split size block_m in the current search loop according to the size of the storage block_m of the local memory corresponding to the multi-core processor and the size of the storage block_m of the shared memory corresponding to the multi-core processor;

[0127] Calculate the corresponding optimization target according to the N-dimensional split size block_n and the M-dimensional split size block_m corresponding to the current search loop; and

[0128] According to the optimization target corresponding to each search cycle in the search space, the N-dimensional split size block_n and the M-dimensional split size block_m corresponding to the optimal optimization target are determined.

[0129] The above loop search process can be expressed in pseudo code as follows:

[0130] for(i=1;i<(search_max_n / pn*align_n);i++);

[0131] temp_block_n=i*pn*align_n;

[0132]

[0133] if(computer_bound):search max DEV utils ;

[0134] if(io_bound):search min io size.

[0135] It can be seen from the above pseudo code that in the maximum search space of N dimensions, in each search process, the method of determining the corresponding temp_block_m based on temp_block_n in the current search loop is similar to the method of determining the maximum search space previously described. That is, the size of the largest block_m that can be accommodated due to the limitation of the local memory l1_cache storage space and the size of the largest block_m that can be accommodated due to the limitation of the shared memory SRAM storage space are used, and then the smaller value of the two is taken as the temp_block_m of the current search loop. Afterwards, according to whether the task type is a computing bottleneck or a memory access bottleneck, the corresponding optimization targets are calculated using formulas (4) and (5) respectively, and the combination of block_n and block_m that optimizes the optimization target is selected.

[0136] Thus, the split sizes block_k, block_n, and block_m that optimize the optimization target can be determined.

[0137] continue Figure 8 In step 850, the group matrix multiplication operation task is split according to the determined split block sizes block_k, block_n, and block_m, so as to be distributed to the operation units in the multi-core processor for parallel execution. Specifically, it can be instructed to split each group of matrix multiplications in the group matrix multiplication operation task into multiple split blocks according to the determined split sizes block_k, block_n, and block_m, and then distribute the corresponding split blocks cyclically to each operation unit for operation according to the grid order of the M and N dimensions of the output matrix block.

[0138] It can be seen from the scheme described above that the scheme splits the group matrix multiplication task from the perspective of an operation unit, which can be more in line with the implementation of the device side and facilitate the task distribution of the device side. When searching for the split size, by predetermining block_k, the search volume of the subsequent optimization parameter search can be greatly reduced, the overhead of the host side can be reduced, and the processing efficiency can be improved. Further, when predetermining block_k, the task types of memory access bottlenecks and computing bottlenecks are distinguished, so that the optimization target can be better matched. Further, in the design of the optimization target, different optimization targets are considered for different task types. For example, for tasks of the computing bottleneck type, the processor core utilization is taken into consideration, so as to make up for the limitations of splitting from the perspective of a single operation unit. In short, the disclosed embodiment provides an optimized task splitting scheme for the implementation of group matrix multiplication type operation tasks on a multi-core processor, which can effectively reduce frequent data exchange and loading, avoid input / output (I / O) bottleneck problems, and make full use of the parallel operation characteristics of the multi-core architecture to improve the processor core utilization.

[0139] Accordingly, the present disclosure also provides a device for implementing task splitting based on a multi-core processor, including a processor and a memory, wherein: the processor is configured to execute program instructions; the memory is configured to store the program instructions; when the program instructions are loaded and executed by the processor, the processor executes the method for implementing task splitting based on a multi-core processor as described in any of the foregoing embodiments. The present disclosure also provides a computer-readable storage medium, in which program instructions are stored, and when the program instructions are loaded and executed by the processor, the processor executes the method for implementing task splitting based on a multi-core processor as described in any of the foregoing embodiments. The present disclosure also provides a computer program product, including a computer program or instruction, and when the computer program or instruction is executed by the processor, the method for implementing task splitting based on a multi-core processor as described in any of the foregoing embodiments is implemented. The present disclosure also provides a processing device, including the device for implementing task splitting based on a multi-core processor as described in the foregoing. The present disclosure also provides a chip, including the device for implementing task splitting based on a multi-core processor as described in the foregoing. The present disclosure also provides a board, including the chip as described in the foregoing.

[0140] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will appreciate that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teaching of the present disclosure, those skilled in the art will appreciate that some of the steps therein may be performed in other orders or simultaneously. Further, those skilled in the art will appreciate that the embodiments described in the present disclosure may be regarded as optional embodiments, i.e., the actions or modules involved therein may not necessarily be necessary for the implementation of one or some of the schemes of the present disclosure. In addition, depending on the different schemes, the present disclosure also has different focuses on the description of some embodiments. In view of this, those skilled in the art may appreciate the parts that are not described in detail in a certain embodiment of the present disclosure, and may also refer to the relevant descriptions of other embodiments.

[0141] In terms of specific implementation, based on the disclosure and teaching of this disclosure, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented by other methods not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this article divides them on the basis of considering logical functions, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. In terms of the connection relationship between different units or components, the connection discussed in the above text in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0142] In the present disclosure, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed on multiple network units. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the scheme described in the embodiments of the present disclosure. In addition, in some scenarios, multiple units in the embodiments of the present disclosure may be integrated into one unit or each unit may exist physically separately.

[0143] In some implementation scenarios, the above-mentioned integrated unit can be implemented in the form of a software program module. If implemented in the form of a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable memory. Based on this, when the scheme of the present disclosure is embodied in the form of a software product (such as a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to enable a computer device (such as a personal computer, a server or a network device, etc.) to perform some or all of the steps of the method described in the embodiment of the present disclosure. The aforementioned memory may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0144] In some other implementation scenarios, the above-mentioned integrated unit can also be implemented in the form of hardware, that is, a specific hardware circuit, which may include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as CPU, GPU, FPGA, DSP and ASIC, etc. Further, the aforementioned storage unit or storage device can be any appropriate storage medium (including magnetic storage medium or magneto-optical storage medium, etc.), which can be, for example, a variable resistive memory (Resistive Random Access Memory, RRAM), a dynamic random access memory (Dynamic Random Access Memory, DRAM), a static random access memory (Static Random Access Memory, SRAM), an enhanced dynamic random access memory (Enhanced Dynamic Random Access Memory, EDRAM), a high bandwidth memory (High Bandwidth Memory, HBM), a hybrid memory cube (Hybrid Memory Cube, HMC), ROM and RAM, etc.

[0145] Although multiple embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art may think of many changes, modifications, and alternatives without departing from the thought and spirit of the present disclosure. It should be understood that in the process of practicing the present disclosure, various alternatives to the embodiments of the present disclosure described herein may be adopted. The attached claims are intended to define the scope of protection of the present disclosure, and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A method for implementing task splitting based on a multi-core processor, wherein: The method is applicable to group matrix multiplication type operation tasks, wherein the group matrix multiplication type operation tasks include multiple groups of matrix multiplications, and the matrix size of the matrix multiplication is variable; the method includes: Obtaining layout information of the group matrix multiplication type operation task; Based on the layout information, determining the task type of the group matrix multiplication task; wherein the task type includes a computing bottleneck type and a memory access bottleneck type; According to the task type, determine the split size block_k of the split block in the K dimension; wherein the K dimension is the column dimension of the left multiplication matrix and the row dimension of the right multiplication matrix; Based on block_k, obtain the split size block_n of the split block in N dimensions and the split size block_m of the split block in M ​​dimensions, so that the split block sizes block_k, block_n, and block_m meet the optimization target; wherein M is the row dimension of the left multiplication matrix, and N is the column dimension of the right multiplication matrix, and the optimization target includes any one of the following: memory access, processor core utilization; and The group matrix multiplication operation task is split according to the determined split block sizes block_k, block_n, and block_m, so as to be allocated to the operation units in the multi-core processor for parallel execution.

2. The method according to claim 1, wherein: The layout information includes: The matrix sizes and transposition information of the multiple groups of matrix multiplications; wherein the matrix sizes are the maximum values ​​m_max, k_max, and n_max of the M, K, and N dimensions, respectively.

3. The method according to claim 2, wherein: Obtaining the maximum value includes any of the following: In response to the dimension data being at the host side, searching for a maximum value therefrom; In response to the dimension data being on the device, obtaining the dimension data from the device and searching for a maximum value; or In response to the dimension data on the device side, the maximum value of each dimension is determined based on prior knowledge.

4. The method according to claim 2, wherein: Determining the split size block_k of the split block in the K dimension includes: In response to the group matrix multiplication operation task being a computational bottleneck task, setting block_k according to the transposition information to meet the efficiency requirements of the instructions for implementing the matrix multiplication operation supported by the multi-core processor; or In response to the group matrix multiplication task being a memory access bottleneck task, block_k is set according to the transposition information to preferentially improve the memory access efficiency of the largest matrix; wherein the largest matrix is ​​determined according to the size of the left multiplication matrix and the size of the right multiplication matrix.

5. The method according to claim 4, further comprising: the situation of the computing bottleneck task is: When the storage dimension where the K dimension is located is the lowest dimension, block_k is set to a multiple of the cache line size; the multiple is a natural number; or When the storage dimension where the K dimension is located is not the lowest dimension, block_k is set to a multiple of the alignment granularity align_k of the matrix multiplication operation instruction; wherein the multiple is a natural number.

6. The method according to claim 4, further comprising, in the case of the memory access bottleneck task: When the storage dimension where the K dimension is located is the lowest dimension, set block_k to a multiple of the cache line size; or When the storage dimension where the K dimension is located is not the lowest dimension, block_k is set to the alignment granularity align_k of the matrix multiplication operation instruction.

7. The method according to claim 2, wherein: The step of determining the task type of the group matrix multiplication task comprises: Determine the computing power limit according to the layout information of the group matrix multiplication operation task, the data bit width of the left multiplication matrix, and the data bit width of the right multiplication matrix; Determine the computing memory access ratio of the multi-core processor according to the bandwidth of the off-chip memory DRAM of the multi-core processor and the number of floating-point operations per second of one trillion times of the multi-core processor; The computing memory access ratio of the multi-core processor is compared with the computing power limit, and the task type of the group matrix multiplication type operation task is determined according to the comparison result.

8. The method according to claim 7, wherein: The step of determining the task type of the group matrix multiplication task according to the comparison result comprises: If the computing power limit is greater than the computing memory access ratio, the task type of the group matrix multiplication task is a computing bottleneck type; If the computing power limit is less than or equal to the computing memory access ratio, the task type of the group matrix multiplication type operation task is a memory access bottleneck type.

9. The method according to claim 1, wherein the method further comprises: A splitting strategy within the operational unit is determined.

10. The method according to claim 9, wherein determining the splitting strategy within a single computing unit comprises: Determine the number of processing cores Ncore in a single computing unit; The task block is split into pm*pn parts, where pm*pn=Ncore, pm is the number of splits in the M dimension, and pn is the number of splits in the N dimension.

11. The method according to claim 10, wherein the splitting comprises any of the following: Evenly distribute pm and pn in M ​​and N dimensions; or In response to any one of the M dimension and the N dimension in the task block being smaller than a corresponding threshold, the other dimension is split.

12. The method according to claim 1, wherein: The steps of obtaining the split block size block_n in N dimensions and the split block size block_m in M ​​dimensions include: Determine the maximum search space in N dimensions; Searching for a block_n that satisfies the optimization objective and determining a corresponding block_m in the maximum search space of the N dimensions; or Determine the maximum search space of M dimensions; In the maximum search space of the M dimension, block_m that meets the optimization goal is searched and the corresponding block_n is determined.

13. The method according to claim 12, wherein: The step of determining the maximum search space comprises: Determine a maximum search space in N dimensions according to the size of the largest block_n that can be accommodated due to the storage space limitation of the local memory of the multi-core processor and the size of the largest block_n that can be accommodated due to the storage space limitation of the shared memory of the multi-core processor; or Determine a maximum search space in the M dimension according to the size of the largest block_m that can be accommodated due to the storage space limitation of the local memory of the multi-core processor and the size of the largest block_m that can be accommodated due to the storage space limitation of the shared memory of the multi-core processor; Among them, the limitation of the storage space of the local memory of the multi-core processor results in that the size of the largest block_n or block_m that can be accommodated is determined based on the split size block_k of the split block in the K dimension, the number pk of blocks_k split when the software pipeline is implemented during the data transfer between the local memory l1_cache and the shared memory SRAM, the left multiplication matrix A, the right multiplication matrix B, the output matrix C, the alignment granularity of the instructions for implementing the matrix multiplication operation in the corresponding dimension, the capacity of the local memory of the multi-core processor, and the splitting strategy within a single operation unit; the limitation of the storage space of the shared memory SRAM of the multi-core processor results in that the size of the largest block_n or block_m that can be accommodated is determined based on the split size block_k of the split block in the K dimension, the left multiplication matrix A, the right multiplication matrix B, the alignment granularity of the instructions for implementing the matrix multiplication operation in the corresponding dimension, the capacity of the shared memory of the multi-core processor, and the splitting strategy within a single operation unit.

14. The method according to claim 12, wherein: The steps of searching include: In the maximum search space, the corresponding block_m or block_n is determined according to the number of searches i in the current search loop, the splitting strategy within a single operation unit, and the alignment granularity of the instructions implementing the matrix multiplication operation in the corresponding dimension; Based on the corresponding block_m, determine the size of the storage block_n of the local memory corresponding to the multi-core processor and the size of the storage block_n of the shared memory corresponding to the multi-core processor, and determine the N-dimensional split size block_n in the current search loop according to the size of the storage block_n of the local memory corresponding to the multi-core processor and the size of the storage block_n of the shared memory corresponding to the multi-core processor; or, based on the corresponding block_n, determine the size of the storage block_m of the local memory corresponding to the multi-core processor and the size of the storage block_m of the shared memory corresponding to the multi-core processor, and determine the M-dimensional split size block_m in the current search loop according to the size of the storage block_m of the local memory corresponding to the multi-core processor and the size of the storage block_m of the shared memory corresponding to the multi-core processor; Calculate the corresponding optimization target according to the N-dimensional split size block_n and the M-dimensional split size block_m corresponding to the current search loop; and According to the optimization target corresponding to each search cycle in the search space, the N-dimensional split size block_n and the M-dimensional split size block_m corresponding to the optimal optimization target are determined.

15. The method according to claim 14, wherein: The optimal optimization objectives include: When the group matrix multiplication operation task is a memory access bottleneck task, the optimal optimization target is the minimum memory access amount in the maximum search space; When the group matrix multiplication type operation task is a computing bottleneck type task, the optimal optimization target is the maximum processor core utilization in the maximum search space.

16. The method according to claim 1, wherein: The memory access amount is determined based on the layout information of the group matrix multiplication operation task, the split size block_n of the split block of the group matrix multiplication operation task in N dimension, the split size block_m of the split block of the group matrix multiplication operation task in M ​​dimension, and the left multiplication matrix and the right multiplication matrix in the group matrix multiplication operation task.

17. The method according to claim 1, wherein: The processor core utilization is determined based on the computational memory access ratio of the multi-core processor itself, the split size block_n of the group matrix multiplication type operation tasks in N dimensions, the split size block_m of the group matrix multiplication type operation tasks in M ​​dimensions, the total number of split blocks of the group matrix multiplication type operation tasks, and the number of operation units of the multi-core processor.

18. A computer-readable storage medium storing program instructions, which, when loaded and executed by a processor, enables the processor to execute the task splitting method based on a multi-core processor according to any one of claims 1-17.

19. A computer program product, comprising a computer program or instructions, wherein when the computer program or instructions are executed by a processor, the method for implementing task splitting based on a multi-core processor according to any one of claims 1-17 is implemented.

20. A device for implementing task splitting based on a multi-core processor, comprising a processor and a memory, wherein: The processor is configured to execute program instructions; The memory is configured to store the program instructions; When the program instructions are loaded and executed by the processor, the processor executes the method for implementing task splitting based on a multi-core processor according to any one of claims 1-17.

Images