A method for data scheduling for a systolic array

Abstract

The application discloses a data scheduling method for a systolic array and belongs to the technical field of neural networks.The application comprises the following steps: step 1, distributing data A to be convolved / matrix-multiplied on m cores; step 2, broadcasting data B to the m cores, and the m cores each obtain the same b to perform calculation; step 3, when the distributed data A on each core is calculated with all the b, the result is written back to the main memory; and step 4, repeating steps 1-3 to perform calculation on the next part of the data A.The application can effectively improve the reuse times of on-chip data, reduce the bandwidth requirement, and effectively improve the performance of a hardware accelerator.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of neural networks, and particularly relates to a data scheduling method for a systolic array. BACKGROUND

[0002] With the increasing size of deep neural network models and data sets, the demand for hardware memory bandwidth is increasing dramatically, and the memory bottleneck is becoming increasingly severe. One of the common methods in academia and industry is to use a systolic array to accelerate matrix multiplication / convolution operations. The characteristic of the systolic array is that data flows between computing units in the array, which can effectively increase the number of data reuse, reduce the amount of memory access, and thus reduce the bandwidth demand.

[0003] Based on the Sunway-AI acceleration chip structure, a multi-style scheduling strategy for a systolic array is proposed. The Sunway-AI chip has a two-level array structure: m processing cores, each core has an n*n dimensional systolic array calculation. The two-level array structure provides a multi-style mapping space for convolution and matrix multiplication algorithm mapping calculation. However, there is currently a lack of a data scheduling method that further increases the on-chip data reuse rate and reduces repeated data interaction with off-chip data to further reduce bandwidth demand and improve hardware accelerator performance. SUMMARY

[0004] The purpose of the present application is to solve the problems existing in the prior art, and to provide a data scheduling method for a systolic array, which can effectively improve the reuse number of on-chip data, reduce the bandwidth demand, and effectively improve the performance of the hardware accelerator.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] A data scheduling method for a systolic array, defining the formula of convolution calculation / matrix multiplication calculation as A*B=C, A, B, C are all multi-dimensional vectors, the data scheduling method comprising:

[0007] Step 1, distributing the data A to be convolved / matrix multiplied on m cores;

[0008] Step 2, broadcasting data B to m cores, and m cores get the same b for calculation each round;

[0009] Step 3, when the distributed data A on each core is calculated with all b, write the result back to the main memory;

[0010] Step 4, repeating steps 1-3 to calculate the next part of data A.

[0011] The above method achieves the highest reuse rate for distributed data A, meaning no repeated off-chip memory accesses occur. While broadcast data B does require repeated off-chip memory accesses, its memory access mode is broadcast, requiring less bandwidth. Alternatively, data A can be broadcast while data B is distributed.

[0012] As a preferred embodiment of the present invention, if the data A merging direction includes n rows / columns of data, the data scheduling method includes:

[0013] Step 1: Divide the m cores into g groups, with n groups of cores as a unit. The number of cores in each group corresponds to the number of data A in each row / column.

[0014] Step 2: distribute the data A to be convolved / matrix multiplied on m cores;

[0015] Step 3: broadcast each column / row data B to g / n units respectively, and broadcast data B among n groups of cores in the same unit;

[0016] Step 4: After each group of cores completes a round of calculation, data A is exchanged between the cores in the group until the complete calculation result is obtained;

[0017] In step 5, each core writes the complete calculation results back to the main memory.

[0018] The above method applies only to cases where data A is large in the merging direction. Considering the merging direction of data A (e.g., in convolution calculations, grouping feature maps by channel direction), each core cannot obtain a complete computation result from the distributed data A retrieved from main memory. Distributed data A must be exchanged between cores to continue the merging calculation, and the complete result must be written back to main memory. Compared to off-chip memory access bandwidth, the on-chip network has a higher memory access bandwidth. Inter-core data exchange can significantly increase data reuse without affecting computational efficiency, reducing off-chip bandwidth requirements. Broadcast data B is partially broadcasted by rows or columns based on the distribution of distributed data A, further reducing off-chip memory access. Similarly, this method also applies only to cases where data B is large in the merging direction.

[0019] As a preferred embodiment of the present invention, if the merging directions of data A and B both include multiple rows / columns of data, the data scheduling method includes:

[0020] Step 1: Let the data A and B to be convolved / matrix multiplied be distributed on m cores;

[0021] Step 2: Each core calculates different A and B respectively;

[0022] Step 3: The calculation results of the m cores are merged through inter-core communication to obtain the complete result and then written back to the main memory.

[0023] The method is aimed at the case that the sizes of the merging direction of data A and data B are large, and the merging direction of both needs to be split, that is, both data need to be distributed on m slave cores, that is, the data obtained by each core is a0b0, a1b1, …, a m-1 b m-1 After the intermediate result is calculated, the complete result is obtained by merging through inter-core communication, and then written back to the main memory.

[0024] The advantage of the present application is that the broadcast mechanism and on-chip memory mechanism are used to reduce the repeated access of off-chip data and the access of intermediate values, increase the reuse times of on-chip data, reduce the bandwidth demand, and effectively improve the performance of the hardware accelerator. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 A schematic diagram of a first strategy of the data scheduling method for the systolic array of the present application;

[0026] Figure 2 A schematic diagram of a second strategy of the data scheduling method for the systolic array of the present application;

[0027] Figure 3 A schematic diagram of a third strategy of the data scheduling method for the systolic array of the present application. DETAILED DESCRIPTION

[0028] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0029] The present application provides a data scheduling method for a systolic array, which is combined with the calculation characteristics of convolution / matrix multiplication on the systolic array, and mainly has three scheduling strategies for the data A and B to be convolved / matrix multiplied: 1, A data distribution+B data broadcast (or A data broadcast+B data distribution); 2, A data group distribution+B data group broadcast (or A data group broadcast+B data group distribution); 3, A data distribution+B data distribution.

[0030] The formula of the convolution calculation / matrix multiplication calculation is defined as A*B=C, and A, B and C are all multi-dimensional vectors. The three scheduling strategies will be introduced respectively.

[0031] Taking the first scheduling strategy as an example, the first scheduling strategy includes the following steps: Figure 1

[0032] Step 1, distribute the data A to be convolved / matrix multiplied on m cores, that is, the data obtained by each core is a0, a1, …, a m-1 ;

[0033] ​Step 2, broadcast data B to m cores, m cores get the same b in each round to calculate, that is, the calculation process of each core is a0*B=c0, a1*B=c1 … a m-1 *B=c m-1 .

[0034] Step 3, when the distributed data A on each core and all b are calculated, write the result back to the main memory;

[0035] Step 4, repeat steps 1-3 to calculate the next part of data A, for example, continue a0'*B=c0', a1'*B=c1' … a m-1 '*B=c m-1 '.

[0036] The above method has the highest reuse rate for distributed data A, that is, there is no repeated off-chip memory access, so data A is generally a relatively large matrix data; Broadcast data B appears repeated off-chip memory access, so data B is generally a relatively small matrix data, and considering its memory access mode is broadcast, the demand for memory bandwidth is lower.

[0037] The second scheduling strategy is for the case where the size of the merging direction of data A is large. The merging direction of data A is divided (for example, in convolution calculation, the feature map is divided in the channel direction for grouping), and the distributed data A obtained by each core from the main memory cannot obtain complete calculation results. It needs to exchange the distributed data A between cores to continue the calculation in the merging direction, and then write the complete result back to the main memory. Take Figure 2 for example, the complete data A includes two rows of data, which are divided into two groups along the merging direction, that is, a0-a3 is a group and a4-a7 is a group. Two groups of cores calculate 1 / 2 of data A respectively. This method includes the following steps:

[0038] Step 1, divide m cores into g groups, and take n groups of cores as a unit. The number of cores in each group corresponds to the number of data A on each row / column; Figure 2 Take two groups of cores as an example, cores 0, 1, 2, and 3 form a group, and cores 4, 5, 6, and 7 form a group, which form a unit.

[0039] Step 2, distribute the data A to be convolved / matrix multiplied on m cores, that is, cores 0-7 are distributed to data a0-a7.

[0040] Step 3, broadcast each column / row of data B to g / n units, and broadcast data B between n groups of cores in the same unit; that is, cores 0-3 get data B1-B3, and through broadcast between the two groups, cores 4-7 also get data B1-B3.

[0041] Step 4, after each group of cores completes a round of calculation, data A is exchanged between the cores in the group until the complete calculation result is obtained; the following is four rounds of calculation performed by cores 0-3 in turn:

[0042] First round of calculation, a0*B0, a1*B1', a2*B2'', a3*B3''''; after that, data A is exchanged between the cores, for example, data A is shifted right as a whole;

[0043] Second round of calculation, a3*B3, a0*B0', a1*B1'', a2*B2''''; data A continues to be shifted right as a whole;

[0044] Third round of calculation, a2*B2, a3*B3', a0*B0'', a1*B1''''; data A continues to be shifted right as a whole;

[0045] Fourth round of calculation, a1*B1, a2*B2', a3*B3'', a0*B0'''''.

[0046] Step 5, each core writes the complete calculation result back to the main memory; after the above four rounds of calculation, the four cores have obtained the complete calculation result, which can be written back to the main memory, and the final calculation result of data A is obtained by combining the calculation results of the above two groups of cores.

[0047] Data A in the above method is generally a picture matrix, which has a large matrix and requires a high memory bandwidth. We use inter-core exchange to reuse data A, because the on-chip network has a larger memory bandwidth compared with the off-chip memory bandwidth, and the inter-core data exchange can greatly improve the data reuse rate and reduce the off-chip bandwidth requirement without affecting the calculation efficiency. Data B is generally a convolution kernel matrix, which has a small matrix and requires a low memory bandwidth, so we use off-chip broadcasting to reuse data B, and perform partial row broadcasting or column broadcasting according to the distribution mode of distributed data A to further reduce the off-chip memory.

[0048] The third scheduling strategy is for the case where the merging direction size of data A and B is large, for example, Figure 3 The third scheduling strategy includes the following steps:

[0049] Step 1, distribute data A and B to be convolved / matrix multiplied on m cores, that is, each core obtains data A as a0, a1… a m-1 , and obtains data B as b0, b1… b m-1 ;

[0050] Step 2, each core calculates different A and B, that is, the calculation performed on each core is a0b0, a1b1… a m-1 b m-1That is, only partial values can be calculated on each core, and neither can obtain complete results;

[0051] Step 3: the calculation results of the m cores are merged through inter-core communication to obtain complete results, and then written back to the main memory.

[0052] The above method mainly uses inter-core communication to merge the intermediate results obtained by calculation to obtain complete results.

[0053] The above described is only a preferred specific embodiment of the present application, which is an implementation based on the overall concept of the present application, and the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data scheduling method for a systolic array, defining a convolution calculation / matrix multiplication calculation formula as A*B=C, where A, B, and C are all multidimensional vectors, characterized in that: Data A is an image matrix, the merging direction of data A includes n rows / columns of data, and data B is a convolution kernel matrix. The data scheduling method includes: Step 1: Divide the m cores into g groups, with n groups of cores as a unit. The number of cores in each group corresponds to the number of data A in each row / column. Step 2: distribute the data A to be convolved / matrix multiplied on m cores; Step 3: broadcast each column / row data B to g / n units respectively, and broadcast data B among n groups of cores in the same unit; Step 4: After each group of cores completes a round of calculation, data A is exchanged between the cores in the group until the complete calculation result is obtained; In step 5, each core writes the complete calculation results back to the main memory.

Images