Method for splitting a convolution operation

Abstract

A method for splitting a convolution operation includes obtaining input data tensors and input weight tensors from a high bandwidth memory, splitting said tensors into several parts, reserving the necessary amount of matrix memory, successively loading the input data parts from the high bandwidth memory into a vector memory, splitting the input data parts into several sub-parts taking into account the reserved amount of memory, and loading the sub-parts into the memory. The weight parts are successively loaded into the memory at the same time as the loading of the input data sub-parts, the input data and the weights, read from the matrix memory, are processed, and sub-parts of a resulting tensor are obtained. The parts of the resulting tensor are loaded into the high bandwidth memory and at the same the input data sub-parts and the weight parts are processed by an array multiplier based on systolic arrays. The result is an improvement in the tiling efficiency of a convolution operation.

Description

METHOD OF SEPARATING THE CONVOLUTION OPERATION AREA OF TECHNOLOGY

[0001] The claimed solution relates to the field of parallel computer computing, in particular to tensor computing and the devices that implement it. LEVEL OF TECHNOLOGY

[0002] The convolution operation (also known as "convolution" [1]), as well as the matrix multiplication operation (as a special case of the convolution operation), are key tools in the operation of modern neural networks. These operations are functions of two arguments: "features" (from English, "feature") - input data from the user or from other operations; "weights" - a matrix of multiplication coefficients obtained during the neural network training phase.

[0003] Systolic array-based devices are designed to execute this type of operation several orders of magnitude faster than modern general-purpose processors (CPUs). In modern tasks involving executing large language models (LLMs) and image recognition and generation, the dimensions of the input data are many times larger than the internal memory (cache) capacity of the execution devices.

[0004] The prior art discloses the use of tiling in systolic arrays to optimize computations in tasks such as image and video processing, where the volume of processed data often exceeds the memory capacity, as well as the influence of tiling on ensuring maximum performance of systolic arrays [2].

[0005] A disadvantage of the prior art is the insufficient efficiency of memory optimization for computing large-sized convolutions. ESSENCE OF THE INVENTION

[0006] The proposed solution allows solving the technical problem of memory optimization for computing large-sized convolutions by ensuring uninterrupted operation of systolic arrays, allowing simultaneous load data for the next tile into the matrix multiplier memory and take into account the specific features of the memory in the vector computing device.

[0007] The technical result is an increase in the productivity of the process of dividing (tiling) a convolution operation, performed on at least one computing device with matrix multipliers based on systolic arrays.

[0008] The claimed technical result is achieved by implementing a method for dividing a convolution operation, performed on at least one computing device with matrix multipliers based on systolic arrays, and containing the following stages: • obtain tensors of input data and input weights, which are arguments of functions for the convolution operation, from high-bandwidth memory, • divide the input data and weight tensors into several parts taking into account the volume of vector and matrix memory, respectively, • reserve the required amount of matrix memory for subsequent loading of parts of the scales, • alternately load parts of the input data from high-bandwidth memory into vector memory, • carry out additional division of parts of the input data into several sub-parts, taking into account the volume of matrix memory reserved for the weights, and alternately load the specified sub-parts of the input data into the matrix memory, • parts of the scales are loaded into the matrix memory one by one, and the loading of parts of the scales into the matrix memory is carried out in parallel with the loading of subparts of the input data into the matrix memory, • processing of subparts of the input data and parts of the weights read from the matrix memory is carried out by at least one matrix multiplier based on systolic arrays, as a result of which subparts of the resulting tensor are obtained, • loading portions of the resulting tensor into high-bandwidth memory in parallel with processing subsequent sub-portions of the input data and portions of the weights by at least one matrix multiplier based on systolic arrays.

[0009] In one particular example of implementation, at the stage of obtaining the input weight tensors, the weights of at least one convolution operation are converted into the required format for matrix memory.

[0010] In another particular example of implementation, during the processing of subparts of the input data and parts of the weights by at least one matrix multiplier based on systolic arrays, the next part of the input data is loaded in parallel from the high-bandwidth memory into the vector memory, as well as the next part of the weights from the high-bandwidth memory into the matrix memory. [OOP] In another particular implementation example, obtaining a subpart of the resulting tensor additionally contains: • obtaining the results of a matrix multiplier based on systolic arrays and their sequential summation by the first scalar operations module, followed by loading into the memory of at least one scalar operations module, • storing the result of the summation, as a subpart of the resulting tensor, in vector memory.

[0012] In another particular example of implementation, summing the results of the matrix multiplier based on systolic arrays further comprises adding the first result of the matrix multiplier based on systolic arrays with a constant offset using a first module of scalar operations.

[0013] In another particular implementation example, during the summation of the last result of a matrix multiplier based on systolic arrays, an activation function and redundant addition are performed on a subpart of the resulting tensor using a second scalar operations module. BRIEF DESCRIPTION OF DRAWINGS

[0014] Fig. 1 shows the general diagram of a vector computing device.

[0015] Fig. 2 shows a block diagram of a method for dividing a convolution operation performed on at least one computing device with matrix multipliers based on systolic arrays. IMPLEMENTATION OF THE INVENTION

[0016] Below, concepts and terms necessary for understanding the present invention will be described.

[0017] Tiling is the division of an operation's input data into parts and the execution of operations equivalent to the original parts on these parts to simplify the operation. Tiling is necessary under limited memory conditions and is also used to improve performance by pipelining tiles.

[0018] A tile (from English, tile) is a part of the input data of an operation that is obtained during the division (tiling) process.

[0019] A convolution operation (or convolution) is an operation applied to an input tensor (image) that returns a tensor whose each value for a given coordinate represents a weighted sum of the input tensor's values ​​in the neighborhood of that coordinate. The weights for calculating the weighted sum are computed during the neural network training phase.

[0020] A systolic array is a homogeneous network of tightly connected data processing units (DPUs), called cells or nodes. Each node independently and in parallel computes a partial result as a function of the data received from its upstream neighbors, stores the result internally, and transmits it to downstream nodes [3].

[0021] A tensor is a multidimensional data structure.

[0022] A vector processor is a processor in which the operands of some commands can be ordered arrays of data—vectors [4].

[0023] High Bandwidth Memory (HBM) is a high-performance RAM interface for DRAM with a multilayer arrangement of crystals in a microassembly [5].

[0024] An Address Pattern Generator (APG) is an addressing mechanism that allows a single instruction / command to provide read and write access cyclically over specified step values ​​and step counts.

[0025] Fig. 1 shows a general view of a computing device with matrix multipliers based on systolic arrays, with the help of which a method for dividing the convolution operation is implemented.

[0026] In general, a computing device is vector-based, meaning it supports simultaneous computation of an array (vector) of data. The vector size Vector Length (VL) is typically a power of 2 (16, 32, 64, 128), meaning the device can execute VL floating-point operations in one machine cycle. The vector computing unit contains at least two matrix multipliers based on systolic arrays (SA), each capable of processing VL data vectors simultaneously for matrix multiplication, and the SA can calculate VL*VL (e.g., for VL=128: 128*128=16384) floating-point multiplication and addition operations in one machine cycle.

[0027] The vector computing device consists of at least the following modules (Fig. 1):

[0028] Vector Memory (VE MEM) is shared SRAM memory between computing modules (VSa, VSb), typically several tens of megabytes in size. Vector addressing is supported via the APG (Address Pattern Generator), allowing different ports to simultaneously read and write data to different memory locations (speed up to 1 clock cycle per vector memory).

[0029] Matrix Engine Memory (ME_MEM) is SRAM memory for SA data, typically smaller in size than VE MEM, designed to allow SAs to access required data more quickly. Each SA has a separate ME_MEM. The access speed is one data vector per machine cycle. Matrix memory can be written from both HBM and VE MEM.

[0030] - Systolic array based matrix multiplier (Systolic Array (SA), which performs matrix multiplication of input data and weights. The input data and weights are read from matrix memory (ME MEM). Generally, a minimum of VL machine cycles is required to initialize (load) the weights. Weights can also be updated while the results of previous weights are being calculated, thereby reducing the VL cycle downtime between instructions. To calculate matrix multiplication and convolution operations on SA, the input channels (the last dimension of the input data) and output channels (the last dimension of the result) must be multiples of VL (i.e., equal to 1VL, 2VL, 3VL, etc.).

[0031] - High Bandwidth Memory (HBM) is a high-speed external memory with a capacity of several gigabytes and sequential addressing. The memory controller also allows addressing 1 data vector per 1 machine cycle (for example, for VL=128 and data in FP24 format [6], the required HBM memory speed for a 1 GHz device is: VL * 3 bytes * 10 л 9 = 384 GB / s). The vector computing device supports reading from high-bandwidth memory (HBM) to vector (VE_MEM) and matrix (ME MEM) memory, while writing to HBM is supported only from vector memory (VE_MEM). Unlike vector and / or matrix memories, simultaneous reading and writing from HBM <=> VE MEM is not supported. Reading and writing from / to HBM is performed by the DMA (Direct Memory Access) controller.

[0032] - Vector Subunit (VS). In the vector computing unit, after executing the matrix multiplication SA, the ability to pipeline data to at least two sequential scalar subunits (VSa, VSb) is implemented. The first scalar subunit (VSa) is used to add the result of the convolution operation with a constant bias (bias), as well as to sum the results with SA to obtain the final convolution result. The second scalar subunit (VSb) is most often used to calculate activation (RELU, Sigmoid, etc.) and addition with "redundant linkage," where the data for "redundant linkage" is fed from vector memory (VE_MEM) to the second scalar subunit (VSb) during execution.

[0033] - VS MEM memory is SRAM memory for each scalar operation module, storing intermediate results of SA calculations and ensuring fast and virtually conflict-free data access. This memory also efficiently stores the bias value to enable efficient addition. The size of VS MEM memory is comparable to the size of vector memory (VE MEM), but the size of VS MEM memory can be reduced by a factor of 2, since the output data size is approximately the same as the input data size, but pipelining within VS_MEM memory is not required. VS_MEM memory is integrated within each scalar operation module (VSa, VSb).

[0034] Each of the specified modules of the vector computing device has a hardware embodiment, for example, in the form of a system on a chip (System-on-a-Chip, SoC).

[0035] When describing the operation of the convolution splitting method, the following conventions are used:

[0036] NHWC - characterizes the dimensionality of the input tensor of the convolution operation. For example, for the image set NHWC={2, 480, 640, 3}, N=2 denotes the number of images (for the example under consideration, 2 images), H=480 characterizes the height of the image, W=640 means the width of the image, C=3 characterizes the number of channels (for the example under consideration - 3 channels).

[0037] For text data, three-dimensional NHC tensors are used, where N is the batch, H is the word / token number in the text, and C is the token characteristic. Without loss of generality, three-dimensional NHC tensors can be considered four-dimensional NHWC tensors, where W = 1.

[0038] KRSC - characterizes the dimension of the input weight tensor, for example, KRSC = {64,7,7,3}. K denotes the output channels, R characterizes the height size of the convolution kernel, S is the width size of the convolution kernel, the last dimension coincides with "C" of the NHWC input data tensor and denotes the number of channels.

[0039] NPQK - characterizes the size of the resulting tensor, where P is the height of the result, Q is the width of the result, K characterizes the output channels, N characterizes the batch (for computer vision tasks, N is the number of images, for text processing tasks, N is the number of text fragments).

[0040] The convolution operation is also characterized by additional parameters: • Stride along the H / W axes: sH, sW. • Dilation: step within the convolution kernel: dH, dW • Paddings: pad begin (pbH / pbW - paddings on the top and left), pad_end (peH / peW - paddings on the right and bottom).

[0041] For convenience of calculations in vectors, since the computing device with matrix multipliers based on systolic arrays works exclusively with data vectors (in the preferred embodiment, with VL = 128 numbers in each), additional notations are introduced: Kv and Cv are the sizes of K and C in vectors, respectively. For example, for the input tensor NHWC (1x112x112x256), we assume that Cv = 256 / 128 = 2. Similarly, for K: weight KRSC = 512x3x3x256, we assume that Kv = 512 / 128 = 4.

[0042] In the context of the above, it should be noted that the use of an address generator (APG) allows addressing a subtensor for a data buffer located at sequential addresses. A data buffer (weights, input data, result) refers to a memory area. characterized by the starting address of the buffer in memory and the size of the buffer. For example, if the current input data buffer is { 10, 20, 30}, to address the sub-tensor {5, 10, 15} from it, you need to specify an APG of the form (5, 60)(10, 30)(15, 1), where 5, 10, 15 are the sizes of the new tensor, 60 is the step for the first (main / outer) significant axes (=20*30), 30 is the step along the second significant axis, 1 is the step along the last (innermost) axis.

[0043] Within the framework of this application, APG can be effectively used for: • addressing part of the input data NHWCv when copying from vector (VE_MEM) to matrix (ME_MEM) memory “NHWcv”, where cv is part of the input channels Cv (for example Cv= 4, cv=2); • when addressing parts of the NPQkv result for writing to high-bandwidth memory (HBM) as NPQKv in one instruction.

[0044] For existing modern convolution operations (e.g., with an input size of 1x2048x2048x128), it is necessary to develop a tiling mechanism into multiple convolution operations, taking into account the memory sizes of VE MEM, VS MEM, and ME MEM. It is also necessary to consider that the weights in convolution operations are obtained during the neural network training stage and do not change depending on the input data (i.e., they are constant) during subsequent use.

[0045] It should be noted that for the case where all tensors fit in internal memory (VE_MEM, VS MEM, ME_MEM), tiling the convolution operation is not required. However, for very large tensors that do not fit in ME_MEM or even VE MEM, the pipelined convolution calculation mechanism becomes more complex.

[0046] Fig. 2 shows a method for tiling a convolution operation performed on at least one computing device with matrix multipliers based on systolic arrays. The computing device with matrix multipliers based on systolic arrays can be, but is not limited to, the vector computing device disclosed above with reference to Fig. 1.

[0047] At the first stage (101), tensors of input data (features) and input weights, which are the arguments of the functions for the convolution operation, are obtained from high-bandwidth memory (HBM).

[0048] In a particular implementation example, at the stage of obtaining the input weight tensors, the weights of at least one convolution operation are converted into the required format for matrix memory (ME_MEM). In the preferred embodiment, the required format is, but is not limited to, the BF16 (bfloatl6) format [7].

[0049] Since the weights do not change depending on the input data, they can be pre-converted to the required format for more efficient subsequent access. Pre-processing the convolution weights into the required format for internal memory reduces the weight loading time into the matrix memory (ME MEM) by approximately 2 times, further improving the performance of the convolution tiling process.

[0050] At step (102), the input data and weight tensors are divided into several parts, taking into account the volume of vector (VE MEM) and matrix (ME_MEM) memories, respectively, in such a way that writing some parts (tiles) is possible simultaneously with reading other parts (tiles) without memory conflicts.

[0051] Since only sequential access to the HBM is supported, it is only possible to read parts of { 1, 1, 1, c_step}, or { 1, 1, w_step, C}, or { 1, h_step, W, C}, or { n_step, H, W, C}. In other words, it is impossible to read a 2x2x128 sub-tensor from a 2x2x256 tensor with a single HBM instruction, since it is located in memory as a non-sequential buffer.

[0052] At step (103), the required amount of matrix memory (ME_MEM) is reserved for subsequent loading of scale parts. Since loading scale parts and input data subparts into matrix memory (ME_MEM) occurs in parallel, a step of reserving the required amount of matrix memory (ME_MEM) for scale parts is necessary to avoid memory conflicts in matrix memory (ME_MEM).

[0053] At step (104), parts of the input data are sequentially loaded from the high-bandwidth memory (HBM) into the vector memory (VE_MEM). In a particular implementation example, parts of the input data are stored in the vector memory (VE_MEM) in FP24 format [6].

[0054] At step (105), an additional division of the input data parts into several sub-parts is carried out taking into account the volume of matrix memory (ME MEM) reserved for the weights, and the specified sub-parts of the input data are loaded one by one into the matrix memory (ME MEM).

[0055] Additionally, the division into subparts of the input data is related to the fact that high-bandwidth memory (HBM) does not support APG addressing, i.e. from HBM it is impossible to read a subpart of the input data (NHWC_0_0) with one instruction; it is possible to read only part of the input data (NHWC_0).

[0056] Thus, the division (cutting) of weight tensors and input data is carried out across all supported dimensions, taking into account the size of internal memories (for example, by reserving the required amount of matrix memory) and specific features of the memories (for example, the lack of APG support in HBM), which helps to optimize the operation of the division (tiling) method of the convolution operation and, as a result, improve its performance.

[0057] At step (106), the weight subsets are sequentially loaded into the matrix memory (ME_MEM), wherein the loading of the weight subsets into the matrix memory (ME_MEM) is performed in parallel with the loading of the input data subsets into the matrix memory (ME_MEM). The parallel loading of the weight subsets and the input data subsets into the matrix memory (ME_MEM) improves the performance of the convolution operation tiling method by optimizing the loading (filling) of the matrix memory (i.e., ensuring that the total of the input data subset and the weight subset do not exceed the volume of the ME_MEM). This advantage is achieved by reserving the required amount of matrix memory for the weight subsets and taking this reservation into account when further dividing the input data subsets into subsets.

[0058] At step (107), subparts of the input data and parts of the weights read from the matrix memory (ME_MEM) are processed by at least one matrix multiplier based on systolic arrays (SA), resulting in subparts of the resulting tensor (NPQK). In a preferred embodiment of the claimed solution, the processing at step (107) is performed by two matrix multipliers based on systolic arrays (SA).

[0059] In one particular example of implementation, during the processing of subparts of the input data and parts of the weights by at least one matrix multiplier based on systolic arrays (SA), the next part of the input data is loaded in parallel from the high-bandwidth memory (HBM) into the vector memory (VE_MEM), as well as the next part of the weights from the high-bandwidth memory (HBM) into the matrix memory (ME MEM). Loading the next part of the input data from the HBM into the VE MEM and the next part of the weights from the HBM into the ME MEM in parallel with the processing at step (107) makes it possible to reduce the downtime of the at least one matrix multiplier based on systolic arrays (SA) due to the continuous receipt of data for processing, which additionally helps to improve the performance of the tiling process of the convolution operation.

[0060] In a particular embodiment, obtaining a subpart of the resulting tensor (NPQK) additionally comprises: • obtaining the results of the matrix multiplier based on systolic arrays (SA) and their sequential summation by the first module of scalar operations (VSa) with subsequent loading into the memory (VS_MEM) of at least one module of scalar operations (VS), • storing the result of the summation, as a subpart of the resulting tensor (NPQK), in vector memory (VE MEM).

[0061] It should be noted that VS MEM does not store all the results of the matrix multiplier (e.g., a, b, c, d, e), but only the sum of all the results of the matrix multiplier (first a, then a+b, then a+b+c, etc.). Thus, the memory in VS_MEM is only needed to store one result of the matrix multiplier.

[0062] In one particular example of implementation, the summation of the results of a matrix multiplier based on systolic arrays (SA) further comprises the summation of the first result of the matrix multiplier based on systolic arrays (SA) with a constant bias using the first scalar operations module (VSa).

[0063] In another particular implementation example, during the summation of the last result of the matrix multiplier based on systolic arrays (SA), an activation function and addition with a residual connection (residual connection [8]) are performed on a subpart of the resulting tensor (NPQK) using the second scalar operations module (VSb).

[0064] At step (108), parts of the resulting tensor (NPQK) are loaded into a high-bandwidth memory (HBM) in parallel with the processing of the following subparts of the input data and parts of the weights by at least one matrix multiplier based on systolic arrays (SA).

[0065] Ensuring the continuity of the tiling process of the convolution operation by implementing the loading of a part of the resulting tensor (NPQK) into the HBM in parallel with the processing of the following subparts of the input data and parts of the weights by at least one matrix multiplier on SA helps to improve the performance of the tiling process of the convolution operation.

[0066] The method for splitting the convolution operation continues as long as input data tensors and input weights continue to arrive from the high-bandwidth memory (HBM). Steps (106)–(108) are repeated for new parts of the input data.

[0067] To evaluate the effectiveness of the claimed method for dividing the convolution operation, the following criteria are used.

[0068] For the input NHWC, weights KRSC and result NPQK, the number of machine cycles required to compute the convolution operation on one SA without stalling is given by: CyclessA = N*P*Q*K v *R*S*Cv

[0069] Or, for simplicity, for matrix multiplication (H=W=P=Q=R=S=1), N>128 (otherwise there will be downtime when switching weights): CyclessA = N * K v * C v

[0070] In the case of two SAs, the number of cycles is reduced by 2 times.

[0071] When dividing the input data into kF parts and the weights into kW parts, the load on different memories will be: Weights (HBM->ME_MEM, reset kF times): Cycles w HBM = K v * C v * kF * VL Input data (HBM->VE_MEM + VE_MEM->HBM): Cyclesr nvm = N * C v + N * K v VE_MEM->ME_MEM: Cyclesr MEMEM = N * C v - always less than Cyclesr nvm

[0072] From the formulas it is clear that it is possible to compensate for the load on memory during useful calculations only if K v > 1 and C v> 1 , otherwise it will not be possible to compensate for the load on the NVM for the input data. It is also necessary that kF * VL < N, otherwise it will not be possible to compensate for the load on the NVM for the weights.

[0073] In the case of a convolution operation, the formulas for calculating the clock cycles are similar (for non-unit kernels, in most cases, overlaps in H and W can be neglected without significant loss of accuracy): Cyclesw_HBM = K v *R*S*Cv*kF* VL CyclesF ​​HBM = N*H*W*C v + N*P*Q*Kv

[0074] In this case, requirement C v > 1 weakens to C v * R * S > 1, and kF*VL < N*P*Q. For large convolution steps, sH > 1 or sW > 1 will result in H*W > P*Q and the H*W requirement must be met. <P * Q *Kv*R* S, что не всегда возможно обеспечить.

[0075] Results of experiments.

[0076] For a typical convolution operation in the ResNet50 network with parameters VL=128, NHWC={N, 56, 56, 128}, KRSC={ 128, 3, 3, 128}, NPQK={ 1,56,56,128}, sH=sW=dH=dW=pbH=beH=pbW=peW=l, the load on the HBM will be CyclesF ​​HBM = N * 56 * 56 * 1 + N * 56 * 56 * 1 = 6272*N, the load on the weights (kF*VL=N) will be CyclesW_HBM = 128*3*3*N = 1152*N, CyclessA = N * 56 * 56 * 1 * 3 * 3 * 1 = 28224*N, or 14112 * N for two SA.

[0077] From the above experimental data it follows that for this convolution operation it is possible to compensate the load on the HVM with a large margin.

[0078] Experiments on a cycle-by-cycle model of the device (using a specialized program - a cycle-by-cycle simulator), taking into account downtime when switching weights and memory conflicts, showed an efficiency of 96% for this example with N = 64: CyclessA = 903168, the total number of cycles on the cycle-by-cycle simulator taking into account loading the offset, the first part of the weights and unloading the last part of the results = 950970 cycles.

[0079] Without implementing this method, when calculating each portion of input data (features) separately (taking into account the parallelization of calculations on two SAs), the total number of cycles on the cycle-by-cycle simulator was approximately 1.86 million cycles, i.e. the declared method of dividing (tiling) the convolution operation with achieving maximum load is approximately 1860 / 950.97 = 1.9 times more efficient.

[0080] Below is a step-by-step description of the algorithms used.

[0081] 1. Algorithm for selecting the NPQK / KRSC tensor slicing coefficients along the axes. Coefficients {dst_n, dst_p, dst_q, dst_k} are the output tensor slicing (dst - destination), {w_k, w_r, w_s, w_c} are the slicing along the weights (w-weights).

[0082] It's important to note that it's not the input NHWC tensor that's sliced, but the output NPQK tensor. This significantly simplifies the slicing algorithm, since it doesn't require ensuring that each tile fits an integer amount of output data. After that, it calculates what portion of the input NHWC data needs to be sliced ​​to obtain the desired NPQK result.

[0083] For example, for input data (features) 0,1,2,3,4,5,6,7,8,9,10 and convolution with kernel r=5 and stride=3, pad_begin=2, pad_end=0 for output data you need to load into ME_MEM output: _,_,0,1,2 output1: 1,2, 3,4, 5 output2: 4, 5, 6, 7, 8 output: 7,8,9,10 - 5 elements are no longer collected.

[0084] This means that the output data (output! and output2) partially share the same input data (4 and 5), and if looping over the input data, this overlap must be taken into account. It's also important to consider that at the end of the loop, the number of elements collected may not be sufficient, and the loop will terminate early. Considering that "r=5" can also be divided into "3+2," slicing the loop by input data (features) becomes much more cumbersome. Therefore, tiling by output dimensions was chosen.

[0085] 1.1 Basic functions

[0086] Dst_to_f is a function for calculating the amount of input data (features) by H / W required to obtain a certain amount of output data by P / Q.

[0087] Dst_min_p is a function for estimating the minimum P value of a tile, taking into account network padding. If the padding is large enough, the initial results are obtained without using input data (features), which leads to reading zero tensors. We calculate the minimum P / Q so that a minimal portion of the input data (features) is still used in the calculations, since it is impossible to pass empty input data (features) to the SA instruction.

[0088] Dst_min_q - similar to dst_min_p, but for the Q value of the tile.

[0089] Is_dst_effective is a tile efficiency evaluation function. Slicing is effective if N*P*Q > VL; otherwise, additional downtime within the SA may occur when switching weight tiles.

[0090] 1.2. Slicing weights {w_k, w_r, w_s, w_c}

[0091] Problem: For the weight tensor W {K, R, S, C}, the tile sizes (slicings) wk, w_r, w_s, w_c are selected so that the tensor {w_k, w_r, w_s, w_c} fits efficiently into ME_MEM.

[0092] Algorithm: • Initially set w_k=VL, w_r=l, w_s=l, w_c = VL; • The maximum size according to C (w_c) is calculated so that 2 tiles fit, and at least space remains for the effective size of the input data (also 2 tiles), but not less than dst_min_p / q. After calculating w_c, we assume that the input data tile will also be w_c. The selection is performed by the enumeration method from C to VL so that w_c is a divisor of C; • - If w_c = C, then we similarly continue searching for a slice for w_s; after searching for w_s, the minimum size of the input data must also be w_s; • If w_s == S, then we select w_r; • w_k can always be left equal to VL, since there is no impact on performance and therefore the minimum tile can be taken.

[0093] 1.3 Slicing input data (features) {dst n, dst_p, dst_q, dst_k}

[0094] Problem: For the input data tensor F {N, H, W, C}, tile sizes (slicing) are selected such that one part fits efficiently into ME_MEM. Since slicing is performed based on the output tensor NPQK, i.e., the output tile sizes {dst_n, dst_p, dst_q, dst_k} are calculated such that the corresponding input data sizes fit efficiently into ME_MEM.

[0095] The input data slicing coefficients are selected using a binary search method. Unlike the weight tile selection, which no longer takes into account the possible (or impossible) future slicing efficiency, we attempt to occupy the maximum remaining space in ME_MEM.

[0096] The algorithm also selects tiles starting from the innermost dimension: • dst_k=VL (equal to w_k); • dst_q is selected, using it we cut into W (feat_W, using dst_to_f), using binary search we select so that 2 tiles fit into ME_MEM together with two weight tiles; • if dst q < Q, then the search stops, for the remaining dimensions we leave the minimum values, otherwise we continue the search by P (dst_p and feat_H); • if dst_P < P, then we consider dst_n to be minimal (dst_n=l), otherwise we continue searching for dst_n.

[0097] As a result of the algorithm's operation, the most efficient slicing by the output tensor and by weights is returned.

[0098] 2. Tiling algorithm. Cycle over N with step dst n Loop along P with step dst_p Cycle along Q with step dst q / / I1: Instruction for loading part of the input data from the HBM into the VE MEM, using the dst_to_f function / / I2 - instruction to load the "excess tensor for addition", Cycle along K with step VL / / Change the 'L' line (0 or 1) to which SA the data will go Loop over R with step w_r Cycle along S with step w_s Cycle along C with step w_c / / FROM: ^Instruction to load a portion of the weight {VL,w_r,w_s,w_c} from HBM to ME MEM on line '> / / I4: <Instruction to load part of the input data {dst_n, p_sizes[p_idx], q_sizes[q_idx], w_c} from VE_MEM to ME_MEM on line 'L'> / / I5: Instruction to start the calculation of matrix multiplication + instructions to add the results in VS_MEM / / On the last iteration of {r_idx / s_idx / c} - I6: redirection instruction to VSb or instruction to write the result to VE_MEM. In case of redirection, activation and addition instructions with redundant link are added to VSb Cycle through C - end Cycle on S - end Cycle on R - end Cycle by K - end / / I7: instruction to save the intermediate result in NVM Cycle on Q - end Cycle by P - end Loop on N - end

[0099] An important point for achieving performance is further mixing of instructions so that instructions that can be executed in parallel are located one after another, i.e. instructions IZ and I4 for loading part of the weight / input data should be located next to instruction I5 (on SA), which calculates the matrix multiplication of the previous part of the already loaded weights / input data, which eliminates downtime during the SA operation.

[0100] The submitted application materials disclose preferred examples of the implementation of the technical solution and should not be interpreted as limiting other, particular examples of its implementation that do not go beyond the scope the requested legal protection, which are obvious to specialists in the relevant field of technology.

[0101] Sources of information: [1] Convolutional neural network - Wikipedia. https: / / ru.wikipedia.o^LLtk! / Convolutional_neural_network#Convolution_layer [2] Udit Kumar Agarwal “Resilience Assessment of Machine Learning Applications under Hardware Faults” (August 2023), section “2.3.2 Operation Tiling in Systolic Arrays”. https: / / open.library.ubc.ca / media / stream / pdf / 24 / l.0435699 / 4 [3] Systolic mass - Wikipedia. https: / / ru.wikipedia.org / wiki / CHCTOHH4ecKHfi_MaccHB [4] Vector processor - Wikipedia. https: / / ru.wikipedia.org / wiki / BeKTopHbffl_npo4eccop [5] High-bandwidth memory - Wikipedia. https: / / ru.wikipedia.org / wiki / High-bandwidth memory [6] Minifloat - Wikipedia. https: / / en.wikipedia.org / wiki / Minifloat [7] bfloatl6 floating-point format - Wikipedia. https: / / en.wikipedia.org / wiki / Bfloatl6_floating-point_format [8] Notes on residual connections, https: / / www.kaggle.com / code / residentmario / notes-on- residual-connections

Claims

FORMULA 1. A method for dividing a convolution operation, performed on at least one computing device with matrix multipliers based on systolic arrays, comprising the steps of: • obtain tensors of input data and input weights, which are arguments of functions for the convolution operation, from high-bandwidth memory, • divide the input data and weight tensors into several parts taking into account the volume of vector and matrix memory, respectively, • reserve the required amount of matrix memory for subsequent loading of parts of the scales, • alternately load parts of the input data from high-bandwidth memory into vector memory, • carry out additional division of parts of the input data into several sub-parts, taking into account the volume of matrix memory reserved for the weights, and alternately load the specified sub-parts of the input data into the matrix memory, • parts of the scales are loaded into the matrix memory one by one, and the loading of parts of the scales into the matrix memory is carried out in parallel with the loading of subparts of the input data into the matrix memory, • processing of subparts of the input data and parts of the weights read from the matrix memory is carried out by at least one matrix multiplier based on systolic arrays, as a result of which subparts of the resulting tensor are obtained, • loading portions of the resulting tensor into high-bandwidth memory in parallel with processing subsequent sub-portions of the input data and portions of the weights by at least one matrix multiplier based on systolic arrays.

2. The method according to paragraph 1, characterized in that at the stage of obtaining the input weight tensors, the weights of at least one convolution operation are converted into the required format for the matrix memory.

3. The method according to paragraph 1, characterized in that during the processing of subparts of the input data and parts of the weights by at least one matrix multiplier 18 SUBSTITUTE SHEET Based on systolic arrays, the next part of the input data is loaded in parallel from high-bandwidth memory into vector memory, as well as the next part of the weights from high-bandwidth memory into matrix memory.

4. The method according to paragraph 1, characterized in that obtaining a subpart of the resulting tensor additionally comprises: • obtaining the results of a matrix multiplier based on systolic arrays and their sequential summation by the first scalar operations module, followed by loading into the memory of at least one scalar operations module, • storing the result of the summation, as a subpart of the resulting tensor, in vector memory.

5. The method according to paragraph 4, characterized in that summing the results of the matrix multiplier based on systolic arrays further comprises adding the first result of the matrix multiplier based on systolic arrays with a constant offset using a first module of scalar operations.

6. The method according to any one of paragraphs 4, 5, characterized in that during the summation of the last result of the matrix multiplier based on systolic arrays, the activation function and addition with redundant connection to a subpart of the resulting tensor are performed using a second module of scalar operations. 19 SUBSTITUTE SHEET

Images