Computational optimization method and device for shallow depth model based on systolic array

Abstract

The present disclosure provides a computational optimization method and apparatus for a shallow depth model based on a systolic array. By determining the standard input depth / standard output depth of the systolic array and the input depth and output depth of the shallow depth model, the input splicing number and the output splicing number are determined. Based on the input splicing number and the output splicing number, input data and / or corresponding output data are spliced ​​to obtain input data in batches and provide them to the systolic array in parallel, and / or multiple output data corresponding to multiple input data are cached in an output buffer and then output in batches, thereby improving the efficiency of moving input data and / or output data and also improving the utilization efficiency of the systolic array.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and more particularly, to a method and apparatus for optimizing the computation of a shallow depth model based on a systolic array. Background Art

[0002] In the field of data processing, many data processing processes, such as deep learning processes, can be performed through convolutional neural networks (CNNs), which require a large number of computational steps, most of which are multiplication and accumulation operations.

[0003] Systolic array architectures are currently being used to implement a large number of multiplication-accumulation operations. These structures consist of computational units (PEs) located at nodes in each row and column, each PE including a multiplication-accumulation unit (MCU) or similar computing device, or a storage unit. Their advantage lies in the high-speed, parallel pipeline processing of operands and intermediate results involved in internal calculations. This means that each piece of input data (input to each row) and the computational results of each PE are simply and efficiently transmitted systolically throughout the array. This improves data reuse, reduces off-chip data movement, and reduces bandwidth requirements. Consequently, single or multiple two-dimensional systolic arrays are widely used as the computational cores for multiplication-accumulation operations in artificial intelligence (AI) processors, such as the Tensor Processing Unit (TPU).

[0004] In the calculation process based on the systolic array structure, the utilization efficiency of the systolic array structure, the efficiency of moving input data to the input buffer, and the efficiency of retrieving the output data calculated by the systolic array from the corresponding output buffer of the systolic array are factors that often need to be considered in the calculation process. When applied to the calculation process of shallow depth models, the utilization efficiency of the systolic array structure and the data movement efficiency are often not high.

[0005] Therefore, it is necessary to design a computational optimization method based on a systolic array that is applicable to shallow depth models and at the same time enables the systolic array structure to have high utilization efficiency and input / output data transfer efficiency. Summary of the Invention

[0006] In order to solve the above problems, the embodiments of the present disclosure provide a computational optimization method, a computational device, and a computer-readable storage medium based on a systolic array.

[0007] According to one aspect of the present disclosure, a computational optimization method for a shallow depth model based on a systolic array is provided, comprising: determining a standard input depth and a standard output depth of the systolic array, and determining a target input depth and a target output depth of the shallow depth model; determining a number of input splices for the systolic array based on the target input depth and the standard input depth, and determining a number of output splices for the systolic array based on the target output depth and the standard output depth, wherein the number of input splices and / or the number of output splices are integers greater than 1; when the number of input splices is an integer greater than 1, generating input data of the input splices, each having the target input depth, and batch-wise inputting the input data of the input splices without overlapping each other into an input buffer of the systolic array; and / or when the number of output splices is an integer greater than 1, generating output data of the output splices, each having the target output depth, and batch-wise outputting the output data of the output splices without overlapping each other through the output buffer of the systolic array.

[0008] According to another aspect of the present disclosure, a computing device is provided. The computing device may include: a controller, a data acquirer, a systolic array, an input buffer, and an output buffer. The controller determines a standard input depth and a standard output depth of the systolic array, and determines a target input depth and a target output depth of the shallow depth model; the controller determines an input splice number for the systolic array based on the target input depth and the standard input depth, and determines an output splice number for the systolic array based on the target output depth and the standard output depth, wherein the input splice number and / or the output splice number are integers greater than 1; if the input splice number is an integer greater than 1, the controller controls the data acquirer to acquire and generate input data of the input splice number, and controls the input data of the input splice number to be inputted into an input buffer of the systolic array in batches without overlapping each other, wherein each input data has the target input depth; and / or if the output splice number is an integer greater than 1, the controller controls the systolic array to generate output data of the output splice number, and controls the output buffer of the systolic array to output the output data of the output splice number in batches without overlapping each other, wherein each output data has the target output depth.

[0009] According to yet another aspect of the present disclosure, a computer-readable storage medium is provided, on which instructions or codes are stored. When the instructions or codes are executed by a processor, the operations performed by the controller in the computing device as described above are implemented.

[0010] According to various embodiments of the present disclosure, by determining the number of input splices and the number of output splices of the systolic array, input data and / or corresponding output data are spliced, so that multiple input data (each input data having a depth) are provided to the systolic array in parallel, and / or multiple output data corresponding to the multiple input data are buffered in an output buffer and then output in batches. This improves the efficiency of moving the input data and / or output data, and also improves the utilization efficiency of the systolic array. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for describing the embodiments. The drawings described below are only exemplary embodiments of the present disclosure.

[0012] Figure 1 A schematic structural diagram of a computing device based on a systolic array according to an embodiment of the present disclosure is shown;

[0013] Figure 2 1 shows a flow chart of a method 200 for optimizing calculation of a shallow depth model based on a systolic array according to an embodiment of the present disclosure;

[0014] Figure 3 Shown Figure 2 An example flow diagram of the steps of the method shown;

[0015] Figures 4A-4B A specific example is shown;

[0016] Figure 5 Shown Figure 2 An example flow diagram of the steps of the method shown;

[0017] Figure 6 A specific example is shown;

[0018] Figure 7 Shown Figure 2 An example flow diagram of the steps of the method shown;

[0019] Figures 8A-8B Shown Figure 7 A situation of the process shown and a corresponding specific example;

[0020] Figures 9A-9D Shown Figure 7 Another situation of the process shown and another corresponding specific example; and

[0021] Figures 10A-10D Shown Figure 7 Another situation of the process shown and a corresponding specific example. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the present disclosure more apparent, the following will describe in detail exemplary embodiments of the present disclosure with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0023] In this specification and the accompanying drawings, substantially the same or similar steps and elements are denoted by the same or similar reference numerals, and repeated descriptions of these steps and elements will be omitted. At the same time, in the description of the present disclosure, the terms "first", "second", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance or ranking.

[0024] Deep learning (also known as deep structured learning or hierarchical learning) is part of a broader family of machine learning methods based on artificial neural networks. Learning can be supervised, semi-supervised, or unsupervised.

[0025] As mentioned above, deep learning can be performed, for example, through convolutional neural networks (CNNs) for data processing. A convolutional neural network (CNN) is mainly composed of an input layer, a convolutional layer, an activation function, a pooling layer, a fully connected layer, and a loss function, and there can be multiple layers in the middle (such as convolutional layers and pooling layers). In the calculation of the convolutional layer, a large number of multiplication and accumulation operations of data are involved. Therefore, a systolic array structure can be used to implement the calculation process of the convolutional layer. It should be noted that although the context of this disclosure describes the application of the systolic array structure to the convolutional layer calculation as an example, the systolic array structure can also be applied to any matrix calculation.

[0026] Before describing the embodiments of the present disclosure in detail, some of the terms used herein are briefly introduced.

[0027] Shallow depth model: A lightweight model used in deep learning to generate output data based on input data, typically with a low depth of input and output data. The computational process of this shallow depth model is based on a systolic array, as described below.

[0028] Systolic arrays: Systolic arrays have multiple computational units arranged in rows and columns. These units include computational elements such as multiply-accumulators (implemented, for example, as multiply-accumulators) to perform matrix multiplication, the fundamental computational unit of neural networks. These units are interconnected by wires, and after integrating necessary internal buffers, they are connected to external buffers. The name systolic matrix is ​​derived from the contractile function of the heart: data is "pumped" through these computational elements, much like blood flow.

[0029] Figure 1FIG. 4 is a schematic structural diagram of a computing device 100 based on a systolic array according to an embodiment of the present disclosure.

[0030] like Figure 1 As shown, the computing device 100 includes a controller 101 , a data acquirer 102 , a systolic array 103 , an input buffer 104 and an output buffer 105 .

[0031] In the embodiments of the present disclosure, for the convenience of description, Figure 1 Although only one controller 101 is shown in the figure, those skilled in the art will appreciate that a controller may be a collective term for multiple control modules that coordinately control all operations of the computing device 100. The controller 101 herein may refer to a global controller for global control, as well as one or more of various types of control logic or control circuits that control the operations of the data acquirer, input buffer, and output buffer. It may also refer to a combination of multiplier-accumulators within a computing unit. For example, the global controller may be configured with functions such as instruction parsing and parameter configuration to provide operating parameters to the data acquirer 102, input buffer 104, and output buffer 105. Furthermore, the control logic corresponding to the input buffer 104 may control the transmission of cached input data to the systolic array 103. Similarly, the control logic corresponding to the output buffer may control the caching of output data from the systolic array 103 into the output buffer and the output of the output data from the output buffer.

[0032] The systolic array 103 may be a computing unit array (hereinafter also used interchangeably with the systolic array) and may have a standard input depth and a standard output depth, wherein the number of rows and the number of columns of the computing unit array correspond to the standard input depth and the standard output depth, respectively. In some embodiments, the number of rows and the number of columns of the computing unit array may be equal to the standard input depth and the standard output depth, respectively. In other embodiments, the number of rows and the number of columns of the computing unit array may be greater than the standard input depth and the standard output depth, respectively, so that the computing units on some rows and columns are optional or redundant. The present disclosure is not limited to this. It should be noted that in different cases, the standard input depth and the standard output depth of the computing unit array may be correspondingly different. For example, for different types of AI processors including computing unit arrays, the computing unit array may be selected as 16×32, 32×32, and so on.

[0033] The computing unit array (systolic array) 103 may include at least one row of computing units, each row of computing units includes at least two computing units 103 a, and two adjacent computing units 103 a in each row of computing units are connected.

[0034] The input buffer 104 buffers at least one input data item from the data acquirer 102 to be computed by the systolic array, as well as various computation parameters used for the computation. The computation parameters may be provided directly to the systolic array 103 by the data acquirer 102 or provided to the systolic array 103 via the input buffer 104. Each input data item may also have a depth (which may be considered to include multiple pieces of input data or multiple channels of input data). Furthermore, the input buffer 104 may include multiple sub-input buffers (not shown) (e.g., equal to the number of rows in the systolic array), each sub-input buffer being configured to provide a piece of input data to a computation unit in a row of the systolic array. For example, when the depth of a piece of input data item is m (m is an integer greater than or equal to 1), the piece of input data item may be considered to include m pieces of input data or m channels of input data. The m pieces of input data item are buffered in m sub-input buffers according to the input format and timing required by the systolic array. Thus, the m sub-input buffers input the m pieces of input data into m rows of the systolic array according to the timing of the systolic array.

[0035] The controller 101 (e.g., the control logic corresponding to the input buffer 104) can control the input data in the input buffer 104 to be input to the computing units 103a in the corresponding row. The computing units 103a in each row transmit each received input data item within the computing units 103a in the corresponding row. For example, upon receiving a piece of input data from the first sub-input buffer, the computing unit 103a in the first column of the first row transmits the received input data item to the computing unit in the second column of the first row, and so on. The timing at which the first computing units 103a in adjacent rows receive each piece of input data item differs by one clock.

[0036] Each computational unit 103a in the systolic array 103 is configured to perform computations on a received input data item and corresponding computational parameters (e.g., weight coefficients of convolution kernels in a CNN model). For each column of the systolic array 103, the computational results of each computational unit 103a in that column are superimposed (i.e., cumulatively summed) (from bottom to top in the figure), resulting in output data at the last computational unit 103a in each column (the mth row of each column), which is then output to the output buffer 105. It should be noted that these computational parameters can be updated as needed, as long as they are available to each computational unit in the systolic array when performing computational tasks. For example, depending on the configuration of the systolic array, the computational parameters for each computational unit can be pre-stored in the computational unit for each input data calculation, or they can be pulsatically transmitted to each computational unit during the computation process, using the same clock as the systolic transmission of the input data but in different directions (perpendicular to each other), consistent with the general principles of systolic arrays.

[0037] Similarly, output buffer 105 may include multiple sub-output buffers for buffering the output data from the last computing unit 103a in each column. Controller 101 (e.g., control logic corresponding to the output buffer) may control the batch output of this output data from output buffer 105 (or multiple sub-output buffers), for example, to provide this output data to a subsequent storage device, or to move the data from output buffer 105 to input buffer 104 for subsequent computation.

[0038] In the embodiment of the present disclosure, taking the CNN model as an example, each input data input to each computing unit and the convolution kernel provided at the computing unit can be a two-dimensional data matrix. For example, taking the image processing process using the CNN model and the convolution kernel being the feature weight in the CNN model as an example, when performing the convolution calculation, the input data is the image feature map tensor (image feature map tensor, stored in NHWC format, where N is the number of images in this batch of images, H is the number of pixels in the vertical direction of the image, W represents the number of pixels in the horizontal direction, and C represents the number of channels). That is, each input data is feature data extracted from the image and can be represented by the number of pixels in the vertical direction H, the number of pixels in the horizontal direction W, and the number of channels C. Each input data of the input data (depth C) can be a 10 (H) × 10 (W) data matrix, that is, the data matrix is ​​divided into 10 rows, each row contains 10 data elements; correspondingly, each convolution kernel can be a 3 × 3 weight matrix, that is, the weight matrix is ​​divided into 3 rows, each row contains 3 weight elements, and each weight element is a weight value used to multiply the data element. It should be noted that the above input data and convolution kernel are only illustrated by two-dimensional data. In actual applications, the above input data and convolution kernel can also be three-dimensional data.

[0039] In the computing unit array shown in the embodiment of the present disclosure, each computing unit in a row of computing units has its own computing parameters pre-stored. When the computing unit array 103 performs calculations, an input data corresponding to a row of computing units can be input into each computing unit in the row of computing units in sequence according to a pipeline manner, and calculations are performed in each computing unit respectively with the corresponding computing parameters.

[0040] Combined with the above Figure 1 The present invention describes a schematic diagram of the structure of a computing device 100 based on a systolic array and a general computing process according to an embodiment of the present disclosure. The computing device 100 can perform multiple depths of parallel computing on input data, thereby accelerating multiply-accumulate operations such as convolution.

[0041] However, for example, in actual applications, the input data depth and output data depth may be different between different models, and there is a certain range of variation. For example, the depth of the input data and the depth of the output data of the lightweight model (shallow depth model) are relatively small (referred to as the input depth of the shallow depth model and the output depth of the shallow depth model), such as 16, 8, etc. At this time, if the standard input depth and standard output depth of the systolic array are, for example, 32, then based on the reference Figure 1 The described systolic array-based computation process significantly reduces systolic array utilization. For example, when the standard input and output depths of the systolic array are both 32, and the input and output depths of the shallow-depth model are both 8, then only the computational results of the computational units in 8 rows and 8 columns of the 32x32 systolic array are valid (the remaining 24 rows of input data are padded with zeros). As a result, the utilization of the computational units in the systolic array is reduced to 1 / 16, severely impacting performance. Furthermore, only one input data of depth 8 is fed into the systolic array at a time, and the 8-way output data of the shallow-depth model (calculated based on the systolic array) is written to the output buffer with a parallelism of 32. Therefore, for every 32 output data items written, 24 are invalid. Therefore, when subsequent processing requires data from the output buffer, all 32 output data items are retrieved from the output buffer simultaneously, resulting in a large amount of invalid data being moved, which also reduces the efficiency of the transfer.

[0042] Therefore, in order to solve the above problems, the embodiments of the present disclosure propose a calculation optimization method and device for a shallow depth model based on a systolic array. The embodiments of the present disclosure splice different input data and / or different output data according to the target input depth and target output depth of the shallow depth model, so that different input data can be input into the input buffer in parallel and then provided to the systolic array, and different output data can be output from the output buffer in batches, thereby improving the efficiency of input / output data transfer. In addition, since the data of each row is systolically transferred between adjacent computing units and the calculation results of the computing units of each column are systolically transferred and accumulated between adjacent computing units, most computing units in the systolic array can be in a computing state at all times, and in the calculation of the shallow depth model, different input data and corresponding output data can utilize computing units corresponding to different row sets and column sets, thereby more effectively improving the utilization efficiency of the systolic array, and can improve throughput and reduce computing latency, thereby reducing TCO (total cost of ownership, used to evaluate the operating costs of data centers).

[0043] The following will be combined Figure 2-10D A method and apparatus for computational optimization of a shallow depth model based on a systolic array according to an embodiment of the present disclosure are described.

[0044] Figure 2 FIG2 shows a flow chart of a method 200 for optimizing a shallow depth model based on a systolic array according to an embodiment of the present disclosure.

[0045] like Figure 2 As shown, in step S210, the standard input depth and the standard output depth of the systolic array are determined, and the target input depth and the target output depth of the shallow depth model are determined.

[0046] As described above, the systolic array can be a computing unit array, and the number of rows and columns of the computing unit array can correspond to the standard input depth and the standard output depth, respectively. In some embodiments, the number of rows and columns of the computing unit array can be equal to the standard input depth and the standard output depth, respectively. In other embodiments, the number of rows and columns of the computing unit array can be greater than the standard input depth and the standard output depth, respectively, so that the computing units on some rows and columns are used as alternatives or redundant. In the following text, the present disclosure takes the rows and columns of the computing unit array being equal to the standard input depth and the standard output depth, respectively, as an example, but it is also feasible that they are not equal, and the present disclosure does not limit this.

[0047] The shallow depth model is a depth model that satisfies at least one of the following conditions: the input depth is less than the standard input depth of the systolic array; and the output depth is less than the standard output depth of the systolic array.

[0048] In some embodiments of the present disclosure, considering the actual situation of computer data processing, the standard input depth / output depth of the systolic array is usually a power of 2, and the target input depth and target output depth of the shallow depth model are also powers of 2.

[0049] However, this is not required, and other values ​​of standard input depth / output depth for the systolic array, and other values ​​of target input depth and target output depth for the shallow depth model are also feasible. For example, the standard input depth / output depth for the systolic array may be 48 and 96, respectively, and the target input depth and target output depth for the shallow depth model may be 24 and 32, or 31 and 42, respectively, without limitation in this disclosure.

[0050] In the detailed description below, for ease of description, a case where the depths of input data input to a systolic array and the corresponding output data are both powers of 2 is described as an example. However, other cases are based on the same principle.

[0051] In step S220 , the number of input tiles for the systolic array is determined based on the target input depth and the standard input depth, and the number of output tiles for the systolic array is determined based on the target output depth and the standard output depth, wherein the number of input tiles and / or the number of output tiles are integers greater than 1.

[0052] For example, to ensure the consistency of input depth and output depth, the standard input depth and standard output depth of the systolic array can be equal (most cases in current AI processors), for example, both are 32, and the target input depth and target output depth of the shallow depth model are both 8, then the number of input splices can be determined to be 4, and the number of output splices can also be 4; or, the target input depth of the shallow depth model is 16, and the target output depth is 8, then the number of input splices can be determined to be 2, and the number of output splices can be determined to be 4.

[0053] Optionally, if the target input depth of the shallow depth model is 32 and the target output depth is 8, then the number of input splicing can be determined to be 1, that is, the depth of each input data input to the shallow depth model for calculation is 32 (i.e., 32 data input channels), and the number of output splicing is 4. Alternatively, if the target input depth of the shallow depth model is 8 and the target output depth is 32, then the number of input splicing can be determined to be 4 and the number of output splicing can be determined to be 1, that is, the depth of the output data output after each input data is calculated by the shallow depth model is 32 (i.e., 32 data output channels).

[0054] In addition, for the case where the target input depth and target output depth of the shallow depth model are not a power of 2, the target input depth and target output depth of the shallow depth model can be determined based on the values ​​of the standard input depth and standard output depth of the systolic array and the values ​​of the target input depth and target output depth of the shallow depth model. For example, if the standard input depth and standard output depth are 32 and 32 respectively and the target input depth and target output depth are 15 and 7 respectively, then the number of input splices can be 2 and the number of output splices can be 4. The computing units on the remaining rows and columns of the systolic array are not used for this calculation, that is, the systolic array can be regarded as an array with 30 rows and 28 columns.

[0055] It should be noted that when the number of input splicing or the number of output splicing is greater than 1, it is necessary to ensure that there are multiple input data so that data splicing can be performed at the input or output of the systolic array to perform various systolic array-based computing operations in the embodiments of the present disclosure.

[0056] In one example, if only one initial input data is obtained (an image feature map tensor (such as the NHWC format described above, N=1)), the initial input data can be tiled in the horizontal direction H and the vertical direction W as needed to obtain multiple input data. If the depth of each input data after segmentation is greater than the target input depth, the segmented input data is segmented in the depth direction according to the target input depth (when it cannot be evenly divided, the data of the channel corresponding to the insufficient depth of the last segment is padded with 0). In addition, for each input data, if segmentation is performed in the depth direction to obtain multiple segmented input data, multiple rounds of operations can be performed on the segmented input data, and the subsequent processing unit can obtain the values ​​of multiple output data corresponding to the multiple segmented input data from the output buffer, and then sum the multiple output data to obtain the value of the output data corresponding to the input data.

[0057] On the other hand, if multiple initial input data (image feature map tensors (such as the NHWC format described above, N is an integer greater than 1)) are obtained in batches, then each initial input data (image feature map tensor) can be used as an input data to be input into the systolic array to obtain multiple input data, and for each input data, if the depth is also greater than the target input depth, the input data can be segmented again according to the target input depth.

[0058] In step S230, when the input splicing number is an integer greater than 1, input data of the input splicing number are generated, each input data has a target input depth, and the input data of the input splicing number are input to the input buffer of the systolic array in batches without overlapping each other; and / or, when the output splicing number is an integer greater than 1, output data of the output splicing number are generated, each output data has a target output depth, and the output data of the output splicing number are output in batches without overlapping each other through the output buffer of the systolic array.

[0059] For example, when the input splicing number is an integer greater than 1, it means that a plurality of (the input splicing number) input data can be provided in batches to the input buffer of the systolic array, and then the input buffer can provide each input data to a different set of rows of the systolic array according to the timing. For example, each of 4 input data (each input data has a depth of 8) can be input into the input buffer and provided to 8 different rows of the systolic array.

[0060] For example, when the output splicing number is an integer greater than 1, it indicates that multiple output data, that is, the output splicing number of output data, can be output from the systolic array at different times, and the output splicing number of output data can be cached in the output buffer of the systolic array, and then the output splicing number of output data can be obtained from the output buffer in batches. For example, four output data of four 8-column outputs of the systolic matrix (the depth of each output data is 8) can be written into the output buffer, and then these four output data can be read from the output buffer at one time.

[0061] By reference Figure 2 The described computational optimization method for shallow depth models based on systolic arrays can, for shallow depth models, move multiple input data (with the same depth) from other devices to an input buffer in batches and use them together as inputs of the systolic array via the input buffer, thereby improving the efficiency of moving the input data. At the same time, multiple output data (with the same depth) can also be cached in the output buffer, so that multiple output data can be output in batches from the output buffer, thereby also improving the efficiency of moving the output data.

[0062] The following combination Figure 3-4B The computational optimization method of the shallow depth model based on the systolic array according to the embodiment of the present disclosure is further described. Figure 3 Shown Figure 2 The specific flow chart of step S230 of the method 200 is shown. Figures 4A-4B Shown Figure 2 A specific example of method 200 is shown.

[0063] Figure 3-4B This corresponds to the case where the input splicing number is equal to 1 and the output splicing number is greater than 1. In this case, the method 200 may further include the following steps.

[0064] In step S310 , calculation parameters of each calculation unit in the calculation unit array are set.

[0065] For example, setting the calculation parameters of each calculation unit may include: pre-storing the required calculation parameters at each calculation unit before the calculation process of each input data, or providing the calculation parameters to each calculation unit in a pulsating manner based on the clock adopted in the transmission process of the input data during the calculation process for each input data, so as to perform calculations at each calculation unit.

[0066] For example, a systolic array can be used for the calculation of a convolution layer in a CNN model, so that the calculation unit can include a convolution kernel, so that the convolution kernel parameters at the calculation unit can be set (for example, the 3×3 weight matrix described above). In this embodiment, the calculation parameters of each calculation unit are determined with respect to each input data, which will be described in subsequent steps.

[0067] In step S320, the output splicing quantity of input data is generated as a group of input data, wherein the output data corresponding to different input data in the group of input data correspond to non-overlapping column sets in the computing unit array and to non-overlapping cache areas of the output buffer.

[0068] For example, Figure 4A As shown, the standard input / output depth of the systolic array is 32 and the target input depth of the shallow depth model is 32 / target output depth is 8, which determines that the number of input splices is 1 and the number of output splices is 4. Furthermore, four input data (each having a depth of 32) are generated as a set of input data to be input into the input buffer at different times (sequentially) and thus provided to respective rows of the systolic matrix. Output data corresponding to the first input data (i.e., data after calculation by the systolic array) can be output from columns 1-8 of the systolic array (i.e., corresponding to the columns 1-8) and cached in a first cache region of the output buffer. Output data corresponding to the second input data can be output from columns 9-16 of the systolic array (i.e., corresponding to the columns 9-16) and cached in a second cache region of the output buffer. Output data corresponding to the third input data can be output from columns 17-24 of the systolic array (i.e., corresponding to the columns 17-24) and cached in a third cache region of the output buffer. Output data corresponding to the fourth input data can be output from columns 25-32 of the systolic array (i.e., corresponding to the columns 25-32) and cached in a fourth cache region of the output buffer. Different regions of the output buffer do not overlap. Furthermore, as described above, the output buffer may include multiple sub-output buffers, and the number of the sub-output buffers may be the same as the number of columns of the systolic array. Therefore, the output data output by the last computing unit on each column in each column set of the systolic array is cached in multiple sub-output buffers, the number of which is equal to the number of columns in the column set. These multiple sub-buffers correspond to a cache area of ​​the output buffer described above.

[0069] In step S330 , each input data in the set of input data is sequentially input into the input buffer, so that the set of input data is sequentially provided to the computing unit array via the input buffer.

[0070] For example, assuming that the set of input data includes 4 input data (the depth of each input data is 32, and the depth of the corresponding output data is 8), Figure 4A As shown, one input data item is provided to the input buffer at a time, and the input data item is then provided to the systolic array via the input buffer (multiple (multi-channel) input data items of the input data item are systolic-input to corresponding rows of the systolic array according to a specific timing sequence). The systolic array outputs the output data corresponding to each input data item. At least after the input buffer provides one input data item to the systolic array, another input data item is input to the input buffer.

[0071] In step S340 , for each input data in the set of input data, the output data corresponding to the input data is output to the corresponding buffer area in the output buffer.

[0072] Optionally, in the present embodiment, and according to the working principle of the systolic array, the input buffer inputs one input data (for example, a depth of 32) to the systolic array in each calculation process, that is, according to a specific timing, the 32 input data, for example, are input to 32 rows of the systolic array, and each computing unit on each row performs calculations based on the input data and calculation parameters input to the row, and accumulates the calculation results with the calculation results of the computing units in the previous row on the same column. In this embodiment, the depth of the output data corresponding to the input data is relatively small (for example, Figure 4A The depth of the output data shown is 8). Therefore, the output data corresponding to the input data need only be output from a portion of the columns (e.g., 8 columns), and the calculation parameters related to the input data need to be set at the calculation units at the intersections of these columns (e.g., 8 columns) and the rows of the systolic array corresponding to the depth of the input data (e.g., 32 rows). For example, the calculation parameters can be pre-stored at each calculation unit or systolically provided to each calculation unit according to a timing sequence during the calculation process. Of course, other methods for providing calculation parameters are also possible, as long as they can provide calculation parameters for the current calculation and do not affect the previous calculation results of other input data. This disclosure is not limited to this.

[0073] Similarly, for each other input data, it is also necessary to set the calculation parameters related to the input data in the calculation unit at the intersection of the corresponding column set (8 columns) and the row of the pulsation array corresponding to the depth of the input data (for example, 32 rows).

[0074] Therefore, in this embodiment, for each input data, it is necessary to set the calculation parameters at a specific part of the calculation units. For example, for the first input data, the specific part of the calculation units is the calculation units at the intersection of columns 1-8 and rows 1-32. The calculation parameters can be set at each calculation unit in the calculation unit array, for example, Figure 4A , the calculation parameters required for the first input data calculation are set at the calculation cells at the intersection of the 1-8 columns and the 1-32 rows (the first zone Z1), the calculation parameters required for the second input data calculation are set at the calculation cells at the intersection of the 9-16 columns and the 1-32 rows (the second zone Z2), the calculation parameters required for the third input data calculation are set at the calculation cells at the intersection of the 17-24 columns and the 1-32 rows (the third zone Z3), and the calculation parameters required for the fourth input data calculation are set at the calculation cells at the intersection of the 25-32 columns and the 1-32 rows (the fourth zone Z4).

[0075] However, considering the operating principle of the systolic array, if no other adjustments are made, when each input data is input into the systolic array, since each computing unit in the computing unit array is also configured with computing parameters required for computing other input data, the input data will be calculated with these computing parameters (for example, the third input data will be calculated with the computing parameters for the second input data at the computing units in columns 9-16). As a result, other output data will be output in columns other than the set of columns corresponding to the output data corresponding to the input data (for example, the third input data will be calculated with the computing parameters at the computing units in columns 9-16, and the corresponding output data will be output from columns 9-16). This data will then be cached in the output buffer, thereby overwriting the output data corresponding to the previous input data. Furthermore, when subsequent input data is input, the output data corresponding to the input data in the output buffer will also be overwritten. This is unacceptable.

[0076] In an embodiment of the present disclosure, this problem can be solved by prohibiting, during the calculation of each input data set, writing to other buffer regions in the output buffer corresponding to output data corresponding to other input data in the set of input data. For example, the controller can prohibit output from the output ports of the systolic array (e.g., by disabling output enable signals), or prohibiting writing to certain regions of the output buffer (e.g., by disabling write enable signals corresponding to write channels corresponding to these regions of the output buffer).

[0077] In this way, each output data corresponding to a plurality of input data can be cached in different cache areas of the output buffer, and the output data do not overlap and do not affect each other.

[0078] In step S350 , for the set of input data, the output data of the output splicing quantity in each buffer area of ​​the output buffer are output in batches through the output buffer without overlapping with each other.

[0079] For example, Figure 4A As shown, the four output data corresponding to the four input data are respectively stored in four non-overlapping cache areas of the output buffer (each cache area includes 8 sub-output buffers), and the four output data will be moved from the output buffer to other devices (for example, memory or input buffer) without overlapping each other.

[0080] In reference Figure 3-4A In the described embodiment, in order to achieve batch output of the output data of the output splicing quantity in each buffer area of ​​the output buffer without overlapping each other, a prerequisite is that there are multiple input data, so that there can be multiple output data for batch output. For example, the multiple input data can be divided into at least one group, wherein each group of input data includes the output data of the output splicing quantity, and the remaining input data can be combined with the input data with a value of zero to form a group of input data for input. For example, if there are 10 input data and the output splicing quantity is 4, then the 10 inputs can be divided into three groups to respectively execute the above reference Figure 3-4A In the described method, one of the three sets of input data includes two input data with values ​​of 0.

[0081] In one example, if only one image feature map tensor is obtained (such as the NHWC format described above, N=1), the image feature map tensor can be split in the H and W directions to obtain multiple input data, such as Figure 4B As shown. And: if the value of the depth C of the tensor is equal to the target input depth (denoted by Ds), which is also the standard input depth in this example, then each input data (each data matrix) included in each input data can be input exactly to each row of the systolic array; if the value of C is less than Ds, it is necessary to supplement each input data with (Ds-C) input data with a value of 0 so that the depth of the input data is Ds; and if the value of C is greater than Ds ( Figure 4B), each input data needs to be further segmented into multiple segmented input data according to Ds in the C direction, and if there is a remaining input data with a depth less than Ds, then (Ds-(C-nDs)) pieces of input data (data matrix) with a value of 0 are supplemented to the remaining input data, where n is greater than or equal to 1. In addition, for each input data, if segmentation is performed in the C direction, multiple rounds of operations can be performed on the segmented input data. The subsequent processing unit can obtain the values ​​of multiple output data corresponding to the multiple segmented input data segmented by the input data from the output buffer, and then sum the multiple output data to obtain the value of the output data corresponding to the input data.

[0082] On the other hand, if multiple image feature map tensors are acquired in batches (such as the NHWC format described above, where N is an integer greater than 1), each image feature map tensor can be used as an input data. Similarly, based on the relationship between the value of the depth C of the tensor and the target input depth (represented by Ds), each tensor as each input data can be further segmented or supplemented so that the depth of each input data is Ds. Obviously, the multiple input data corresponding to the multiple tensors can be divided into at least one group of input data (each group includes the input data of the output splicing quantity), and for each group of input data, the above reference can be used. Figure 3 Describe the methods to perform calculations on these input data.

[0083] Similarly, in the various embodiments described below, the acquisition of multiple input data can still be similar to that described here, that is, the acquired initial input data is split in the horizontal and vertical directions and possibly segmented in the depth direction, or each of the initial input data acquired in batches is taken as an input data, so it will not be repeated below.

[0084] By reference Figure 3-4B The described computational optimization method for the shallow depth model based on the systolic array can batch-take out the output data corresponding to a set of input data (input data of the output splicing number) from the output buffer when the input splicing number is equal to 1 and the output splicing number is greater than 1, thereby improving the efficiency of moving the output data.

[0085] The following combination Figure 5-6 The computational optimization method of the shallow depth model based on the systolic array according to the embodiment of the present disclosure is further described. Figure 5 Shown Figure 2 The specific flow chart of step S230 of the method 200 is shown. Figure 6 A specific example is shown.

[0086] Figure 5-6This corresponds to the case where the input splicing quantity is greater than 1 and the output splicing quantity is equal to 1. In this case, step S230 of the method 200 may further include the following steps.

[0087] In step 510, the input data of the input splicing quantity is generated as a set of input data, wherein different input data in the set of input data correspond to a set of non-overlapping rows in the computing unit array, and the output data corresponding to the different input data in the set of input data all correspond to all columns in the computing unit array.

[0088] Optionally, the output data corresponding to each input data is output from all columns of the systolic array to the output buffer. If the storage space of the output buffer can only store the output data corresponding to one input data, the output data corresponding to each input data is output to the entire cache area of ​​the output buffer. After the current output data in the output buffer is retrieved, the output buffer caches the output data corresponding to the next input data. Alternatively, if the storage space of the output buffer is large enough to store the output data corresponding to multiple input data, the output data obtained for each input data can be arranged in a depth-contiguous manner in the output buffer. For example, the output data corresponding to the first input data can occupy the cache area indicated by addresses 0-7 in the output buffer, the output data corresponding to the next input data can occupy the cache area indicated by addresses 8-15 in the output buffer, and so on.

[0089] For example, Figure 6As shown, the standard input / output depth of the systolic array is 32, and the target input depth of the shallow depth model is 8 / target output depth is 32. Therefore, the number of input splices is determined to be 4 and the number of output splices is 1. Furthermore, four input data (each with a depth of 8) are generated as a set of input data to be input into an input buffer and then provided in parallel to different row sets of the systolic matrix. For example, the first input data is provided to rows 1-8 of the systolic array via the input buffer (i.e., corresponding to the row set consisting of rows 1-8), the second input data is provided to rows 9-16 of the systolic array via the input buffer (i.e., corresponding to the row set consisting of rows 9-16), the third input data is provided to rows 17-24 of the systolic array via the input buffer (i.e., corresponding to the row set consisting of rows 17-24), and the fourth input data is provided to rows 25-32 of the systolic array via the input buffer (i.e., corresponding to the row set consisting of rows 25-32). Furthermore, the output data corresponding to each of the four input data (i.e., data after calculation by the systolic array) can be output from columns 1-32 of the systolic array and buffered in the entire or at least a portion of the buffer area of ​​the output buffer (depending on the storage space of the output buffer). Furthermore, as previously described, the output buffer can include multiple sub-output buffers. Thus, the output data output by the last computing unit in each column of the systolic array is buffered in multiple sub-output buffers. These multiple sub-buffers correspond to the entire or at least a portion of the buffer area of ​​the output buffer (depending on the storage space of the output buffer).

[0090] In step S520, the calculation parameters of each calculation unit required to calculate each input data in the group of input data are set in sequence in each calculation unit, wherein, for each input data, the calculation parameters of the calculation unit on the row set corresponding to the input data are valid values, and the calculation parameters of the remaining calculation units are zero.

[0091] Likewise, the manner of setting the calculation parameters at the calculation units may include pre-storing at the corresponding calculation units or providing them to the calculation units in a pulsating manner according to a timing sequence.

[0092] For example, as described above, the systolic array accumulates the calculation results of the calculation units on the same column in the column direction during the calculation process and then outputs them. Therefore, for the group of input data (depth is 8) provided in parallel to the systolic array, in the calculation process for each input data, it is necessary to set the values ​​of the calculation parameters of the calculation units on the rows other than the row set corresponding to the input data to 0, so that the calculation results of the input data are valid values ​​only at the calculation units on the corresponding row set, and the calculation results at other calculation units are all 0. Therefore, when the calculation results on the same column are superimposed, only the calculation results of the calculation units on the column and the corresponding row set are superimposed, thereby avoiding the erroneous superposition of the calculation results for different input data on the same column (for example, the calculation units on each column can perform calculations on 4 input data in one calculation process, so the calculation results are wrong). Based on this, calculations can be performed on only one input data each time the group of input data is input. For example, Figure 6 In the example, the calculation parameters of the computation units in rows 1-8 are first set to valid values, while the values ​​of the other computation units are set to 0. Then, the input data is input in parallel for the first time. After the systolic array is calculated, the output data (depth 32) corresponding to the first input data (depth 8) is output across all columns of the systolic array. Next, the calculation parameters of the computation units in rows 9-16 are set to valid values, while the values ​​of the other computation units are set to 0. Then, the input data is input in parallel for the second time. After the systolic array is calculated, the output data (depth 32) corresponding to the second input data (depth 8) is output across all columns of the systolic array. Figure 6 Only two input data are shown in FIG. 1 , and similar operations are performed on other input data until all the output data corresponding to the group of input data are output.

[0093] In step S530 , the group of input data is input into the input buffer in batches, so as to be provided to the computing unit array in parallel via the input buffer.

[0094] For example, Figure 6 As shown in , the group of input data (4 input data) is input into the input buffer (either simultaneously or at different times), that is, the input buffer buffers the group of input data at the same time, and the input buffer can provide the group of input data to the systolic array in parallel according to a predetermined timing (controlled by the controller and based on the working principle of the systolic array), for example, the first input data is provided to rows 1-8 of the systolic array, the second input data is provided to rows 9-15 of the systolic array, the third input data is provided to rows 16-24 of the systolic array, and the fourth input data is provided to rows 25-32 of the systolic array.

[0095] In step S540 , for the set of input data, the output data corresponding to each input data is sequentially output to the output buffer and output through the output buffer.

[0096] For example, Figure 6 As shown in , the output data corresponding to the first input data is first output to the cache area of ​​the output buffer, and at least after the output buffer is free again (that is, the output data cached therein is taken out) or when the remaining storage space is sufficient to cache the output data corresponding to the next input data, the output data corresponding to the second input data is output to the entire cache area or at least a part of the cache area (depending on the storage space of the output buffer) of the output buffer.

[0097] It should be noted that in the detailed description of each embodiment of the present disclosure, the description of the method uses a flowchart to illustrate each step, but this does not necessarily mean that these methods must be performed in the order shown, but can be performed in an interleaved manner. Figure 5-6 In the embodiment, in step S520, the calculation parameters of each calculation unit for the calculation of the first input data are set (the calculation parameters of the calculation units in rows 1-8 are valid values, and the calculation parameters of the calculation units in the remaining rows are 0). Then, step S530 can be executed to provide the group of input data in parallel to the systolic array for calculation to obtain output data corresponding to the first input data. After step S530 is completed, the calculation parameters of each calculation unit for the calculation of the second input data can be set in step S520 (the calculation parameters of the calculation units in rows 9-16 are valid values, and the calculation parameters of the calculation units in the remaining rows are 0). Then, step S530 is executed again to provide the group of input data in parallel to the systolic array for calculation to obtain output data corresponding to the second input data, and so on. Therefore, the execution order of each step can be flexibly adjusted according to the principle of the technical solution of the present disclosure.

[0098] By reference Figure 5-6The described computational optimization method for shallow-depth models based on a systolic array can cache a set of input data (the input data of the input splicing number) in an input buffer and provide them to the systolic array in parallel when the number of input splices is greater than 1 and the number of output splices is equal to 1. Since the input provided to the systolic array by the input buffer for the calculation of each input data is the same, the input data only needs to be moved once (for example, from other memories to the input buffer), thereby improving the efficiency of moving the input data. On the other hand, if the computational optimization method described in the embodiments of the present disclosure is not adopted, the plurality of input data included in the group of input data requires multiple transfer processes to the input buffer (in this case, the data of the channels corresponding to the depth of each input data that is insufficient relative to the standard input depth of the systolic array is padded with 0), but the process of loading the computational parameters into the computational unit only needs to be performed once; with the above-mentioned computational optimization method, only one transfer process to the input buffer is required. Although the computational parameters are loaded into the computational unit multiple times (once for each input data) in this case, since the data size of the computational parameters is generally much smaller than the data size of the input data, the time required to load the computational parameters is much less than the time required to transfer the input data. Therefore, in terms of overall time consumption, relatively good performance is still achieved, thereby improving transfer efficiency.

[0099] The following combination Figure 7-10D The computational optimization method of the shallow depth model based on the systolic array according to the embodiment of the present disclosure is further described. Figure 7 Shown Figure 2 An example flow diagram of the steps of the method is shown. Figures 8A-8B Shows the case where the number of input splices is equal to the number of output splices Figure 7 More details on the computational optimization method. Figures 9A-9D Shows the case where the number of input splices is greater than the number of output splices Figure 7 For more details on the computational optimization method, Figures 10A-10D Shows the case where the number of input splices is less than the number of output splices Figure 7 More details on the computational optimization method.

[0100] exist Figure 7 In the embodiment, the input splicing number is greater than 1, and the output splicing number is greater than 1, so step S230 of the calculation optimization method 200 further includes the following steps.

[0101] In step S710 , a comprehensive splicing quantity is determined based on the input splicing quantity and the output splicing quantity.

[0102] Optionally, as will be described in detail later, the comprehensive splicing number can be the larger or smaller value of the input splicing number and the output splicing number depending on the size and multiple relationship of the input splicing number and the output splicing number, or when the input splicing number and the output splicing number are equal, the comprehensive splicing number is equal to the input splicing number or the output splicing number.

[0103] In step S720 , a comprehensive splicing quantity of input data is generated as a set of input data.

[0104] In step S730, based on the input splicing number and the output splicing number, the group of input data is divided into at least one sub-group, wherein each sub-group of input data corresponds to a set of rows of the input splicing number that do not overlap with each other in the computing unit array, and the output data corresponding to different input data in the sub-group of input data corresponds to a set of columns that do not overlap with each other in the computing unit array and to non-overlapping cache areas of the output buffer.

[0105] Optionally, the number of subgroups may be 1 or an integer greater than 1. Each subgroup may include one or more input data.

[0106] For example, the standard input depth / standard output depth of the systolic array are both 32, and the target input depth of the shallow depth model is 16 and the target output depth is 8, thereby determining that the number of input splices is 2, the number of output splices is 4, and the comprehensive number of splices is 4. In addition, the group of input data includes 4 input data, but because the number of input splices is 2, only 2 input data (depth is 16) can be provided to the systolic array at a time, so the 4 input data are divided into two subgroups, each subgroup includes 2 input data, and one subgroup of input data is provided to the systolic array each time.

[0107] In step S740 , when calculating each sub-group of input data, calculation parameters of each calculation unit in the calculation unit array are set for the sub-group of input data.

[0108] The specific method for setting the calculation parameters of the calculation unit will be described in further detail later.

[0109] Figures 8A-8B Shows the case where the number of input splices is equal to the number of output splices Figure 7 More details on the computational optimization method. Figure 7 The method may further include the following steps.

[0110] exist Figures 8A-8B In this case, the number of input splices is equal to the number of output splices (4 in the figure). In this case, it can be regarded as having only one subgroup, and each subgroup includes input data of the number of input splices (the number of output splices).

[0111] exist Figure 8A In step S810, each subgroup of input data is input into the input buffer in batches, so as to be provided to the computing unit array in parallel via the input buffer, wherein the number of subgroups is 1.

[0112] Similarly, each subgroup of input data corresponds to a set of rows of the input splicing quantity that do not overlap with each other in the computing unit array, and the output data corresponding to different input data in the group of input data corresponds to a set of columns that do not overlap with each other in the computing unit array and to cache areas that do not overlap with each other in the output buffer.

[0113] For example, Figure 8B As shown, the standard input depth and the standard output depth of the systolic array are both 32, and the target input depth and the target output depth of the shallow depth model are 8, thereby determining that the number of input splices is 4 and the number of output splices is also 4. In addition, 4 input data (each with a depth of 8) are generated as a set of input data to be input to the input buffer and then provided in parallel to different row sets of the systolic matrix, and their respective corresponding output data are also output from different column sets of the systolic array. For example, a first input data is provided to rows 1-8 of the systolic array (i.e., corresponding to the row set consisting of the rows 1-8) via the input buffer, and the corresponding output data is output from columns 1-8 of the systolic array. A second input data is provided to rows 1-8 of the systolic array (i.e., corresponding to the row set consisting of the rows 9-16) via the input buffer, and the corresponding output data is output from columns 9-16 of the systolic array. A third input data is provided to rows 17-24 of the systolic array (i.e., corresponding to the row set consisting of the rows 17-24) via the input buffer, and the corresponding output data is output from columns 17-24 of the systolic array. A fourth input data is provided to rows 25-32 of the systolic array (i.e., corresponding to the row set consisting of the rows 25-32) via the input buffer, and the corresponding output data is output from columns 25-32 of the systolic array. That is, the set of rows corresponding to the four input data and the set of columns corresponding to the four output data included in the subgroup can be represented as: a first region Z1 (1-8, 1-8); a second region Z2 (9-16, 9-16); a third region Z3 (17-24, 17-24); and a fourth region Z4 (25-32, 25-32). Furthermore, as previously described, the output buffer may include multiple sub-output buffers. Thus, the output data output by the last computing unit in each column of the systolic array is cached in multiple sub-output buffers. These multiple sub-buffers correspond to the entire or at least a portion of the cache area of ​​the output buffer described above (based on the storage space of the output buffer).

[0114] In step S820, for the calculation of the sub-group input data, the calculation parameters of the calculation units at the intersection of the row sets corresponding to different input data in the sub-group input data and the column sets corresponding to the output data corresponding to different input data are set to valid values, and the calculation parameters of the remaining calculation units on the corresponding column sets are set to zero.

[0115] That is, since the systolic array accumulates the calculation results of the calculation units on the same column in the column direction and then outputs them during the calculation process, for each input data in each sub-group of input data provided in parallel to the systolic array, during the calculation process for the input data, the calculation parameters of the calculation units that are not related to the calculation need to be set to zero to ensure the accuracy and correspondence of the output data. For example, Figure 8B As shown, the four zones (Z1-Z4) are as follows Figure 8B As shown in the shaded areas, the calculation parameters of the calculation units for calculating the corresponding input data are set in these shaded areas, and the calculation parameters of other calculation units on each column are set to 0 (such as the blank box area in the figure).

[0116] In step S830, for each input data in the sub-group input data, the output data corresponding to the input data is output to the corresponding cache area in the output buffer, that is, the output data corresponding to the sub-group input data will be cached in the output buffer and output in batches through the output buffer.

[0117] For example, as shown in FIG8B , the output buffer caches the output data (four output data) corresponding to the respective input data of the subgroup of input data, so the four output data can be retrieved from the output buffer in batches.

[0118] However, in some embodiments, there may be another situation where the input splice number is greater than 1, and the output splice number is greater than 1, and the output splice number is greater than or less than the input splice number.

[0119] Figure 9A Shows the case where the output splicing number is an integer multiple of the input splicing number greater than 1 Figure 7 More details on the computational optimization method.

[0120] In this case, the combined splice quantity is the output splice quantity, and each subgroup input data includes the input data of the input splice quantity, and for this case reference is made to Figure 7 The described computational optimization method may further comprise the following steps.

[0121] In step S910 , the integrated splicing quantity of input data is sequentially input into the input buffer in subgroups, so that each subgroup of input data is provided to the computing unit array in parallel via the input buffer.

[0122] For example, Figure 9B As shown in FIG, if the standard input depth and the standard output depth of the systolic array are 32, and the target input depth of the shallow depth model is 16 and the target output depth is 8, the number of input splices can be 2, the number of output splices is 4, and the number of integrated splices is 4. A set of input data (4 input data) generated is divided into two subgroups, each subgroup includes 2 input data. Each subgroup of input data (including 2 input data) is sequentially input to the input buffer, and each subgroup of input data is provided to the systolic array in parallel.

[0123] In step S920, for the calculation of each subgroup input data, the calculation parameters of the calculation units at the intersection of the row set corresponding to the different input data in the subgroup input data and the column set corresponding to the output data corresponding to the different input data are set to valid values, and the calculation parameters of the other calculation units on the column set corresponding to the output data corresponding to the different input data in the subgroup input data are set to zero. This step is equivalent to Figure 7 Step S740 in .

[0124] For example, Figure 9BAs shown, the row set corresponding to the first subgroup (including the first input data and the second input data) and the column set corresponding to the corresponding output data are respectively represented as the first region Z1 (1-16, 1-8) and the second region Z2 (17-32, 9-16). Before calculating the first subgroup, it is necessary to set the calculation parameters of the calculation units at the row set and column set. The row set corresponding to the second subgroup (including the third input data and the fourth input data) and the column set corresponding to the corresponding output data are respectively represented as the third region Z3 (1-16, 17-24) and the fourth region Z4 (17-32, 25-32). Similarly, before calculating the second subgroup, it is necessary to set the calculation parameters of the calculation units at the row set and column set. Since the areas covered by these row and column sets do not overlap, a simple approach is to set the computational parameters of the computational units required for the two sub-group calculations together. Furthermore, as previously described, taking into account the computational principles of the systolic array, when calculating each sub-group of input data, only the cache area in the output buffer corresponding to the output data of that sub-group of input data is written, while writing to other cache areas in the output buffer is prohibited. That is, when calculating the first sub-group of input data, only the output data corresponding to the two input data are outputted from the two column sets (columns 1-16 inclusive), and when calculating the second sub-group of input data, only the output data corresponding to the last two input data are outputted from the two column sets (columns 17-32 inclusive).

[0125] In step S930 , the output data corresponding to the group of input data is output to the output buffer and output in batches through the output buffer.

[0126] For example, Figure 9B As shown, the four output data corresponding to the two sub-groups of input data are output from the 1st to 32nd columns of the systolic array (the two output data corresponding to the first sub-group are first output from the 1st to 16th columns, and the two output data corresponding to the second sub-group are then output from the 17th to 32nd columns), that is, output to the entire or at least a portion of the cache area of ​​the output buffer, so that an external device or the input buffer can retrieve these output data in batches from the output buffer.

[0127] Figure 9C The output splicing number is greater than the input splicing number but is not an integer multiple of it. Figure 7 In this case, the number of integrated splices is the number of input splices, and the number of subgroups is 1, i.e., the generated set of input data includes only one subgroup.

[0128] In step S910 ′, the group of input data is input into the input buffer in batches, so as to be provided to the computing unit array in parallel via the input buffer.

[0129] like Figure 9D As shown, if the standard input depth and the standard output depth of the systolic array are 36 and 64, and the target input depth of the shallow depth model is 12 and the target output depth is 16, the number of input splices can be 3, the number of output splices is 4, and the number of integrated splices is 3. A set of generated input data (3 input data) is input into the input buffer in batches, and thus provided to the systolic array in parallel.

[0130] In step S920', for the calculation of the group of input data, the calculation parameters of the calculation units at the intersection of the row sets corresponding to different input data in the group of input data and the column sets corresponding to the output data corresponding to different input data are set to valid values, and the calculation parameters of the remaining calculation units in the calculation unit array are set to zero.

[0131] For example, Figure 9D As shown, the row set corresponding to the group of input data (including the first to third input data) and the column set corresponding to the corresponding output data are respectively represented as: (1-12, 1-16), (13-24, 17-32) and (25-36, 33-48). The calculation parameters of the calculation units at the row set and column set can be set to valid values, and the calculation parameters of the remaining calculation units can be set to 0 (including the calculation units on the 49th to 64th columns). Since the areas covered by these row sets and column sets do not overlap, the calculation parameters of the calculation units required for the calculation of the two subgroups of input data can be set together.

[0132] In step S930 ′, the output data corresponding to the set of input data is output to the entire buffer area of ​​the output buffer and is output in batches through the output buffer, wherein at least a portion of the value in the entire buffer area is 0.

[0133] For example, Figure 9D As shown, the three output data corresponding to the group of input data are output from the 1st to 48th columns of the systolic array, and the computing units on the 49th to 64th columns also output the calculated 0s. That is, the cache areas corresponding to the 1st to 48th columns in the output buffer store valid data, while the remaining cache areas store 0s, so that these output data and 0s can be taken out from the output buffer in batches.

[0134] The above reference Figures 9A-9D Describes the situation where the number of input splices is greater than 1 and the number of output splices is greater than 1, and the number of output splices is greater than the number of input splices.

[0135] The following description focuses on the case where the input splicing number is greater than 1 and the output splicing number is greater than 1, and the input splicing number is greater than the output splicing number.

[0136] In some cases, the number of input splices is an integer multiple of the number of output splices that is greater than 1.

[0137] Figures 10A-10B Shows the case where the number of input splices is an integer multiple of the number of output splices greater than 1 Figure 7 In this case, the integrated number of splices is the input number of splices, and each subgroup of input data includes input data for the output number of splices.

[0138] For example, if the standard input depth and standard output depth of the systolic array are 32, and the target input depth of the shallow depth model is 8 and the target output depth is 16, the number of output splices can be 2, the number of input splices is 4, and the number of integrated splices is 4. Each subgroup input data includes two input data.

[0139] against Figure 7 The method may further include the following steps.

[0140] In step S1010 , a comprehensive splicing quantity of input data is input into the input buffer in batches, so that the comprehensive splicing quantity of input data is provided to the computing unit array in parallel via the input buffer.

[0141] For example, Figure 10B As shown, 4 input data are output to the input buffer in batches and provided to the computing unit array in parallel.

[0142] In step S1020, for the calculation of each subgroup of input data, the calculation parameters of the calculation units at the intersection of the row set corresponding to the different input data in the subgroup of input data and the column set corresponding to the output data corresponding to the different input data are set to valid values, and the calculation parameters of the remaining calculation units in the calculation unit array are set to zero. This step corresponds to Figure 7 Step S740.

[0143] For example, Figure 10BAs shown, the row set corresponding to the first subgroup (including the first input data and the second input data) and the column set corresponding to the corresponding output data are respectively represented as: first zone Z1 (1-8, 1-16), second zone Z2 (9-16, 17-32), third zone Z3 (17-24, 1-16), and fourth zone Z4 (25-32, 17-32). The calculation parameters of the calculation units in the first zone Z1 (1-8, 1-16) used for the calculation of the first subgroup input data are set to valid values, the calculation parameters of the calculation units in the second zone Z1 (9-16, 17-32) are set to valid values, and the calculation parameters of the remaining calculation units are set to 0. When calculating the second sub-group input data, the two sub-group input data are input in parallel, but the calculation parameters of the calculation units in the third area Z3 (17-24, 1-16) are set to valid values, the calculation parameters of the calculation units in the fourth area Z4 (25-32, 17-32) are set to valid values, and the calculation parameters of the remaining calculation units are 0.

[0144] In step S1030 , for the group of input data including the integrated spliced ​​number of input data, the output data corresponding to each subgroup of input data is sequentially output to the output buffer, and is sequentially output through the output buffer.

[0145] For example, in Figure 10B In the embodiment, after the calculation of the first subgroup of input data is completed, the output data corresponding to the first input data and the second input data included therein are output to the output buffer via all columns of the systolic array, and after the output buffer is free again or when the remaining storage space is sufficient to cache the output data corresponding to the next input data, the calculation result of the subsequent second subgroup of input data is output to the output buffer.

[0146] Furthermore, in some cases, the input splice number is greater than the output splice number but is not an integer multiple of it.

[0147] Figures 10C-10D Shows the case where the number of input splices is greater than the number of output splices but is not an integer multiple of it Figure 7 In this case, the number of synthesized splices is the number of output splices, and the number of subgroups is 1, i.e., the generated set of input data includes only one subgroup.

[0148] For example, if the standard input depth and the standard output depth of the systolic array are 48 and 36, and the target input depth of the shallow depth model is 12 and the target output depth is 12, the number of output splices may be 3, the number of input splices may be 4, and the number of integrated splices may be 3. The group of input data and the subgroup of input data include three input data.

[0149] against Figure 7 The method may further include the following steps: Figure 10C shown.

[0150] In step S1010 ′, N input data padded with zeros are added to the set of input data, where the value of N is equal to the input splicing number minus the input-output splicing number.

[0151] For example, Figure 10D As shown, an input data (with a depth of 12) is added to match the number of rows of the systolic array.

[0152] In step S1020 ′, the group of input data to which N input data are added is input into the input buffer in batches, so as to be provided to the computing unit array in parallel via the input buffer.

[0153] For example, Figure 10D As shown, four input data are provided to the systolic array in parallel.

[0154] In step S1030', before calculating the set of input data, the calculation parameters of the calculation units at the intersection of the row sets corresponding to different input data and the column sets corresponding to the output data corresponding to different input data in the set of input data are set to valid values, and the calculation parameters of the remaining calculation units in the calculation unit array are set to zero. This step corresponds to Figure 7 Step S740 in .

[0155] For example, Figure 10D As shown, the row set corresponding to the input data set and the column set corresponding to the output data are respectively represented as: the first region Z1 (1-12, 1-12), the second region Z2 (13-24, 13-24), and the third region Z3 (25-36, 25-36). It is necessary to set the calculation parameters of the calculation cells at the row set and column set used to perform calculations on this group. The calculation parameters of the calculation cells in the regions of these row sets and column sets are set to valid values, while the calculation parameters of the remaining calculation cells (including the calculation cells on rows 37-48 corresponding to the added 0 input data) are set to 0.

[0156] In step S1040 ′, the output data corresponding to each input data in the group of input data is output to the entire buffer area of ​​the output buffer, and is output in batches through the output buffer.

[0157] For example, Figure 10DAs shown, since the calculation results of all calculation units on the corresponding 37th to 48th rows of the supplemented 0 input data are 0, it does not affect other calculation results on each column. The output data corresponding to the three input data are all cached to the entire or at least a part of the cache area of ​​the output buffer to be output from the output buffer in batches.

[0158] By reference Figure 7-10D The computational optimization method for shallow-depth models based on a systolic array is described. When the number of input splices is greater than 1 and the number of output splices is greater than 1 (including various situations where the two are equal or unequal), a set of input data (the number of input data of the output splice) can be cached in an input buffer and provided to the systolic array in parallel. In addition, a plurality of output data corresponding to the set of input data output by the systolic array can be cached in an output buffer and output in batches from the output buffer, thereby improving the efficiency of moving input data and output data. In addition, the systolic array can perform calculations on at least two input data in the set of input data during one calculation process, relative to the reference Figure 1 The existing calculation process only performs calculations on one input data, which improves the calculation efficiency and the utilization efficiency of the calculation units in the systolic array.

[0159] According to another aspect of the present disclosure, a computing device is also provided. Figure 1 The architecture of the computing device 100 described is substantially the same.

[0160] like Figure 1 As shown, the computing device 100 includes a controller 101 , a data acquirer 102 , a systolic array 103 , an input buffer 104 , and an output buffer 105 .

[0161] According to an embodiment of the present disclosure, the controller 101 determines a standard input depth and a standard output depth of the systolic array 103, and determines a target input depth and a target output depth of the shallow depth model. The controller 101 then determines the number of input tiles for the systolic array based on the target input depth and the standard input depth, and determines the number of output tiles for the systolic array based on the target output depth and the standard output depth, wherein the number of input tiles and / or the number of output tiles are integers greater than 1. When the input splicing number is an integer greater than 1, the controller 101 controls the data acquirer 102 to acquire and generate input data of the input splicing number, and controls the input data of the input splicing number to be input into the input buffer of the systolic array 103 in batches without overlapping each other, and controls the input buffer to provide the input data of the input splicing number to the computing unit array in parallel, wherein each input data has a target input depth; and / or when the output splicing number is an integer greater than 1, the controller 101 controls the systolic array 103 to generate output data of the output splicing number, and the controller 101 controls the output buffer of the systolic array 103 to output the output data of the output splicing number in batches without overlapping each other, wherein each output data has a target output depth.

[0162] According to an embodiment of the present disclosure, the systolic array 103 is a computing unit array, and the number of rows and columns of the computing unit array correspond to (are equal to or greater than) the standard input depth and the standard output depth, respectively. The depth of the input buffer 104 corresponds to the standard input depth, and the depth of the output buffer 105 corresponds to (are equal to or greater than) the standard output depth.

[0163] According to an embodiment of the present disclosure, when the input splicing number is equal to 1 and the output splicing number is greater than 1, the controller 101 controls the setting of calculation parameters (for example, the convolution kernel in the CNN model) at each computing unit in the computing unit array, controls the data acquirer 102 to generate the output splicing number of input data as a group of input data, wherein the output data corresponding to different input data in the group of input data correspond to non-overlapping column sets in the computing unit array 103 and correspond to non-overlapping cache areas of the output buffer 105, controls the input buffer 104 to input the group of input data into the input buffer in sequence, and for each input data in the group of input data, the controller 101 controls the computing unit array to output the output data corresponding to the input data to the corresponding cache area in the output buffer, and then for the group of input data, the controller 101 controls the output buffer 105 to output the output splicing number of output data in each area of ​​the output buffer 105 in batches without overlapping each other.

[0164] According to an embodiment of the present disclosure, when the controller 101 controls the output of the output data corresponding to the input data to the corresponding cache area in the output buffer 105, it also prohibits writing to other cache areas in the output buffer 105 corresponding to the output data corresponding to other input data in the group of input data.

[0165] According to an embodiment of the present disclosure, when the input splicing quantity is greater than 1 and the output splicing quantity is equal to 1, the controller 101 may: control the data acquirer 102 to acquire and generate the input data of the input splicing quantity as a group of input data, wherein different input data in the group of input data correspond to a set of rows in the computing unit array that do not overlap with each other, and the output data corresponding to the different input data in the group of input data all correspond to all columns in the computing unit array and correspond to the entire or at least a portion of the cache area of ​​the output buffer; control the data acquirer 102 to input the group of input data into the input buffer 104; control each input data in the group of input data to be received by the controller 101; The calculation parameters of each calculation unit required for calculation are sequentially set in each calculation unit, wherein, for each input data, the calculation parameters of the calculation units on the row set corresponding to the input data are valid values, and the calculation parameters of the remaining calculation units are zero; the input buffer is controlled to provide the input data of the input splicing quantity to the calculation unit array 103 in parallel; and for this group of input data, the output data corresponding to each input data is controlled to be output sequentially to the entire or at least a part of the cache area of ​​the output buffer (according to the storage space of the output buffer), and is output sequentially or in batches through the output buffer (according to the storage space of the output buffer).

[0166] According to an embodiment of the present disclosure, when the input splicing number is greater than 1 and the output splicing number is greater than 1, the controller 101 can: determine the comprehensive splicing number based on the input splicing number and the output splicing number; control the data acquirer 102 to acquire and generate the input data of the comprehensive splicing number as a group of input data; control the data acquirer 102 to divide the group of input data into at least one sub-group based on the input splicing number and the output splicing number, wherein each input data in each sub-group input data corresponds to a set of rows of the input splicing number that do not overlap with each other in the computing unit array, and the output data corresponding to different input data in the sub-group input data correspond to a set of columns that do not overlap with each other in the computing unit array and correspond to non-overlapping cache areas of the output buffer 105; and for the calculation of each sub-group input data, control the calculation parameters of each computing unit for the sub-group input data to be set in the computing unit array 103.

[0167] In some embodiments, the input splicing number is greater than 1, the output splicing number is greater than 1, and the input splicing number is equal to the output splicing number. In this case, the comprehensive splicing number is the input splicing number or the output splicing number, and the number of subgroups is 1. The controller 101 can also: control the data acquirer 102 to input the subgroup input data into the input buffer in batches, and control the input buffer 104 to provide the subgroup input data to the computing unit array 103 in parallel; for the calculation of the subgroup input data, control the calculation parameters of the computing units at the intersection of the row set corresponding to different input data in the subgroup input data and the column set corresponding to the output data corresponding to different input data to be set to valid values, and set the calculation parameters of the remaining computing units in the computing unit array 103 to zero; and control the systolic array to output the output data corresponding to each input data in the subgroup input data to the output buffer 105, and control the output buffer 105 to output these output data in batches.

[0168] In other embodiments, the input splicing number is an integer multiple of the output splicing number greater than 1. In this case, the comprehensive splicing number is the input splicing number, and each subgroup input data includes the input data of the output splicing number. The controller 101 can also: control the data acquirer 102 to input the input data of the comprehensive splicing number into the input buffer 104 in batches, and control the input buffer 104 to provide the input data of the comprehensive splicing number to the computing unit array 103 in parallel; for the calculation of each subgroup input data, control the calculation parameters of the computing units at the intersection of the row sets corresponding to different input data in the subgroup input data and the column sets corresponding to the output data corresponding to different input data to be set to valid values, and set the calculation parameters of the remaining computing units in the computing unit array to zero; for the group of input data including the input data of the comprehensive splicing number, control the computing unit array 103 to output the output data corresponding to each subgroup input data to the output buffer 105 in sequence, and output them in sequence or in batches (according to the storage space of the output buffer) through the output buffer.

[0169] In some other embodiments, the input splicing number is greater than the output splicing number but is not an integer multiple of the output splicing number. In this case, the comprehensive splicing number is the output splicing number, and the number of subgroups is 1. The controller 101 can also: control the data acquirer 102 to add N input data padded with zeros to the group of input data, where the value of N is equal to the input splicing number minus the input and output splicing number; control the data acquirer 102 to input the group of input data with the added N input data into the input buffer 104 in batches, and control the input buffer to provide it to the computing unit array 103 in parallel; for the calculation of the group of input data, control the calculation parameters of the computing units at the intersection of the row sets corresponding to different input data in the group of input data and the column sets corresponding to the output data corresponding to different input data to be set to valid values, and set the calculation parameters of the remaining computing units in the computing unit array to zero; and control the computing unit array 103 to output the output data corresponding to each input data in the group of input data to the output buffer 105, and output it in batches through the output buffer 105.

[0170] In some other embodiments, the output splicing quantity is an integer multiple of the input splicing quantity greater than 1. In this case, the comprehensive splicing quantity is the output splicing quantity, and each subgroup input data includes the input data of the input splicing quantity. The controller 101 can also: control the data acquirer 102 to input the input data of the comprehensive splicing quantity into the input buffer 104 in sequence according to the subgroup, and control the input buffer 104 to provide each subgroup input data to the computing unit array in parallel; for the calculation of each subgroup input data, control the different inputs in the subgroup input data to be processed in parallel; The calculation parameters of the calculation units at the intersection of the row sets corresponding to the input data and the column sets corresponding to the output data corresponding to different input data are set to valid values; the calculation unit array 103 is controlled to output the output data corresponding to the group of input data (including all sub-group input data) to the output buffer 105, and output them in batches through the output buffer 105, wherein, when calculating each sub-group of input data, only the cache area corresponding to the output data of the sub-group input data in the output buffer is written, and writing to other cache areas in the output buffer is prohibited.

[0171] In some other embodiments, when the output splicing number is greater than the input splicing number but is not an integer multiple thereof, in this case, the comprehensive splicing number is the input splicing number, and the number of subgroups is 1, the controller 101 can also: control the data acquirer 102 to input the group of input data into the input buffer 104 in batches, and control the input buffer 104 to provide it in parallel to the computing unit array; for the calculation of the group of input data, control the calculation parameters of the computing units at the intersection of the row sets corresponding to different input data in the group of input data and the column sets corresponding to the output data corresponding to different input data to be set to valid values, and set the calculation parameters of the remaining computing units in the computing unit array to zero; and control the computing unit array 103 to output each output data corresponding to the group of input data to the output buffer 105, and output them in batches through the output buffer, wherein the value of at least a part of the entire cache area is 0.

[0172] More details of the above operations of the controller 101 can be found in the previous Figure 2-10D The description content will not be repeated here.

[0173] The form of one or more controllers can include a computing device capable of performing any of the above operations described herein, such as a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components or any combination thereof to implement or perform. The general-purpose processor can be a microprocessor, microcontroller or the like. Any of the above operations described herein can be implemented in hardware, software executed by a processor, firmware or any combination thereof. If implemented in software executed by a controller in the form of a processor, these operations can be stored on a computer-readable storage medium or transmitted by encoding as one or more instructions or codes on a computer-readable medium.

[0174] Therefore, an embodiment of the present disclosure further provides a computer-readable storage medium having instructions or codes stored thereon, which implement the operations performed by the controller as described above when executed by a processor.

[0175] The computer-readable storage medium may be a memory containing instructions or codes. The codes or instruction sets may be executed by a controller (processor) to complete all or part of the steps performed by the controller (including components such as the control input buffer and the DSP in the computing unit) in the computing device. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0176] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations of the methods and devices according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of the code, and the module, program segment, or a part of the code contains at least one executable instruction for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0177] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will appreciate that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A computational optimization method for a shallow depth model based on a systolic array, comprising: determining a standard input depth and a standard output depth of the systolic array, and determining a target input depth and a target output depth of the shallow depth model; determining a number of input tiles for the systolic array based on the target input depth and the standard input depth, and determining a number of output tiles for the systolic array based on the target output depth and the standard output depth, wherein the number of input tiles and / or the number of output tiles are integers greater than 1; When the input splicing number is equal to 1 and the output splicing number is an integer greater than 1, a group of input data is determined based on at least the output splicing number, wherein output data corresponding to different input data in the group of input data correspond to non-overlapping cache areas of the output buffer of the systolic array, and when calculating each input data, writing to other cache areas in the output buffer corresponding to output data corresponding to other input data in the group of input data is prohibited; and output data of the output splicing number is generated, each output data having the target output depth, and the output data of the output splicing number are output in batches through the output buffer without overlapping with each other.

2. The computational optimization method according to claim 1, wherein: The systolic array is a computing unit array, and the number of rows and the number of columns of the computing unit array correspond to the standard input depth and the standard output depth respectively; The depth of the input buffer of the systolic array corresponds to the standard input depth, and the depth of the output buffer corresponds to the standard output depth.

3. The computational optimization method according to claim 2, wherein: When the input splicing quantity is equal to 1 and the output splicing quantity is greater than 1, the calculation optimization method further includes: Setting calculation parameters of each calculation unit in the calculation unit array; generating the output spliced ​​quantity of input data as the set of input data, wherein the output data corresponding to different input data in the set of input data correspond to non-overlapping column sets in the computing unit array; inputting each input data in the set of input data into the input buffer in sequence, thereby providing the set of input data to the computing unit array in sequence via the input buffer; For each input data in the set of input data, outputting the output data corresponding to the input data to the corresponding buffer area in the output buffer; For the set of input data, the output data of the output splicing quantity in each area of ​​the output buffer are output in batches through the output buffer without overlapping with each other.

4. The computational optimization method according to claim 2, further comprising: When the input splicing number is an integer greater than 1, input data of the input splicing number is generated, each input data has the target input depth, and the input data of the input splicing number are input into the input buffer in batches without overlapping each other.

5. The computational optimization method according to claim 4, wherein: When the input splicing quantity is greater than 1 and the output splicing quantity is equal to 1, the calculation optimization method further includes: generating the input data of the input splicing quantity as the set of input data, wherein the output data corresponding to different input data in the set of input data all correspond to all columns in the computing unit array; sequentially setting calculation parameters of each calculation unit required for calculation of each input data in the set of input data, wherein, for each input data, the calculation parameters of the calculation units on the row set corresponding to the input data are valid values, and the calculation parameters of the remaining calculation units are zero; Inputting the set of input data into the input buffer in batches, thereby providing the set of input data to the computing unit array in parallel via the input buffer; For the group of input data, the output data corresponding to each input data is sequentially output to the output buffer and output through the output buffer.

6. The computational optimization method according to claim 4, wherein: When the input splicing number is greater than 1 and the output splicing number is greater than 1, the calculation optimization method further includes: Determining a comprehensive splicing quantity based on the input splicing quantity and the output splicing quantity; generating the integrated spliced ​​quantity of input data as the set of input data; Dividing the group of input data into at least one subgroup based on the input splicing number and the output splicing number, wherein each input data in each subgroup of input data corresponds to a set of non-overlapping rows of the input splicing number in the computing unit array, and output data corresponding to different input data in the subgroup of input data corresponds to a set of non-overlapping columns in the computing unit array; For the calculation of each subgroup of input data, the calculation parameters of each calculation unit in the calculation unit array for the subgroup of input data are set.

7. The computational optimization method according to claim 6, wherein: In a case where the input splicing number is equal to the output splicing number, the comprehensive splicing number is the input splicing number or the output splicing number, and the number of subgroups is 1, the calculation optimization method further includes: Inputting subgroups of input data into the input buffer in batches, thereby providing the subgroups of input data to the computing unit array in parallel via the input buffer; For the calculation of the subgroup input data, the calculation parameters of the calculation units at the intersections of the row sets corresponding to different input data in the subgroup input data and the column sets corresponding to the output data corresponding to the different input data are set to valid values, and the calculation parameters of the remaining calculation units in the calculation unit array are set to zero; The output data corresponding to each input data in the subgroup of input data is output to the output buffer and output in batches through the output buffer.

8. The computational optimization method according to claim 6, wherein: In a case where the input splicing number is an integer multiple of the output splicing number that is greater than 1, the comprehensive splicing number is the input splicing number, and each subgroup of input data includes input data of the output splicing number, the calculation optimization method further includes: Inputting the integrated splicing quantity of input data into the input buffer in batches, thereby providing the integrated splicing quantity of input data to the computing unit array in parallel via the input buffer; For the calculation of each subgroup of input data, the calculation parameters of the calculation units at the intersection of the row sets corresponding to different input data in the subgroup of input data and the column sets corresponding to the output data corresponding to the different input data are set to valid values, and the calculation parameters of the remaining calculation units in the calculation unit array are set to zero; For the group of input data including the integrated spliced ​​number of input data, the output data corresponding to each subgroup of input data is sequentially output to the output buffer and output through the output buffer.

9. The computational optimization method according to claim 6, wherein: In a case where the input splicing number is greater than the output splicing number but is not an integer multiple thereof, the comprehensive splicing number is the output splicing number, and the number of subgroups is 1, wherein the calculation optimization method further includes: Add N input data padded with zeros to the set of input data, where the value of N is equal to the number of input splicing minus the number of input and output splicing; inputting the group of input data added with N input data into the input buffer in batches, so as to be provided to the computing unit array in parallel via the input buffer; For the calculation of the set of input data, setting the calculation parameters of the calculation units at the intersections of the row sets corresponding to different input data in the set of input data and the column sets corresponding to the output data corresponding to the different input data to valid values, and setting the calculation parameters of the remaining calculation units in the calculation unit array to zero; The output data corresponding to each input data in the group of input data is output to the output buffer and output in batches through the output buffer.

10. The computational optimization method according to claim 6, wherein: In a case where the output splicing number is an integer multiple of the input splicing number that is greater than 1, the comprehensive splicing number is the output splicing number, the input data of each subgroup includes input data of the input splicing number, and the calculation optimization method further includes: inputting the integrated spliced ​​quantity of input data into the input buffer in sequence according to subgroups, so that each subgroup of input data is provided to the computing unit array in parallel via the input buffer, For the calculation of each subgroup of input data, the calculation parameters of the calculation units at the intersection of the row sets corresponding to different input data in the subgroup of input data and the column sets corresponding to the output data corresponding to different input data are set to valid values, and the calculation parameters of the remaining calculation units on the column sets corresponding to the output data corresponding to different input data in the subgroup of input data are set to zero; Outputting the output data corresponding to the set of input data to the output buffer and outputting the output data in batches through the output buffer, When calculating each subgroup of input data, only the buffer area corresponding to the output data corresponding to the subgroup of input data in the output buffer is written, and writing to other buffer areas in the output buffer is prohibited.

11. The computational optimization method according to claim 6, wherein: In a case where the output splicing number is greater than the input splicing number but is not an integer multiple thereof, the comprehensive splicing number is the input splicing number, and the number of subgroups is 1, the calculation optimization method further includes: Inputting the set of input data into the input buffer in batches, so as to be provided to the computing unit array in parallel via the input buffer; For calculation of the set of input data, setting calculation parameters of calculation units at intersections of row sets corresponding to different input data in the set of input data and column sets corresponding to output data corresponding to different input data to valid values, and setting calculation parameters of remaining calculation units in the calculation unit array to zero; and The output data corresponding to the group of input data is output to the output buffer and output in batches through the output buffer, wherein the value of at least a portion of the buffer area of ​​the output buffer is 0.

12. A computing device for a shallow depth model based on a systolic array, comprising: A controller, a data acquirer, a systolic array, an input buffer, and an output buffer, wherein the systolic array is an array of computing units, wherein: The controller determines a standard input depth and a standard output depth of the systolic array, and determines a target input depth and a target output depth of the shallow depth model; The controller determines an input splice number for the systolic array based on the target input depth and a standard input depth, and determines an output splice number for the systolic array based on the target output depth and a standard output depth, wherein the input splice number and / or the output splice number are integers greater than 1; When the input splicing number is equal to 1 and the output splicing number is an integer greater than 1, the controller determines a group of input data based at least on the output splicing number, wherein the output data corresponding to different input data in the group of input data correspond to non-overlapping cache areas of the output buffer, and when calculating each input data, prohibiting writing to other cache areas in the output buffer corresponding to output data corresponding to other input data in the group of input data; and controls the systolic array to generate the output splicing number of output data, and controls the output buffer to output the output splicing number of output data in batches without overlapping with each other, wherein each output data has the target output depth.

13. The computing device according to claim 12, wherein: When the input splicing quantity is equal to 1 and the output splicing quantity is greater than 1, The controller controls the setting of calculation parameters of each calculation unit in the calculation unit array; The controller controls the data acquirer to acquire and generate the output splicing quantity of input data as the set of input data, wherein the output data corresponding to different input data in the set of input data correspond to non-overlapping column sets in the computing unit array; The controller controls the input data to be sequentially input to the input buffer, and controls the input buffer to provide the input data to the computing unit array in sequence; For each input data in the set of input data, the controller controls the computing unit array to output the output data corresponding to the input data to the corresponding cache area in the output buffer; For the group of input data, the controller controls the output buffer to output the output splicing quantity of output data in each buffer area of ​​the output buffer in batches without overlapping each other.

14. The computing device according to claim 12, wherein: When the input splicing quantity is an integer greater than 1, The data acquirer is controlled to acquire and generate the input data of the input splicing quantity, and the input data of the input splicing quantity are controlled to be input into the input buffer in batches without overlapping each other, wherein each input data has the target input depth.

15. The computing device of claim 14, wherein: When the input splicing quantity is greater than 1 and the output splicing quantity is equal to 1, The controller controls the data acquirer to acquire and generate the input data of the input splicing quantity as the set of input data, wherein different input data in the set of input data correspond to different sets of rows in the computing unit array that do not overlap with each other, and output data corresponding to different input data in the set of input data all correspond to all columns in the computing unit array; The controller controls sequentially setting calculation parameters of each calculation unit required for calculating each input data in the set of input data, wherein, for each input data, the calculation parameters of the calculation units on the row set corresponding to the input data are valid values, and the calculation parameters of the remaining calculation units are zero; The controller controls the group of input data to be input into the input buffer in batches, and controls the input buffer to provide the input splicing quantity of input data to the computing unit array in parallel; For each input data in the set of input data, the controller controls the systolic array to output the output data corresponding to the input data to the output buffer, and controls the output buffer to output the output data corresponding to each input data.

16. The computing device of claim 14, wherein: When the input splicing number is greater than 1 and the output splicing number is greater than 1, The controller determines a comprehensive splicing quantity based on the input splicing quantity and the output splicing quantity; The controller controls the data acquirer to acquire and generate the comprehensive splicing quantity of input data as the set of input data; The controller controls the data acquirer to divide the group of input data into at least one subgroup based on the input splicing number and the output splicing number, wherein each input data in each subgroup of input data corresponds to a set of rows of the input splicing number that do not overlap with each other in the computing unit array, and output data corresponding to different input data in the subgroup of input data corresponds to a set of columns in the computing unit array that do not overlap with each other; For calculation of each subgroup of input data, the controller controls the calculation parameters of each calculation unit in the calculation unit array to be set for the subgroup of input data.

Images